fix(outputs.azure_monitor): Prevent infinite send loop for outdated metrics

[srebhan]

Summary

The Azure Monitor service has limitations on the time-range of accepted metrics. Metrics older than 30 minutes and more than 4 minutes into the future (or 20 min in the past and 5 min in the future according to documentation) are not accepted and the service responds with a 400 HTTP error code. The current code propagates this error up into the model and the model code then assumes it must reschedule the batch or metrics due to a retryable error. Usually metrics are filtered according to those limits but due to latency during transmission etc. the error might still be triggered.

This PR handles the 400 HTTP error code as a non-retryable error and fixes other non-retryable errors paths on the way.
It furthermore introduces two new configuration settings to better control the accepted metric timeframe allowing a more robust limit for filtering the metrics. Please note: The default values of the parameters (30 min into the past until 1 minute into the past) are representing the previous behavior, where the upper limit is in the past (contrary to 4 min into the future) to allow aggregation of metrics to happen before sending those aggregates.

Checklist

• No AI generated code was used in this PR

Related issues

resolves #15908

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

⚠️ This pull request increases the Telegraf binary size by 1.01 % for linux amd64 (new size: 283.4 MB, nightly size 280.6 MB)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz