Filter Azure monitor logs when customer does not subscribe for the categories

[manikantanallagatla]

Issue describing the changes in this PR

For a few SKUs in functions, we want to filter azure monitor logs based on the categories subscribed by the user. This PR checks for the env variable for the categories and skips logging.

Pull request checklist

IMPORTANT: Currently, changes must be backported to the in-proc branch to be included in Core Tools and non-Flex deployments.

• Backporting to the in-proc branch is not required
  • Otherwise: Link to backporting PR
• My changes do not require documentation changes
  • Otherwise: Documentation issue linked to PR
• My changes should not be added to the release notes for the next release
  • Otherwise: I've added my notes to release_notes.md
• My changes do not need to be backported to a previous version
  • Otherwise: Backport tracked by issue/PR #issue_or_pr
• My changes do not require diagnostic events changes
  • Otherwise: I have added/updated all related diagnostic events and their documentation (Documentation issue linked to PR)
• I have added all required tests (Unit tests, E2E tests)

Additional information

Additional PR information

[jviau]

Can you give an example of these categories? Are they regular ILogger categories?

If so, we should use built in filtering with Microsoft.Extensions.Logging. We can either transform configuration, or we can simply call:

Part 1: log level filter

// condition on if this is the right sku
services.AddOptions<LoggerFilterOptions>()
    .Configure<IConfiguration>((options, config) =>
    {
        string setting = config[EnvironmentSettingNames.AzureMonitorCategories];
        options.AddFilter<AzureMonitorDiagnosticLoggerProvider>((category, level) =>
        {
            // return false when AzureMonitorCategories does not contain FunctionAppLogs
        });
    });

Part 2: IsEnabled update

// AzureMonitorDiagnosticLogger.cs
// Update the below method to return false when AzureMonitorCategories does not contain FunctionAppLogs
public bool IsEnabled(LogLevel logLevel)
{
    // We want to instantiate this Logger in placeholder mode to warm it up, but do not want to log anything.
    return !string.IsNullOrEmpty(_hostNameProvider.Value) && !_environment.IsPlaceholderModeEnabled();
}

Although I have to ask - who sets EnvironmentSettingNames.AzureMonitorCategories? Is it from the platform, or does the user explicitly set it?

[RohitRanjanMS]

I agree with @jviau that providing more context about the scenario would be beneficial.
A simple list of categories alone might not be enough; the log level will also be necessary.
The environment variable is applicable only to LinuxContainerEventGenerator and does not extend to AzureMonitor/Diagnostic Settings integrations for other skus.
If these are specified by the customer, can the existing logging configuration in host.json be leveraged instead?

[manikantanallagatla]

Can you give an example of these categories? Are they regular ILogger categories?

If so, we should use built in filtering with Microsoft.Extensions.Logging. We can either transform configuration, or we can simply call:

Part 1: log level filter

// condition on if this is the right sku
services.AddOptions<LoggerFilterOptions>()
    .Configure<IConfiguration>((options, config) =>
    {
        string setting = config[EnvironmentSettingNames.AzureMonitorCategories];
        options.AddFilter<AzureMonitorDiagnosticLoggerProvider>((category, level) =>
        {
            // return false when AzureMonitorCategories does not contain FunctionAppLogs
        });
    });

Part 2: IsEnabled update

// AzureMonitorDiagnosticLogger.cs
// Update the below method to return false when AzureMonitorCategories does not contain FunctionAppLogs
public bool IsEnabled(LogLevel logLevel)
{
    // We want to instantiate this Logger in placeholder mode to warm it up, but do not want to log anything.
    return !string.IsNullOrEmpty(_hostNameProvider.Value) && !_environment.IsPlaceholderModeEnabled();
}

Although I have to ask - who sets EnvironmentSettingNames.AzureMonitorCategories? Is it from the platform, or does the user explicitly set it?

I have removed all the changes. We already have category filter in the host code. I just had to called it in isenabled class

[jviau]

@manikantanallagatla I believe you will need to also add the filter callback I suggested. It may seem redundant but they have different purposes:

1. filter: avoid running templating on the log message
2. IsEnabled: give Logger users a hook to wrap their log statements in incase they have pre-templating code they want to skip.

I also am wondering about the IsAzureMonitorEnabled implementation as is being used in this new code paths. These are hot-paths, we need to make sure they are as performant as possible. IsAzureMonitorEnabled will be allocating an array and perform string comparison on each run. Can we optimize that? Cache the results of it somewhere?

Also this change will affect all skus - is the behavior of that env variable consistent across all skus?

[vivekjilla]

If we put this in else, would the provider still gets added as a singleton.

[vivekjilla]

Did we validate the logs are not being emitted in placeholder mode..

[vivekjilla]

Can this impact other SKUs? should we scope down these changes only to Legion.

[vivekjilla]

we already have a method called IsConsumptionOnLegion which we can use instead of adding new method