Built-in check command: ifw-api

[Al2Klimov]

To do for IfW to work with this

• Get-IcingaCheckCommandConfig: provide vars.ifw_api_arguments in each command icinga-powershell-framework#633

[Al2Klimov]

Colleagues, please have a look ASAP (only) at the newly introduced stuff exposed to the DSL. Just whether we can "cement" this stuff. The idea is: a template like https://www.netways.de/blog/2016/03/21/check_by_ssh-mit-icinga-2/ which you import at the bottom of checkables and it runs s/PS/PS ReST API/.

[julianbrost]

If we do this, I'd not do this as something Windows-specific but rather as a generic solution for #8995 (i.e. not name things icinga4win and probably also allow more flexible requests).

[Al2Klimov]

That was my idea. CVs method, URL, headers and body for the request and a CV function for the response. But you can't have function CVs.

Anyway: even if we'd do this (other construction area!), I'd base a specific command for Win PS on the generic one. Which leads us back to the question: is that DSL/config stuff OK and "cementable"?

[Al2Klimov]

Discussed with @LordHepipud just now

Goal

There is a check command template similar to plugin-check-command named ifw-api-check-command and a check command template named "IfW API" (probably not in the ITL, but where also "PowerShell Base" is now) which replaces "PowerShell Base" in Director imports in IfW check commands. The latter may be done by the user in Director and is ideally the user's only TODO. ifw-api-check-command takes these custom vars:

• ifw_api_command
• ifw_api_arguments
• ifw_api_host
• ifw_api_port
• ifw_api_fallback_command
• ifw_api_fallback_arguments

ifw-api-check-command connects to $ifw_api_host$:$ifw_api_port$, requires the Icinga CA (@LordHepipud What about the TLS host name? $ifw_api_cn$ with a sane default?) and requests to run $ifw_api_command$ with $ifw_api_arguments$. It awaits the response (using our ASIO event loop) and produces the check result.

In case the API is down, if technically possible (TODO @Al2Klimov), the command falls back to running $ifw_api_fallback_command$ with $ifw_api_fallback_arguments$ locally as usual.

OK?

[julianbrost]

There is a check command template similar to plugin-check-command named ifw-api-check-command and a check command template named "IfW API" (probably not in the ITL, but where also "PowerShell Base" is now) which replaces "PowerShell Base" in Director imports in IfW check commands. The latter may be done by the user in Director and is ideally the user's only TODO.

I don't understand why there are two check command templates. How do they differ?

ifw-api-check-command takes these custom vars:
* ifw_api_command
* ifw_api_arguments
* ifw_api_host
* ifw_api_port
* ifw_api_fallback_command
* ifw_api_fallback_arguments

What would you pass in the argument variables (i.e. type and content)?

requires the Icinga CA (@LordHepipud What about the TLS host name? $ifw_api_cn$ with a sane default?)

So the Powershell framework is then supposed to also read and use the agent certificate and key? Is this intended as a workaround for authentication on localhost? Or do you actually plan to use this over the network? Does the client (i.e. icinga2) also authenticate itself?

In case the API is down, if technically possible (TODO @Al2Klimov), the command falls back to running $ifw_api_fallback_command$ with $ifw_api_fallback_arguments$ locally as usual.

What's the point of the fallback here? Do you plan to use this in a way that's more useful than a fixed error result? If so, how?

[Al2Klimov]

I don't understand why there are two check command templates. How do they differ?

ifw-api-check-command shall (as its colleagues) only import built-in C++ stuff into the DSL. IfW API will at least import ifw-api-check-command and may do additional DSL depending on what I'll dig out while playing around with all this stuff.

What would you pass in the argument variables (i.e. type and content)?

Pretty much like in check_by_ssh.

So the Powershell framework is then supposed to also read and use the agent certificate and key? Is this intended as a workaround for authentication on localhost? Or do you actually plan to use this over the network? Does the client (i.e. icinga2) also authenticate itself?

Good questions!

@LordHepipud If I can install the PS daemon, but not Icinga 2 on a box – where do cert+key come from?

What's the point of the fallback here?

If the PS daemon dies, your checks shall still work.

[julianbrost]

Results of some discussion with @LordHepipud (please correct me if I described something wrong or if you have to add something):

Authentication

Authentication will remain as-is in the PowerShell framework. So the PowerShell service using a TLS connection where it presents a certificate. For client authentication, the default is listening on localhost only and trusting connections by default with the option to also listen on external interfaces and adding basic auth.

This implies that all PowerShell checks must be written in a way that they don't reveal additional information to local users, but if the PowerShell framework ensures this, this is fine for me. Additional data point: the PowerShell API service should be enabled by default right now, so on systems where it's installed, the checks are already accessible for all local users.

If I can install the PS daemon, but not Icinga 2 on a box – where do cert+key come from?

Users have to provide them themselves in this situation.

Config Options

The following things should be configurable for now:

• host + port
• username + password (optional, for HTTP basic auth)
• TLS validation options: hostname to check against (should probably default to either host or the endpoint name, TBD), also TBD: do we want options to disable verification or specify another CA (this could be useful to enable checks from multiple environments)

[Not 100% sure that what's described in this paragraph will work, so that's subject to discussion/change] For passing the command and arguments, my preferred solution would be to have the CheckCommand.execute function just use CheckCommand.command and CheckCommand.arguments directly. This would have two advantages: with some change to how the CheckCommand objects for PowerShell checks are generated, everything that specific to whether the check is done by CreateProcess() or API call can be done inside a template, two of those could be provided and the one used in the CheckCommand determines the execution method. Additionally, this would avoid the host/service templates, that in the current form, have to be imported at the end of the objects, which isn't supported in Director.

I would not add a fallback right now, that could be a future extension but it's not required right now.

@Al2Klimov If I've missed some open question, please point it out.

[Al2Klimov]

What exactly does the plugin output when called on command line and what does the daemon API return if you call it by yourself (as with curl)?

[LordHepipud]

Another more security related topic:

In case the API returns an error for SecureStrings, we should always transform the error message to something more generic and never return the full error during the Icinga plugin execution.

In Icinga for Windows, the error is handled during the Icinga execution as follows:

if ($ExErrorId -Like "*ParameterArgumentTransformationError*" -And $ExMsg.Contains('System.Security.SecureString')) {
        $ExMsg = [string]::Format(
            'Cannot bind parameter {0}. Cannot convert the provided value for argument "{0}" of type "System.String" to type "System.Security.SecureString".',
            $ArgName
        );

...

For local development reasons and custom API-calls, this is fine in my oppinion to have the error as outputed. Icinga should however not return this error. Instead, we should change the message to the following, in case the API returns something containing System.Security.SecureString for non-objects:

Failed to execute plugin call over REST-Api. One or multiple arguments defined as "System.String" were tried to be converted to "System.Security.SecureString" without proper transformation. Please ensure you are running the latest configuration for your plugins.

[LordHepipud]

Calling the plugin locally will, will return this:

[OK] CPU Load
| 'total::ifw_cpu::load'=0.246928%;;;0;100 'load15m'=6.474504%;;;0;100 'load1m'=4.588769%;;;0;100 'load3m'=4.484824%;;;0;100 'load5m'=5.796771%;;;0;100 '0::ifw_cpu::load'=0%;;;0;100 'load15m'=15.131862%;;;0;100 'load1m'=9.035778%;;;0;100 'load3m'=6.542370%;;;0;100 'load5m'=10.018513%;;;0;100 '1::ifw_cpu::load'=0%;;;0;100 'load15m'=8.399096%;;;0;100 'load1m'=7.446934%;;;0;100 'load3m'=9.838786%;;;0;100 'load5m'=9.673718%;;;0;100 '2::ifw_cpu::load'=0.334744%;;;0;100 'load15m'=4.742161%;;;0;100 'load1m'=2.346604%;;;0;100 'load3m'=2.611210%;;;0;100 'load5m'=4.276676%;;;0;100 '3::ifw_cpu::load'=0%;;;0;100 'load15m'=2.275388%;;;0;100 'load1m'=0.248665%;;;0;100 'load3m'=0.239317%;;;0;100 'load5m'=1.080430%;;;0;100 '4::ifw_cpu::load'=1.237365%;;;0;100 'load15m'=7.237302%;;;0;100 'load1m'=5.364076%;;;0;100 'load3m'=4.536041%;;;0;100 'load5m'=6.480357%;;;0;100 '5::ifw_cpu::load'=0%;;;0;100 'load15m'=2.714305%;;;0;100 'load1m'=3.089743%;;;0;100 'load3m'=2.120856%;;;0;100 'load5m'=2.127818%;;;0;100 '6::ifw_cpu::load'=0%;;;0;100 'load15m'=5.122571%;;;0;100 'load1m'=1.801306%;;;0;100 'load3m'=1.667468%;;;0;100 'load5m'=3.759925%;;;0;100 '7::ifw_cpu::load'=0%;;;0;100 'load15m'=2.784228%;;;0;100 'load1m'=4.153756%;;;0;100 'load3m'=2.316715%;;;0;100 'load5m'=3.053943%;;;0;100 '8::ifw_cpu::load'=0%;;;0;100 'load15m'=4.667609%;;;0;100 'load1m'=1.947001%;;;0;100 'load3m'=1.497210%;;;0;100 'load5m'=2.685528%;;;0;100 '9::ifw_cpu::load'=0.715632%;;;0;100 'load15m'=2.202579%;;;0;100 'load1m'=1.600142%;;;0;100 'load3m'=1.023877%;;;0;100 'load5m'=1.104869%;;;0;100 '10::ifw_cpu::load'=0%;;;0;100 'load15m'=3.677235%;;;0;100 'load1m'=6.155566%;;;0;100 'load3m'=3.660855%;;;0;100 'load5m'=3.534219%;;;0;100 '11::ifw_cpu::load'=0%;;;0;100 'load15m'=4.954133%;;;0;100 'load1m'=0.691924%;;;0;100 'load3m'=0.695369%;;;0;100 'load5m'=2.073820%;;;0;100 '12::ifw_cpu::load'=14.51751%;;;0;100 'load15m'=11.813533%;;;0;100 'load1m'=9.091800%;;;0;100 'load3m'=9.679714%;;;0;100 'load5m'=11.505569%;;;0;100 '13::ifw_cpu::load'=0%;;;0;100 'load15m'=2.064079%;;;0;100 'load1m'=0.895355%;;;0;100 'load3m'=0.687991%;;;0;100 'load5m'=2.386542%;;;0;100 '14::ifw_cpu::load'=3.455384%;;;0;100 'load15m'=5.877652%;;;0;100 'load1m'=4.998814%;;;0;100 'load3m'=6.148866%;;;0;100 'load5m'=5.197742%;;;0;100 '15::ifw_cpu::load'=0%;;;0;100 'load15m'=19.066950%;;;0;100 'load1m'=11.425888%;;;0;100 'load3m'=16.986604%;;;0;100 'load5m'=18.056543%;;;0;100 '16::ifw_cpu::load'=6.028974%;;;0;100 'load15m'=4.758413%;;;0;100 'load1m'=1.833932%;;;0;100 'load3m'=2.731817%;;;0;100 'load5m'=3.705972%;;;0;100 '17::ifw_cpu::load'=0.155785%;;;0;100 'load15m'=11.366588%;;;0;100 'load1m'=13.264995%;;;0;100 'load3m'=13.236616%;;;0;100 'load5m'=13.897197%;;;0;100 '18::ifw_cpu::load'=0%;;;0;100 'load15m'=10.809771%;;;0;100 'load1m'=7.463446%;;;0;100 'load3m'=5.236495%;;;0;100 'load5m'=5.472646%;;;0;100 '19::ifw_cpu::load'=0%;;;0;100 'load15m'=5.480742%;;;0;100 'load1m'=6.529680%;;;0;100 'load3m'=7.456943%;;;0;100 'load5m'=8.882515%;;;0;100 '20::ifw_cpu::load'=10.55765%;;;0;100 'load15m'=8.822701%;;;0;100 'load1m'=6.144199%;;;0;100 'load3m'=4.494725%;;;0;100 'load5m'=10.555931%;;;0;100 '21::ifw_cpu::load'=0%;;;0;100 'load15m'=1.859089%;;;0;100 'load1m'=0.334002%;;;0;100 'load3m'=0.390957%;;;0;100 'load5m'=1.137375%;;;0;100 '22::ifw_cpu::load'=0%;;;0;100 'load15m'=5.204355%;;;0;100 'load1m'=1.342644%;;;0;100 'load3m'=1.394436%;;;0;100 'load5m'=4.943521%;;;0;100 '23::ifw_cpu::load'=2.358489%;;;0;100 'load15m'=4.357303%;;;0;100 'load1m'=2.944484%;;;0;100 'load3m'=2.439848%;;;0;100 'load5m'=3.511765%;;;0;100

By using the API, the result is as follows:

{
    "Invoke-IcingaCheckCPU": {
        "exitcode": 0,
        "checkresult": "[OK] CPU Load",
        "perfdata": [
            "'total::ifw_cpu::load'=7.118014%;;;0;100 'load15m'=6.643090%;;;0;100 'load1m'=7.931634%;;;0;100 'load3m'=6.509641%;;;0;100 'load5m'=5.564334%;;;0;100",
            "'0::ifw_cpu::load'=11.66598%;;;0;100 'load15m'=14.599008%;;;0;100 'load1m'=14.81294%;;;0;100 'load3m'=9.237693%;;;0;100 'load5m'=8.512385%;;;0;100",
            "'1::ifw_cpu::load'=8.919567%;;;0;100 'load15m'=8.753730%;;;0;100 'load1m'=8.61643%;;;0;100 'load3m'=8.633471%;;;0;100 'load5m'=9.519103%;;;0;100",
            "'2::ifw_cpu::load'=6.208435%;;;0;100 'load15m'=5.604229%;;;0;100 'load1m'=8.684490%;;;0;100 'load3m'=6.620467%;;;0;100 'load5m'=4.903369%;;;0;100",
            "'3::ifw_cpu::load'=2.243811%;;;0;100 'load15m'=2.230745%;;;0;100 'load1m'=2.990866%;;;0;100 'load3m'=1.776193%;;;0;100 'load5m'=1.121639%;;;0;100",
            "'4::ifw_cpu::load'=6.830565%;;;0;100 'load15m'=7.539819%;;;0;100 'load1m'=7.717305%;;;0;100 'load3m'=5.855016%;;;0;100 'load5m'=5.356999%;;;0;100",
            "'5::ifw_cpu::load'=7.412375%;;;0;100 'load15m'=3.787795%;;;0;100 'load1m'=10.833543%;;;0;100 'load3m'=5.950030%;;;0;100 'load5m'=4.482164%;;;0;100",
            "'6::ifw_cpu::load'=4.410651%;;;0;100 'load15m'=5.310580%;;;0;100 'load1m'=5.536898%;;;0;100 'load3m'=4.295507%;;;0;100 'load5m'=3.158966%;;;0;100",
            "'7::ifw_cpu::load'=2.6487%;;;0;100 'load15m'=2.840038%;;;0;100 'load1m'=3.386922%;;;0;100 'load3m'=2.913623%;;;0;100 'load5m'=2.896971%;;;0;100",
            "'8::ifw_cpu::load'=6.132501%;;;0;100 'load15m'=4.759969%;;;0;100 'load1m'=8.092874%;;;0;100 'load3m'=5.325559%;;;0;100 'load5m'=3.702488%;;;0;100",
            "'9::ifw_cpu::load'=2.363927%;;;0;100 'load15m'=2.155526%;;;0;100 'load1m'=3.657888%;;;0;100 'load3m'=2.052258%;;;0;100 'load5m'=1.699246%;;;0;100",
            "'10::ifw_cpu::load'=3.647085%;;;0;100 'load15m'=4.033311%;;;0;100 'load1m'=5.533991%;;;0;100 'load3m'=3.200901%;;;0;100 'load5m'=3.739074%;;;0;100",
            "'11::ifw_cpu::load'=7.171406%;;;0;100 'load15m'=4.637360%;;;0;100 'load1m'=8.437233%;;;0;100 'load3m'=4.824223%;;;0;100 'load5m'=3.062289%;;;0;100",
            "'12::ifw_cpu::load'=11.90565%;;;0;100 'load15m'=11.682973%;;;0;100 'load1m'=12.699554%;;;0;100 'load3m'=12.003024%;;;0;100 'load5m'=10.869960%;;;0;100",
            "'13::ifw_cpu::load'=6.045875%;;;0;100 'load15m'=3.087188%;;;0;100 'load1m'=8.569662%;;;0;100 'load3m'=5.797095%;;;0;100 'load5m'=3.632981%;;;0;100",
            "'14::ifw_cpu::load'=4.44508%;;;0;100 'load15m'=5.665241%;;;0;100 'load1m'=6.093470%;;;0;100 'load3m'=4.421081%;;;0;100 'load5m'=5.141157%;;;0;100",
            "'15::ifw_cpu::load'=19.35953%;;;0;100 'load15m'=18.973055%;;;0;100 'load1m'=13.634061%;;;0;100 'load3m'=15.808177%;;;0;100 'load5m'=14.067095%;;;0;100",
            "'16::ifw_cpu::load'=5.8132%;;;0;100 'load15m'=5.166282%;;;0;100 'load1m'=6.134066%;;;0;100 'load3m'=5.404336%;;;0;100 'load5m'=4.172912%;;;0;100",
            "'17::ifw_cpu::load'=13.79048%;;;0;100 'load15m'=11.983098%;;;0;100 'load1m'=12.636845%;;;0;100 'load3m'=14.35672%;;;0;100 'load5m'=12.149911%;;;0;100",
            "'18::ifw_cpu::load'=5.914064%;;;0;100 'load15m'=8.509475%;;;0;100 'load1m'=6.307726%;;;0;100 'load3m'=5.402937%;;;0;100 'load5m'=5.684196%;;;0;100",
            "'19::ifw_cpu::load'=10.32908%;;;0;100 'load15m'=7.247427%;;;0;100 'load1m'=11.353410%;;;0;100 'load3m'=9.189203%;;;0;100 'load5m'=8.741450%;;;0;100",
            "'20::ifw_cpu::load'=10.98479%;;;0;100 'load15m'=9.407397%;;;0;100 'load1m'=7.591742%;;;0;100 'load3m'=10.941266%;;;0;100 'load5m'=8.178106%;;;0;100",
            "'21::ifw_cpu::load'=1.236017%;;;0;100 'load15m'=1.846647%;;;0;100 'load1m'=1.176588%;;;0;100 'load3m'=1.103994%;;;0;100 'load5m'=0.800458%;;;0;100",
            "'22::ifw_cpu::load'=8.1916%;;;0;100 'load15m'=5.890120%;;;0;100 'load1m'=11.984778%;;;0;100 'load3m'=8.158067%;;;0;100 'load5m'=5.148411%;;;0;100",
            "'23::ifw_cpu::load'=3.195939%;;;0;100 'load15m'=3.724442%;;;0;100 'load1m'=3.790864%;;;0;100 'load3m'=2.919653%;;;0;100 'load5m'=2.757258%;;;0;100"
        ]
    }
}

The multi output in general is designed to have the first value with the full index, like 23::ifw_cpu::load. Every other value written behind is interpreted to belong to the first metric. Which means Icinga in general is replacing ::load with ::load15m for example.

[Al2Klimov]

Perfdata

My last two commits fixed it. Please cross-check. https://git.icinga.com/packaging/windows-icinga2/-/jobs/358086

"IfW" API response

{
        "lolcat": {
                "exitcode": 0,
                "perfdata": [ "animal::cat::lolcat=1 grumpycat=2" ],
                "checkresult": "OK"
        }
}

/v1/objects/services?pretty=1

Before

                    "exit_status": 0,
                    "output": "OK",
                    "performance_data": [
                        "animal::cat::lolcat=1 grumpycat=2"
                    ],

After

                    "exit_status": 0,
                    "output": "OK",
                    "performance_data": [
                        "animal::cat::lolcat=1",
                        "animal::cat::grumpycat=2"
                    ],

SecureString

I see Icinga/icinga-powershell-framework#633 (comment) already handled this, so no action is required on our side.

[julianbrost]

@LordHepipud

Another more security related topic:

In case the API returns an error for SecureStrings, we should always transform the error message to something more generic and never return the full error during the Icinga plugin execution.

What exactly is SecureString and in particular, what should Icinga 2 know about it? At first glance, the SecureString documentation reads more like it's an implementation detail how a .NET/PowerShell application stores strings (basically, a fancy string class that zeros out the value when freed or something like that).

That screenshot also looks like the API response can also be a JSON string that's supposed to be an error message? Is there a full specification of the response? The /v1/checker documentation does not state this (nor did it contain a hint about the multiple values per string in the perfdata array).

[julianbrost]

Just had a meeting with @LordHepipud and @Al2Klimov. The key results are summarized here.

Actions for Icinga 2

• Add a feature flag that can easily be checked in the configuration to allow a nicer fallback mechanism than the current extra template with the try/import/except construct, something like:

  if (!Icinga.HasFeatureIfwApi) {
    object CheckCommand "ifw-api" {
      // Empty fallback object to allow `import "ifw-api"` on versions
      // older than Icinga 2.14 which will behave like a no-op there.
    }
  }
  

  Exact name and location (as in namespace) of the flag are still subject to discussion, above is my suggestion.
• Update documentation accordingly.

Actions for Icinga for Windows

• Error messages are redacted in PowerShell, i.e. the handling Built-in check command: ifw-api #9062 (comment) is done in a way that affects the API response (possibly by checking if the user agent matches Icinga/*, that's up to @LordHepipud but Icinga 2 will always use the message from the response for the check result without extra filtering).
• Additionally, the API shall always return a JSON object representing a check result, including in all error cases (including those where currently a JSON string is returned instead of a JSON object). Maybe not immediately as the current handling of these errors is acceptable, even if not super nice, possibly also based on the user agent, again up to @LordHepipud.

[julianbrost]

Add a feature flag for use in the config as described in #9062 (comment) (after agreeing on a name).

[Al2Klimov]

My counter-suggestion

System.ApplicationFeatures.IfwApi = true

next to

<1> => System.Application
System.ApplicationType    System.ApplicationVersion
<1> => System.Application

[Al2Klimov]

Fellows, it is a pleasure to report that we don't have to re-invent the wheel. This works even in Icinga2 console:

<1> => get_template
{
	arguments = [ "type", "name" ]
	deprecated = false
	name = "System#get_template"
	side_effect_free = true
	type = "Function"
}
<2> => get_template(CheckCommand, "plugin-check-command")
{
	location = {
		first_column = 2.000000
		first_line = 18.000000
		last_column = 94.000000
		last_line = 18.000000
		path = "methods-itl.conf"
	}
	name = "plugin-check-command"
	type = "CheckCommand"
}
<3> => get_template(CheckCommand, "ifw-api-check-command")
null
<4> =>

@LordHepipud You already have a patched Director, please test the condition under <3> =>

[julianbrost]

Overall state of the PR: again, fine for me if fine for @LordHepipud, espcially:

get_template(CheckCommand, "ifw-api-check-command")

Somewhat of an implementation detail, but already exposed on /v1/templates/checkcommands anyways. So if that works for the planned use in Director, I guess that's fine for me.

[LordHepipud]

Hello,

Thanks for the input. I just patched the Icinga Director with the code provided and it works perfectly fine.
Looks good to me 👍