Prometheus add nodes gauge for SQS mode

[phuhung273]

Issue #, if available:
Close #785

Description of changes:

• Count nodes/instances being tracked. Eg:
  k get node return 5
  aws ec2 describe-instances return 2
  Identify 3 nodes no longer under NTH control

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[phuhung273]

@stevehipwell can you please give me a help on this. Now i only stuck at writing unit test for opentelemetry.go since it doesn
t have any test.

Do you think i should refactor this file along with this PR or should I open another separate PR (eg: make opentelemetry.go testable)

[stevehipwell]

@phuhung273 I'm not a maintainer here but I like to see untestable code refactored to be testable, following the boy scout rule. If you do the work in this PR you can always split the refactoring to a separate PR before it's merged.

@LikithaVemulapalli what do you think?

[LikithaVemulapalli]

Yes I agree with @stevehipwell here, let's separate refactoring PR to have a clear idea on the changes made. @phuhung273 if you want to test your changes for this PR let me know I will approve and run so for the future commits you can verify if the existing tests are working, if there are any conflicts or not, once you change the PR status to ready I will run the workflow, appreciate for your contribution. Thanks!

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want this PR to never become stale, please ask a maintainer to apply the "stalebot-ignore" label.

[phuhung273]

/remove-lifecycle stale

[github-actions]

This PR has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you want this PR to never become stale, please ask a maintainer to apply the "stalebot-ignore" label.

[tiationg-kho]

Hi @phuhung273, could you resolve the conflicts in pkg/node/node_test.go?

[phuhung273]

Thanks and Happy new year @tiationg-kho. Conflicts resolved

[tiationg-kho]

Hi @phuhung273,

Thanks for resolving the conflicts. I have left some comments.
Also, we can run the e2e test in local (docker desktop) to make sure our modification is valid.

• Run all e2e test cases: make e2e-test
• Run certain e2e test case: ./test/k8s-local-cluster-test/run-test -a ./test/e2e/<test-case> -d -b e2e-test

[tiationg-kho]

Could we add a filter for instance state here?

[phuhung273]

Of course we can, but I wonder if we should ? I saw cases where instances enter Stopped state instead of Terminated . Without filterring user can discover such case.

[tiationg-kho]

Thanks for addressing the comments.

Maybe we could rename nodesGauge and instancesGauge to let these 2 metrics more self-explain?

Then we could decide we need a filter here or not. And would also know should separate these 2 metrics, or filter one based on another (opentelemetry.go).

[tiationg-kho]

We could add nil check for both results. (instanceIdsMap, nodeInstanceIds)

We should not use instanceIdsMap in second block if we got error from GetInstanceIdsMapByTagKey.

Consider we filter the nodes result based on instances result, we would not have a chance to record any result (nodes > instances). Do you think this is a potential issue?

[phuhung273]

Thanks, I have addressed the 2 nil check.

About the edge cases, I did think of if but could not find anyway to do 2 metrics separately:

• For instances we already have a NTH managed tag so we can filter base on it
• For nodes i cannot see any label/annotation that we can filter upon

Therefore, I decided to filter nodes based on instances result. Do you know any info we can use to filter nodes independently