added test for multi-container pod

Author: harshit-splunk

test/README.md

@@ -26,5 +26,7 @@
 
 # How metric tests works
 Test collects test data from metric_data, each data contains metric name, field selector for metric and list of asserions.
-For example, for `kube.node.uptime` test, selector value is `node`. So test will search `| mstats max(kube.node.uptime) where index=ci_metric by node`. It will collect uptime value for each node and test will assert each values to be greater than zero.
-All selector fields are listed in [selector.conf](https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/test/k8s_metrics_tests/metric_data/selector.conf) file.
+
+Generic search query used in metric tests: `| mstats max(<metric-name>) where index=<index-name> by <selector field>`
+
+All selector fields are listed in [selector.md](https://github.com/splunk/splunk-connect-for-kubernetes/blob/develop/test/k8s_metrics_tests/metric_data/selector.md)

test/k8s_metrics_tests/metric_data/selector.conf

@@ -0,0 +1,19 @@
+This file is for documentation purposes only. It contains which selector field used in each test
+
+
+
+| Metric type | selector for stats | selector for summary | selector for cAdvisor | selector for aggregated |
+| ----------- | ------------------ | -------------------- | --------------------- | ----------------------- |
+| container   | -                  | container-name       | container_name        | name                    |
+| pod         | -                  | pod-name             | pod_name              | name                    |
+| node        | node               | node                 | -                     | node                    |
+| namespace   | -                  | -                    | -                     | name                    |
+| cluster     | -                  | -                    | -                     | name                    |
+
+Selector field suggest unique identifier field for given metric. 
+Here, `"-"` means the given type of metric doesn't exist in metric source(i.e stats/summary/cAdvisor)
+
+For example, `kube.container.uptime` summary metric suggest container uptime. Each containers will have different uptime. So, we can group all `kube.container.uptime` by container's name. Summary scrapper will set the container's name as `container-name`. For this metric, since it is container metric and it's coming from summary api, the selector field will be `container-name`. 
+
+Test will search following query in splunk for example test: `| mstats max(kube.container.cpu.uptime) where index=<index> by container-name`
+

test/k8s_metrics_tests/metric_data/selector.md

@@ -185,3 +185,39 @@ def test_namespace_limit_and_requests(setup, index_metrics):
                 + actual_data["splunk-fluentd-k8s-objects"]
                 + actual_data["splunk-fluentd-k8s-metrics-agg"]
             )
+
+def test_multi_container_pod_limit_and_request(setup, index_metrics):
+    selector = "name"
+    for resource in ["cpu", "memory"]:
+        for metric in ["limit", "request"]:
+            logger.info(
+                "testing {} metric".format(f"kube.container.{resource}.{metric} for multi-container pods")
+            )
+            container_events = collect_metric_from_splunk(
+                f"kube.container.{resource}.{metric}",
+                index_metrics,
+                selector,
+                url=setup["splunkd_url"],
+                user=setup["splunk_user"],
+                password=setup["splunk_password"],
+                func="avg",
+            )
+            if not "pod-wo-index-wo-ns-index-dup" in container_events:
+                pytest.fail("pod 'pod-wo-index-wo-ns-index-dup' not found" )
+
+            assert container_events["pod-wo-index-wo-ns-index"] == 50
+            assert container_events["pod-wo-index-wo-ns-index-dup"] == 50
+            
+            pod_events = collect_metric_from_splunk(
+                f"kube.pod.{resource}.{metric}",
+                index_metrics,
+                selector,
+                url=setup["splunkd_url"],
+                user=setup["splunk_user"],
+                password=setup["splunk_password"],
+                func="avg",
+            )
+
+            for pod_name, metric_value in pod_events.items():
+                if "pod-wo-index-wo-ns-index" in pod_name:
+                    assert metric_value == 100, f"{resource}.{metric} of pod '{pod_name}' should be 100"

test/k8s_metrics_tests/test_metric_aggr_plugin.py

@@ -145,4 +145,20 @@ spec:
     spec:
       containers:
       - name: pod-wo-index-wo-ns-index
-        image: rock1017/log-generator:2.2.6
+        image: rock1017/log-generator:2.2.6
+        resources:
+          limits:
+           cpu: 50m
+           memory: 50Mi
+          requests:
+            cpu: 50m
+            memory: 50Mi
+      - name: pod-wo-index-wo-ns-index-dup
+        image: rock1017/log-generator:2.2.6
+        resources:
+          limits:
+           cpu: 50m
+           memory: 50Mi
+          requests:
+            cpu: 50m
+            memory: 50Mi