Feature/more assertions (rancher#6, rancher#11, helm-unittest#44, hel…

…m-unittest#52, helm-unittest#74) (rancher#12)

* Add IsSubsetValidator, including test verification (rancher#6).

* Add raw assertions for NOTES.txt (rancher#11), including tests.

* Fix resolving and running templates in subfolders, including test verification. (helm-unittest#44)

* Add Counter to contains assertion to validate the number of times a value appears in an array. (helm-unittest#52)

* Add Any (boolean) to contains assertion to ignore the other values to be validated within an array. Added testverification and updated documentation. (helm-unittest#74)

* Update Makefile with unittest, add additional logging when a test fails in test_runner.

* Uniform Raw output. Update snapshots with correct line endings.

* Fix matchSnapshotRaw tests and improve errorhandling. Testresult fails with error when loading is failed, instead of ignoring the handling.

* Add buffer when a testrunner test failes.

* Add Counter to contains assertion to validate the number of times a value appears in an array. (helm-unittest#52)

* Add IsSubsetValidator, including test verification (rancher#6).

* Improve testabillity and lower code duplication.

DOCUMENT.md

@@ -114,7 +114,7 @@ Available assertion types are listed below:
 | `notEqual` | **path**: *string*. The `set` path to assert.<br/>**value**: *any*. The value expected not to be. | Assert the value of specified **path** NOT equal to the **value**. | <pre>notEqual:<br/>  path: metadata.name<br/>  value: my-deploy</pre> |
 | `matchRegex` | **path**: *string*. The `set` path to assert, the value must be a *string*. <br/>**pattern**: *string*. The regex pattern to match (without quoting `/`). | Assert the value of specified **path** match **pattern**. | <pre>matchRegex:<br/>  path: metadata.name<br/>  pattern: -my-chart$</pre> |
 | `notMatchRegex` | **path**: *string*. The `set` path to assert, the value must be a *string*. <br/>**pattern**: *string*. The regex pattern NOT to match (without quoting `/`). | Assert the value of specified **path** NOT match **pattern**. | <pre>notMatchRegex:<br/>  path: metadata.name<br/>  pattern: -my-chat$</pre> |
-| `contains` | **path**: *string*. The `set` path to assert, the value must be an *array*. <br/>**content**: *any*. The content to be contained. | Assert the array as the value of specified **path** contains the **content**. |<pre>contains:<br/>  path: spec.ports<br/>  content:<br/>    name: web<br/>    port: 80<br/>    targetPort: 80<br/>    protocle:TCP</pre> |
+| `contains` | **path**: *string*. The `set` path to assert, the value must be an *array*. <br/>**content**: *any*. The content to be contained.<br/>**count**: *int, optional*. The count of content to be contained.<br/>**any**: *bool, optional*. ignores any other values within the found content. | Assert the array as the value of specified **path** contains the **content**. |<pre>contains:<br/>  path: spec.ports<br/>  content:<br/>    name: web<br/>    port: 80<br/>    targetPort: 80<br/>    protocle:TCP</pre> |
 | `notContains` | **path**: *string*. The `set` path to assert, the value must be an *array*. <br/>**content**: *any*. The content NOT to be contained. | Assert the array as the value of specified **path** NOT contains the **content**. |<pre>notContains:<br/>  path: spec.ports<br/>  content:<br/>    name: server<br/>    port: 80<br/>    targetPort: 80<br/>    protocle: TCP</pre> |
 | `isNull` | **path**: *string*. The `set` path to assert. | Assert the value of specified **path** is `null`. |<pre>isNull:<br/>  path: spec.strategy</pre> |
 | `isNotNull` | **path**: *string*. The `set` path to assert. | Assert the value of specified **path** is NOT `null`. |<pre>isNotNull:<br/>  path: spec.replicas</pre> |

Makefile

@@ -16,8 +16,12 @@ install: bootstrap build
 .PHONY: hookInstall
 hookInstall: bootstrap build
 
+.PHONY: unittest
+unittest:
+	go test ./unittest/... -v -cover
+
 .PHONY: build
-build:
+build: unittest
 	go build -o untt -ldflags $(LDFLAGS) ./main.go
 
 .PHONY: dist

__fixtures__/v2/basic/tests/__snapshot__/notes_test.yaml.snap

@@ -0,0 +1,20 @@
+should pass the notes file with ingress enabled:
+  1: |
+    |
+      1. Get the application URL by running these commands:
+        http://chart-example.local
+should pass the notes file with service type LoadBalancer:
+  1: |
+    |
+      1. Get the application URL by running these commands:
+           NOTE: It may take a few minutes for the LoadBalancer IP to be available.
+                 You can watch the status of by running 'kubectl get svc -w RELEASE-NAME-basic'
+        export SERVICE_IP=$(kubectl get svc --namespace NAMESPACE RELEASE-NAME-basic -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+        echo http://$SERVICE_IP:9999
+should pass the notes file with service type NodePort:
+  1: |
+    |
+      1. Get the application URL by running these commands:
+        export NODE_PORT=$(kubectl get --namespace NAMESPACE -o jsonpath="{.spec.ports[0].nodePort}" services RELEASE-NAME-basic)
+        export NODE_IP=$(kubectl get nodes --namespace NAMESPACE -o jsonpath="{.items[0].status.addresses[0].address}")
+        echo http://$NODE_IP:$NODE_PORT

__fixtures__/v2/basic/tests/notes_test.yaml

@@ -0,0 +1,35 @@
+suite: test notes
+templates:
+  - NOTES.txt
+tests:
+  - it: should pass the notes file with ingress enabled
+    set:
+      ingress.enabled: true
+    asserts:
+      - equalRaw:
+          value: | 
+            1. Get the application URL by running these commands:
+              http://chart-example.local
+      - matchSnapshotRaw: {}
+
+  - it: should pass the notes file with service type NodePort
+    set:
+      service.type: NodePort
+    asserts:
+      - equalRaw:
+          value: | 
+            1. Get the application URL by running these commands:
+              export NODE_PORT=$(kubectl get --namespace NAMESPACE -o jsonpath="{.spec.ports[0].nodePort}" services RELEASE-NAME-basic)
+              export NODE_IP=$(kubectl get nodes --namespace NAMESPACE -o jsonpath="{.items[0].status.addresses[0].address}")
+              echo http://$NODE_IP:$NODE_PORT
+      - matchSnapshotRaw: {}
+
+  - it: should pass the notes file with service type LoadBalancer
+    set:
+      service.type: LoadBalancer
+      service.externalPort: 9999
+    asserts:
+      - matchRegexRaw:
+          pattern: http://\$SERVICE_IP:9999
+      - matchSnapshotRaw: {}
+

__fixtures__/v2/basic/tests_failed/notes_test.yaml

@@ -0,0 +1,31 @@
+suite: test notes
+templates:
+  - NOTES.txt
+tests:
+  - it: should pass the notes file with ingress enabled
+    set:
+      ingress.enabled: true
+    asserts:
+      - notEqualRaw:
+          value: | 
+            1. Get the application URL by running these commands:
+              http://chart-example.local
+
+  - it: should pass the notes file with service type NodePort
+    set:
+      service.type: NodePort
+    asserts:
+      - notEqualRaw:
+          value: | 
+            1. Get the application URL by running these commands:
+              export NODE_PORT=$(kubectl get --namespace NAMESPACE -o jsonpath="{.spec.ports[0].nodePort}" services MY-RELEASE)
+              export NODE_IP=$(kubectl get nodes --namespace NAMESPACE -o jsonpath="{.items[0].status.addresses[0].address}")
+              echo http://$NODE_IP:$NODE_PORT
+              
+  - it: should pass the notes file with service type LoadBalancer
+    set:
+      service.type: LoadBalancer
+      service.externalPort: 9999
+    asserts:
+      - matchRegexRaw:
+          pattern: http://\$SERVICE_IP:80

__fixtures__/v2/with-subchart/charts/child-chart/tests/__snapshot__/notes_test.yaml.snap

@@ -0,0 +1,20 @@
+should pass the notes file with ingress enabled:
+  1: |
+    |
+      1. Get the application URL by running these commands:
+        http://chart-example.local
+should pass the notes file with service type LoadBalancer:
+  1: |
+    |
+      1. Get the application URL by running these commands:
+           NOTE: It may take a few minutes for the LoadBalancer IP to be available.
+                 You can watch the status of by running 'kubectl get svc -w RELEASE-NAME-child-chart'
+        export SERVICE_IP=$(kubectl get svc --namespace NAMESPACE RELEASE-NAME-child-chart -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+        echo http://$SERVICE_IP:9999
+should pass the notes file with service type NodePort:
+  1: |
+    |
+      1. Get the application URL by running these commands:
+        export NODE_PORT=$(kubectl get --namespace NAMESPACE -o jsonpath="{.spec.ports[0].nodePort}" services RELEASE-NAME-child-chart)
+        export NODE_IP=$(kubectl get nodes --namespace NAMESPACE -o jsonpath="{.items[0].status.addresses[0].address}")
+        echo http://$NODE_IP:$NODE_PORT

__fixtures__/v2/with-subchart/charts/child-chart/tests/notes_test.yaml

@@ -0,0 +1,35 @@
+suite: test notes
+templates:
+  - NOTES.txt
+tests:
+  - it: should pass the notes file with ingress enabled
+    set:
+      ingress.enabled: true
+    asserts:
+      - equalRaw:
+          value: | 
+            1. Get the application URL by running these commands:
+              http://chart-example.local
+      - matchSnapshotRaw: {}
+
+  - it: should pass the notes file with service type NodePort
+    set:
+      service.type: NodePort
+    asserts:
+      - equalRaw:
+          value: | 
+            1. Get the application URL by running these commands:
+              export NODE_PORT=$(kubectl get --namespace NAMESPACE -o jsonpath="{.spec.ports[0].nodePort}" services RELEASE-NAME-child-chart)
+              export NODE_IP=$(kubectl get nodes --namespace NAMESPACE -o jsonpath="{.items[0].status.addresses[0].address}")
+              echo http://$NODE_IP:$NODE_PORT
+      - matchSnapshotRaw: {}
+
+  - it: should pass the notes file with service type LoadBalancer
+    set:
+      service.type: LoadBalancer
+      service.externalPort: 9999
+    asserts:
+      - matchRegexRaw:
+          pattern: http://\$SERVICE_IP:9999
+      - matchSnapshotRaw: {}
+

__fixtures__/v2/with-subchart/templates/NOTES.txt

@@ -4,16 +4,16 @@
   http://{{ . }}
 {{- end }}
 {{- else if contains "NodePort" .Values.service.type }}
-  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ template "parant-chart.fullname" . }})
+  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ template "parent-chart.fullname" . }})
   export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
   echo http://$NODE_IP:$NODE_PORT
 {{- else if contains "LoadBalancer" .Values.service.type }}
      NOTE: It may take a few minutes for the LoadBalancer IP to be available.
-           You can watch the status of by running 'kubectl get svc -w {{ template "parant-chart.fullname" . }}'
-  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ template "parant-chart.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+           You can watch the status of by running 'kubectl get svc -w {{ template "parent-chart.fullname" . }}'
+  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ template "parent-chart.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
   echo http://$SERVICE_IP:{{ .Values.service.externalPort }}
 {{- else if contains "ClusterIP" .Values.service.type }}
-  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app={{ template "parant-chart.name" . }},release={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
+  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app={{ template "parent-chart.name" . }},release={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
   echo "Visit http://127.0.0.1:8080 to use your application"
   kubectl port-forward $POD_NAME 8080:{{ .Values.service.internalPort }}
 {{- end }}

__fixtures__/v2/with-subchart/tests/__snapshot__/notes_test.yaml.snap

@@ -0,0 +1,20 @@
+should pass the notes file with ingress enabled:
+  1: |
+    |
+      1. Get the application URL by running these commands:
+        http://chart-example.local
+should pass the notes file with service type LoadBalancer:
+  1: |
+    |
+      1. Get the application URL by running these commands:
+           NOTE: It may take a few minutes for the LoadBalancer IP to be available.
+                 You can watch the status of by running 'kubectl get svc -w RELEASE-NAME-parent-chart'
+        export SERVICE_IP=$(kubectl get svc --namespace NAMESPACE RELEASE-NAME-parent-chart -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+        echo http://$SERVICE_IP:9999
+should pass the notes file with service type NodePort:
+  1: |
+    |
+      1. Get the application URL by running these commands:
+        export NODE_PORT=$(kubectl get --namespace NAMESPACE -o jsonpath="{.spec.ports[0].nodePort}" services RELEASE-NAME-parent-chart)
+        export NODE_IP=$(kubectl get nodes --namespace NAMESPACE -o jsonpath="{.items[0].status.addresses[0].address}")
+        echo http://$NODE_IP:$NODE_PORT

__fixtures__/v2/with-subchart/tests/notes_test.yaml

@@ -0,0 +1,35 @@
+suite: test notes
+templates:
+  - NOTES.txt
+tests:
+  - it: should pass the notes file with ingress enabled
+    set:
+      ingress.enabled: true
+    asserts:
+      - equalRaw:
+          value: | 
+            1. Get the application URL by running these commands:
+              http://chart-example.local
+      - matchSnapshotRaw: {}
+
+  - it: should pass the notes file with service type NodePort
+    set:
+      service.type: NodePort
+    asserts:
+      - equalRaw:
+          value: | 
+            1. Get the application URL by running these commands:
+              export NODE_PORT=$(kubectl get --namespace NAMESPACE -o jsonpath="{.spec.ports[0].nodePort}" services RELEASE-NAME-parent-chart)
+              export NODE_IP=$(kubectl get nodes --namespace NAMESPACE -o jsonpath="{.items[0].status.addresses[0].address}")
+              echo http://$NODE_IP:$NODE_PORT
+      - matchSnapshotRaw: {}
+
+  - it: should pass the notes file with service type LoadBalancer
+    set:
+      service.type: LoadBalancer
+      service.externalPort: 9999
+    asserts:
+      - matchRegexRaw:
+          pattern: http://\$SERVICE_IP:9999
+      - matchSnapshotRaw: {}
+

__fixtures__/v2/with-subfolder/.helmignore

@@ -0,0 +1,23 @@
+# Patterns to ignore when building packages.
+# This supports shell glob matching, relative path matching, and
+# negation (prefixed with !). Only one pattern per line.
+.DS_Store
+# Common VCS dirs
+.git/
+.gitignore
+.bzr/
+.bzrignore
+.hg/
+.hgignore
+.svn/
+# Common backup files
+*.swp
+*.bak
+*.tmp
+*~
+# Various IDEs
+.project
+.idea/
+*.tmproj
+
+tests

__fixtures__/v2/with-subfolder/Chart.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+description: A advanced example chart to demonstrate unittest plugin
+name: with-subfolder
+version: 0.1.0

__fixtures__/v2/with-subfolder/templates/NOTES.txt

@@ -0,0 +1,19 @@
+1. Get the application URL by running these commands:
+{{- if .Values.ingress.enabled }}
+{{- range .Values.ingress.hosts }}
+  http://{{ . }}
+{{- end }}
+{{- else if contains "NodePort" .Values.service.type }}
+  export NODE_PORT=$(kubectl get --namespace {{ .Release.Namespace }} -o jsonpath="{.spec.ports[0].nodePort}" services {{ template "with-subfolder.fullname" . }})
+  export NODE_IP=$(kubectl get nodes --namespace {{ .Release.Namespace }} -o jsonpath="{.items[0].status.addresses[0].address}")
+  echo http://$NODE_IP:$NODE_PORT
+{{- else if contains "LoadBalancer" .Values.service.type }}
+     NOTE: It may take a few minutes for the LoadBalancer IP to be available.
+           You can watch the status of by running 'kubectl get svc -w {{ template "with-subfolder.fullname" . }}'
+  export SERVICE_IP=$(kubectl get svc --namespace {{ .Release.Namespace }} {{ template "with-subfolder.fullname" . }} -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
+  echo http://$SERVICE_IP:{{ .Values.service.externalPort }}
+{{- else if contains "ClusterIP" .Values.service.type }}
+  export POD_NAME=$(kubectl get pods --namespace {{ .Release.Namespace }} -l "app={{ template "with-subfolder.name" . }},release={{ .Release.Name }}" -o jsonpath="{.items[0].metadata.name}")
+  echo "Visit http://127.0.0.1:8080 to use your application"
+  kubectl port-forward $POD_NAME 8080:{{ .Values.service.internalPort }}
+{{- end }}

__fixtures__/v2/with-subfolder/templates/_helpers.tpl

@@ -0,0 +1,16 @@
+{{/* vim: set filetype=mustache: */}}
+{{/*
+Expand the name of the chart.
+*/}}
+{{- define "with-subfolder.name" -}}
+{{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
+{{- end -}}
+
+{{/*
+Create a default fully qualified app name.
+We truncate at 63 chars because some Kubernetes name fields are limited to this (by the DNS naming spec).
+*/}}
+{{- define "with-subfolder.fullname" -}}
+{{- $name := default .Chart.Name .Values.nameOverride -}}
+{{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
+{{- end -}}

__fixtures__/v2/with-subfolder/templates/db/deployment.yaml

@@ -0,0 +1,42 @@
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: {{ template "with-subfolder.fullname" . }}-db
+  labels:
+    app: {{ template "with-subfolder.name" . }}
+    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
+    release: {{ .Release.Name }}
+    heritage: {{ .Release.Service }}
+spec:
+  replicas: 1
+  template:
+    metadata:
+      labels:
+        app: {{ template "with-subfolder.name" . }}
+        release: {{ .Release.Name }}
+      annotations:
+        some_template: |
+          ---
+          apiVersion: ...
+          this: is test for old separator workaround bug
+    spec:
+      containers:
+        - name: {{ .Chart.Name }}
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          ports:
+            - containerPort: {{ .Values.service.dbPort }}
+          livenessProbe:
+            httpGet:
+              path: /
+              port: {{ .Values.service.internalPort }}
+          readinessProbe:
+            httpGet:
+              path: /
+              port: {{ .Values.service.internalPort }}
+          resources:
+{{ toYaml .Values.resources | indent 12 }}
+    {{- if .Values.nodeSelector }}
+      nodeSelector:
+{{ toYaml .Values.nodeSelector | indent 8 }}
+    {{- end }}

__fixtures__/v2/with-subfolder/templates/webserver/deployment.yaml

@@ -0,0 +1,38 @@
+---
+apiVersion: extensions/v1beta1
+kind: Deployment
+metadata:
+  name: {{ template "with-subfolder.fullname" . }}
+  labels:
+    app: {{ template "with-subfolder.name" . }}
+    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
+    release: {{ .Release.Name }}
+    heritage: {{ .Release.Service }}
+spec:
+  replicas: {{ .Values.replicaCount }}
+  template:
+    metadata:
+      labels:
+        app: {{ template "with-subfolder.name" . }}
+        release: {{ .Release.Name }}
+    spec:
+      containers:
+        - name: {{ .Chart.Name }}
+          image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
+          imagePullPolicy: {{ .Values.image.pullPolicy }}
+          ports:
+            - containerPort: {{ .Values.service.internalPort }}
+          livenessProbe:
+            httpGet:
+              path: /
+              port: {{ .Values.service.internalPort }}
+          readinessProbe:
+            httpGet:
+              path: /
+              port: {{ .Values.service.internalPort }}
+          resources:
+{{ toYaml .Values.resources | indent 12 }}
+    {{- if .Values.nodeSelector }}
+      nodeSelector:
+{{ toYaml .Values.nodeSelector | indent 8 }}
+    {{- end }}