Skip to content

Commit

Permalink
Merge pull request #37 from bats-core/#34
Browse files Browse the repository at this point in the history 
#34 Support regex validation of property values
  • Loading branch information
@vincent-zurczak
vincent-zurczak authored Aug 8, 2022
2 parents e56f901 + 7927bb1 commit 0d71702
Show file tree
Hide file tree
Showing 18 changed files with 1,484 additions and 97 deletions.
75 changes: 69 additions & 6 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ This kind of test is the ultimate set of verifications to run for a project, lon
- [Syntax Reference](#syntax-reference)
- [Counting Resources](#counting-resources)
- [Verifying Property Values](#verifying-property-values)
- [Using Regular Expressions](#using-regular-expressions)
- [Property Names](#property-names)
- [Errors](#errors)
- [Error Codes](#error-codes)
Expand Down Expand Up @@ -167,6 +168,11 @@ try "at most 5 times every 30s " \
try at most 5 times every 30s \
to get svc named "'nginx'" \
and verify that "'.spec.ports[*].targetPort'" is "'8484'"

# Regular expressions can also be used
try at most 5 times every 30s \
to get svc named 'nginx' \
and verify that '.spec.ports[*].targetPort' matches '[[:digit:]]+'
```

If you work with OpenShift and would prefer to use **oc** instead of **kubectl**...
Expand Down Expand Up @@ -304,9 +310,7 @@ try "at most <number> times every <number>s \
For services, you may directly use the simple count assertions.

This is a checking loop.
It breaks the loop if as soon as the assertion is verified. If it reaches the end of the loop
without having been verified, an error is thrown. Please, refer to [this section](#property-names) for details
about the property names.
It breaks the loop if as soon as the assertion is verified. If it reaches the end of the loop without having been verified, an error is thrown. Please, refer to [this section](#property-names) for details about the property names.


### Verifying Property Values
Expand Down Expand Up @@ -334,6 +338,65 @@ about the property names.
But unlike the assertion type to [count resources](#counting-resources), you do not verify _how many instances_ have this value. Notice however that **if it finds 0 item verifying the property, the assertion fails**.


### Using Regular Expressions

It is also possible to verify property values against a regular expression.
This can be used, as an example, to verify values in a JSON array.

```bash
# Verifying a property
verify "'<property-name>' matches '<regular-experession>' for <resource-type> named '<regular-expression>'"

# Finding elements with a matching property
try "at most <number> times every <number>s \
to get <resource-type> named '<regular-expression>' \
and verify that '<property-name>' matches '<regular-experession>'"

# Counting elements with a matching property
try "at most <number> times every <number>s \
to find <number> <resource-type> named '<regular-expression>' \
with '<property-name>' matching '<regular-expression>'"
```

The regular expression used for property values relies on
[BASH regexp](https://en.wikibooks.org/wiki/Regular_Expressions/POSIX-Extended_Regular_Expressions).
More exactly, it uses extended regular expressions (EREs). You can simulate the result of such an assertion
with `grep`, as it is the command used internally. Hence, you can use `echo your-value | grep -E your-regex`
to prepare your assertions.

> Unlike the assertions with the verb « to be », those with the verb « to match » are case-sensitive.
All the assertions using the verb « to be » make case-insensitive comparison.
It means writing `is 'running'` or `is 'Running'` does not change anything.
If you want case-sensitive equality, then use a regular expression, i.e. write
`matches '^Running$'`.

If for some reasons, one needs case-insensitive matches, you can set the DETIK_CASE_INSENSITIVE_PROPERTIES
property to `true` in your test. All the retrieved values by the DETIK_CLIENT will be lower-cased. It means
you can write a pattern that only considers lower-case characters. The following sample illustrates this situation:

```bash
# Assuming the status of the POD is "Running"...
# ... then the following assertion will fail.
verify "'status' matches 'running' for pods named 'nginx'"

# Same for...
verify "'status' matches '^running$' for pods named 'nginx'"

# This is because the value returned by the client starts with an upper-case letter.
# For case-insensivity operations with a regular expression, just use...
DETIK_CASE_INSENSITIVE_PROPERTIES="true"
verify "'status' matches 'running' for pods named 'nginx'"

# The assertion will now be verified.
# Just make sure the pattern ONLY INCLUDES lower-case characters.

# If you set DETIK_CASE_INSENSITIVE_PROPERTIES directly in a "@test" function,
# there is no need to reset it for the other tests. Its scope is limited to the
# function that defines it. It is recommended to NOT make this variable a global one.
```


### Property Names

In all assertions, *property-name* is one of the column names supported by K8s.
Expand Down Expand Up @@ -428,9 +491,9 @@ DEBUG_DETIK=""

### Linting

Because Bash is not a compiled language, it is easy to make mistakes.
Even if the library was designed to be simple. This is why a linter was created, to help to
locate syntax errors when writing DETIK assertions. You can use it with BATS in your tests.
Despite the efforts to make the DETIK syntax as simple as possible, BASH remains a non-compiled
language and mistakes happen. To prevent them, a linter was created to help locating
syntax errors when writing DETIK assertions. You can use it with BATS in your tests.

```bash
#!/usr/bin/env bats
Expand Down
104 changes: 84 additions & 20 deletions lib/detik.bash
View file
Original file line number Diff line number Diff line change
Expand Up @@ -38,18 +38,30 @@ try() {
property=""
expected_value=""
expected_count=""
verify_strict_equality="true"

if [[ "$exp" =~ $try_regex_verify ]]; then
if [[ "$exp" =~ $try_regex_verify_is ]]; then

# Extract parameters
times="${BASH_REMATCH[1]}"
delay="${BASH_REMATCH[2]}"
resource=$(to_lower_case "${BASH_REMATCH[3]}")
name="${BASH_REMATCH[4]}"
property="${BASH_REMATCH[5]}"
expected_value=$(to_lower_case "${BASH_REMATCH[6]}")
expected_value="${BASH_REMATCH[6]}"

elif [[ "$exp" =~ $try_regex_find ]]; then
elif [[ "$exp" =~ $try_regex_verify_matches ]]; then

# Extract parameters
times="${BASH_REMATCH[1]}"
delay="${BASH_REMATCH[2]}"
resource=$(to_lower_case "${BASH_REMATCH[3]}")
name="${BASH_REMATCH[4]}"
property="${BASH_REMATCH[5]}"
expected_value="${BASH_REMATCH[6]}"
verify_strict_equality="false"

elif [[ "$exp" =~ $try_regex_find_being ]]; then

# Extract parameters
times="${BASH_REMATCH[1]}"
Expand All @@ -58,7 +70,19 @@ try() {
resource=$(to_lower_case "${BASH_REMATCH[4]}")
name="${BASH_REMATCH[5]}"
property="${BASH_REMATCH[6]}"
expected_value=$(to_lower_case "${BASH_REMATCH[7]}")
expected_value="${BASH_REMATCH[7]}"

elif [[ "$exp" =~ $try_regex_find_matching ]]; then

# Extract parameters
times="${BASH_REMATCH[1]}"
delay="${BASH_REMATCH[2]}"
expected_count="${BASH_REMATCH[3]}"
resource=$(to_lower_case "${BASH_REMATCH[4]}")
name="${BASH_REMATCH[5]}"
property="${BASH_REMATCH[6]}"
expected_value="${BASH_REMATCH[7]}"
verify_strict_equality="false"
fi

# Do we have something?
Expand All @@ -73,7 +97,7 @@ try() {
for ((i=1; i<=times; i++)); do

# Verify the value
verify_value "$property" "$expected_value" "$resource" "$name" "$expected_count" && code=$? || code=$?
verify_value "$verify_strict_equality" "$property" "$expected_value" "$resource" "$name" "$expected_count" && code=$? || code=$?

# Break the loop prematurely?
if [[ "$code" == "0" ]]; then
Expand Down Expand Up @@ -158,7 +182,20 @@ verify() {
name="${BASH_REMATCH[4]}"

echo "Valid expression. Verification in progress..."
verify_value "$property" "$expected_value" "$resource" "$name"
verify_value true "$property" "$expected_value" "$resource" "$name"

if [[ "$?" != "0" ]]; then
return 3
fi

elif [[ "$exp" =~ $verify_regex_property_matches ]]; then
property="${BASH_REMATCH[1]}"
expected_value="${BASH_REMATCH[2]}"
resource=$(to_lower_case "${BASH_REMATCH[3]}")
name="${BASH_REMATCH[4]}"

echo "Valid expression. Verification in progress..."
verify_value false "$property" "$expected_value" "$resource" "$name"

if [[ "$?" != "0" ]]; then
return 3
Expand All @@ -172,6 +209,7 @@ verify() {


# Verifies the value of a column for a set of elements.
# @param {boolean} true to verify equality, false to match a regex
# @param {string} A K8s column or one of the supported aliases.
# @param {string} The expected value.
# @param {string} The resouce type (e.g. pod).
Expand All @@ -183,11 +221,12 @@ verify() {
verify_value() {

# Make the parameters readable
property="$1"
expected_value=$(to_lower_case "$2")
resource="$3"
name="$4"
expected_count="$5"
verify_strict_equality=$(to_lower_case "$1")
property="$2"
expected_value="$3"
resource="$4"
name="$5"
expected_count="$6"

# List the items and remove the first line (the one that contains the column names)
query=$(build_k8s_request "$property")
Expand Down Expand Up @@ -224,22 +263,47 @@ verify_value() {
for line in $result; do

# Keep the second column (property to verify)
# and put it in lower case
value=$(to_lower_case "$line" | awk '{ print $2 }')
value=$(echo "$line" | awk '{ print $2 }')
element=$(echo "$line" | awk '{ print $1 }')
if [[ "$value" != "$expected_value" ]]; then
echo "Current value for $element is $value..."
invalid=$((invalid + 1))

# Compare with an exact value (case insensitive)
if [[ "$verify_strict_equality" == "true" ]]; then
value=$(to_lower_case "$value")
expected_value=$(to_lower_case "$expected_value")
if [[ "$value" != "$expected_value" ]]; then
echo "Current value for $element is $value..."
invalid=$((invalid + 1))
else
echo "$element has the right value ($value)."
valid=$((valid + 1))
fi

# Verify a regex (we preserve the case)
else
echo "$element has the right value ($value)."
valid=$((valid + 1))
# We do not want another syntax for case-insensitivity
if [ "$DETIK_REGEX_CASE_INSENSITIVE_PROPERTIES" = "true" ]; then
value=$(to_lower_case "$value")
fi

reg=$(echo "$value" | grep -E "$expected_value")
if [[ "$?" -ne 0 ]]; then
echo "Current value for $element is $value..."
invalid=$((invalid + 1))
else
echo "$element matches the regular expression (found $reg)."
valid=$((valid + 1))
fi
fi
done

# Do we have the right number of elements?
if [[ "$expected_count" != "" ]]; then
if [[ "$valid" != "$expected_count" ]]; then
echo "Expected $expected_count $resource named $name to have this value ($expected_value). Found $valid."
if [[ "$verify_strict_equality" == "true" ]]; then
echo "Expected $expected_count $resource named $name to have this value ($expected_value). Found $valid."
else
echo "Expected $expected_count $resource named $name to match this pattern ($expected_value). Found $valid."
fi
invalid=101
else
invalid=0
Expand Down Expand Up @@ -279,7 +343,7 @@ build_k8s_client_with_options() {

client_with_options="$DETIK_CLIENT_NAME"
if [[ -n "$DETIK_CLIENT_NAMESPACE" ]]; then
# eval does not like '-n'
# eval does not "like" the '-n' syntax
client_with_options="$DETIK_CLIENT_NAME --namespace=$DETIK_CLIENT_NAMESPACE"
fi

Expand Down
31 changes: 23 additions & 8 deletions lib/linter.bash
View file
Original file line number Diff line number Diff line change
Expand Up @@ -132,15 +132,24 @@ check_line() {
part=$(clean_regex_part "${BASH_REMATCH[2]}")
context="$context\nRegex part: $part"

verify_against_pattern "$part" "$try_regex_verify"
p_verify="$?"
verify_against_pattern "$part" "$try_regex_verify_is"
p_verify_is="$?"

verify_against_pattern "$part" "$try_regex_find"
p_find="$?"
verify_against_pattern "$part" "$try_regex_verify_matches"
p_verify_matches="$?"

verify_against_pattern "$part" "$try_regex_find_being"
p_find_being="$?"

verify_against_pattern "$part" "$try_regex_find_matching"
p_find_matching="$?"

# detik_debug "p_verify=$p_verify, p_find=$p_find, part=$part"
if [[ "$p_verify" != "0" ]] && [[ "$p_find" != "0" ]]; then
handle_error "Invalid TRY statement at line $line_number." "$context"
if [[ "$p_verify_is" != "0" ]] && \
[[ "$p_verify_matches" != "0" ]] && \
[[ "$p_find_being" != "0" ]] && \
[[ "$p_find_matching" != "0" ]]; then
handle_error "Invalid TRY statement at line $line_number." "$context"
fi

# We have "verify" or "run verify" followed by something
Expand All @@ -159,9 +168,15 @@ check_line() {
verify_against_pattern "$part" "$verify_regex_property_is"
p_prop="$?"

verify_against_pattern "$part" "$verify_regex_property_matches"
p_matches="$?"

# detik_debug "p_is=$p_is, p_are=$p_are, p_prop=$p_prop, part=$part"
if [[ "$p_is" != "0" ]] && [[ "$p_are" != "0" ]] && [[ "$p_prop" != "0" ]] ; then
handle_error "Invalid VERIFY statement at line $line_number." "$context"
if [[ "$p_is" != "0" ]] && \
[[ "$p_are" != "0" ]] && \
[[ "$p_prop" != "0" ]] && \
[[ "$p_matches" != "0" ]] ; then
handle_error "Invalid VERIFY statement at line $line_number." "$context"
fi
fi
}
Expand Down
17 changes: 14 additions & 3 deletions lib/utils.bash
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,16 @@


# The regex for the "try" key word
try_regex_verify="^at +most +([0-9]+) +times +every +([0-9]+)s +to +get +([a-z]+) +named +'([^']+)' +and +verify +that +'([^']+)' +is +'([^']+)'$"
try_regex_find="^at +most +([0-9]+) +times +every +([0-9]+)s +to +find +([0-9]+) +([a-z]+) +named +'([^']+)' +with +'([^']+)' +being +'([^']+)'$"
try_regex_verify_is="^at +most +([0-9]+) +times +every +([0-9]+)s +to +get +([a-z]+) +named +'([^']+)' +and +verify +that +'([^']+)' +is +'([^']+)'$"
try_regex_verify_matches="^at +most +([0-9]+) +times +every +([0-9]+)s +to +get +([a-z]+) +named +'([^']+)' +and +verify +that +'([^']+)' +matches +'([^']+)'$"
try_regex_find_being="^at +most +([0-9]+) +times +every +([0-9]+)s +to +find +([0-9]+) +([a-z]+) +named +'([^']+)' +with +'([^']+)' +being +'([^']+)'$"
try_regex_find_matching="^at +most +([0-9]+) +times +every +([0-9]+)s +to +find +([0-9]+) +([a-z]+) +named +'([^']+)' +with +'([^']+)' +matching +'([^']+)'$"

# The regex for the "verify" key word
verify_regex_count_is="^there +is +(0|1) +([a-z]+) +named +'([^']+)'$"
verify_regex_count_are="^there +are +([0-9]+) +([a-z]+) +named +'([^']+)'$"
verify_regex_property_is="^'([^']+)' +is +'([^']+)' +for +([a-z]+) +named +'([^']+)'$"

verify_regex_property_matches="^'([^']+)' +matches +'([^']+)' +for +([a-z]+) +named +'([^']+)'$"


# Prints a string in lower case.
Expand Down Expand Up @@ -73,3 +75,12 @@ reset_detik_debug() {
reset_debug
fi
}


# Dumps the argument and return the previous error code.
# @return the previous error code
ddump() {
res="$?"
echo "$1"
return $res
}
Loading

0 comments on commit 0d71702

Please sign in to comment.