Skip to content

Release Testing

Jump to bottom
Erik Jaegervall edited this page Jul 27, 2023 · 22 revisions

This page intends to list the tests that we shall/must do before we do a release for this repository. The intention of the release test is to do a reasonable regression test for components listed as Beta/Production/Stable/Mature in the KUKSA Component Maturity Overview. It is assumed that new features of individual components have been thoroughly tested, the focus of the release test is to verify that there are no regressions. For capacity reasons the tests have no intention to cover all possible use cases.

Prerequisites

  • No outstanding Pull Requests that may affect component behavior.
  • CI Docker builds on latest master has succeeded
  • Pre-release of kuksa-client exists on PyPI

Basic testing with Databroker (Insecure, No TLS)

erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker -- --metadata ../data/vss-core/vss_release_4.0.json --insecure

Python Client test

(kuksa-client) erik@debian3:~/kuksa.val/kuksa-client$ kuksa-client --ip 127.0.0.1 --port 55555 --protocol grpc  --insecure


Test Client> setValue Vehicle.Speed 33
OK

Test Client> getValue Vehicle.Speed

Databroker CLI

  • NOTE: Verify that correct version number is shown in the "hello message!*
erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker-cli
...
sdv.databroker.v1 > feed Vehicle.Speed 44
[feed]  OK  
sdv.databroker.v1 > get Vehicle.Speed
[get]  OK  
Vehicle.Speed: 44.00

GO Client

Start Client

erik@debian3:~/kuksa.val/kuksa_go_client$ go run .

Start Python client (or Databroker CLI) and verify that DTCList has been set by Go client:

Test Client> getValue Vehicle.OBD.DTCList
{
    "path": "Vehicle.OBD.DTCList",
    "value": {
        "value": {
            "values": [
                "dtc1, dtc2",
                "dtc2",
                "dtc3, dtc3",
                "dtc4"
            ]
        },
        "timestamp": "2023-07-20T13:03:46.314802+00:00"
    }
}

Set Vehicle.Speed in Python client and verify that Go client gets a notification

Test Client> setValue Vehicle.Speed 65

2023/07/20 15:04:14 Vehicle.Speed Subscribed: timestamp:{seconds:1689858254 nanos:509989582} float:65

Feeder smoke tests

DBC2VAL

They rely on the client, so some testing is necessary. Two options exist, either build and install kuksa-client locally, or upload an alpha package to pypi. Testing with an uploaded package is preferable as that also tests that Pypi packaging works as expected.

Start DBC Feeder using default settings

erik@debian3:~/kuksa.val.feeders/dbc2val$ ./dbcfeeder.py

Let it run for a while, verify with Python client (or databroker CLI) that Vehicle.Speed change now and then.

Test Client> getValue Vehicle.Speed                          
{
    "path": "Vehicle.Speed",
    "value": {
        "value": 77.0,
        "timestamp": "2023-07-20T13:17:26.946733+00:00"
    }
}

Test Client> getValue Vehicle.Speed
{
    "path": "Vehicle.Speed",
    "value": {
        "value": 54.0,
        "timestamp": "2023-07-20T13:17:33.924064+00:00"
    }
}

GPS2VAL

This is somewhat tricky. the trick seems to be to make sure that you use the default Python version of your distribution, i.e. not something you have sideloaded. Also using a different port is needed. To test:

Run gpsfake:

erik@debian3:~/kuksa.val.feeders/gps2val$ gpsfake -t -P 2949 simplelog_example.nmea
Processing simplelog_example.nmea
gpsfake: log cycle of simplelog_example.nmea begins.
gpsfake: log cycle of simplelog_example.nmea begins.

Run gps2val:

erik@debian3:~/kuksa.val.feeders/gps2val$ python gpsd_feeder.py --protocol grpc --port 55555 --insecure true --gpsd_port 2949
2023-07-20 15:26:32,525 INFO gpsfeeder: No configuration file found. Using default values.
2023-07-20 15:26:32,526 INFO gpsfeeder: Init kuksa client...
2023-07-20 15:26:32,578 INFO gpsfeeder: No token information provided, subsequent errors expected if Server/Databroker requires authentication!
2023-07-20 15:26:32,579 INFO gpsfeeder: Connection status is: False
2023-07-20 15:26:32,579 INFO kuksa_client.grpc: No Root CA present, it will not be posible to use a secure connection!
2023-07-20 15:26:32,579 INFO kuksa_client.grpc.aio: Establishing insecure channel
2023-07-20 15:26:32,579 INFO gpsfeeder: Init gpsd client...
2023-07-20 15:26:32,580 INFO gpsfeeder: Trying to connect gpsd at 127.0.0.1 port 2949
2023-07-20 15:26:32,580 INFO gpsfeeder: Using mapping
2023-07-20 15:26:32,582 INFO gpsfeeder: gpsd receive loop started
gRPC channel connected.
2023-07-20 15:26:32,601 INFO gpsfeeder: Number of VSS messages sent so far: 1
2023-07-20 15:26:32,620 INFO gpsfeeder: Number of VSS messages sent so far: 2
2023-07-20 15:26:33,640 INFO gpsfeeder: Number of VSS messages sent so far: 4

verify with KUKSA Client that lat and/or lon is updated

Test Client> getValue Vehicle.CurrentLocation.Latitude
{
    "path": "Vehicle.CurrentLocation.Latitude",
    "value": {
        "value": 48.779966667,
        "timestamp": "2023-07-20T13:27:09.618384+00:00"
    }
}

Test Client> getValue Vehicle.CurrentLocation.Latitude
{
    "path": "Vehicle.CurrentLocation.Latitude",
    "value": {
        "value": 48.780183333,
        "timestamp": "2023-07-20T13:27:13.012102+00:00"
    }
}

Basic testing with Databroker (Tokens and TLS)

erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker -- --metadata ../data/vss-core/vss_release_4.0.json --tls-cert ../kuksa_certificates/Server.pem --tls-private-key ../kuksa_certificates/Server.key --jwt-public-key ../kuksa_certificates/jwt/jwt.key.pub

Kuksa-client

Verify with Kuksa-client that connection works

erik@debian3:~/kuksa.val/kuksa-client$ kuksa-client --ip 127.0.0.1 --port 55555 --protocol grpc  --cacertificate ../kuksa_certificates/CA.pem --tls-server-name Server
...
Secure gRPC channel connected.
Test Client> authorize ../jwt/provide-all.token
"Authenticated"

Test Client> setValue Vehicle.Speed 12         
OK

Test Client> getValue Vehicle.Speed   
{
    "path": "Vehicle.Speed",
    "value": {
        "value": 12.0,
        "timestamp": "2023-07-21T09:12:42.968598+00:00"
    }
}

Test Client>

Databroker-cli

erik@debian3:~/kuksa.val/kuksa_databroker$ cargo run --bin databroker-cli -- --token-file ../jwt/provide-all.token --ca-cert ../kuksa_certificates/CA.pem
    Finished dev [unoptimized + debuginfo] target(s) in 0.20s
     Running `/home/erik/kuksa.val/target/debug/databroker-cli --token-file ../jwt/provide-all.token --ca-cert ../kuksa_certificates/CA.pem`

...

sdv.databroker.v1 > feed Vehicle.Speed 5
[feed]  OK  
sdv.databroker.v1 > get Vehicle.Speed
[get]  OK  
Vehicle.Speed: 5.00
sdv.databroker.v1 >

Go Client

NOTE: Go Client does not support TLS, test skipped

Databroker Docker

Verify first that Databroker:master is recently built and there is no pending/queued builds

erik@debian3:~/kuksa.val/kuksa_databroker$ docker pull ghcr.io/eclipse/kuksa.val/databroker:master
erik@debian3:~/kuksa.val/kuksa_databroker$ docker run --rm -it  -p 55555:55555/tcp -v /home/erik/kuksa.val/kuksa_certificates:/certs   ghcr.io/eclipse/kuksa.val/databroker:master --tls-cert /certs/Server.pem --tls-private-key /certs/Server.key --jwt-public-key /certs/jwt/jwt.key.pub

Kuksa-client (native)

Test as above for TLS + Token

Kuksa-client (Docker)

Note: Verify that correct version is shown!

erik@debian3:~/kuksa.val/kuksa-client$ docker run --rm -it -v /home/erik/kuksa.val/kuksa_certificates:/certs -v /home/erik/kuksa.val/jwt:/jwt --net=host ghcr.io/eclipse/kuksa.val/kuksa-client:master --ip 127.0.0.1 --port 55555 --protocol grpc  --cacertificate /certs/CA.pem --tls-server-name Server
Welcome to Kuksa Client version 0.1.6
...
Secure gRPC channel connected.
Test Client> authorize /jwt/provide-all.token
"Authenticated"

Test Client> setValue Vehicle.Speed 63
OK

Test Client> getValue Vehicle.Speed
{
    "path": "Vehicle.Speed",
    "value": {
        "value": 63.0,
        "timestamp": "2023-07-21T11:31:28.461371+00:00"
    }
}

Test Client>

KUKSA Server (native, TLS + cert)

This is the default setting if using default config

./kuksa-val-server  --vss ./vss_release_4.0.json

KUKSA Client Docker

Do as below, checks given as comments with #

erik@debian3:~/kuksa.val$ docker run --rm -it --net=host kuksa-client:latest
Welcome to Kuksa Client version 0.4.0a5.post9+git.77a1d363

# Verify that version is as expected

Websocket connected securely.

Test Client> getValue Vehicle.Speed

# Shall give authorization error

Test Client> authorize /kuksa-client/kuksa_certificates/jwt/super-admin.json.token

# Shall be accepted

Test Client> setValue Vehicle.Speed 54

# OK

Test Client> getValue Vehicle.Speed

# OK

Test Client>

KUKSA Server Docker

docker pull ghcr.io/eclipse/kuksa.val/kuksa-val:master

KUKSA-Client (docker or not)

Test with kuksa-client like above

DBCFeeder

Change config to connect to KUKSA.val, typically use:


[general]

server_type = kuksa_val_server
port = 8090
tls = True
tls_server_name=localhost
token=../../kuksa.val/kuksa_certificates/jwt/super-admin.json.token

Run as ./dbcfeeder.py

Verify that Vehicle.Speed changes

Test after creating new kuksa-client PyPI package (TLS, token)

Start databroker as above. Install kuksa-client and run it. Make sure you do not run from the directory where kuksa-client resides

erik@debian3:~$ pip install -U kuksa-client
...
Successfully installed kuksa-client-0.4.0

erik@debian3:~$ kuksa-client --ip 127.0.0.1 --port 55555 --protocol grpc  --cacertificate kuksa.val/kuksa_certificates/CA.pem --tls-server-name Server  
...
Welcome to Kuksa Client version 0.4.0
Test Client> authorize kuksa.val/jwt/provide-all.token 
"Authenticated"

Test Client> setValue Vehicle.Speed 43
OK

Test Client> getValue Vehicle.Speed   
{
    "path": "Vehicle.Speed",
    "value": {
        "value": 43.0,
        "timestamp": "2023-07-26T09:21:14.432932+00:00"
    }
}

Test Client>

Tests after creating new docker containers on ghcr.io

In general, download containers:

docker pull ghcr.io/eclipse/kuksa.val/databroker:0.4

Databroker

erik@debian3:~/kuksa.val/kuksa_databroker$ docker run --rm -it  -p 55555:55555/tcp -v /home/erik/kuksa.val/kuksa_certificates:/certs ghcr.io/eclipse/kuksa.val/databroker:0.4 --tls-cert /certs/Server.pem --tls-private-key /certs/Server.key

Test with Python Client and CLI. Check that correct version is shown

erik@debian3:~/kuksa.val$ docker run --rm -it --net=host -v /home/erik/kuksa.val/kuksa_certificates:/certs ghcr.io/eclipse/kuksa.val/kuksa-client:0.4  --ip 127.0.0.1 --port 55555 --protocol grpc --cacertificate /certs/CA.pem --tls-server-name Server 
Welcome to Kuksa Client version 0.4.0

Secure gRPC channel connected.
Test Client> setValue Vehicle.Speed 3
OK

Test Client> getValue Vehicle.Speed
{
    "path": "Vehicle.Speed",
    "value": {
        "value": 3.0,
        "timestamp": "2023-07-27T07:56:44.898774+00:00"
    }
}
Clone this wiki locally