Data Type Change in Adapter Causes Data Storage Failure in InfluxDB

[tenthe]

Body

Problem

If you edit the adapter and change the data type, the data will no longer be saved in InfluxDB.

How to Reproduce:

1. Create an adapter with a temperature field as an integer and create a pipeline to store it in the data lake.
2. Change the data type of the temperature field to double.
3. An exception appears in the data lake sink because it tries to write into the same measurement "temperature" with two different data types.

StreamPipes Committer

I acknowledge that I am a maintainer/committer of the Apache StreamPipes project.

[tenthe]

When data is written to the data lake, an entry is created in CouchDB with the schema information and the data. If a user adjusts the event schema of an adapter, this is also updated in CouchDB, but the old data remains unchanged in InfluxDB.

Problem

• Adding new properties or renaming a property works without issues.
• Changing the data type of a property causes a runtime error when writing the data, due to the column in the database having a different data type.

Potential Solutions

• Migrate old entries in the database to match the new schema.
• Adapt the schema internally to prevent the issue. This requires checking for pipelines that write data to the DataLake when editing the adapter.

Current Workaround for Users

• For now we will show a warning when the user changes the data type
• Change the name of the data type.
• Write data to a new measurement.
• Truncate the old data in the database.