Import logbooks, tags and properties

[georgweiss]

To facilitate creation/population of data - logbooks, tags and properties - the service should support an import (and export?) mechanism using plain files (yaml, json or xml).

As the service supports an authentication mechanism one could also consider adding management of these entities in the client UI.

[gregoryleblanc]

Adding links to https://github.com/shroffk/olog-data-migration and https://github.com/gregoryleblanc/olog-data-migration for future reference.

[konrad-gajewski]

We plan to start using phoebus-olog and would like to import data from our old logbooks to olog.
Looking at the code in olog-data-migration I've noticed that at some point it was stated that you need to prepare the list of log entries in ascending order of ids https://github.com/shroffk/olog-data-migration/blob/7c7cbf8c1378752173bfbeef10724fead5b06e96/src/main/java/org/phoebus/old/olog/OldOlogLogRetrieval.java#L120

My question is: are there any constrains on relation between id and createdDate fields? For example, that createdDate must both increase with every new log entry.

In our transition scenario we are considering the option to start using olog and later on import the old logbooks. But then we would have the situation that we have lower createdDate values and higher id for the imported entries (I assume that id will be incremented every time a new entry is created, would it be the new entry or the entry imported from the old lobook with preserved original creation date).
Is the above scenario possible or do we need to import the old logbooks first before adding the new entries?

[georgweiss]

There is no relation at all between ids and dates. You can import data at any time.

Note that in order to preserve the original date in the migrated entries you need to run a modified version of the service as the date is always created on the service when a new entry is added. This is intentional: clients should not be able to set the date.

[konrad-gajewski]

@georgweiss, thanks for the quick response. Yes, I'm aware of the fact that we would need to modify the service to support setting the arbitrary createdDate value. It looks to me that it will be easier for us to modify the REST API of the service and write a relatively simple python program to read our logbooks and create new entries using the REST API rather than write a java program based on olog-data-migration.

[shroffk]

I just pushed some updates for the elastic 8.x version of the olog-data-migration.

Modifying the service would need some java code too :)...but if you find it easier then go for it.

Also remember to update the sequence generation index to match the next unique index expected post transfer

[konrad-gajewski]

I was thinking about adding an optional property createdDate to "PUT https://localhost:8181/Olog/logs/multipart" request and if its defined use it instead of Instant.now() when processing the request. I hope the updating the index is done in the processing of PUT request. Please correct me if I'm missing something. I have now experience with programming this kind of stuff.