Reading samples timestamp back.

[pablocael]

Hello, Im trying to read the sample data along with the timestamp where the data was measured. Is it possible to retrive the measurement time? Thanks,

[ihedvall]

I'm not sure if we are on the same page.

After an initial call to the ReadEverythingButData(), the MDF files configuration is now known. Each channel group shall have a so-called master channel. This master channel is normally the relative time in seconds. When you set up the channel subscriptions, you should always include this master channel along with the data channels you want. The ReadData() call will read all samples and store these sample values into the channel subscribers.

If you want the absolute start time of the measurement(s), then it's the timestamp in the header block. Note that some recorders set this time when the file was created and not when the measurement was started.

[pablocael]

Thanks for your quck reply!

Would it be possible to provide a simple example on how to read back the samples with the timestamps? Im writting data like this:

  void WriteRecordsToFile(const std::vector<Record>& records) const override {
    // Cleanup
    std::remove("/tmp/mdf4.mf4");

    // Create an MDF4 writer
    auto writer = MdfFactory::CreateMdfWriter(MdfWriterType::Mdf4Basic);
    writer->Init("/tmp/mdf4.mf4");

    /* // Create a data group for metadata */
    auto* metaGroup = writer->CreateDataGroup();
    metaGroup->Description("the main data group container");

    auto* channelGroup = writer->CreateChannelGroup(metaGroup);
    channelGroup->Description("the main channel group container");

    // First create the channels
    for (const Record& record : records) {
      // Retrieve or create metadata channels
       getOrCreateStringChannel(channelGroup, record.topic_name);
    }

    writer->InitMeasurement();
    writer->StartMeasurement(nowNS());
    for (const Record& record : records) {
      // Retrieve or create metadata channels
       std::cout << ">>> writing record: " << record.topic_name << " payload is " << record.payload << std::endl;
       auto* channel = getStringChannel(channelGroup, record.topic_name);
       channel->SetChannelValue(record.payload);
       writer->SaveSample(*channelGroup, record.timestamp_ns);

    }
    // Assume data writting took 1000ns
    writer->StopMeasurement(nowNS());
    writer->FinalizeMeasurement();

Basically there is a list of our records, I write the samples. Now I want to read back with the timestamps.

The getOrCreateStringChannel will basically create a channel like this:

  IChannel* getOrCreateStringChannel(IChannelGroup* group, const std::string& channelName) const {
      // Check if the channel exists
    auto* channel = group->GetChannel(channelName);
    if (channel != nullptr) {
      return channel;
    }
    channel = group->CreateChannel(channelName);
    // For string payloads
    channel->Type(ChannelType::VariableLength); 
    channel->DataType(ChannelDataType::StringUTF8);
    return channel;
  }

The get channel only retrieves the channel from the group.

  // Function to get or create a channel within a given data group
  IChannel* getStringChannel(IChannelGroup* group, const std::string& channelName) const {
      // Check if the channel exists
    return group->GetChannel(channelName);
  }

Thanks in advance.

[ihedvall]

First some comments regarding the creation of the MDF channel. Note that inside the mdflib_test/src directory, there are many unit tests. The testwrite.cpp unit tests, first creates a file and then reads/checks it.

The timestamp channel must be created and its flags should be set as the master channel. I think you commented away that channel. It's mainly correct but it should be a double float type instead of an integer. The writer uses a sample cache that is sorted in absolute time order. It's the SaveSample() function so the record.timestamp_ns must be larger than the NowNS otherwise there will be no samples saved.

You create a number of string values in your channel group and this is OK. You should set all channel values and then call the SaveSample() function for each sample. It's a background thread that actually save the sample onto the disc and it is then the absolute timestamp is converted to a relative time.

Note that MDF sucks on storing string values so avoid this in real applications.

Now to reading. It involves 5 steps.

1. First you need to create a reader object and associate it with a file name.
2. The next step is to read in channel and channel group information.
3. The third step is to create channel observers (subscription) for each channel you wants data. The channel observers purpose is to store sample values/channel in primary memory.
4. Then you need to call the ReadData() function that fills the channel observer with data.
5. The last step is to iterate through the (observer) values. What to do with the values is your business.

There is a help manual for the reader at MDF Reader

[pablocael]

Thank you very much!

[pablocael]

Update: I managed to work. Just need to convert back to unint64 and need to add start time. Seems times are relative to start time.

Original question:

Thanks, I think that solves my problem. One last question: is it possible to store master channel timestamps in uint64 instead of float? Im storing measurements using uint64 and I would like to read it back as uint64. Thanks!

[ihedvall]

The absolute time of a sample is the start time + relative time. There is no obvious solution if you have more than one measurement in the file. The start time should be stored in the measurement (DG) block and not in the header (HD) block but we probably have to wait for the MDF 4.3 version. In my previous project, I stored the relative time but also stored the absolute time (uint64_t ns since 1970) as normal data channel. Note that the time channel is related to the channel group not to a channel.

When storing the time channel it required to be in chronological order, why your application is responsible for doing that. You should however note that the MDF reader is quite generic but the MDF writers are not. Each writer assume a certain external behavior. Currently the writer assume that the samples comes in time order. If your application receives the values out-of-order, a new writer needs to be made. For example if you have a converter file situation, the built in cache is just annoying.

The sample cache solves 2 problem. When you call InitMeasurement(), it starts a thread that is responsible of the cache. Post-mortem
measurement, you set up a pre-trig time for example 1 minute. When you call StartMeasurement(), the thread will store the cache samples including the 1 minute pre-trig time. The relative time will be negative, starting at -60 s. The other purpose is to save disc access. The cache is saved each 10 second i.e. open file, save cache and then close file. As the file is closed, another application can read the file during logging.

If the current writer or reader needs some improvement, please note that there exist a project for MDF related issues. Currently working with the Python interface.

[pablocael]

Thanks for all the info.
I have manage to make read/write work. I had one issue though, that setting PreTrigTime(0) and using sequential incremental timestamp like i=0 to N for calling SaveSample will not write ANY sample if i starts from 0. But if i starts from 1, it will write all samples... Is that due to the PreTrigTime? I expected it to maybe clip the first sample, but not all of them.

Also, just writing string payload and time into two different channels into two separate channel groups, and reading back, is pretty slow for large number of samples.

Let me show the writting code. The reading code is basically same as in the docs (using observers):

  void WriteRecordsToFile(const std::vector<Record> &records) const override {
    std::remove(kFilePath);
    mkdir(kTestDir, 0755);

    auto writer = MdfFactory::CreateMdfWriter(MdfWriterType::Mdf4Basic);
    writer->Init(kFilePath);

    /* // Create a data group for metadata */
    auto *dataGroup = writer->CreateDataGroup();
    dataGroup->Description("the main data group container");

    for (auto &record : records) {
      if (!getOrCreateChannelGroup(writer.get(), dataGroup,
                                   record.topic_name)) {
        std::cerr << "Failed to create or get channel group for topic: "
                  << record.topic_name << std::endl;
        return;
      }
    }

    uint64_t i = 1;
    writer->PreTrigTime(0);
    writer->InitMeasurement();
    writer->StartMeasurement(i);
    for (auto &record : records) {
      auto channelGroup =
          getOrCreateChannelGroup(writer.get(), dataGroup, record.topic_name);
      auto *timeChannel = channelGroup->Channels()[TIME_CHANNEL_INDEX];
      auto *dataChannel = channelGroup->Channels()[DATA_CHANNEL_INDEX];

      timeChannel->SetChannelValue((uint64_t)record.timestamp_ns);
      dataChannel->SetChannelValue(record.payload);

      writer->SaveSample(*channelGroup, i);
      i++;
    }
    writer->StopMeasurement(i);
    writer->FinalizeMeasurement();
  }

For reading:

    size_t result = 0;
    MdfReader reader(kFilePath);
    ChannelObserverList observerList;
    readFileChannelObservers(reader, observerList);
    CHECK(observerList.size() > 0);

    for (auto &observer : observerList) {
      auto *channel = &observer->Channel();
      if (channel->Name() != "Time") {
        // Only time channels
        continue;
      }
      auto nofSamples = observer->NofSamples();
      for (size_t sample = 0; sample < nofSamples; sample++) {
        uint64_t channel_value;
        const auto valid = observer->GetChannelValue(sample, channel_value);
        CHECK(valid);
        result += channel_value;
      }
    }

I have a question:

It seems MDF4 format is somehow like a linked list of data, you append at the end and then point the data block to that offset for a channel. Does that mean that writting to multiple channels can fragment the file and make it slower for reading as we would need more seeks to load a data block for one sample?

What can explain the slow data scan time?

Is there anything we can change in the code to make it faster? I remember you mentioned working with variable length data is slow, can you elaborate a bit on that?

Thank you in advance!

[ihedvall]

I want you to rename the variable "i" to "sample_time". The sample time 0 is an invalid time (midnight 1970-01-01) so stop using that time, it save you and me a lot of trouble. Note that sample_time is related to the record.timestamp_ns so your write loop has some time order issue.

The MDF 4 file format is an international standard so there is little I can do about that.

The MDF 4 format is designed to simplify writing of the cost of reading the file. You can sort the MDF 4 sample after it is stopped, making the reading faster. Note that you still reading from a file and that file is stored on a NAS (1GBit/s). You can improve your reading time in the range of twice the time.

The next speed improvement is to just read a partial part of the samples. There is a ReadData() function that reads all samples, a ReadVlsdData() that reads a range of VLSD data and last the ReadPartialData() that reads a range of samples. The last function is not yet checked in. The two latter function are significant faster for example if you have 10000 samples and you only need one VLSD sample, the read is 100000 times faster (well in that are).

The third read speed improvement is to use a database for indexing the measurements. You still stores the data in the MDF files but the database store with index, where in the file the values are stored. For a single file, this solution is overkill but if you have >100 files, you need to consider this solution.

Back to basic MDF writing. The MDF file doesn't append each channel separately, it append each channel group data. Note call to SaveSample(with time stamp) on the channel group not on the channel. The channel group stores it channel data in a byte array of fixed size. This doesn't work with variable length data so the channel group stores an offset to a separate BLOB where the VSLD data is stored.

There is basically 2 ways to store this signal data blob. In your current configuration, you actually store the strings in a separate SD block. As you only can append a file with data, the SD block is temporary stored in primary memory until you stop the measurement. The problem with this approach is that it consumes primary memory so it's not a recommended way of storing strings.

To solve the above problem, you can configure a channel group so it stores the BLOB in a channel group. This channel group doesn't have any channels. Instead it stores variable length block in that channel group. Bus loggers that typical stores CAN messages uses this technique. It's little bit tricky to configure why there exist a specialized MDF writer for this purpose. I think your use case(s) require this type of solution maybe with a combination of sorting when finalizing the measurement.

Well, I might have complicated everything for you but you did ask for it. You getting closer to a specialized writer but that require some better planning in form of use cases (requirements) or a functional specification. I'm on LinkedIn if you need to contact me. This GitHub might be to public for that matter.

Best Regards
Ingemar Hedvall

[pablocael]

Thanks for your reply! Your anwser is exactly what I was looking for. I already had experience before in reading fragmented MDF4 files. By that time I used MDF Sorter to defragment the file and reading speed got much better.
As I also understand and you confirmed, MDF files are designed for fast writting, but not particularly for fast reading as the read is usually intended for non real time usecases, as writing usually needs to happen in real time in high through put systems.

I will take a deeper look into your suggestions and try to optimize my code.

Thanks again!