Skip to content

How Does Datamaster Work

awadell1 edited this page Feb 28, 2017 · 12 revisions

The overall goal of Datamaster is to enable the user to quickly analyze large numbers of log files using MATLAB. However, as the file format used by MoTeC is closed source, the information stored in the log files must first be converted into a format that MATLAB/ Datamaster can access. The conversion process in handled by the DataReporter object and consists of the following steps:

  1. The Google Drive is scanned for new MoTeC Log files which are then downloaded to the local machine
  2. Using MoTeC i2Pro the Log files are opened and exported to a temporary *.MAT File
  3. DataReporter parses the Log files *.LDX file for the information stored as Details in the log file
  4. The Exported *.MAT file is saved to the Datastore Folder and an entry is added to Master Directory

MoTeC Log File Structure

Each MoTeC Log File is made up of two separate files:

  • LD: Stores the channel data that was logged
  • LDX: Stores log file maths, details and inserted or modified beacons

Diagram showing structure of a MoTeC Log file

Structure of the *.LD File

At present the underlying structure of the *.LD file is unknown, however, some information is suspected:

  • The LD file is approximately half the size of the exported MAT file suggesting:
    • The LD file resembles the exported MAT file but uses single point precision (MATLAB uses double by default)
    • The LD file records the sampling rate instead of a time vector containing the time of each sample
    • The LD file records the readings as a 16-bit number (Per ADL3 Manual) with is later converted into a floating point number
  • Channel Names appear to be recorded using ASCII
  • Channel Data appears to be encoded into some other format

Structure of the *.LDX File

The LDX file is an XML file used to store log file maths, details and inserted or modified beacons. As shown below in a sample file (Some fields have been removed).

<?xml version="1.0"?>
<LDXFile Locale="English_United States.1252" DefaultLocale="C" Version="1.6">
 <Layers>
  <Details>
   <String Id="Event" Value="DYNO"/>
   <String Id="Venue" Value="Dyno-LAB"/>
   <String Id="Driver" Value="hg294"/>
   <String Id="Vehicle Id" Value="ARG15"/>
   <DateTime Id="Log Date" Value="22/12/2016"/>
   <DateTime Id="Log Time" Value="16:41:36"/>
   <Numeric Id="Fuel Tank Capacity" Value="7.6" Unit="l" DPS="1"/>
  </Details>
 </Layers>
</LDXFile>

At present DataReporter only extracts the data stored inside the <Detail> block, however it is not unfeasible that information contained in other blocks also be extracted. At present there does not appear to be a need and thus it has not been pursued.

Datasource Structure

Diagram showing the structure of a Datasource

Each datasource (The Datamaster equivalent to MoTeC's Log File) also consists of two parts:

  • A MAT File that stores the channel data
  • SQLite Database used to store details and a list of logged channeled

MAT File

Master Directory: SQLite Database