Skip to content

Commit

Permalink
fixed a bug with noDataLoading reading
Browse files Browse the repository at this point in the history
  • Loading branch information
@ratal
ratal committed Dec 17, 2017
1 parent 73e057e commit 4b4c88f
Show file tree
Hide file tree
Showing 2 changed files with 10 additions and 10 deletions.
2 changes: 1 addition & 1 deletion mdfreader/mdf3reader.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -873,7 +873,7 @@ def read3(self, fileName=None, info=None, multiProc=False, channelList=None,
if info['DGBlock'][dataGroup]['numberOfChannelGroups'] > 0 and \
(channelSet is None or
len(channelSet & info['ChannelNamesByDG'][dataGroup]) > 0): # data exists
if minimal > 1: # load CG, CN and CC block info
if minimal > 1 and not self._noDataLoading: # load CG, CN and CC block info
info.readCGBlock(info.fid, dataGroup, minimal=minimal)
# Pointer to data block
pointerToData = info['DGBlock'][dataGroup]['pointerToDataRecords']
Expand Down
18 changes: 9 additions & 9 deletions mdfreader/mdf4reader.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -1205,7 +1205,7 @@ def returnField(obj, field):
if not info['DG'][dataGroup]['dg_data'] == 0 and \
(channelSet is None or
len(channelSet & info['ChannelNamesByDG'][dataGroup]) > 0): # there is data block and channel in
if minimal > 1: # load CG, CN and CC block info
if minimal > 1 and not self._noDataLoading: # load CG, CN and CC block info
info.readCGBlock(info.fid, dataGroup, channelSet, minimal=minimal)
if info['CG'][dataGroup][0]['cg_cycle_count']: # data exists
# Pointer to data block
Expand Down Expand Up @@ -1512,8 +1512,8 @@ def write4(self, fileName=None):
pointers['CG'] = {}
pointers['CN'] = {}
DG_flag = 0
for ndataGroup, dataGroup in enumerate(self.masterChannelList):
masterChannel = list(self.masterChannelList.keys())[dataGroup]
for dataGroup, masterChannel in enumerate(self.masterChannelList):
#masterChannel = list(self.masterChannelList.keys())[dataGroup]
# writes dataGroup Block
temp = DGBlock()
pointers['DG'][dataGroup] = temp.write(fid)
Expand Down Expand Up @@ -1870,23 +1870,23 @@ def valueRangeToTextConv(vect, cc_val, cc_ref):
cc_ref[ref] = lambdify(X, cc_ref[ref]['cc_ref']['Comment']
, modules='numpy', dummify=False)
elif cc_ref[ref]['cc_type'] == 1: # linear conversion
cc_ref[ref] = lambdify(X, '{0}* X + {1}'.format(cc_val[1], cc_val[0])
cc_ref[ref] = lambdify(X, '{0} * X + {1}'.format(cc_val[1], cc_val[0])
, modules='numpy', dummify=False)
else: # identity, non conversion
cc_ref[ref] = lambdify(X, 'X'
else: # identity, no conversion
cc_ref[ref] = lambdify(X, '1 * X'
, modules='numpy', dummify=False)
# Otherwise a string
# look up in range keys
temp = []
for Lindex in range(len(vect)):
for value in vect:
key_index = val_count # default index if not found
for i in range(val_count):
if key_min[i] < vect[Lindex] < key_max[i]:
if key_min[i] <= value <= key_max[i]:
key_index = i
break
if callable(cc_ref[key_index]):
# TXBlock string
temp.append(cc_ref[key_index](vect[i]))
temp.append(cc_ref[key_index](value))
else: # scale to be applied
temp.append(cc_ref[key_index])
return asarray(temp)
Expand Down

13 comments on commit 4b4c88f

@danielhrisca
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hello Aymeric,

did you change something with the blosc compression? I can't run benchmarks with 32bit Python (Python 2.7 and 3.6), multiprocessing being used, but without multiprocessing I get no errors. No problems with 64 bit Python

@ratal
Copy link
Owner Author

@ratal ratal commented on 4b4c88f Dec 18, 2017 via email

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@danielhrisca
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm using a different PC with blosc 1.5.1 installed in all Python distributions. This is most likely a blosc issue

@cristi-neagu
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Daniel, as an aside, is there any particular reason why you prefer blosc over hdf5 format compression? Tool chain requirements?

@danielhrisca
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure what you mean.

@cristi-neagu
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I misunderstood something. Are you using blosc to compress data to disk, or to compress in memory data?

@danielhrisca
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Neither: I'm just benchmarking mdfreader with compression='blosc' init argument.

@cristi-neagu
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok. I see.

@danielhrisca
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@cristi-neagu do you use hdf5 to store measurements instead of MDF?

@cristi-neagu
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not currently, but it is something I was considering.

@ratal
Copy link
Owner Author

@ratal ratal commented on 4b4c88f Dec 19, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For info, it is used at my job to manipulate big endurance data. HDF5 is easily readable by Matlab and many other tools, shows good performance (can be even used as database, multiprocessing, etc.), flexible, widely used, a good competitor to mdf in fact (I do not understand why it is not used also in automotive industry as standard... with some standardised attributes, you could store sorted and unsorted data blocks, have attachments, SI, etc.). Nowadays, mdf can also be pared by Matlab with vehicle network toolbox.

@cristi-neagu
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect that the reason behind the widespread use of the MDF format in the Automotive industry is that ETAS still uses it as a native format, and automotive engineers are not software engineers, so they don't really care about data efficiency. They mostly want their tools working consistently.

@danielhrisca
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You forget that Vector is quite a bit player in this field, and they developed this standard. There are many automotive companies in the ASAM membership list.

Please sign in to comment.