Data damage?

[YangSiri]

Hi, @xingyul , thanks for your sharing codes and it is great. But i got a little trouble while testing it, could you or anyone help me?
I downloaded the processed data of flyingtings3d which is around 11GB, i extracted it and put the folder under data_processing, and then i run 'sh command_train.sh" as you suggested.There is the error :

2019-09-30 13:38:13.146499: W tensorflow/core/framework/allocator.cc:108] Allocation of 33554432 exceeds 10% of system memory.
-- 010 / 032 --
mean loss: 1.026778
Traceback (most recent call last):
File "train.py", line 283, in
train()
File "train.py", line 155, in train
train_one_epoch(sess, ops, train_writer)
File "train.py", line 201, in train_one_epoch
batch_data, batch_label, batch_mask = get_batch(TRAIN_DATASET, train_idxs, start_idx, end_idx)
File "train.py", line 173, in get_batch
pc1, pc2, color1, color2, flow, mask1 = dataset[idxs[i+start_idx]]
File "/home/py001/cyz/flownet3d/flying_things_dataset.py", line 36, in getitem
data = np.load(fp)
File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 426, in load
pickle_kwargs=pickle_kwargs)
File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 182, in init
_zip = zipfile_factory(fid)
File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 112, in zipfile_factory
return zipfile.ZipFile(file, *args, **kwargs)
File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/zipfile.py", line 1026, in init
self._RealGetContents()
File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/zipfile.py", line 1093, in _RealGetContents
raise BadZipFile("File is not a zip file")
zipfile.BadZipFile: File is not a zip file
Exception ignored in: <bound method NpzFile.del of <numpy.lib.npyio.NpzFile object at 0x7f3f12e80630>>
Traceback (most recent call last):
File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 219, in del
self.close()
File "/home/py001/anaconda3/envs/cyzpy35/lib/python3.5/site-packages/numpy/lib/npyio.py", line 210, in close
if self.zip is not None:
AttributeError: 'NpzFile' object has no attribute 'zip

Is there anything wrong with the data i downloaded ?

[xingyul]

Can you print out the filename of the .npz file that cause the problem?

[YangSiri]

Hi @xingyul , Thanks for answering me. But I think none file is in processing... Doesn't this problem occur as long as the first data file is read?

[xingyul]

Hi, if you take a look at the dataset file, the code read each npz file one by one. From the error message you showed, it seems like there is an error during the reading of the npz file. It would be helpful to check if the npz filename is correct.

[YangSiri]

Hi @xingyul thanks for your attention.
This is the processed data folder, i dont spot any mistake?