Generator does not consider .h5 files of varying lengths

[vitoriapacela]

gen_from_data() seems to assume that all the HDF5 files in the given directory path have the same number of samples.

[DannyWeitekamp]

Hmm I definitely didn't hard code this constraint. However it will make sure that in each collection (i.e. HCal, Ecal) in a single h5 file you have the same number of samples. Can you send a snippet of what you are trying to do and a stack trace? This shouldn't be an issue if everything is being used properly, but there are always bugs. I'm going to be traveling the next few days. But I'll look at this closer on Sunday or Monday.

…

On Thu, Mar 22, 2018 at 2:12 PM Vitória Barin Pacela < ***@***.***> wrote: gen_from_data() seems to assume that all the HDF5 files in the given directory path have the same number of samples. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFODxGhs2a60lk_JPdlix8YX-VfDBVTFks5tg-magaJpZM4S3jBJ> .

[vitoriapacela]

I was doing model.fit_generator() and I got the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-15-f1360b97f12e> in <module>()
      4                            nb_val_samples=val_samples, verbose=1,
      5                            callbacks=[EarlyStopping(monitor='val_loss', patience=8, verbose=1, mode='min')
----> 6                            , ModelCheckpoint(filepath='/nfshome/vitoriabp/gpu-4-culture-plate-sm/new_ds_notebooks/fix_pi0_dnn.h5', monitor='val_loss', verbose=0, save_best_only=True, mode='min')]
      7                             )

/usr/local/lib/python2.7/dist-packages/keras/engine/training.pyc in fit_generator(self, generator, samples_per_epoch, nb_epoch, verbose, callbacks, validation_data, nb_val_samples, class_weight, max_q_size, nb_worker, pickle_safe, initial_epoch)
   1530                                          '(x, y, sample_weight) '
   1531                                          'or (x, y). Found: ' +
-> 1532                                          str(generator_output))
   1533                     if len(generator_output) == 2:
   1534                         x, y = generator_output

ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None

I checked the shapes of the gen_from_data objects I had created and they were correct. I got rid of the error after removing from my train directory the files that contained less samples.

[DannyWeitekamp]

I have a feeling that the file you removed might have been corrupted, since Maurizio creates his datasets from a distributed system sometimes a machine fails and spits out a file that isn't formatted quite right. Try double checking that this isn't the case. If this is the case then I think the thing to do would be to just make the error checking a little more incisive so that its clear when this happens.

…

On Fri, Mar 23, 2018 at 3:35 AM Vitória Barin Pacela < ***@***.***> wrote: I was doing model.fit_generator() and I got the following error: ValueError Traceback (most recent call last) <ipython-input-15-f1360b97f12e> in <module>() 4 nb_val_samples=val_samples, verbose=1, 5 callbacks=[EarlyStopping(monitor='val_loss', patience=8, verbose=1, mode='min') ----> 6 , ModelCheckpoint(filepath='/nfshome/vitoriabp/gpu-4-culture-plate-sm/new_ds_notebooks/fix_pi0_dnn.h5', monitor='val_loss', verbose=0, save_best_only=True, mode='min')] 7 ) /usr/local/lib/python2.7/dist-packages/keras/engine/training.pyc in fit_generator(self, generator, samples_per_epoch, nb_epoch, verbose, callbacks, validation_data, nb_val_samples, class_weight, max_q_size, nb_worker, pickle_safe, initial_epoch) 1530 '(x, y, sample_weight) ' 1531 'or (x, y). Found: ' + -> 1532 str(generator_output)) 1533 if len(generator_output) == 2: 1534 x, y = generator_output ValueError: output of generator should be a tuple (x, y, sample_weight) or (x, y). Found: None I checked the shapes of the gen_from_data objects I had created and they were correct. I got rid of the error after removing from my training directory the files that contained less samples. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#5 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFODxArpQdOBnQ-eB27YrljHsAQvf5AYks5thKXQgaJpZM4S3jBJ> .