get_spectra fails to process more than 3500 files at a time

[FedeMassetti]

Whenever using get_spectra on more than 3500 asd files at a time the result is an empty object. Any smaller number of files doesn't cause any issue and the code runs smoothly.

[pierreroudier]

Hum. Is there any error or warning issued? Is that on Windows?

[FedeMassetti]

Correct, on windows 10, R 3.6.1.
I tested it on two different computers one of which particularly powerful thinking it might be because of the lack of computational power of the other machine.
There is no error. It looks like it drops the operation all together immediately after confirming the file selection.

[pierreroudier]

I've checked, and it is working fine on my Linux machine... so I might need your help to debug this as I don't have a windows machine!

This is the code I've used:

library(asdreader)

tmp_dir <- tempdir()
lapply(1:3501, function(x) { 
fn <- paste0(tempfile(tmpdir = tmp_dir), '.asd')
file.copy(from = asd_file(), to = fn)
})

d <- get_spectra(list.files(tmp_dir, full.names = TRUE))
dim(d)
spectacles::big.head(d)

Could you please try and run that code please? Checking if that could be an issue with one of your ASD files.

[FedeMassetti]

After running

d <- get_spectra(list.files(tmp_dir, full.names = TRUE))

I get the error

Error in readBin(con, what = md$data_format, n = md$channels, endian = "little") :
invalid 'n' argument

I don't think it's a problem with my files. For my data I actually managed to move on just repeating the procedure multiple times and then merging the results to have a full data frame. So as long as I select less than about 3500 files everything is fine.

[pierreroudier]

That is so weird. I can't reproduce the issue on my Linux machine unfortunately, so that will be a tricky one for me to debug alone, unless you have an interest in giving a hand on that one.

[Bidhan-ghimire]

After running

d <- get_spectra(list.files(tmp_dir, full.names = TRUE))

I get the error

Error in readBin(con, what = md$data_format, n = md$channels, endian = "little") :
invalid 'n' argument

I don't think it's a problem with my files. For my data I actually managed to move on just repeating the procedure multiple times and then merging the results to have a full data frame. So as long as I select less than about 3500 files everything is fine

After running

d <- get_spectra(list.files(tmp_dir, full.names = TRUE))

I get the error

Error in readBin(con, what = md$data_format, n = md$channels, endian = "little") :
invalid 'n' argument

I don't think it's a problem with my files. For my data I actually managed to move on just repeating the procedure multiple times and then merging the results to have a full data frame. So as long as I select less than about 3500 files everything is fine.

Hello. Were you able to solve this issue? If so, can you please let me know how did you do it.

[pierreroudier]

@Bidhan-ghimire If you encounter the same issue, can you put some more details here please (version of the package, operating system, etc)? I could not reproduce the issue posted by @FedeMassetti , and therefore I never managed to think of a way to potentially fix it.

Thanks!