Problem configuring an SFTP Bulk source

[isisgoliveira]

Hey guys!

I have a problem configuring an SFTP Bulk source.

On my server there are two types of files together at the root, but I would like to ingest them separately into two streams.
In the SFTP source settings, I should be able to pass a filename filter in the optional "Globs" field, but it is not accepting anything other than the generic "**" pattern which ends up reading everything and mixing two different data schemes. I didn't want to have to deal with the separation of this raw data only in the refinement process, as I would already be able to resolve it when reading the source.

Could anyone help me with this?

[adamb53]

I am having this same issue

[isisgoliveira]

I'm sorry you're having the same problem. Unfortunately in my case I had to deal with data separation during refinement, as this glob filter did not work during ingestion. It was the only way I found to deliver the data to the client :/
It seems that the community here is not that engaged.

[skouzini]

I am also in this same boat. I have briefly gone through the source code of the connector, although it was not all that insightful at the depth I went. I intend to dig deeper and see what is going on soon.

[TimFinucane]

Found the issue! The SFTP Bulk code doesn't check whether the provided Folder Path already ends in a slash (which is the case by default), and may double it up. All you have to do is add that double slash to the glob where appropriate.

E.g. in @isisgoliveira's case if you have not set a folder path and your files are at the root you will need a glob pattern like '//my_prefix_*.csv'

Also note if your files are in a folder, you need to include that folder in your glob too. E.g. for files in a folder 'my_folder', your glob will need to be 'my_folder/my_prefix_*.csv' even when your folder path is 'my_folder' (if you don't set a folder path, it'll need to be '//my_folder/my_prefix_*.csv'!!).

I tried this for my use-case and it seems to be working fine now!

[jessica-bauer]

This helped me immensely as I was stuck with this issue for a while. Thank you so much!