Implement filesystem watcher input

[henrikschristensen]

No description provided.

[henrikschristensen]

resolves #649

[gregfurman]

Is there a chance we can use the testing/fstest instead of actually creating files/dirs on the OS?

[henrikschristensen]

I dont think that is possible unless we write a mock for fsnotify too that can produce filesystem events when changes are made to the MapFS in testing/fstest.
I have not been able to find any examples of doing that.
I am aware of the dilemma that the tests rely on the actual OS that it is executed on, but abstracting that part out into some kind of mock would be a lot of work i fear.
I assume fsnotify itself is tested cross platform.

[henrikschristensen]

@gregfurman

It is a known problem that fsnotify produces a lot of redundant events when f.ex. a file is saved to a dir. Then it will typical produce a CREATE event followed by several WRITE events depending on file size and maybe a CHMOD event. This makes it hard to detect when a file done being written to the disk and as a result u risk reading parts of the file or empty content.
The suggested solution from the fsnotify project is a simple deduplication/debouncing like this example https://github.com/fsnotify/fsnotify/blob/main/cmd/fsnotify/dedup.go.

I have tried to do something similar with existing bento components but I think it makes the resulting yaml very complicated and error prone.
So I think the best approach for this input is to implement something like the suggested dedup code.
An alternative could be to refactor the code into using this project https://github.com/sgtdi/fswatcher witch has buildin support for dedup and batching of events. But it is a single developer project and maybe not a good option for Bento.

Any thoughts on this?

[jem-davies]

#181

[jem-davies]

Fixes #232