Fails in multiprocessing even with 'r' mode

[Jwink3101]

As one would expect with a normal dictionary, it cannot be used (easily) with multiprocessing for writing values. However, a normal diction can be used for reading.

sqlitedict fails even with flag='r' in multiprocessing. This is almost certainly due to the threaded connection, but that connection is not needed for flag='r' since you can concurrently read an sqlite file.

Is not too hard to fix, but would add a major change in code flow

[menshikh-iv]

@Jwink3101 thanks for the report, can you add a code example that reproduces your issue + maybe some tip about fixing if you have it.

[Jwink3101]

It was a long time ago that I figured it out but I seem to recall it was the thread for queuing the SQLite operations. That didn’t work properly in multiprocessing. I pulled it out and it worked but I assume there was a more important use case envisioned for having the threading. (Avoiding lockouts presumably)

…

On Jan 16, 2019, at 8:54 PM, Ivan Menshikh ***@***.***> wrote: @Jwink3101 thanks for the report, can you add a code example that reproduces your issue + maybe some tip about fixing if you have it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[piskvorky]

Yes, multiprocessing affects threads. AFAIK only the main thread remains running in a forked process.

Not sure this is a bug; it definitely has nothing to do with r mode. If we want to support transparent forking of SqliteDict objects, we'd need to think carefully about how to handle/restart running threads (not trivial).

OTOH, SqliteDict seeks to prevent multithreading issues with plain Sqlite (ProgrammingError: SQLite objects created in a thread can only be used in that same thread.). If you're using multiple processes instead of threads, either instantiate a separate SqliteDict in each, or just use plain sqlite?