Scientific Metadata Search Engine (Fulltext) implementation for PostgreSQL 🔎

[Kezzsim]

This feature adds a rudimentary scientific metadata search engine, implemented purely through Postgresql's native ts_vector and ts_query operations. Documented as part of the Tiled client's FullText query.

Tasks:

• Add fulltext logic to adapter.py to enable functionality
• Create a new computed index metadata_search for storing jsonb_to_tsvector data
• Generate Alembic migration to move existing tiled collections to the new standard
• Add new tests that ensure the index is being invoked

Resolves #457

[danielballan]

Very excited to see this!

[danielballan]

Also: our FullText query accepts a case_sensitive parameter. I added that somewhat speculatively, when I was more naively about what is involved in supporting both options. As we're still in alpha, I wonder if it makes sense to remove that for now---being case insensitive only---and add it back later if there is demand for it.

There are tradeoffs in insert speed and database size that make me question whether case sensitive search is useful and important enough to justify that.

tiled/tiled/queries.py

Lines 42 to 50 in fca4330

	Parameters
	----------
	text : str
	case_sensitive : bool, optional
	Default False (case-insensitive).
	"""
	text: str
	case_sensitive: bool = False

[dylanmcreynolds]

I don't know how much to expect of this index, but could we do partial word search? e.g. urple

[danielballan]

“light urple!”

[danielballan]

I think support is limited, but robust fuzzy text search is next up for @Kezzsim.

[Kezzsim]

Current C.I. issues are caused by the caching we set up previously, ORM and alembic need to be able to handle if an index already exists in that cache somehow.

[Kezzsim]

Also: our FullText query accepts a case_sensitive parameter. I added that somewhat speculatively, when I was more naively about what is involved in supporting both options. As we're still in alpha, I wonder if it makes sense to remove that for now---being case insensitive only---and add it back later if there is demand for it.

There are tradeoffs in insert speed and database size that make me question whether case sensitive search is useful and important enough to justify that.

tiled/tiled/queries.py

Lines 42 to 50 in fca4330

	Parameters
	----------
	text : str
	case_sensitive : bool, optional
	Default False (case-insensitive).
	"""
	text: str
	case_sensitive: bool = False

I will remove the case sensitive flag from both the source code and the documentation, minimizing the API surface impact by ignoring any other kwargs sent to query other than text.

The postgresql documentation writes:

A lexeme is a string, just like a token, but it has been normalized so that different forms of the same word are made alike. For example, normalization almost always includes folding upper-case letters to lower-case