filename: INTERFACES.md description: MetaStore idea: One such file per repository (services, plugins, frontends, ...) as a proposal for DEM developers. ...
This document aims to answer questions on how to configure external dependencies and which public interfaces are offered by MetaStore in a comprehensive way. It is meant to be used for getting an overview and guidance in addition to the official documentation, which is available at the official MetaStore Web page.How can this software be connected with other software? This is the question this document aims to answer.
ℹ️ Note: This document applies to the MetaStore version it is shipped with. If you have a specific version running, please refer to
INTERFACE.mdof this particular release.
External dependencies are third-party services that are required for MetaStore to work properly or that can be added optionally to provide additional functionality. Typically, external dependencies require additional software to be installed and configured, before they can be included in the MetaStore configuration, which is typically done via the main configuration file application.properties. If you do not want to lose your default settings, we recommend that you make a copy of "application.properties" and move it to the "config" subfolder. Remove all properties you want to keep from the new file. All properties in "config/application.properties" override the settings in "application.properties".
External dependencies are third-party services that are required for MetaStore to work properly or that can be added optionally to provide additional functionality. Typically, external dependencies require
additional software to be installed and configured, before they can be included in the MetaStore configuration, which is should be done via the configuration file config/application.properties.
A relational database is required by MetaStore to store administrative metadata for metadata documents/schemas. If not configured properly, MetaStore will fail to start.
- H2 In-Memory (driver included, used for testing only, not recommended for production) Example
- H2 File-Based (driver included, used for basic Docker setup or for 'quick and dirty' tests, not recommended for production) Example
- Postgres (driver included, requires a running PostgreSQL server, recommended for production) PostgreSQL, Example
ℹ️ Note: Other relational databases, such as MariaDB, SQLite, or Oracle, may also work, but require additional steps. To allow MetaStore to connect, the source code repository must be cloned, an appropriate JDBC driver has to be added to
build.gradleand MetaStore has to be compiled. Proper JDBC drivers are typically provided on the database's web page. Afterwards, the database can be configured inconfig/application.propertiessimilar to PostgreSQL but with database-specific property naming. Please refer to the driver documentation for details.
MetaStore requires access to the local file system in order to store and manage uploaded metadata (schema) documents. MetaStore requires access to two folders, which can be located on the local hard drive or mounted via NFS. By default, two subfolders named 'metadata' and 'schema' are created in the installation directory.
- see
application.properties
ℹ️ Note: The file path to the folders have to start with three '/'. If you overwrite the default settings we recommend to create or edit 'config/application.properties'. e.g.:
metastore.schema.schemaFolder:file:///data/metastore/schema metastore.metadata.metadataFolder:file:///data/metastore/metadataIf the folders do not exist, they will be created.
AMQP-based messaging is an optional feature of MetaStore, which allows MetaStore to emit messages about creation, modification, and deletion events related to metadata documents. These messages can be received by registered consumers and processed in an asynchronous way.
- RabbitMQ (dependencies included, serves as messaging distributor, requires a running RabbitMQ server)
By default, MetaStore offers basic search based on administrative metadata via RESTful API by certain query parameters. Optionally, enhanced search via a search index can be enabled and used for fine-grained and facetted search operations.Therefor all metadata documents were indexed using an elasticsearch instance. (JSON metadata documents are provided by default. XML documents should be transformed to JSON)
- Messaging
- Indexing Service
- Elasticsearch Messaging is used to inform indexing service about any updates regarding metadata documents. Indexing service will fetch the document, transform it (if mapping available) and send it to elasticsearch for indexing.
By default, MetaStore itself is open for all kinds of operations, i.e., read and write, where write access should be restricted on the user interface level, e.g., by a password-protected area for critical operations. Optionally, authentication and authorization via JSON Web Tokens (JWT) issued by a Keycloak instance, can be configured, which allows a fine-granulated access management on document level.
Public Interfaces are used to access MetaStore in order to obtain its contents, typically this happens via HTTP/REST. Depending on the interface, special clients or protocols must be used to access a specific public interface.
Interfaces which trigger functionality or return information on request.
A REST interface which allows to access the service functionality, like creating/registering, updating and validating metadata documents/schemas on request.
- OpenAPI / or via running instance, as described in the Readme.
- Usage with Examples
If Enhanced Search is enabled, an additional REST endpoint becomes available, which allows to tunnel search queries to the underlying Elasticsearch instance. The advantage for proxying Elasticsearch access is, that access restrictions enabled via Access Control are included in the query such that only results accessible by the caller are returned.
OAI-PMH is a standardized harvesting protocol that allows to build up external search indices that can be kept up to data by regular harvesting changes from an OAI-PMH source. For MetaStore only XML metadata documents are supported.
DOIP is a novel protocol to provide generic access to digital resources. Instead of using HTTP-based communication, the protocol acts directly on top of TCP/IP and is therefore mainly relevant for special ecosystems. Not finalized yet!
Not yet available!