diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e490f03..64aba67 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,18 +1,17 @@ Contributing to `mongo_fdw` =========================== -Following these guidelines helps to facilitate relevant discussion in pull -requests and issues so the developers managing and developing this open source -project can address patches and bugs as efficiently as possible. +Following these guidelines helps to facilitate relevant discussion in +pull requests and issues so the developers managing and developing this +open source project can address patches and bugs as efficiently as +possible. Using Issues ------------ -`mongo_fdw`'s maintainers prefer that bug reports, feature requests, and pull -requests are submitted as [GitHub Issues][1]. If you think you require personal -assistance, please **do not** open an issue: email `mongo_fdw` `@` `enterprisedb.com` -instead. +`mongo_fdw`'s maintainers prefer that bug reports, feature requests, and +pull requests are submitted as [GitHub Issues][1]. Bug Reports @@ -21,20 +20,22 @@ Bug Reports Before opening a bug report: 1. Search for a duplicate issue using GitHub's issue search - 2. Check whethe the bug remains in the lasest `master` or `develop` commit - 3. Create a reduced test case: remove code and data not relevant to the bug + 2. Check whether the bug remains in the latest `master` or `develop` + commit + 3. Create a reduced test case: remove code and data not relevant to + the bug -A contributor should be able to begin work on your bug without asking too many -followup questions. If you include the following information, your bug will be -serviced more quickly: +A contributor should be able to begin work on your bug without asking +too many followup questions. If you include the following information, +your bug will be serviced more quickly: * Short, descriptive title * Your OS * Versions of dependencies * Any custom modifications -Once the background information is out of the way, you are free to present the -bug itself. You should explain: +Once the background information is out of the way, you are free to +present the bug itself. You should explain: * Steps you took to exercise the bug * The expected outcome @@ -44,42 +45,44 @@ bug itself. You should explain: Feature Requests ---------------- -We are open to adding features but ultimately control the scope and aims of the -project. If a proposed feature is likely to incur high testing, maintenance, or -performance costs it is also unlikely to be accepted. If a _strong_ case exists -for a given feature, we may be persuaded on merit. Be specific. +We are open to adding features but ultimately control the scope and aims +of the project. If a proposed feature is likely to incur high testing, +maintenance, or performance costs it is also unlikely to be accepted. +If a _strong_ case exists for a given feature, we may be persuaded on +merit. Be specific. Pull Requests ------------- -Well-constructed pull requests are very welcome. By _well-constructed_, we mean -they do not introduce unrelated changes or break backwards compatibility. Just -fork this repo and open a request against `develop`. +Well-constructed pull requests are very welcome. By _well-constructed_, +we mean they do not introduce unrelated changes or break backwards +compatibility. Just fork this repo and open a request against `develop`. -Some examples of things likely to increase the likelihood a pull request is -rejected: +Some examples of things likely to increase the likelihood a pull request +is rejected: * Large structural changes, including: - * Refactoring for its own sake + * Re-factoring for its own sake * Adding languages to the project - * Unnecesary whitespace changes + * Unnecessary whitespace changes * Deviation from obvious conventions * Introduction of incompatible intellectual property -Please do not change version numbers in your pull request: they will be updated -by the project owners prior to the next release. +Please do not change version numbers in your pull request: they will be +updated by the project owners prior to the next release. License ------- -By submitting a patch, you agree to allow the project owners to license your -work under the terms of the [`LICENSE`][2]. Additionally, you grant the project -owners a license under copyright covering your contribution to the extent -permitted by law. Finally, you confirm that you own said copyright, have the -legal authority to grant said license, and in doing so are not violating any -grant of rights you have made to third parties, including your employer. +By submitting a patch, you agree to allow the project owners to license +your work under the terms of the [`LICENSE`][2]. Additionally, you grant +the project owners a license under copyright covering your contribution +to the extent permitted by law. Finally, you confirm that you own said +copyright, have the legal authority to grant said license, and in doing +so are not violating any grant of rights you have made to third parties, +including your employer. [1]: https://github.com/EnterpriseDB/mongo_fdw/issues [2]: LICENSE diff --git a/README.md b/README.md index 82badd1..4c3984a 100644 --- a/README.md +++ b/README.md @@ -1,128 +1,178 @@ # MongoDB Foreign Data Wrapper for PostgreSQL -This [MongoDB][1] extension implements the PostgreSQL's Foreign Data Wrapper. +This PostgreSQL extension implements a Foreign Data Wrapper (FDW) for +[MongoDB][1]. -Please note that this version of mongo_fdw works with PostgreSQL and EDB Postgres Advanced Server 9.5, 9.6, 10, 11, 12, and 13. +Please note that this version of mongo_fdw works with PostgreSQL and EDB +Postgres Advanced Server 9.5, 9.6, 10, 11, 12, and 13. Installation ------------ -The MongoDB FDW depends on the official MongoDB C Driver version 0.8 and includes it as a git submodule. If you are cloning this repository for the first time, be sure to pass the --recursive option to git clone in order to initialize the driver submodule to a useable state. +The MongoDB FDW depends on the official MongoDB C Driver version 0.8 and +includes it as a git submodule. If you are cloning this repository for +the first time, be sure to pass the --recursive option to git clone in +order to initialize the driver submodule to a usable state. -If checked out this project before and for some reason your submodule is not up-to-date, run git submodule update --init. +If checked out this project before and for some reason your submodule +is not up-to-date, run git submodule update --init. -When you type `make`, the C driver's source code also gets automaticallycompiled and linked. +When you type `make`, the C driver's source code also gets automatically +compiled and linked. -Note: Make sure you have permission to "/usr/local" (default installation location) folder. +Note: Make sure you have permission to "/usr/local" +(default installation location) folder. -Note that we have verified the `mongo_fdw` extension only on MacOS X, -Fedora and Ubuntu systems. If you run into issues on other systems, please [let us know][3] +If you run into any issues, please [let us know][2]. Enhancements ----------- -The following enhancements are added to the latest version of mongo_fdw - -Write-able FDW --------------- -The previous version was only read-only, the latest version provides the write capability. -The user can now issue an insert / update and delete statements for the foreign tables using the mongo_fdw. - -Connection Pooling ------------------- -The latest version comes with a connection pooler that utilizes the same mango database connection for all the queries in the same session. The previous version would open a new [MongoDB][1] connection for every query. This is a performance enhancement. - -New MongoDB C Driver Support ----------------------------- -The third enhancement is to add a new [MongoDB][1]' C driver. The current implementation is based on the legacy driver of MongoDB. But [MongoDB][1] is provided completely new library for driver called MongoDB's Meta Driver. So I have added support of that driver. Now compile time option is available to use legacy and Meta driver. I am sure there are many other benefits of the new Mongo-C-driver that we are not leveraging but we will adopt those as we learn more about the new C driver. +The following enhancements are added to the latest version of mongo_fdw: + +### Write-able FDW +The previous version was only read-only, the latest version provides the +write capability. The user can now issue an insert, update, and delete +statements for the foreign tables using the mongo_fdw. + +### Connection Pooling +The latest version comes with a connection pooler that utilizes the +same mango database connection for all the queries in the same session. +The previous version would open a new [MongoDB][1] connection for every +query. This is a performance enhancement. + +### New MongoDB C Driver Support +The third enhancement is to add a new [MongoDB][1]' C driver. The +current implementation is based on the legacy driver of MongoDB. But +[MongoDB][1] is provided completely new library for driver called +MongoDB's meta driver. Added support for the same. Now compile time +option is available to use legacy and meta driver. In order to use MongoDB driver 1.0.0+, take the following steps: - * clone `libbson` version 1.0.0+ (https://github.com/mongodb/libbson). Follow install directions on that project's README. - * clone `libmongoc` version 1.0.0+ (https://github.com/mongodb/mongo-c-driver). Follow the install directions, except make sure to also run `./configure --with-libbson=system` after running automake but before running make. This should be the default behavior, but to be certain include this step. + * clone `libbson` version 1.0.0+ (https://github.com/mongodb/libbson). + Follow install directions on that project's README. + * clone `libmongoc` version 1.0.0+ + (https://github.com/mongodb/mongo-c-driver). Follow the + install directions, except make sure to also run `./configure + --with-libbson=system` after running automake but before running make. + This should be the default behavior, but to be certain include this + step. * ensure pkg-config / pkgconf is installed on your system. * run `make -f Makefile.meta && make -f Makefile.meta install` - * if you get an error when trying to `CREATE EXTENSION mongo_fdw;`, then try running `ldconfig` + * if you get an error when trying to `CREATE EXTENSION mongo_fdw;`, + then try running `ldconfig` Compilation script ----------------- -Number of manual steps needs to be performed to compile and install different type of MongoDB drivers and supported libraries. If you want to avoid the manual steps, there is a shell script available which will download and install the appropriate drivers and libraries for you. +Number of manual steps needs to be performed to compile and install +different type of MongoDB drivers and supported libraries. If you want +to avoid the manual steps, there is a shell script available which will +download and install the appropriate drivers and libraries for you. Here is how it works : Build with [MongoDB][1]'s legacy branch driver * autogen.sh --with-legacy -Build [MongoDB][1]'s master branch driver +Build [MongoDB][1]'s master branch driver * autogen.sh --with-master -The script will do all the necessary steps to build with legacy and metadriver accordingly. +The script will do all the necessary steps to build with legacy and +meta driver accordingly. Usage ----- The following parameters can be set on a MongoDB foreign server object: - * **`address`**: the address or hostname of the MongoDB server Defaults to `127.0.0.1` - * **`port`**: the port number of the MongoDB server. Defaults to `27017` - * **`authentication_database`**: database against which user will be authenticated against. Only valid with password based authentication. Defaults to per same database as the MongoDB collection database. - * **`replica_set`**: replica set the server is member of. If set, driver will auto-connect to correct primary in the replica set when writing. - * **`read_preference`**: primary [default], secondary, primaryPreferred, secondaryPreferred, or nearest (meta driver only). Defaults to `primary` - * **`ssl`**: false [default], true to enable ssl (meta driver only). See http://mongoc.org/libmongoc/current/mongoc_ssl_opt_t.html to understand the options. - * **`pem_file`**: SSL option; - * **`pem_pwd`**: SSL option; - * **`ca_file`**: SSL option; - * **`ca_dir`**: SSL option; - * **`crl_file`**: SSL option; - * **`weak_cert_validation`**: SSL option; + * `address`: Address or hostname of the MongoDB server. Defaults to + `127.0.0.1` + * `port`: Port number of the MongoDB server. Defaults to `27017`. + +The following options are only supported with meta driver: + + * `authentication_database`: Database against which user will be + authenticated against. Only valid with password based authentication. + * `replica_set`: Replica set the server is member of. If set, + driver will auto-connect to correct primary in the replica set when + writing. + * `read_preference`: primary [default], secondary, primaryPreferred, + secondaryPreferred, or nearest. + * `ssl`: false [default], true to enable ssl. See + http://mongoc.org/libmongoc/current/mongoc_ssl_opt_t.html to + understand the options. + * `pem_file`: SSL option. + * `pem_pwd`: SSL option. + * `ca_file`: SSL option. + * `ca_dir`: SSL option. + * `crl_file`: SSL option. + * `weak_cert_validation`: SSL option. The following parameters can be set on a MongoDB foreign table object: - * **`database`**: the name of the MongoDB database to query. Defaults to `test` - * **`collection`**: the name of the MongoDB collection to query. Defaults to the foreign table name used in the relevant `CREATE` command + * `database`: Name of the MongoDB database to query. Defaults to + `test`. + * `collection`: Name of the MongoDB collection to query. Defaults to + the foreign table name used in the relevant `CREATE` command. -As an example, the following commands demonstrate loading the `mongo_fdw` -wrapper, creating a server, and then creating a foreign table associated with -a MongoDB collection. The commands also show specifying option values in the -`OPTIONS` clause. If an option value isn't provided, the wrapper uses the -default value mentioned above. +The following parameters can be supplied while creating user mapping: -`mongo_fdw` can collect data distribution statistics will incorporate them when estimating costs for the query execution plan. To see selected execution plans for a query, just run `EXPLAIN`. + * `username`: Username to use when connecting to MongoDB. + * `password`: Password to authenticate to the MongoDB server. -Examples with [MongoDB][1]'s equivalent statments. +As an example, the following commands demonstrate loading the +`mongo_fdw` wrapper, creating a server, and then creating a foreign +table associated with a MongoDB collection. The commands also show +specifying option values in the `OPTIONS` clause. If an option value +isn't provided, the wrapper uses the default value mentioned above. -```sql +`mongo_fdw` can collect data distribution statistics will incorporate +them when estimating costs for the query execution plan. To see selected +execution plans for a query, just run `EXPLAIN`. + +Examples +-------- + +Examples with [MongoDB][1]'s equivalent statements. +```sql -- load extension first time after install CREATE EXTENSION mongo_fdw; -- create server object CREATE SERVER mongo_server - FOREIGN DATA WRAPPER mongo_fdw - OPTIONS (address '127.0.0.1', port '27017'); + FOREIGN DATA WRAPPER mongo_fdw + OPTIONS (address '127.0.0.1', port '27017'); -- create user mapping CREATE USER MAPPING FOR postgres - SERVER mongo_server - OPTIONS (username 'mongo_user', password 'mongo_pass'); + SERVER mongo_server + OPTIONS (username 'mongo_user', password 'mongo_pass'); -- create foreign table -CREATE FOREIGN TABLE warehouse( - _id NAME, - warehouse_id int, - warehouse_name text, - warehouse_created timestamptz) -SERVER mongo_server - OPTIONS (database 'db', collection 'warehouse'); - --- Note: first column of the table must be "_id" of type "NAME". +CREATE FOREIGN TABLE warehouse + ( + _id name, + warehouse_id int, + warehouse_name text, + warehouse_created timestamptz + ) + SERVER mongo_server + OPTIONS (database 'db', collection 'warehouse'); + +-- Note: first column of the table must be "_id" of type "name". -- select from table SELECT * FROM warehouse WHERE warehouse_id = 1; + _id | warehouse_id | warehouse_name | warehouse_created +--------------------------+--------------+----------------+--------------------------- + 53720b1904864dc1f5a571a0 | 1 | UPS | 2014-12-12 12:42:10+05:30 +(1 row) - _id | warehouse_id | warehouse_name | warehouse_created -------------------------+----------------+--------------------------- -53720b1904864dc1f5a571a0| 1 | UPS | 12-DEC-14 12:12:10 +05:00 - - -db.warehouse.find({"warehouse_id" : 1}).pretty() +db.warehouse.find +( + { + "warehouse_id" : 1 + } +).pretty() { "_id" : ObjectId("53720b1904864dc1f5a571a0"), "warehouse_id" : 1, @@ -130,98 +180,99 @@ db.warehouse.find({"warehouse_id" : 1}).pretty() "warehouse_created" : ISODate("2014-12-12T07:12:10Z") } - -- insert row in table -INSERT INTO warehouse values (0, 1, 'UPS', '2014-12-12T07:12:10Z'); +INSERT INTO warehouse VALUES (0, 2, 'Laptop', '2015-11-11T08:13:10Z'); db.warehouse.insert ( - { - "warehouse_id" : NumberInt(1), - "warehouse_name" : "UPS", - "warehouse_created" : ISODate("2014-12-12T07:12:10Z") - } -); + { + "warehouse_id" : NumberInt(2), + "warehouse_name" : "Laptop", + "warehouse_created" : ISODate("2015-11-11T08:13:10Z") + } +) -- delete row from table -DELETE FROM warehouse where warehouse_id = 3; - -> db.warehouse.remove({"warehouse_id" : 2}) +DELETE FROM warehouse WHERE warehouse_id = 2; +db.warehouse.remove +( + { + "warehouse_id" : 2 + } +) -- update a row of table -UPDATE warehouse set warehouse_name = 'UPS_NEW' where warehouse_id = 1; +UPDATE warehouse SET warehouse_name = 'UPS_NEW' WHERE warehouse_id = 1; db.warehouse.update ( - { - "warehouse_id" : 1 - }, - { - "warehouse_id" : 1, - "warehouse_name" : "UPS_NEW" - } + { + "warehouse_id" : 1 + }, + { + "warehouse_id" : 1, + "warehouse_name" : "UPS_NEW", + "warehouse_created" : ISODate("2014-12-12T07:12:10Z") + } ) -- explain a table EXPLAIN SELECT * FROM warehouse WHERE warehouse_id = 1; QUERY PLAN - ----------------------------------------------------------------- - Foreign Scan on warehouse (cost=0.00..0.00 rows=1000 width=44) +----------------------------------------------------------------- + Foreign Scan on warehouse (cost=0.00..0.00 rows=1000 width=84) Filter: (warehouse_id = 1) Foreign Namespace: db.warehouse - Planning time: 0.671 ms -(4 rows) +(3 rows) --- collect data distribution statistics` +-- collect data distribution statistics ANALYZE warehouse; -select mongo_fdw_version(); - mongo_fdw_version -------------------- - 50100 -(1 row) - - ``` Limitations ----------- - * If the BSON document key contains uppercase letters or occurs within a - nested document, `mongo_fdw` requires the corresponding column names to be - declared in double quotes. + * If the BSON document key contains uppercase letters or occurs within + a nested document, `mongo_fdw` requires the corresponding column names + to be declared in double quotes. - * Note that PostgreSQL limits column names to 63 characters by default. If - you need column names that are longer, you can increase the `NAMEDATALEN` - constant in `src/include/pg_config_manual.h`, compile, and reinstall. + * Note that PostgreSQL limits column names to 63 characters by + default. If you need column names that are longer, you can increase the + `NAMEDATALEN` constant in `src/include/pg_config_manual.h`, compile, + and re-install. Contributing ------------ -Have a fix for a bug or an idea for a great new feature? Great! Check out the contribution guidelines [here][4]. For all other types of questions or comments about the wrapper please contact us at `mongo_fdw` `@` `enterprisedb.com`. +Have a fix for a bug or an idea for a great new feature? Great! Check +out the contribution guidelines [here][3]. Support ------- -This project will be modified to maintain compatibility with new PostgreSQL and EDB Postgres Advanced Server releases. +This project will be modified to maintain compatibility with new +PostgreSQL and EDB Postgres Advanced Server releases. + +If you need commercial support, please contact the EnterpriseDB sales +team, or check whether your existing PostgreSQL support provider can +also support mongo_fdw. -If you need commercial support, please contact the EnterpriseDB sales team, or check whether your existing PostgreSQL support provider can also support mongo_fdw. License ------- Portions Copyright (c) 2004-2020, EnterpriseDB Corporation. Portions Copyright © 2012–2014 Citus Data, Inc. -This program is free software: you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free -Software Foundation, either version 3 of the License, or (at your option) any later version. +This program is free software: you can redistribute it and/or modify it +under the terms of the GNU Lesser General Public License as published by +the Free Software Foundation, either version 3 of the License, or (at +your option) any later version. -See the [`LICENSE`][5] file for full details. +See the [`LICENSE`][4] file for full details. [1]: http://www.mongodb.com -[2]: http://www.citusdata.com/blog/51-run-sql-on-mongodb -[3]: https://github.com/enterprisedb/mongo_fdw/issues/new -[4]: CONTRIBUTING.md -[5]: LICENSE -[6]: https://github.com/mongodb/mongo-c-driver-legacy -[7]: https://github.com/mongodb/mongo-c-driver +[2]: https://github.com/enterprisedb/mongo_fdw/issues/new +[3]: CONTRIBUTING.md +[4]: LICENSE