Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
fca6bb0
Connection Pooler for mongodb similar to PostgreSQL FDW.
ibrarahmed1974 Jun 2, 2014
955350c
Plemove DeserializeDocument and SerializeDocument functions which are
ibrarahmed1974 Jun 2, 2014
d8d8f48
Write support (INSERT/UPDATE/DELETE).
ibrarahmed1974 Jun 2, 2014
5f881c9
MongoDB's Meta Driver support added.
ibrarahmed1974 Jun 2, 2014
3dd8ef6
Added files missed during last commits.
ibrarahmed1974 Jun 2, 2014
aacb0c8
Restrict first column name of MongoDB's foreign table to "_id".
ibrarahmed1974 Jun 2, 2014
b67102c
README Changes
ibrarahmed1974 Jun 2, 2014
a501e6d
README file typos and cosmetic changes.
ibrarahmed1974 Jun 2, 2014
4ead1e4
Compiler warning of unused variable fixed.
ibrarahmed1974 Jun 2, 2014
2e5a491
Default Makefile to compile FDW with MongoDB's legacy Driver.
ibrarahmed1974 Jun 2, 2014
e31a193
Licence information update
ibrarahmed1974 Jun 2, 2014
0fe3122
README file changes.
ibrarahmed1974 Jun 2, 2014
9f971d9
Numeric array support for MongoDB's legacy and meta driver added
ibrarahmed1974 Jun 4, 2014
bce5fc3
README file changes
ibrarahmed1974 Jun 5, 2014
1fd94bc
README file changes.
ibrarahmed1974 Jun 6, 2014
2007c72
Throw error in case type of first column of MongoDB's foreign Table n…
ibrarahmed1974 Jun 6, 2014
990bdf1
Wrong BSON object was passed in case of Meta Drive
ibrarahmed1974 Jun 6, 2014
87b026c
The "analyze" statement throwing error while retrieving rows from Mon…
ibrarahmed1974 Jun 9, 2014
9f7e125
Misc changes to readme
Jun 24, 2014
19f16f4
Update CONTRIBUTING.md
Jun 24, 2014
08bbee6
Authentication using USER mapping
ibrarahmed1974 Jul 10, 2014
f4f6c42
README Changes for user mapping
ibrarahmed1974 Jul 10, 2014
9b66014
Merge commit 'c632ac480e6fffc79c32845a34488574fc5bc685'
truongsinh Sep 24, 2014
0223b23
Merge pull request #2 from truongsinh/master
ibrarahmad Sep 25, 2014
9da15a0
License header change
ibrarahmad Sep 30, 2014
b6aa5e1
PostgreSQL Version 9.5 support
ibrarahmad Sep 30, 2014
f814283
PostgreSQL Version 9.5 support
ibrarahmad Sep 30, 2014
c9b3461
README File changes
ibrarahmad Sep 30, 2014
e5d1ec0
Error message typos fix
ibrarahmad Oct 1, 2014
8167398
README typo fix
ibrarahmad Oct 1, 2014
a4b043c
Function name change in connection.c and option. c
ibrarahmad Oct 1, 2014
da603d5
README Changes
ibrarahmad Oct 1, 2014
8e6ca32
read/write binary data (bytea)
truongsinh Oct 2, 2014
486e7d6
Merge pull request #4 from truongsinh/master
ibrarahmad Oct 2, 2014
7a3d32e
support write text[] (TEXTARRAYOID)
truongsinh Oct 28, 2014
9872f6f
Merge pull request #6 from truongsinh/master
ibrarahmad Oct 30, 2014
c25671b
Error #7: Fixed compiler warning
ibrarahmad Oct 30, 2014
5794527
Error #8: Using serverid and userid for connection pooling instead of…
ibrarahmad Oct 30, 2014
005b342
Fixed some formating issues.
ibrarahmad Oct 30, 2014
2ad3238
Change the Option's variable name from mongoFdwOptions to options
ibrarahmad Oct 30, 2014
cc070a8
write JSON (object and array)
truongsinh Nov 7, 2014
5bae0f8
$date and $oid
truongsinh Nov 7, 2014
9b2679d
NAMEARRAYOID
truongsinh Nov 11, 2014
4421151
Ignore the value of first column which is row identifier in MongoDb (…
truongsinh Nov 14, 2014
ff7236d
allow RETURNING
truongsinh Nov 26, 2014
46d45ea
fix bug when writing array of numeric, text or name
truongsinh Nov 28, 2014
80bb2ab
fix BsonAppendBool
truongsinh Dec 3, 2014
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
[submodule "mongo-c-driver"]
path = mongo-c-driver
url = ../../mongodb/mongo-c-driver.git
[submodule "json-c"]
path = json-c
url = https://github.com/json-c/json-c.git
4 changes: 2 additions & 2 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ Using Issues

`mongo_fdw`'s maintainers prefer that bug reports, feature requests, and pull
requests are submitted as [GitHub Issues][1]. If you think you require personal
assistance, please **do not** open an issue: email `engage` `@` `citusdata.com`
assistance, please **do not** open an issue: email `mongo_fdw` `@` `enterprisedb.com`
instead.


Expand Down Expand Up @@ -81,5 +81,5 @@ permitted by law. Finally, you confirm that you own said copyright, have the
legal authority to grant said license, and in doing so are not violating any
grant of rights you have made to third parties, including your employer.

[1]: https://github.com/citusdata/mongo_fdw/issues
[1]: https://github.com/EnterpriseDB/mongo_fdw/issues
[2]: LICENSE
24 changes: 19 additions & 5 deletions Makefile
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
# mongo_fdw/Makefile
#
# Copyright (c) 2012-2014 Citus Data, Inc.
# Portions Copyright © 2004-2014, EnterpriseDB Corporation.
#
# Portions Copyright © 2012–2014 Citus Data, Inc.
#

MODULE_big = mongo_fdw
Expand All @@ -10,14 +12,16 @@ MODULE_big = mongo_fdw
# on another platform, change env_posix.os in MONGO_OBJS with the appropriate
# environment object file.
#

MONGO_DRIVER = mongo-c-driver
MONGO_PATH = $(MONGO_DRIVER)/src
MONGO_OBJS = $(MONGO_PATH)/bson.os $(MONGO_PATH)/encoding.os $(MONGO_PATH)/md5.os \
$(MONGO_PATH)/mongo.os $(MONGO_PATH)/numbers.os $(MONGO_PATH)/env.os

PG_CPPFLAGS = --std=c99 -I$(MONGO_PATH)
OBJS = mongo_fdw.o mongo_query.o $(MONGO_OBJS)
LIBJSON = json-c
LIBJSON_OBJS = $(LIBJSON)/json_util.o $(LIBJSON)/json_object.o $(LIBJSON)/json_tokener.o \
$(LIBJSON)/json_object_iterator.o $(LIBJSON)/printbuf.o $(LIBJSON)/linkhash.o \
$(LIBJSON)/arraylist.o $(LIBJSON)/random_seed.o $(LIBJSON)/debug.o
PG_CPPFLAGS = --std=c99 -I$(MONGO_PATH) -I$(LIBJSON)
OBJS = connection.o option.o mongo_wrapper.o mongo_fdw.o mongo_query.o $(MONGO_OBJS) $(LIBJSON_OBJS)

EXTENSION = mongo_fdw
DATA = mongo_fdw--1.0.sql
Expand All @@ -28,6 +32,8 @@ REGRESS_OPTS = --inputdir=test --outputdir=test \

$(MONGO_DRIVER)/%.os:
$(MAKE) -C $(MONGO_DRIVER) $*.os
#$(LIBJSON)/json.o:
# $(MAKE) -C $(LIBJSON)

#
# Users need to specify their Postgres installation path through pg_config. For
Expand All @@ -37,3 +43,11 @@ $(MONGO_DRIVER)/%.os:
PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

ifndef MAJORVERSION
MAJORVERSION := $(basename $(VERSION))
endif

ifeq (,$(findstring $(MAJORVERSION), 9.3 9.4 9.5))
$(error PostgreSQL 9.3, 9.4 or 9.5 is required to compile this extension)
endif
46 changes: 46 additions & 0 deletions Makefile.meta
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
# mongo_fdw/Makefile.meta
#
# Portions Copyright © 2004-2014, EnterpriseDB Corporation.
#
# Portions Copyright © 2012–2014 Citus Data, Inc.
#

MODULE_big = mongo_fdw

#
# We assume we are running on a POSIX compliant system (Linux, OSX). If you are
# on another platform, change env_posix.os in MONGO_OBJS with the appropriate
# environment object file.
#

MONGO_DRIVER = mongo-c-meta-driver
MONGO_PATH = $(MONGO_DRIVER)/src/mongoc
MONGO_INCLUDE = -I$(MONGO_DRIVER)/src/libbson/src/bson/ -I$(MONGO_PATH)
PG_CPPFLAGS = --std=c99 $(MONGO_INCLUDE)
SHLIB_LINK = -L$(MONGO_DRIVER)/.libs -lmongoc-1.0
SHLIB_LINK += -L$(MONGO_DRIVER)/src/libbson/.libs -lbson-1.0

OBJS = connection.o option.o mongo_wrapper_meta.o mongo_fdw.o mongo_query.o

EXTENSION = mongo_fdw
DATA = mongo_fdw--1.0.sql

$(MONGO_DRIVER)/%.os:
$(MAKE) -C $(MONGO_DRIVER) $*.os

#
# Users need to specify their Postgres installation path through pg_config. For
# example: /usr/local/pgsql/bin/pg_config or /usr/lib/postgresql/9.1/bin/pg_config
#

PG_CONFIG = pg_config
PGXS := $(shell $(PG_CONFIG) --pgxs)
include $(PGXS)

ifndef MAJORVERSION
MAJORVERSION := $(basename $(VERSION))
endif

ifeq (,$(findstring $(MAJORVERSION), 9.3 9.4 9.5))
$(error PostgreSQL 9.3, 9.4 or 9.5 is required to compile this extension)
endif
173 changes: 122 additions & 51 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,12 @@
MongoDB FDW for PostgreSQL
==========================
# MongoDB Foreign Data Wrapper for PostgreSQL

This PostgreSQL extension implements a Foreign Data Wrapper (FDW) for
[MongoDB][1]. For an example demonstrating this wrapper's use, see [our blog
post][2]. Please also note that this version of `mongo_fdw` only works with
PostgreSQL 9.2 or 9.3.
This [MongoDB][1] extension implements the PostgreSQL's Foreign Data Wrapper.

Please note that this version of `mongo_fdw` only works with
PostgreSQL Version **9.3** and greater.

Installation
------------

The MongoDB FDW depends on the official MongoDB C Driver version 0.8 and
includes it as a git submodule. If you are cloning this repository for the
first time, be sure to pass the --recursive option to git clone in order to
Expand All @@ -22,33 +19,47 @@ not up-to-date, run git submodule update --init.
When you type `make`, the C driver's source code also gets automatically
compiled and linked.

To build on POSIX-compliant systems (like Linux and OS X), you need to ensure
the `pg_config` executable is in your path when you run `make`. This executable
is typically in your PostgreSQL installation's `bin` directory. For example:

```sh
PATH=/usr/local/pgsql/bin/:$PATH make
sudo PATH=/usr/local/pgsql/bin/:$PATH make install
```
Note: Make sure you have permission to "/usr/local" (default installation location) folder.

Note that we have tested the `mongo_fdw` extension only on Fedora and Ubuntu
systems. If you run into issues on other systems, please [let us know][3].
Note that we have verified the `mongo_fdw` extension only on MacOS X,
Fedora and Ubuntu systems. If you run into issues on other systems, please [let us know][3]

Enhancements
-----------
The following enhancements are added to the latest version of mongo_fdw

Write-able FDW
--------------
The previous version was only read-only, the latest version provides the write capability.
The user can now issue insert/update and delete statements for the foreign tables using the mongo_fdw.

Connection Pooling
------------------
The latest version comes with a connection pooler that utilises the same mongo
database connection for all the queries in the same session. The previous version
would open a new mongodb connection for every query.
This is a performance enhancement.

New MongoDB C Driver Support
----------------------------
The third enhancement is to add a new [MongoDB][1]' C driver. The current implementation is
based on the legacy driver of MongoDB. But [MongoDB][1] is provided completely new library
for driver called MongoDB's Meta Driver. So I have added support of that driver.
Now compile time option is available to use legacy and Meta driver. I am sure there
are many other benefits of the new Mongo-C-driver that we are not leveraging but we
will adopt those as we learn more about the new C driver.

Usage
-----

The following parameters can be set on a MongoDB foreign server object:

* `address`: the address or hostname of the MongoDB server.
Defaults to `127.0.0.1`
* `port`: the port number of the MongoDB server. Defaults to `27017`
* **`address`**: the address or hostname of the MongoDB server Defaults to `127.0.0.1`
* **`port`**: the port number of the MongoDB server. Defaults to `27017`

The following parameters can be set on a MongoDB foreign table object:

* `database`: the name of the MongoDB database to query. Defaults to `test`
* `collection`: the name of the MongoDB collection to query. Defaults to
the foreign table name used in the relevant `CREATE` command
* **`database`**: the name of the MongoDB database to query. Defaults to `test`
* **`collection`**: the name of the MongoDB collection to query. Defaults to the foreign table name used in the relevant `CREATE` command

As an example, the following commands demonstrate loading the `mongo_fdw`
wrapper, creating a server, and then creating a foreign table associated with
Expand All @@ -60,46 +71,104 @@ default value mentioned above.
estimating costs for the query execution plan. To see selected execution plans
for a query, just run `EXPLAIN`.

We also currently use the internal PostgreSQL `NAME` type to represent the BSON
object identifier type (the `_id` field).
Examples with [MongoDB][1]'s equivalent statments.

```sql

-- load extension first time after install
CREATE EXTENSION mongo_fdw;

-- create server object
CREATE SERVER mongo_server FOREIGN DATA WRAPPER mongo_fdw
OPTIONS (address '127.0.0.1', port '27017');
CREATE SERVER mongo_server
FOREIGN DATA WRAPPER mongo_fdw
OPTIONS (address '127.0.0.1', port '27017');

-- create user mapping
CREATE USER MAPPING FOR postgres
SERVER mongo_server
OPTIONS (username 'mongo_user', password 'mongo_pass');

-- create foreign table
CREATE FOREIGN TABLE customer_reviews
CREATE FOREIGN TABLE warehouse(
_id NAME,
warehouse_id int,
warehouse_name text,
warehouse_created timestamptz)
SERVER mongo_server
OPTIONS (database 'db', collection 'warehouse');

-- Note: first column of the table must be "_id" of type "NAME".

-- select from table
SELECT * FROM warehouse WHERE warehouse_id = 1;

_id | warehouse_id | warehouse_name | warehouse_created
------------------------+----------------+---------------------------
53720b1904864dc1f5a571a0| 1 | UPS | 12-DEC-14 12:12:10 +05:00


db.warehouse.find({"warehouse_id" : 1}).pretty()
{
"_id" : ObjectId("53720b1904864dc1f5a571a0"),
"warehouse_id" : 1,
"warehouse_name" : "UPS",
"warehouse_created" : ISODate("2014-12-12T07:12:10Z")
}


-- insert row in table
INSERT INTO warehouse values (0, 1, 'UPS', to_date('2014-12-12T07:12:10Z'));

db.warehouse.insert
(
{
"warehouse_id" : NumberInt(1),
"warehouse_name" : "UPS",
"warehouse_created" : ISODate("2014-12-12T07:12:10Z")
}
);

-- delete row from table
DELETE FROM warehouse where warehouse_id = 3;

> db.warehouse.remove({"warehouse_id" : 2})


-- update a row of table
UPDATE warehouse set warehouse_name = 'UPS_NEW' where warehouse_id = 1;

db.warehouse.update
(
_id NAME,
customer_id TEXT,
review_date TIMESTAMP,
review_rating INTEGER,
product_id CHAR(10),
product_title TEXT,
product_group TEXT,
product_category TEXT,
similar_product_ids CHAR(10)[]
{
"warehouse_id" : 1
},
{
"warehouse_id" : 1,
"warehouse_name" : "UPS_NEW"
}
)
SERVER mongo_server
OPTIONS (database 'test', collection 'customer_reviews');

-- collect data distribution statistics
ANALYZE customer_reviews;
```
-- explain a table
EXPLAIN SELECT * FROM warehouse WHERE warehouse_id = 1;
QUERY PLAN
-----------------------------------------------------------------
Foreign Scan on warehouse (cost=0.00..0.00 rows=1000 width=44)
Filter: (warehouse_id = 1)
Foreign Namespace: db.warehouse
Planning time: 0.671 ms
(4 rows)

-- collect data distribution statistics`
ANALYZE warehouse;

```

Limitations
-----------

* If the BSON document key contains uppercase letters or occurs within a
nested document, `mongo_fdw` requires the corresponding column names to be
declared in double quotes. For example, a nested field such as `"review": {
"Votes": 19 }` should be declared as `"review.Votes" INTEGER` in the `CREATE
TABLE` statement.
declared in double quotes.

* Note that PostgreSQL limits column names to 63 characters by default. If
you need column names that are longer, you can increase the `NAMEDATALEN`
Expand All @@ -108,25 +177,25 @@ Limitations

Contributing
------------

Have a fix for a bug or an idea for a great new feature? Great! Check out the
contribution guidelines [here][4]. For all other types of questions or comments
about the wrapper please contact us at `engage` `@` `citusdata.com`.
about the wrapper please contact us at `mongo_fdw` `@` `enterprisedb.com`.


Support
-------

This project will be modified to maintain compatibility with new PostgreSQL
releases. The project owners set aside a day every month to look over open
issues and support emails, but are not engaged in active feature development.
Reported bugs will be addressed by apparent severity.

As with many open source projects, you may be able to obtain support via the public mailing list (`mongo_fdw` `@` `enterprisedb.com`). If you need commercial support, please contact the EnterpriseDB sales team, or check whether your existing PostgreSQL support provider can also support mongo_fdw.

License
-------
Portions Copyright © 2004-2014, EnterpriseDB Corporation.

Copyright © 2012–2014 Citus Data, Inc.
Portions Copyright © 2012–2014 Citus Data, Inc.

This program is free software: you can redistribute it and/or modify it under
the terms of the GNU Lesser General Public License as published by the Free
Expand All @@ -137,6 +206,8 @@ See the [`LICENSE`][5] file for full details.

[1]: http://www.mongodb.com
[2]: http://www.citusdata.com/blog/51-run-sql-on-mongodb
[3]: https://github.com/citusdata/mongo_fdw/issues/new
[3]: https://github.com/enterprisedb/mongo_fdw/issues/new
[4]: CONTRIBUTING.md
[5]: LICENSE
[6]: https://github.com/mongodb/mongo-c-driver-legacy
[7]: https://github.com/mongodb/mongo-c-driver
22 changes: 22 additions & 0 deletions config.h
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
/*-------------------------------------------------------------------------
*
* config.h
* Foreign-data wrapper for remote MongoDB servers
*
* Portions Copyright (c) 2012-2014, PostgreSQL Global Development Group
*
* Portions Copyright (c) 2004-2014, EnterpriseDB Corporation.
*
* Portions Copyright (c) 2012–2014 Citus Data, Inc.
*
* IDENTIFICATION
* config.h
*
*-------------------------------------------------------------------------
*/

/*
* Define if you want to compile the MongoFDW with Meta C Driver, otherwise
* it will compile using MongoDB legacy C Driver
*/
/* #define META_DRIVER */
Loading