pjpersist

A Python PostgreSQL/JSONB Persistence Backend.

Providing transparent persistence of Python objects.

This document outlines the general capabilities of the pjpersist package. pjpersist is a PostgreSQL/JSONB storage implementation for persistent Python objects. It is NOT a storage for the ZODB.

The goal of pjpersist is to provide a data manager that serializes objects to PostgreSQL using JSONB at transaction boundaries. The PJ data manager is a persistent data manager, which handles events at transaction boundaries (see transaction.interfaces.IDataManager) as well as events from the persistency framework (see persistent.interfaces.IPersistentDataManager).

An instance of a data manager is supposed to have the same life time as the transaction, meaning that it is assumed that you create a new data manager when creating a new transaction:

>>> import transaction

Let's now define a simple persistent object:

>>> import persistent

>>> class Person(ReprMixin, persistent.Persistent):
...
...     def __init__(self, name, phone=None, address=None, friends=None,
...                  visited=(), birthday=None):
...         self.name = name
...         self.address = address
...         self.friends = friends or {}
...         self.visited = visited
...         self.phone = phone
...         self.birthday = birthday
...         self.today = datetime.datetime(2014, 12, 4, 12, 30, 0)
...
...     def __str__(self):
...         return self.name

Let's create a new person and store it in PostgreSQL:

>>> stephan = Person(u'Stephan')
>>> dm.root['stephan'] = stephan

By default, persistent objects are stored in a tabke having the Python path of the class. Since table names cannot statewith an underscore and contain dots, we have to escpae the path a little bit. Let's see what got stored in PostgreSQL:

>>> dumpTable('u__main___dot_Person')
[{'data': {u'_py_persistent_type': u'__main__.Person',
           u'address': None,
           u'birthday': None,
           u'friends': {},
           u'name': u'Stephan',
           u'phone': None,
           u'today': {u'_py_type': u'datetime.datetime',
                      u'value': u'2014-12-04T12:30:00.000000'},
           u'visited': []},
  'id': u'0001020304050607080a0b0c0'}]

Let's now add an address for Stephan. Addresses are also persistent objects:

>>> class Address(ReprMixin, persistent.Persistent):
...     _p_pj_table = 'address'
...
...     def __init__(self, city, zip):
...         self.city = city
...         self.zip = zip
...
...     def __str__(self):
...         return '%s (%s)' %(self.city, self.zip)

>>> stephan.address = Address('Maynard', '01754')

We need to commit the transaction, to push the data to PostgreSQL:

>>> transaction.commit()

>>> dumpTable('address')
[{'data': {u'_py_persistent_type': u'__main__.Address',
           u'city': u'Maynard',
           u'zip': u'01754'},
  'id': u'0001020304050607080a0b0c0'}]

As you can see, even the reference to the Address object looks nice and uses the standard PostgreSQL reference construct.

>>> dumpTable('u__main___dot_Person')
[{'data': {u'_py_persistent_type': u'__main__.Person',
           u'address': {u'_py_type': u'DBREF',
                        u'database': u'pjpersist_test',
                        u'id': u'0001020304050607080a0b0c0',
                        u'table': u'address'},
           u'birthday': None,
           u'friends': {},
           u'name': u'Stephan',
           u'phone': None,
           u'today': {u'_py_type': u'datetime.datetime',
                      u'value': u'2014-12-04T12:30:00.000000'},
           u'visited': []},
  'id': u'0001020304050607080a0b0c0'}]

But what about arbitrary non-persistent, but picklable, objects? Well, let's create a phone number object for that:

>>> class Phone(ReprMixin):
...
...     def __init__(self, country, area, number):
...         self.country = country
...         self.area = area
...         self.number = number
...
...     def __str__(self):
...         return '%s-%s-%s' %(self.country, self.area, self.number)

>>> stephan = dm.root['stephan']
>>> stephan.phone = Phone('+1', '978', '394-5124')
>>> transaction.commit()

>>> dumpTable('u__main___dot_Person')
[{'data': {u'_py_persistent_type': u'__main__.Person',
           u'address': {u'_py_type': u'DBREF',
                        u'database': u'pjpersist_test',
                        u'id': u'0001020304050607080a0b0c0',
                        u'table': u'address'},
           u'birthday': None,
           u'friends': {},
           u'name': u'Stephan',
           u'phone': {u'_py_type': u'__main__.Phone',
                      u'area': u'978',
                      u'country': u'+1',
                      u'number': u'394-5124'},
           u'today': {u'_py_type': u'datetime.datetime',
                      u'value': u'2014-12-04T12:30:00.000000'},
           u'visited': []},
  'id': u'0001020304050607080a0b0c0'}]

Let's now set various attributes:

>>> stephan = dm.root['stephan']
>>> stephan.friends = {'roy': Person(u'Roy Mathew')}
>>> stephan.visited = (u'Germany', u'USA')
>>> stephan.birthday = datetime.date(1980, 1, 25)

Push the data to PostgreSQL, and dump the results:

>>> transaction.commit()
>>> dumpTable('u__main___dot_Person')
[{'data': {u'_py_persistent_type': u'__main__.Person',
           u'address': {u'_py_type': u'DBREF',
                        u'database': u'pjpersist_test',
                        u'id': u'0001020304050607080a0b0c0',
                        u'table': u'address'},
           u'birthday': {u'_py_type': u'datetime.date',
                         u'value': u'1980-01-25'},
           u'friends': {u'roy': {u'_py_type': u'DBREF',
                                 u'database': u'pjpersist_test',
                                 u'id': u'0001020304050607080a0b0c0',
                                 u'table': u'u__main___dot_Person'}},
           u'name': u'Stephan',
           u'phone': {u'_py_type': u'__main__.Phone',
                      u'area': u'978',
                      u'country': u'+1',
                      u'number': u'394-5124'},
           u'today': {u'_py_type': u'datetime.datetime',
                      u'value': u'2014-12-04T12:30:00.000000'},
           u'visited': [u'Germany', u'USA']},
  'id': u'0001020304050607080a0b0c0'},
 {'data': {u'_py_persistent_type': u'__main__.Person',
           u'address': None,
           u'birthday': None,
           u'friends': {},
           u'name': u'Roy Mathew',
           u'phone': None,
           u'today': {u'_py_type': u'datetime.datetime',
                      u'value': u'2014-12-04T12:30:00.000000'},
           u'visited': []},
  'id': u'0001020304050607080a0b0c0'}]

Of course all properties can be retrieved as python objects:

>>> stephan = dm.root['stephan']
>>> stephan.address
<Address Maynard (01754)>

>>> stephan.address.city
u'Maynard'

>>> stephan.birthday
datetime.date(1980, 1, 25)

>>> stephan.friends
{u'roy': <Person Roy Mathew>}

>>> stephan.phone
<Phone +1-978-394-5124>

>>> stephan.today
datetime.datetime(2014, 12, 4, 12, 30)

>>> stephan.visited
[u'Germany', u'USA']

See src/pjpersist/README.txt and the other txt files in the package for more details.