Monthly Archives: April 2014

Running OpenERP 8.0-trunk with pypy and psycopg2cffi

Returning from PyCon 2014, my enthusiasm for python is at an all time high.
Many of the talks I attended centered around the advantages of python3 and pypy. Being an OpenERP developper, this really stings since OpenERP stable (7.0) is configured to run on python2.6.

Knowning pypy is limited by its incompatibility with python ctypes, the question I asked myself today is:

Does OpenERP run in pypy?

In order to answer this question, we must first setup an enviroment which works in python2. For this experiment, I will use the trunk of the (presently) unrealeased version 8.0 of OpenERP. I will also be doing this inside of a virtualenv which is different from how I usually deploy OpenERP at Savoir-faire Linux where we use a buildout recipe.

Setting up a virtualenv

At the time of this writing, OpenERP 8.0 is still in the trunk repository.
After its planned release in June 2014, it should be available at lp:~openerp/openobject-server/8.0 and lp:~openerp/openerp-web/8.0

$ cd ~/openerp-to-pypy
$ virtualenv2 python2-openerp/
$ virtualenv-pypy pypy-openerp/
$ # the following command takes a while
$ bzr branch --stacked lp:~openerp/openobject-server/trunk ./openerp-server
$ bzr branch --stacked lp:~openerp/openerp-web/trunk ./openerp-web

For the purposes of this experiment, we won’t touch lp:~openobject-addons

For both virtualenvs, we need to install some dependencies

python2

$ source python2-openerp/bin/activate

$ pip install lxml pyyaml python-dateutil babel pillow simplejson \
              unittest2 psutil werkzeug reportlab mako Jinja2 \
              docutils mock
$ pip install http://download.gna.org/pychart/PyChart-1.39.tar.gz
$ pip install psycopg2  # to be replaced with psycopg2cffi

deactivate

pypy

For pypy, we skip psycop2 because it is not as efficient under pypy, we will substitute it with psycop2cffi which is a cffi implementation of psycop2 which is faster and more elegant than its ctypes counterpart.

$ source pypy-openerp/bin/activate

$ pip install lxml pyyaml python-dateutil babel pillow simplejson \
              unittest2 psutil werkzeug reportlab mako Jinja2 \
              docutils mock
$ pip install http://download.gna.org/pychart/PyChart-1.39.tar.gz
$ pip install psycopg2cffi psycopg2cffi-compat

$ deactivate

 Run comparison

python2

Nothing much to say about running python2, OpenERP is meant to run with python2 so there should be no issues with our setup. We will revisit this virtualenv, though.

$ createdb python2
$ source python2-openerp/bin/activate

$ ./openerp-server/openerp-server -d python2 --addons-path=./openerp-web/addons/

$ deactivate
$ dropdb python2

pypy

This is where the issues start popping up and you can see some of the legacy code of OpenERP holding it back.

$ createdb pypy
$ source pypy-openerp/bin/activate

$ ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/

$ deactivate
$ dropdb pypy

Output

Traceback (most recent call last):
 File "app_main.py", line 72, in run_toplevel
 File "./openerp-server/openerp-server", line 2, in <module>
 import openerp
 File "openerp-server/openerp/__init__.py", line 70, in <module>
 import cli
 File "openerp-server/openerp/cli/__init__.py", line 5, in <module>
 from openerp import tools
 File "openerp-server/openerp/tools/__init__.py", line 27, in <module>
 from convert import *
 File "openerp-server/openerp/tools/convert.py", line 35, in <module>
 import openerp.workflow
 File "openerp-server/openerp/workflow/__init__.py", line 22, in <module>
 from openerp.workflow.service import WorkflowService
 File "openerp-server/openerp/workflow/service.py", line 21, in <module>
 from helpers import Session
 File "openerp-server/openerp/workflow/helpers.py", line 1, in <module>
 import openerp.sql_db
 File "openerp-server/openerp/sql_db.py", line 38, in <module>
 from psycopg2.psycopg1 import cursor as psycopg1cursor
ImportError: No module named psycopg2.psycopg1

Indeed, openerp uses a legacy psycopg1 style cursor in its orm. Unfortunately, the cffi implementation of psycopg2 does away with this backwards compatibility.

The use of this, as far as I can tell, is to have access to the dictfetchall() function which is used quite a bit in server, as well as addons:

$ grep dictfetchall -r ./openerp-server | wc -l
44

Before we lose hope, let’s have a look at psycopg2.psycopg1.cursor

As we can see, it is a small wrapper of psycopg2.extensions.cursor and less than 35 lines of code. We could conceivebly move this legacy code into openerp-server/openerp/sql_db.py itself.

=== modified file 'openerp/sql_db.py'
--- openerp/sql_db.py 2014-04-14 07:59:06 +0000
+++ openerp/sql_db.py 2014-04-25 01:47:09 +0000
@@ -35,7 +35,46 @@
 import psycopg2.extensions
 from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT, ISOLATION_LEVEL_READ_COMMITTED, ISOLATION_LEVEL_REPEATABLE_READ
 from psycopg2.pool import PoolError
-from psycopg2.psycopg1 import cursor as psycopg1cursor
+from psycopg2.extensions import cursor as _2cursor
+
+
+class psycopg1cursor(_2cursor):
+ """psycopg 1.1.x cursor.
+
+Note that this cursor implements the exact procedure used by psycopg 1 to
+build dictionaries out of result rows. The DictCursor in the
+psycopg.extras modules implements a much better and faster algorithm.
+
+source: https://github.com/psycopg/psycopg2/blob/5a6a303d43f385f885c12d87240e86d6cb421463/lib/psycopg1.py#L61
+"""
+
+ def __build_dict(self, row):
+ res = {}
+ for i in range(len(self.description)):
+ res[self.description[i][0]] = row[i]
+ return res
+ 
+ def dictfetchone(self):
+ row = _2cursor.fetchone(self)
+ if row:
+ return self.__build_dict(row)
+ else:
+ return row
+ 
+ def dictfetchmany(self, size):
+ res = []
+ rows = _2cursor.fetchmany(self, size)
+ for row in rows:
+ res.append(self.__build_dict(row))
+ return res
+ 
+ def dictfetchall(self):
+ res = []
+ rows = _2cursor.fetchall(self)
+ for row in rows:
+ res.append(self.__build_dict(row))
+ return res
+
 
 psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)

With reimplemented cursor

Now if we run, we run into a wall

$ ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/

2014-04-25 01:48:35,012 18986 INFO ? openerp.modules.loading: init db
2014-04-25 01:48:35,671 18986 INFO pypy openerp.modules.loading: loading 1 modules...
2014-04-25 01:48:36,271 18986 INFO pypy openerp.modules.module: module base: creating or updating database tables
2014-04-25 01:48:37,856 18986 INFO pypy openerp.osv.orm: Computing parent left and right for table ir_ui_menu...
2014-04-25 01:48:45,002 18986 INFO pypy openerp.osv.orm: Computing parent left and right for table res_partner_category...
2014-04-25 01:48:48,702 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'display_name'
2014-04-25 01:48:48,709 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'views_by_module'
2014-04-25 01:48:48,735 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'reports_by_module'
2014-04-25 01:48:48,763 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'menus_by_module'
2014-04-25 01:48:48,783 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'web_icon_data'
2014-04-25 01:48:48,784 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'needaction_enabled'
2014-04-25 01:48:48,784 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'web_icon_hover_data'
2014-04-25 01:48:48,785 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'global'
2014-04-25 01:48:48,785 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'report_file'
2014-04-25 01:48:48,785 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'image_medium'
2014-04-25 01:48:48,786 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'image_small'
2014-04-25 01:48:48,787 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'commercial_partner_id'
2014-04-25 01:48:48,797 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'crud_model_name'
2014-04-25 01:48:48,798 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'wkf_model_name'
2014-04-25 01:48:48,798 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'logo_web'
2014-04-25 01:48:48,801 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'email'
2014-04-25 01:48:48,808 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'phone'
2014-04-25 01:48:48,824 18986 INFO pypy openerp.modules.loading: module base: loading base_data.xml
2014-04-25 01:48:49,481 18986 INFO pypy openerp.modules.loading: module base: loading res/res_currency_data.xml
2014-04-25 01:48:53,814 18986 INFO pypy openerp.modules.loading: module base: loading res/res_country_data.xml
2014-04-25 01:48:57,337 18986 INFO pypy openerp.modules.loading: module base: loading security/base_security.xml
2014-04-25 01:48:57,693 18986 INFO pypy openerp.modules.loading: module base: loading base_menu.xml
2014-04-25 01:48:57,900 18986 INFO pypy openerp.modules.loading: module base: loading res/res_security.xml
2014-04-25 01:48:57,957 18986 INFO pypy openerp.modules.loading: module base: loading res/res_config.xml
2014-04-25 01:48:58,009 18986 INFO pypy openerp.modules.loading: module base: loading res/res.country.state.csv
2014-04-25 01:48:58,261 18986 INFO pypy openerp.modules.loading: module base: loading ir/ir_actions.xml
2014-04-25 01:48:59,740 18986 INFO pypy openerp.modules.loading: module base: loading ir/ir_config_parameter_view.xml
2014-04-25 01:48:59,998 18986 INFO pypy openerp.modules.loading: module base: loading ir/ir_cron_view.xml
[1] 18986 segmentation fault (core dumped) ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/

I am not sure what causes this fault: pypy or OpenERP itself. A likely culprit would be lxml.

After a bit of digging, it would seem the segfault happens sometimes when calling the function on line 923 of openerp-server/openerp/tools/convert.py

We will skip this step for now by generating the pypy database with our python2 virtualenv

$ dropdb pypy
$ deactivate

$ createdb pypy
$ source python2-openerp/bin/activate

$ ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/ --stop-after-init

$ deactivate

$ source pypy-openerp/bin/activate
$ ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/

And there you have it! OpenERP running under pypy with the help of psycopg2cffi!

But wait…

… there’s more!

Edit: The following issue has been fixed in upstream psycopg2cffi.

Everything seems to be all well and good, besides the segfaulting when loading some models.

Let’s try creating a french database, forcing unicode into the enviroment. OpenERP is notorious for not handling unicode, afterall.

Logging out and selecting manage databases, we create a new database with a default language other than English.

Screenshot from 2014-04-24 22:12:55

2014-04-25 02:10:07,804 20716 INFO None openerp.service.db: Create database `new`.
2014-04-25 02:10:09,565 20716 ERROR None openerp.service.db: CREATE DATABASE failed:
Traceback (most recent call last):
 File "openerp-server/openerp/service/db.py", line 33, in _initialize_db
 with closing(db.cursor()) as cr:
 File "openerp-server/openerp/sql_db.py", line 585, in cursor
 return Cursor(self.__pool, self.dbname, serialized=serialized)
 File "openerp-server/openerp/sql_db.py", line 216, in __init__
 self._cnx = pool.borrow(dsn(dbname))
 File "openerp-server/openerp/sql_db.py", line 478, in _locked
 return fun(self, *args, **kwargs)
 File "openerp-server/openerp/sql_db.py", line 541, in borrow
 result = psycopg2.connect(dsn=dsn, connection_factory=PsycoConnection)
 File "pypy-openerp/site-packages/psycopg2cffi/__init__.py", line 109, in connect
 conn = _connect(dsn, connection_factory=connection_factory, async=async)
 File "pypy-openerp/site-packages/psycopg2cffi/_impl/connection.py", line 873, in _connect
 return connection_factory(dsn)
 File "pypy-openerp/site-packages/psycopg2cffi/_impl/connection.py", line 130, in __init__
 self._connect_sync()
 File "pypy-openerp/site-packages/psycopg2cffi/_impl/connection.py", line 135, in _connect_sync
 self._pgconn = libpq.PQconnectdb(self.dsn)
TypeError: initializer for ctype 'char *' must be a str or list or tuple, not unicode

This is a new error for me. It would seem the cffi declaration for libpq.PQconnectdb did not consider unicode strings (wchar *).

Take away

Switching to pypy was not too complicated thanks to the existance to psycopg2cffi and psycopg2cffi-compat.

Some legacy code needed to be copied over and this should serve as note to core developers to drop the psycopg1.cursor and use the DictCursor in the
psycopg.extras modules which implements a much better and faster algorithm.

Something went terribly wrong with xml loading in pypy causing segfaults. While I haven’t had the time to fully investigate, it would be beneficial to know the root cause to be able to fix either OpenERP, pypy or the other libraries used.

Finally, psycopg2cffi, while beautiful and fast, needs to patch unicode support for libpq.PQconnectdb.

Edit: Thank you to lopuhin who merged my fix to unicode support in psycopg2cffi.