Returning from PyCon 2014, my enthusiasm for python is at an all time high.
Many of the talks I attended centered around the advantages of python3 and pypy. Being an OpenERP developper, this really stings since OpenERP stable (7.0) is configured to run on python2.6.
Knowning pypy is limited by its incompatibility with python ctypes, the question I asked myself today is:
Does OpenERP run in pypy?
In order to answer this question, we must first setup an enviroment which works in python2. For this experiment, I will use the trunk of the (presently) unrealeased version 8.0 of OpenERP. I will also be doing this inside of a virtualenv which is different from how I usually deploy OpenERP at Savoir-faire Linux where we use a buildout recipe.
Setting up a virtualenv
At the time of this writing, OpenERP 8.0 is still in the trunk repository.
After its planned release in June 2014, it should be available at lp:~openerp/openobject-server/8.0 and lp:~openerp/openerp-web/8.0
$ cd ~/openerp-to-pypy $ virtualenv2 python2-openerp/ $ virtualenv-pypy pypy-openerp/ $ # the following command takes a while $ bzr branch --stacked lp:~openerp/openobject-server/trunk ./openerp-server $ bzr branch --stacked lp:~openerp/openerp-web/trunk ./openerp-web
For the purposes of this experiment, we won’t touch lp:~openobject-addons
For both virtualenvs, we need to install some dependencies
python2
$ source python2-openerp/bin/activate
$ pip install lxml pyyaml python-dateutil babel pillow simplejson \
unittest2 psutil werkzeug reportlab mako Jinja2 \
docutils mock
$ pip install http://download.gna.org/pychart/PyChart-1.39.tar.gz
$ pip install psycopg2 # to be replaced with psycopg2cffi
deactivate
pypy
For pypy, we skip psycop2 because it is not as efficient under pypy, we will substitute it with psycop2cffi which is a cffi implementation of psycop2 which is faster and more elegant than its ctypes counterpart.
$ source pypy-openerp/bin/activate
$ pip install lxml pyyaml python-dateutil babel pillow simplejson \
unittest2 psutil werkzeug reportlab mako Jinja2 \
docutils mock
$ pip install http://download.gna.org/pychart/PyChart-1.39.tar.gz
$ pip install psycopg2cffi psycopg2cffi-compat
$ deactivate
Run comparison
python2
Nothing much to say about running python2, OpenERP is meant to run with python2 so there should be no issues with our setup. We will revisit this virtualenv, though.
$ createdb python2 $ source python2-openerp/bin/activate $ ./openerp-server/openerp-server -d python2 --addons-path=./openerp-web/addons/ $ deactivate $ dropdb python2
pypy
This is where the issues start popping up and you can see some of the legacy code of OpenERP holding it back.
$ createdb pypy $ source pypy-openerp/bin/activate $ ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/ $ deactivate $ dropdb pypy
Output
Traceback (most recent call last): File "app_main.py", line 72, in run_toplevel File "./openerp-server/openerp-server", line 2, in <module> import openerp File "openerp-server/openerp/__init__.py", line 70, in <module> import cli File "openerp-server/openerp/cli/__init__.py", line 5, in <module> from openerp import tools File "openerp-server/openerp/tools/__init__.py", line 27, in <module> from convert import * File "openerp-server/openerp/tools/convert.py", line 35, in <module> import openerp.workflow File "openerp-server/openerp/workflow/__init__.py", line 22, in <module> from openerp.workflow.service import WorkflowService File "openerp-server/openerp/workflow/service.py", line 21, in <module> from helpers import Session File "openerp-server/openerp/workflow/helpers.py", line 1, in <module> import openerp.sql_db File "openerp-server/openerp/sql_db.py", line 38, in <module> from psycopg2.psycopg1 import cursor as psycopg1cursor ImportError: No module named psycopg2.psycopg1
Indeed, openerp uses a legacy psycopg1 style cursor in its orm. Unfortunately, the cffi implementation of psycopg2 does away with this backwards compatibility.
The use of this, as far as I can tell, is to have access to the dictfetchall() function which is used quite a bit in server, as well as addons:
$ grep dictfetchall -r ./openerp-server | wc -l 44
Before we lose hope, let’s have a look at psycopg2.psycopg1.cursor
As we can see, it is a small wrapper of psycopg2.extensions.cursor and less than 35 lines of code. We could conceivebly move this legacy code into openerp-server/openerp/sql_db.py itself.
=== modified file 'openerp/sql_db.py'
--- openerp/sql_db.py 2014-04-14 07:59:06 +0000
+++ openerp/sql_db.py 2014-04-25 01:47:09 +0000
@@ -35,7 +35,46 @@
import psycopg2.extensions
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT, ISOLATION_LEVEL_READ_COMMITTED, ISOLATION_LEVEL_REPEATABLE_READ
from psycopg2.pool import PoolError
-from psycopg2.psycopg1 import cursor as psycopg1cursor
+from psycopg2.extensions import cursor as _2cursor
+
+
+class psycopg1cursor(_2cursor):
+ """psycopg 1.1.x cursor.
+
+Note that this cursor implements the exact procedure used by psycopg 1 to
+build dictionaries out of result rows. The DictCursor in the
+psycopg.extras modules implements a much better and faster algorithm.
+
+source: https://github.com/psycopg/psycopg2/blob/5a6a303d43f385f885c12d87240e86d6cb421463/lib/psycopg1.py#L61
+"""
+
+ def __build_dict(self, row):
+ res = {}
+ for i in range(len(self.description)):
+ res[self.description[i][0]] = row[i]
+ return res
+
+ def dictfetchone(self):
+ row = _2cursor.fetchone(self)
+ if row:
+ return self.__build_dict(row)
+ else:
+ return row
+
+ def dictfetchmany(self, size):
+ res = []
+ rows = _2cursor.fetchmany(self, size)
+ for row in rows:
+ res.append(self.__build_dict(row))
+ return res
+
+ def dictfetchall(self):
+ res = []
+ rows = _2cursor.fetchall(self)
+ for row in rows:
+ res.append(self.__build_dict(row))
+ return res
+
psycopg2.extensions.register_type(psycopg2.extensions.UNICODE)
With reimplemented cursor
Now if we run, we run into a wall
$ ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/ 2014-04-25 01:48:35,012 18986 INFO ? openerp.modules.loading: init db 2014-04-25 01:48:35,671 18986 INFO pypy openerp.modules.loading: loading 1 modules... 2014-04-25 01:48:36,271 18986 INFO pypy openerp.modules.module: module base: creating or updating database tables 2014-04-25 01:48:37,856 18986 INFO pypy openerp.osv.orm: Computing parent left and right for table ir_ui_menu... 2014-04-25 01:48:45,002 18986 INFO pypy openerp.osv.orm: Computing parent left and right for table res_partner_category... 2014-04-25 01:48:48,702 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'display_name' 2014-04-25 01:48:48,709 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'views_by_module' 2014-04-25 01:48:48,735 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'reports_by_module' 2014-04-25 01:48:48,763 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'menus_by_module' 2014-04-25 01:48:48,783 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'web_icon_data' 2014-04-25 01:48:48,784 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'needaction_enabled' 2014-04-25 01:48:48,784 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'web_icon_hover_data' 2014-04-25 01:48:48,785 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'global' 2014-04-25 01:48:48,785 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'report_file' 2014-04-25 01:48:48,785 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'image_medium' 2014-04-25 01:48:48,786 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'image_small' 2014-04-25 01:48:48,787 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'commercial_partner_id' 2014-04-25 01:48:48,797 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'crud_model_name' 2014-04-25 01:48:48,798 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'wkf_model_name' 2014-04-25 01:48:48,798 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'logo_web' 2014-04-25 01:48:48,801 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'email' 2014-04-25 01:48:48,808 18986 INFO pypy openerp.osv.orm: storing computed values of fields.function 'phone' 2014-04-25 01:48:48,824 18986 INFO pypy openerp.modules.loading: module base: loading base_data.xml 2014-04-25 01:48:49,481 18986 INFO pypy openerp.modules.loading: module base: loading res/res_currency_data.xml 2014-04-25 01:48:53,814 18986 INFO pypy openerp.modules.loading: module base: loading res/res_country_data.xml 2014-04-25 01:48:57,337 18986 INFO pypy openerp.modules.loading: module base: loading security/base_security.xml 2014-04-25 01:48:57,693 18986 INFO pypy openerp.modules.loading: module base: loading base_menu.xml 2014-04-25 01:48:57,900 18986 INFO pypy openerp.modules.loading: module base: loading res/res_security.xml 2014-04-25 01:48:57,957 18986 INFO pypy openerp.modules.loading: module base: loading res/res_config.xml 2014-04-25 01:48:58,009 18986 INFO pypy openerp.modules.loading: module base: loading res/res.country.state.csv 2014-04-25 01:48:58,261 18986 INFO pypy openerp.modules.loading: module base: loading ir/ir_actions.xml 2014-04-25 01:48:59,740 18986 INFO pypy openerp.modules.loading: module base: loading ir/ir_config_parameter_view.xml 2014-04-25 01:48:59,998 18986 INFO pypy openerp.modules.loading: module base: loading ir/ir_cron_view.xml [1] 18986 segmentation fault (core dumped) ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/
I am not sure what causes this fault: pypy or OpenERP itself. A likely culprit would be lxml.
After a bit of digging, it would seem the segfault happens sometimes when calling the function on line 923 of openerp-server/openerp/tools/convert.py
We will skip this step for now by generating the pypy database with our python2 virtualenv
$ dropdb pypy $ deactivate $ createdb pypy $ source python2-openerp/bin/activate $ ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/ --stop-after-init $ deactivate $ source pypy-openerp/bin/activate $ ./openerp-server/openerp-server -d pypy --addons-path=./openerp-web/addons/
And there you have it! OpenERP running under pypy with the help of psycopg2cffi!
But wait…
… there’s more!
Edit: The following issue has been fixed in upstream psycopg2cffi.
Everything seems to be all well and good, besides the segfaulting when loading some models.
Let’s try creating a french database, forcing unicode into the enviroment. OpenERP is notorious for not handling unicode, afterall.
Logging out and selecting manage databases, we create a new database with a default language other than English.
2014-04-25 02:10:07,804 20716 INFO None openerp.service.db: Create database `new`. 2014-04-25 02:10:09,565 20716 ERROR None openerp.service.db: CREATE DATABASE failed: Traceback (most recent call last): File "openerp-server/openerp/service/db.py", line 33, in _initialize_db with closing(db.cursor()) as cr: File "openerp-server/openerp/sql_db.py", line 585, in cursor return Cursor(self.__pool, self.dbname, serialized=serialized) File "openerp-server/openerp/sql_db.py", line 216, in __init__ self._cnx = pool.borrow(dsn(dbname)) File "openerp-server/openerp/sql_db.py", line 478, in _locked return fun(self, *args, **kwargs) File "openerp-server/openerp/sql_db.py", line 541, in borrow result = psycopg2.connect(dsn=dsn, connection_factory=PsycoConnection) File "pypy-openerp/site-packages/psycopg2cffi/__init__.py", line 109, in connect conn = _connect(dsn, connection_factory=connection_factory, async=async) File "pypy-openerp/site-packages/psycopg2cffi/_impl/connection.py", line 873, in _connect return connection_factory(dsn) File "pypy-openerp/site-packages/psycopg2cffi/_impl/connection.py", line 130, in __init__ self._connect_sync() File "pypy-openerp/site-packages/psycopg2cffi/_impl/connection.py", line 135, in _connect_sync self._pgconn = libpq.PQconnectdb(self.dsn) TypeError: initializer for ctype 'char *' must be a str or list or tuple, not unicode
This is a new error for me. It would seem the cffi declaration for libpq.PQconnectdb did not consider unicode strings (wchar *).
Take away
Switching to pypy was not too complicated thanks to the existance to psycopg2cffi and psycopg2cffi-compat.
Some legacy code needed to be copied over and this should serve as note to core developers to drop the psycopg1.cursor and use the DictCursor in the
psycopg.extras modules which implements a much better and faster algorithm.
Something went terribly wrong with xml loading in pypy causing segfaults. While I haven’t had the time to fully investigate, it would be beneficial to know the root cause to be able to fix either OpenERP, pypy or the other libraries used.
Finally, psycopg2cffi, while beautiful and fast, needs to patch unicode support for libpq.PQconnectdb.
Edit: Thank you to lopuhin who merged my fix to unicode support in psycopg2cffi.
bwrsandman talks code 