Wednesday, March 28, 2012
Problems Creating .pyc Files
Having clojure-py create .pyc files should greatly improve start-up times, hopefully bringing them into the 0.2 sec range. To start with, let me begin by explaining how python generates .pyc files.
pyc files are basically marshaled code objects. As such all data contained in code.co_consts must conform to the restrictions of marshal. This means that constants can only be ints, floats, lists, tuples, etc. No user classes are allowed! However, this is not a restriction in the bytecode compiler. For instance, examine the following clojure code:
(py/print '(1 2))
And similar code in Python:
print (1 2)
Here's how Python compiles the code:
And how clojure-py compiles the code:
LOAD_CONST (1 2)
So the idea is that Clojure builds the tuple every single time it is assigned to a variable. They have to do this because some objects like lists or dicts are mutable. So you actually want each invocation of the above code to return a completely different object. However, in clojure-py we can save thousands of clock cycles simply by creating the list/hashmap/vector at compile-time and then loading it in a single instruction.Sometimes when we know the function to invoke at compile-time we even load functions via LOAD_CONST, this makes our code much faster, but also has one major problem...the code is now non-marshallable.
So what about using Pickle? Well Pickle doesn't support code objects....we could extend Pickle (cPickle can't be extended in the way we need it to be), but that's going to be very slow.
One path I tried going down is to store all consts in globals. That is, we generate code to create const values and then store the results in globals, loading them with LOAD_GLOBAL instead of LOAD_CONST. This kindof worked, but when I was done, it was riddled with bugs, and the compiler had become way more complex. Compilers are already complex, making it worse seemed like a bad idea.
So, now I should write some performance notes. From my testing compiling core.clj is pretty evenly split between three modules:
lispreader.py -- reads lisp objects
compiler.py -- creates bytecode from lisp objects
byteplay.py -- compiles bytecode to actual binary data that Python understands
Now, the output of lispreader can be quite easily run through cPickle. This is the "low hanging fruit" so to speak. With few tweaks, we can pickle our source code and completely remove lispreader.py from the equation.
Secondly I think I'll look into performance improvements in the compiler. There has to be some room for improvement there.
Finally, I'm not actually sure we'll ever be able to compile to .pyc files. And macros are a major reason. Imagine that we have a macro in a.clj and that macro is used in b.clj. If we change the macro, we want to trigger clojure-py to re-create the .pyc files for b.clj. This means we'd have to create some sort of dependency graph for every file, that tracks every module it's dependent on, and does automatic re-compilation.
So anyway, this is why we still don't have .pyc files...and why it's taking some time.