Wednesday, February 29, 2012

Simple Performance Metrics

I've been asked many times recently "How does the performance of clojure-py compare to clojure on the JVM?". I think the best way to go about this is to define what it is that the pypy JIT excels at, this will help us better understand our performance metrics.

The PyPy jit is a tracing jit. I'm not going to go into all the ins and outs of tracing jits, but suffice it to say, it's a totally different way of writing a jit than what is used in Hotspot. What PyPy excels at is optimizing dynamic code, or put in Clojure terms, anytime you think "oops...reflection might be happening here" this is where PyPy can step in and save the day. To start with, let's look at this Clojure (jvm) code:


(ns examples.speedtest)

(definterface IStrValue
    (strvalue []))

(deftype A [x]
    IStrValue
    (strvalue [self] (str x)))

(deftype B [x]
    IStrValue
    (strvalue [self] (str (+ x 1))))

(deftype C [x]
    IStrValue
    (strvalue [self] (str (dec x))))


(defn getstr [obj]
    (.strvalue obj))

(time (dotimes [x 10000000]
    (getstr (A. x))
    (getstr (B. x))
    (getstr (C. x))))

Execution time on my machine is around 36 sec. If we set "warn on reflection" on, the compiler will complain that (.strvalue) is causing reflection. Now, the proper way to handle this is to give a type hint to getstr:

(defn getstr [^IStrValue obj]
    (.strvalue obj))

Now the execution time is only 3 sec! 10x speed boost by a single tag. This isn't bad.

But let's see how clojure-py fares:

(ns examples.speedtest)

(deftype A [x]
    (strvalue [self] (str x)))

(deftype B [x]
    (strvalue [self] (str (+ x 1))))

(deftype C [x]
    (strvalue [self] (str (dec x))))


(defn getstr [obj]
    (.strvalue obj))

(time (dotimes [x 10000000]
    (getstr (A. x))
    (getstr (B. x))
    (getstr (C. x))))

Execution time: 8 sec

To me, the boon here is that the clojure-py code is more dynamic. We don't have to worry about interfaces, inheritance,  or type inference. We simply write our code in an idiomatic way, and let the jit figure the rest out. 

But how does clojure-py compare when it comes to raw number performance? Well there it's not quite so good a story. In most of my tests clojure-py is about 2-8x slower than the JVM version, but almost exactly the same speed as Python code.

So there's the performance numbers. If you're willing to write interfaces, provide type hints, etc. Then the JVM will be faster. But if all you want to do is pound out some code with acceptable performance, then maybe clojure-py is the way to go. And hey...it's not Java!

(sorry, couldn't resist)




No comments:

Post a Comment