Friday, May 9, 2014

Boon 2x faster than Jackson at InputStream and not using index overlay

Boon non-index overlay mode using inputstream4 minutes ago by Richard Hightower
This is with InputStream. According to Tatu comments InputStream is a use case that Boon could not compete in because boon is really just optimized for String. Tatu also said that Boon likely only wins because it uses index overlay.

So here is a test that uses Boon without index overlay and uses InputStream not String.

Benchmark                       Mode Thr     Count  Sec         Mean   Mean error    Units
MainBoonBenchmark.webxml              thrpt  16         6    1   455347.925    46637.751    ops/s
BoonClassicEagerNoLazyParse.webxml    thrpt  16         6    1   401126.575    28331.138    ops/s
JacksonASTBenchmark.webxml            thrpt  16         6    1   233730.506    17868.136    ops/s
MainJacksonObjectBenchmark.webxml     thrpt  16         6    1   227287.992    21363.353    ops/s
BoonReaderSource.webxml               thrpt  16         6    1   216429.247    22538.238    ops/s
BoonAsciiBenchMark.webxml             thrpt  16         6    1   210416.450    10610.062    ops/s
BoonUTF8BenchMark.webxml              thrpt  16         6    1   199869.811     8742.968    ops/s
GSONBenchmark.webxml                  thrpt  16         6    1   168144.639     5311.387    ops/s

MainBoonBenchmark is the index overlay parser it comes in first.
BoonClassicEagerNoLazyParse (the original boon parser) comes in second.

BoonClassicEagerNoLazyParse does not use index overlay so all those comments are just off. You can see for this benchmark index overlay is only about an 11% improvement.

If I include full chop and chop, Jackson will come in fourth.
But then I would have to explain what chop and full chop mean, and then I would have to refer you to the article on InfoQ on index overlay and explain when it makes sense and what use cases you can't use it, and then introduce full chop and chop. :)


Jackson used to come after BoonReaderSource but Tata has been busy so now Jackson is only twice as slow BoonClassicEagerNoLazyParse. So there goes that myth about Index overlay and Boon not doing anything and Boon in only able to handle string and not InputStreams.

LET ME REPEAT THAT: Boon using InputStream, and not using index overlay is faster than Jackson.

GSON comes in last place. I remember when I started GSON would often beat Jackson.

Jackson has improved but even the improved Jackson is twice as slow as Boon for parsing with no index overlay.

RE: Input source. Most commonly cited tests start with Java Strings. Strings are rarely used as input source, because they are JVM constructs -- all external input comes as byte streams.

The benchmark linked which has not changed since this was published and for quite some time covers InputStream, string, byte[], etc. It is in the first paragraph. Next time I will use a blink tag. It did not cover not using index overlay because in only rare cases does index overlay not make sense as a viable alternative to full parse and index overlay is actually better for POJO serialization and REST. SO OF COURSE I included it in the benchmark. To think otherwise is just wrong. But there are cases where index overlay does not make sense (thus chop, full chop, and the full parse version), but that is very nuanced discussion, and an argument hard to make when there is some clear FUD. So instead of muddying the water and defending index overlay, let just keep at this, Boon does not need index overlay to beat Jackson at JSON parsing. PERIOD!

. So when user uses, say, JAX-RS style REST handling, where all JSON data gets bound to a POJO, from an InputStream; and reverse direction goes from another POJO into OutputStream, performance experienced is very different from what a benchmark would suggest.

Actually Boon was designed for exactly the REST / Websocket use case. And it was designed for POJO serialization. And you can see in the benchmarks cited that Boon does better than Jackson at POJO serialization in most cases, and in the article by Andrey (Groovy in that case which has some Boon DNA), and in the Gatling Benchmark and its POJO performance was confirmed by Julien Ponge (the author of Golo julien.ponge.org/blog/revisiting-a-json-benchmark/). Boon used to be consistently twice as fast as Jackson at POJO object serialization but Boon added some features which made it slower and Jackson got faster so now Boon usually wins by 30%, but not always. And once you are doing POJO mapping the index overlay vs non-overlay point is completely moot.

Let's go bigger. webxml, which is from the JSON.org examples is small, let's use a big file.. The catalog which is 170K.

Benchmark                                                     Mode Thr     Count  Sec         Mean   Mean error    Units
i.g.j.inputStream.MainBoonBenchmark.citmCatalog              thrpt  16         6    1     1061.592      106.462    ops/s
i.g.j.inputStream.BoonClassicEagerNoLazyParse.citmCatalog    thrpt  16         6    1      979.794       51.968    ops/s
i.g.j.inputStream.BoonReaderSource.citmCatalog               thrpt  16         6    1      681.072       37.804    ops/s
i.g.j.inputStream.JacksonASTBenchmark.citmCatalog            thrpt  16         6    1      554.181       26.974    ops/s
i.g.j.inputStream.GSONBenchmark.citmCatalog                  thrpt  16         6    1      538.389       43.384    ops/s
i.g.j.inputStream.MainJacksonObjectBenchmark.citmCatalog     thrpt  16         6    1      486.275       70.753    ops/s

So again, the claim was that Boon can't handle things fast unless it uses Strings and unless it uses index overlay is just plain wrong.

This is from an inputstream. Jackson is 4th. Boon index overlay beats. Boon full parser, non lazy version beats it. It beats by a wide margin. When Jackson does beat Boon which happens in some uses cases the margin is usually very small. When Boon wins which happens in most of the test cases, it wins by a wide margin with and without index overlay.

Since no one asked me why its Boon is faster or what it does, I wont say other than I have been talking about it on my blog for the last six months, and I'd like to work on an article for InfoQ about it with Jakob Jenkov and/or Stephane Landelle at some point.

Jackson can be faster than Boon. It probably will be. But today. Today. It is not.

Jackson and Boon have different philosophies.

Jackson is more mature. Jackson is more stable. Jackson integrates with more frameworks. Jackson has different and possibly more features. There are use cases where Jackson will be faster. There are many use cases where JSON parser speed will not matter. Jackson probably has less bugs since it has had more eyeballs for more years.

That said... Boon is faster. Today.

No comments:

Post a Comment

Kafka and Cassandra support, training for AWS EC2 Cassandra 3.0 Training