JSCK Benchmarks

May 11, 2015Giles Bowkett

Panda Strike’s JSON Schema validator, JSCK, is very fast. So, a little while ago, I blogged about JSCK being the fastest JSON Schema validator for Node.js.

Prior to the blog post, of course, we revised, expanded, and improved our benchmarks, because we wanted them to be accurate and comprehensive. They were, and they showed JSCK to be the fastest option. But this attracted a lot of attention to the library, and brought some other JSON Schema validation libraries out of the woodwork.

Eventually, we discovered JSCK isn’t the fastest JSON Schema validator for Node after all. The good news: it looks like we have the best benchmarks.

The New Champion

JSCK had to surrender its title to is-my-json-valid (which I’ll refer to as IMJV). We initially rebuffed IMJV’s claims of greater speed, because IMJV’s implementation had a bunch of flaws. But the library’s creator responded with an intense streak of new development and point releases, and essentially rewrote the library to a higher standard literally overnight. So we had to give IMJV credit as the fastest Node validator out there.

A few other libraries came along, too, although IMJV held onto its crown. We still love JSCK, of course — it’s our baby — and we still find its code easier to read and reason about than the code for its competitors, although we’re obviously biased.

Test The Right Things

Along the way, we’ve seen some alternative benchmarking schemes that we have felt were misguided. Our benchmarks use two schemas to benchmark draft 3, and three schemas for draft 4. These schemas range in complexity from very trivial, basic examples, to complex, fully-fledged schemas which exercise every major feature of the JSON Schema standard. I’m not going to name names, but there’s a JSON Schema validator out there which uses the standard’s compliance tests for its benchmarks.

What that means, if I’m not being clear, is that the JSON Schema project has a set of standard tests which any validator should pass in order to call itself a JSON Schema validator. These tests confirm that a library adheres to the standard.

In fact, if you want to run these tests, we’ve created a Node.js test harness for them. If your library deviates from the JSON Schema standard in any way — most validators do, slightly — this testing harness also includes a very succinct and clear way to stipulate exactly which deviations occur, and why.

These tests are very useful for their intended purpose. But you’re not going to get a realistic estimate of production performance by measuring how many times per second your validator passes a test like maxItems validation, too long is invalid. That test exists only to ensure that your validator correctly implements the maxItems keyword. It doesn’t have anything to do with how people will use the code in production.

That’s why we built realistic benchmarking schemas of varying levels of complexity. The JSCK benchmark schemas model real-world use cases.

Share And Enjoy

Adding a new library to JSCK’s benchmarks is easy. Check out this pull request on GitHub to see how easy it is. All you have to do is add your validation library to the test harness and to JSCK’s package.json. But please make your changes on a topic branch, not master.

JSCK’s benchmarks are fast, comprehensive, and easy to use. We may spin them out into their own project. We currently track nine other libraries, besides our own:

Along the way, we’ve written some brief documentation updates for Themis (PR accepted) and JSV (seems to be abandonware), in the hopes of sharing what we’ve learned with the larger community.

So, if your favorite Node.js JSON Schema validator isn’t yet covered by JSCK’s benchmarks, please file a pull request.