The Fast And The Spurious
After I wrote a blog post announcing that JSCK, Panda Strike’s JSON Schema validator, is the fastest around, we got a ticket and a pull request about another project called is-my-json-valid, which claims to be faster.
The good news for the is-my-json-valid project is they do run through our JSON Schema benchmarks much faster than any other project, including JSCK. The bad news: they’re not doing JSON Schema validation.
is-my-json-valid uses Orderly, an aggressively pruned subset of the JSON Schema standard. Orderly was created in 2009 and abandoned in 2010. (An alternate compiler lasted as long as 2013.) The idea with Orderly is that it’s like a CoffeeScript for JSON Schema; a more concise, readable, human-friendly format which translates back to the more verbose version when necessary.
Its own creator said that Orderly:
The JSON Schema project has created a set of unit tests which any JSON Schema validator can use to verify that it supports a particular draft of the schema. Panda Strike created a Node.js testing harness for this project, which you can install with npm install json-schema-tests. Running is-my-json-valid through this test harness, we find that it passes 32 tests, fails 49, and errors out of 1.
is-my-json-valid fails official JSON Schema tests for additionalItems, additionalProperties, allOf, anyOf, default, dependencies, enum, items, maximum, minimum, not, oneOf, required, and numerous other JSON Schema attributes.
Our test harness allows you to specify very clearly, in code, which aspects of the JSON Schema standard you support. Virtually every JSON Schema validator makes a few exceptions to the standard, and JSCK does too. But you can’t skim through the majority of the standard, electing not to implement support for it, and then claim to be the fastest option.