HTTP And REST Are Great, But We Need A Better Developer Experience

December 15th, 2015Dan Yoder

Developers are struggling to fully leverage HTTP and REST. By definition, we can stipulate that they’re difficult technologies. This is largely because distributed computing is hard. And HTTP and REST are distributed computing technologies. If we want to solve this problem, we need make things easier. In particular, we need better tooling and libraries that make it easier to leverage HTTP and REST by default.

This is why I sometimes get frustrated with companies that develop libraries purporting to address the alleged failures of HTTP and REST. If we put the same resources into making those technologies easier to use, instead of reinventing them, we could make the advantages of HTTP and REST more accessible to a wider audience of developers. This is what we’ve tried to do with our experimental libraries, Patchboard and PBX. And that’s what Joyent, Walmart, and others have tried to do with libraries like Restify and HAPI. It’s not like no one is trying. But we have a long way to go.

But I Can See It Right There

I was reminded of this today when I was working on some API code. I have a resource that accepts a POST request to create new subordinate methods. It returns a 201 Created with a Location header, consistent with the HTTP specification.

However, CORS was hiding the Location header from my Web client. I had forgotten to include the Location header in the Access-Control-Expose-Headers header, so the browser was hiding it. Of course, this is consistent with the CORS specification, but the browser (in this case, Chrome) didn’t give me much of a hint about what was going on. The Location header just wasn’t there.

I thought at first there was a problem with my HTTP client library. Then I thought maybe there was a bug in the new Fetch API. I spent two hours trying to figure out what was going on. I believe this is exactly the kind of developer experience that causes developers to throw up their hands with HTTP and REST. After all, I was using HTTP by the book:

The newly created resource can be referenced by the URI(s) returned in the entity of the response, with the most specific URI for the resource given by a Location header field.

So why isn’t the Location field a simple response header as defined by the CORS spec? For that matter, why is Last-Modified permitted, but not Etag? At best, this is counter-intuitive and makes for a poor developer experience. Sure, the security considerations that inspired CORS are important, but if they result in developers abandoning HTTP and REST wholesale, we’ve likely done more harm than good.

Protocols Are Not The Problem

The reality is that distributed computing is hard. And one of the reasons its hard is because security over an untrusted network is hard. That isn’t the fault of HTTP or CORS. They’re simply network protocols, doing their job. As developers, we can’t keep running away from the problems by fleeing to the latest framework or library which offers a better initial experience by virtue of ignoring those same hard problems.

We Can Do More

At the same time, if we want developers to use HTTP and REST our libraries and tooling need to get better. In both Patchboard and PBX, we have API descriptions. These descriptions include the expected status code. Ideally, we can see that a 201 Created is a possible response, and automatically include Location by default in the Access-Control-Expose-Headers header. This is the kind of detail that would allow us to have our cake and eat it too. We can make the developer experience better without compromising on the hard stuff. Similarly, browsers should do more to clarify what’s going. A simple error message would have saved me two hours of debugging.

Essence And Accidents

Distributed computing will still be difficult. But our tools and libraries need to do more to make it as simple as possible. Fred Brooks once wrote about essence and accidents in software development. The essence is the real, underlying problem we want to solve. The accidents are all the avoidable problems.

Following Aristotle, I divide them into essence, the difficulties inherent in the nature of software, and accidents, those difficulties that today attend its production but are not inherent.

Developers struggle with REST and HTTP as much because of the accidents as because of the essence of distributed computing. Until we create a better developer experience, there will always be another Relay, claiming to “fix” HTTP and REST.

HTTP/2 And Beyond

It’s a bit of a catch-22, since the resources necessary to solve these problems are considerable. I can tell you that from first-hand experience. After all, I’m effectively complaining about our own library. But, on the other hand, I’m not aware of any other library that would have prevented this particular problem. The fact that we chose to write out own libraries is itself an indication of the difficulty of the problem. Even people who agree that HTTP and REST provide a great foundation for distributed computing have trouble coming to consensus on how to best to build on it. Internally at Panda Strike, we’ve had some doozies debating this. If we really want to create more Web, we need to do better.

And it’s not going to get any easier, with the emergence of HTTP/2. There’s a great opportunity here, but also a danger. Features like multiplexing and server push are significant advances. And they’re also an opportunity. But only if we learn from the past and create an equally compelling developer experience around them.