REST in Peace
Before actually describing the API design ideas of meshcaline, I will start with a subject that became more and more apparent, the longer I worked with our users on improvements of our API design:
Let's stop talking about REST(ful) API design!
One of the most misused terms in API development -- if not software development in general -- is REST or RESTful. Most likely only few people have read or fully understood the REST architecture style as described in Roy Fielding's dissertation published back in 2000 (and I can't claim either). Looking back into the early 2000s, after the dust, dispersed by the hype around XML, XML-RPC and its successor SOAP and the ws-posse had settled, many developers where just fed up with interface generators and complex, incomplete and incompatible standard implementations. They wanted something more lightweight, tangible and robust for their client/server communication, especially when the client was a web browser. Most developers found it immediately appealing to apply the same mechanisms commonly used for passing values between dynamic web pages also for they mechanisms they used to interface with dynamic data sources on the server-side. Nobody really cared about the full concept behind Roy Fielding's work, but everyone was just happy that there was now a name for something they already were doing and that did not come with so much overhead and pain. The fact that the term REST was known to be be the underlying concept of the web as such gave this approach an accolade, and nipped any discussion in the bud whether SOAP or anything else would be a valid or even superior alternative. And with the next hype around web 2.0, the acronym REST, originally a quite technical term, became quickly a marketing buzzword -- although barely anyone using it could explain what "Representational State Transfer" really meant. With the rise of successful public web service offerings, and more and more of them claiming to be RESTful, in 2008 Roy Fielding felt compelled to state six rules which an API must fulfill before calling itself REST API. Unfortunately, at that time the term REST was already too important for the marketing buzz, so the inventor of the term had no chance to prevent the ongoing misuse.
But even today, nearly six years after the rules got published, and many successful web service APIs in the market, I am not aware of any popular web service API that fulfills all of those rules. Also most books about REST APIs -- even the ones I find great -- just pick up some of the aspects. So there must be something in those rules that either API developers don't like or at least don't see that they create value for their API. Alternatively the potential users of such APIs didn't like those API designs and then those APIs never get popular.
If your goal is a successful API you better understand how the rules you are following supports you on that. Although I can't claim that my interpretation of the rules is exactly what Roy Fielding had in mind, here comes my analysis:
Rule 1: Protocol Dependencies
A REST API should not be dependent on any single communication protocol, though its successful mapping to a given protocol may be dependent on the availability of metadata, choice of methods, etc. In general, any protocol element that uses a URI for identification must allow any URI scheme to be used for the sake of that identification [Failure here implies that identification is not separated from interaction.]
I would guess barely any API designer spends time on the first sentence of this rule, simply because what most of them want to build are web service APIs, so HTTP as underlying protocol (and its secure sister HTTPS) is set. So why should a developer spend time thinking about the possibility to map his API to some other protocol. While it may be an nice academic exercise to also think about how to offer an API via SMTP (even SOAP in theory supported that approach too), in most cases it does not create real business value. And so this becomes the first rule API designers are more than willing to drop.
The second sentence is often read over although it contains an important conceptual detail: the identity of something referred to in an API protocol should not be bound to the accessibility of that object. The primary purpose of identifiers is to give an object a unique name(space), not necessarily to provide information how to access that object. Many objects might only be accessible in the context of other resources, so you can't provide means to access them directly. But you might still want to identify the object in the context of various resources it is embedded in. Unfortunately, many APIs ignore that rule. Instead of clearly separating between hypertext elements and identifiers, many API designs follow a paradigm that I call "Object-relational mapping over HTTP": Foreign keys of an underlying database model get expressed as "executable" URL, and therefore the separation of identity and accessibility is considered an avoidable overhead.
Rule 2: Protocol Extensions
A REST API should not contain any changes to the communication protocols aside from filling-out or fixing the details of underspecified bits of standard protocols, such as HTTP’s PATCH method or Link header field. Workarounds for broken implementations (such as those browsers stupid enough to believe that HTML defines HTTP’s method set) should be defined separately, or at least in appendices, with an expectation that the workaround will eventually be obsolete. [Failure here implies that the resource interfaces are object-specific, not generic.].
While most API designs do not have to modify the underlying protocol, the question when a protocol is underspecified may be a quite subjective judgment. Just check how many IETF RFC exist that introduce additional HTTP headers, which are only important for a very specific domain of services. When the communication protocol your API is using is anyway set to HTTP(S), and the best way to get a problem solved for your users is by introducing some private header, then being pragmatic might be a better solution than following the book and then miss to meet the needs of your users. I think as long as the extension is a generic one (at least in the context of your complete API, not just for an individual resource) then I'm willing to relax the rule to: don't do this, unless you know why you do this.
Rule 3: Media Types
A REST API should spend almost all of its descriptive effort in defining the media type(s) used for representing resources and driving application state, or in defining extended relation names and/or hypertext-enabled mark-up for existing standard media types. Any effort spent describing what methods to use on what URIs of interest should be entirely defined within the scope of the processing rules for a media type (and, in most cases, already defined by existing media types). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]
I think nobody debates that defining media types is the most important part of any API definition. Unfortunately not every aspect of this rule is precisely defined and leaves room for interpretation and dispute. To me is it not clear what "processing rules for a media type" are allowed to contain. Just two examples for details that I find unclear, although important:
- The rule mentions "methods" should be defined in the processing rules of a media type, but I am not sure how this should work: A media type "image/jpeg" of a given resource does not define if the corresponding API that delivers such a media type in a response of a GET method also supports a PUT method on the same resource.
- It does this not define precisely, if such processing rules might allow parameters for a given resource: Assume you define a new media type for "adjustable images", that allows you to modify various parameters of a returned image (transparency, size, dpi, background color, cropping...). Would it be valid to specify in the processing rules, that you can retrieve a variant of such an "adjustable image" by adding various parameters to such the URL of such a resource, or would it require specific hypertext controls in the link to such a resource which would allow you to specify such parameters.
I fully understand why REST propagates the avoidance of "out of band" information and how this contributes to the success of the web. But the real power of this approach comes into play when you define a generic protocol (where you may have many server implementations and more or less generic clients using them). But this is not what drives most developers when they design an API for their product. It is not their primary goal to allow their users to also use other API products, unless they mimic another popular API to entice its customers away to their solution. To make it worse: The primary customers (the application developers) want an API that solves their specific problem in a simple, long-term-stable way. They are very happy with some "hard coded interaction behavior" in their code, as they believe it is the responsibility of the API provider to maintain that behavior -- and believe me I had many discussions with app developers on that topic.
Rule 4: Resource Addressing
A REST API must not define fixed resource names or hierarchies (an obvious coupling of client and server). Servers must have the freedom to control their own namespace. Instead, allow servers to instruct clients on how to construct appropriate URIs, such as is done in HTML forms and URI templates, by defining those instructions within media types and link relations. [Failure here implies that clients are assuming a resource structure due to out-of band information, such as a domain-specific standard, which is the data-oriented equivalent to RPC's functional coupling].
This again is a tricky rule. From an API developer's point of view, it might be understandable that you want to retain the right to change resource names whenever you have a need for it. On the other hand, using hard wired URI templates in their code is something all applications developers are familiar with and tend to prefer. Just one anecdote from our API project: For some time we did support both, a nice, read to use hypertext URI field in our response, but also a documented URI template that allowed the construction of the actual URL based on the template and other attributes found in the response. And guess what: Nearly 100% of the app developers did use the URI template approach, even when using just the full URL for the response would have been far simpler.
Also forms (or URI templates, which for me is just a low-level variant of a form) as part of the response may sound like a good idea in the first place ("if it works for browsers then is must also work for API)". But keep in mind that forms in HTML do work because they get primarily used to construct a UI for a human beings. Those "consumers" are capable to derive what to enter into the various fields from the context in which the form is presented and the meta-information displayed long with the form (e.g. labels). Humans will find the source for the information they are supposed to provide in the fields. Even when things are not completely clear, we are capable to try out and test how the system behaves and re-try if the resulting behavior doesn't match expectations. That's not the way app developers want to implement the code that interacts with your API. Developers want to know in advance (at coding time) what information is required in what context and then either ask their own users to provide this information (by building a form in their UI), or derive the information from the environment or state the application is in.
Once again I believe that this rule helped the web becoming as successful as it is. But if applied to a specific API product, you might risk to meet the expectations of your customers, and by that fail on the market, unless you are a monopolist who can enforce the usage.
Rule 5: Typed Resources
A REST API should never have “typed” resources that are significant to the client. Specification authors may use resource types for describing server implementation behind the interface, but those types must be irrelevant and invisible to the client. The only types that are significant to a client are the current representation’s media type and standardized relation names. [ditto]
If the previous rule already risks that your API design makes your customers unhappy, that one definitively will. Not only does it not meet the mental model of a developer who calls an API for some specific purpose, and for that the API is expected to respond with some "type" of result. In addition, every API call has a "price". Even when you do not charge for the usage of your API directly, the client application will have to fire the request, wait for the response and then process the response accordingly. So the currency of the price all apps will have to pay is time, and this usually has a direct impact on the responsiveness of their application. Therefore your users don't want to access a resource and wait for the response just to then learn, that either the server can't deliver what they would like, or they can't process what the server responds. They want to know upfront which methods a specific resource supports and what the response will return -- ideally at coding time (to make if simple for the developer) but at least before they decide to fire a request (to avoid negative impact for the app).
Rule 6: Service Discovery
A REST API should be entered with no prior knowledge beyond the initial URI (bookmark) and set of standardized media types that are appropriate for the intended audience (i.e., expected to be understood by any client that might use the API). From that point on, all application state transitions must be driven by client selection of server-provided choices that are present in the received representations or implied by the user’s manipulation of those representations. The transitions may be determined (or limited by) the client’s knowledge of media types and resource communication mechanisms, both of which may be improved on-the-fly (e.g., code-on-demand). [Failure here implies that out-of-band information is driving interaction instead of hypertext.]
Also this rule has questionable value for an API. Your users want to use certain features of your API. Usually they expect to use some URI template from the documentation. If you would follow this rule, your users the would have to fetch the initial URL and then scan the result for some link relation type identifying, the features they want to use. While the advantage for you as API developer is obvious, you must also find an answer for the question how this helps your users. If the flexibility it brings to you doesn't introduce also advantages for your users, then they will not like it, because it (a) requires an avoidable request and (b) adds complexity to the app code -- they first have to parse the response in order to find which relation types are available. And they would even have to prepare for the case, where the relation type they need is no longer available or the response offers multiple options to choose from. When your API does provide functionality beyond "simple links", then you would have to embed into the initial response also the forms needed to construct the request, with all the problems coming with forms described above.
Conclusion
So where did this analysis lead me to? I doubt I will ever design an API that honors all or even most of Roy Fielding's rules. Also I doubt a commercial API product will become successful if it tries to. Although I am deeply convinced of the advantage of the HATEOAS constraint for API, I am equally convinced that a prosperous commercial API product requires a different design paradigm than the one derived from the REST rules.
At the same time I believe it is important to take definitions serious and not to dilute the work of its creator by ignoring the parts you don't like or that do not fit. What is the purpose of a definition, if people ignore it? You would not like an integer variable to contain a float value, because a developer thought that the restriction that came with integer where not appropriate in this context and this little dot can't really hurt. You should only call your API RESTful, when you are willing to accept all the rules. Let the marketing guys put a REST tag to your offering if it helps them promoting your product, but as developers we should be precise.
Realizing that playing RESTful according to Roy Fielding's rules in not my type of game, I decided that its worth to write down the concepts I believe are more appropriate for API based products -- and so the idea for this blog started. But if it is not REST, how should I call the ideas? Without a name you can't precisely state what you are talking about. Initially my ideas for names where still attached to REST, like RESTless, RESTignation, etc. But then I realized that those names seem to indicate that there is something wrong with REST, that needs improvement. And that's definitively not what I want to say. REST is one of the most important architectural concepts in today's IT landscape and this will remain. It's just not the right tool for everything. So I had to find name independent from REST and came up with meshcaline.