Service API Archetypes
API design sits on a distinguished shelf in my brain. To its left is functional programming in the large and to its right is UI menu design. This shelf is the shelf of contentious things with no right answers. It’s the shelf of someone starting a sentence with “in the real world,” of toy examples, and longing gazes at the work of overfunded startups on their way to acquihire. It’s a shelf filled with hard problems that we have neither the wisdom nor the language to solve.
While I was working at Bronto Software, the architecture team I was leading found our organization mired in half-baked APIs. The problem went far beyond aesthetics and into frequent delays and system instability. The most problematic symptoms were:
- Complex cross-team dependencies requiring a chain of API changes
- Client libraries that needed to be updated in coordination with API changes
- Substantial ramp time for developers unfamiliar with an API due to inconsistency and lack of or incorrect documentation
- Poorly managed performance characteristics leading to outage
The situation was frustrating for everyone involved, particularly the developers who often spent hours arguing over the minutiae of their APIs only to have turmoil find them all the same. API design is an area of expertise, and expecting everyone in an engineering organization to develop that expertise is misguided. The only uniform expertise an engineering department needs is expertise around customers and the business — everywhere else we hope to find a diversity of experiences and expertise. Our problem was that we weren’t leveraging that expertise, and we were asking engineers to focus on a problem that wasn’t where they could best contribute.
Based on this observation, we worked to figure out ways to make good and consistent API design easier than bad or inconsistent design. We adopted standards and developed tools that were flexible, but clear and opinionated. API design shifted to being a problem of mapping a business problem into a smaller and more well-understood space of possible designs. The piles of rules could be unwieldy and the new tools needed more development, but the approach was sound. One gap that persisted was language. APIs could be standards compliant, but to understand much more than that required digging into the interface definition. As we worked through all of the API designs to fit them into this new world, we started to see a few patterns emerge.
I’ve broken these patterns down into what I call the four service API archetypes: Transactional, Bulk, Search, and Reporting.
Transactional APIs are the bread-and-butter APIs that make the world turn. They are typically low-latency, used for reads and writes, and offer stricter consistency models. Some are resource-oriented and expose resources that match entities in their customers mental models. Others provide an RPC focused approach and describe the service in terms of actions that can be taken upon it. When designing a transactional API, purity isn’t important, but you should probably focus on one or the other.
For many use cases, a transactional API is all you’ll ever need. A well-designed one is flexible enough to express all of the capabilities of your system in a way that is tolerable even if not optimal. As your system scales in both usage and features, you often hit performance bottlenecks. At a certain point, the full table-scans don’t work, so you add indexes. The indexes get too slow, so you start optimizing them. Eventually you give in and think about adding TTL caching. From here on out, the hacks can pile onto each other. Transactional APIs are versatile, but once you hit performance bottlenecks, you should start looking to other archetypes.
Bulk APIs aren’t used as frequently, but for services that hold large volumes of data they can simplify operations. Bulk APIs either ingest datasets into, or export them from, the service in question. Their asynchronous nature means their view on consistency is, well, complicated.
Search APIs are often used to support high-level user requests, but they can go far beyond that. Search APIs are low-latency, read-only, offer flexible access models, and are typically slightly behind and/or incomplete. The flexible access models provide a multitude of sorting and filtering options that are difficult to support in the transactional API.
Reporting APIs provide read-only aggregated data. Like search APIs, their query interface is low-latency, but their data might be behind. Unlike search APIs, reporting APIs aren’t particularly flexible and tend to provide relatively canned responses.
An API is the ultimate expression of a programmable system’s capabilities, but we are too quick to discard a clear model of our APIs. Without that model, we can’t coherently add new capabilities.
In Information Science, there’s a concept called the DIKW pyramid — Data, Information, Knowledge, Wisdom. Wisdom is the least well-defined and sometimes omitted, a trend I’ll continue here. A similar breakdown occurs in military intelligence where we see the separation of Data, Information, and Intelligence. In both cases, data refers to raw facts and information to collection of data in a way that is usable and intelligible. Knowledge or Intelligence combines pieces of information within a subjective context and then typically summarizes or infers based on that information.
I said earlier that the transactional API is the most flexible across all use-cases. Its most significant value is its costly consistency model. The other three archetypes aren’t nearly so flexible and generally fit a particular set of purposes. When I first read about the Data, Information, Intelligence split it immediately made me think of those three archetypes. A bulk API traffics in raw data. A search API is all about taking that raw data and making it accessible in a way that’s more useful, in the same way that a search engine is much more useful than a CSV dump of the internet. Reporting APIs represent an opinionated view over that data either via summary or just by noting the particular data items that, through analysis, are known to be important.
As I’ve gotten more comfortable with these non-transactional API archetypes, I’ve started to make my transactional APIs leaner. In the past, I might’ve tried to squeeze everything I could out of the transactional API, but I now I give up at the first sign of resistance. The overzealous filter syntax on a transactional collection endpoint still needs to be supported after you add your search API. If you wait to add that search API until your system is melting, you’ll regret the next six months you’ll have to continue supporting it for.
The archetypes have been useful to me over the past few years. I’ve used them to explain why I don’t believe every system needs One True API, but instead might provide several to suit the needs of different types of clients. Having several defined models can allow a team to think more expansively about the possibilities of what it can offer through APIs. Perhaps most importantly, to a pedant like myself, the resultant APIs can at least be internally consistent in their patterns, semantics, and consistency models. That means the time to get up to speed on the APIs is dramatically lessened.
