Why an API Developer Should Care About Big Data Technologies

As developers we have to be careful not to be so narrowly focused on our immediate responsibility that we lose sight of the bigger picture. In pursuit of serving up the almighty contract, care must be taken to not overlook design decisions that are almost certainly going to have negative performance implications. One such implication revolves around fetching and presenting data from a database.

This problem seems rather cut and dry - its been done for decades after all. The consumer sends a request, the API fetches the data from the database and sends it to the consumer. Simple? Agreed, until sub-300 millisecond service level agreements come into play and the database(s) has many millions of artifacts stored in it. Now we have design considerations that are no longer trivial.