API Pagination is your friend

Pagination has a mixed reputation among developers who write applications that consume API data.

Some developers cite
presence of pagination
as justification for
qualifying an API as “bad” or otherwise lacking in design or implementation.

Indeed, pagination does place a higher barrier to entry on inexperienced developers. But for experienced developers, pagination presents only a marginal difficulty.

When working with an API that returns a small amount of data, for example around a few hundred JSON objects, a single network request to the API endpoint URL is sufficient to yield the entire dataset. For example:

However, many APIs offer volumes of data too great to return in a single network request. These APIs usually instruct the developer to pass additional URL parameters to the API endpoint to designate which subset of data to return. And their responses usually include metadata about the identifiers of the current, previous, next, and/or last page(s).

The pagination pattern requires more developer effort because it requires multiple network requests, each yielding an incremental subset of the data, in order to yield the entire dataset.

The experienced developer leverages a scripting language like JavaScript, Ruby, Python, et. al. to loop through multiple requests in order to retrieve the entire dataset. Pagination strategies differ from API to API because each offers its own URL structures, URL parameters, and response structures.

# a ruby script excerpt which uses a 'client' from the "open_fec_api" ruby gem to make requests...options={:page=>1}response=client.candidates(options)# the first API callwhileresponse.page<response.pagesdo# comparison of the current page number to the last page number, to determine whether or not to place future calls...options.merge!({:page=>response.page+1})# incrementing of page numberresponse=client.candidates(options)# subsequent API call(s)end

Paginated APIs do place somewhat of a higher burden on developers, but usually not high enough to outweigh the benefits of transmitting high volumes of data. And usually the difficulty is marginal. So criticism of API Pagination is largely undeserved.