If you are a developer using Intrinio's API you will have noticed that for many data types, the results are limited to a specific number per page. This article answers these questions about API paging limits:
- What are API paging limits?
- What are Intrinio's paging limits?
- How can you write code to get all of the pages?
- What does an example of this look like using R?
What are API paging limits?
When you make an API call in your code, you are requesting data. Paging limits put a threshold on the amount of data you can get in a single API call. For example, you can use this API call to request Apple's stock price history from Intrinio:
There are more than 30 years of daily price history for Apple. In order for our API to provide timely and efficient service to every user, we will only return 100 price entries at a time. This results in multiple "pages" of data, allowing developers to request all of the data one page at a time instead of all at once.
What are Intrinio's paging limits?
Intrinio's API documentation shows three different paging limits depending on the data type you want to access:
- Historical Data: 1,000 results per page
- File Downloads to CSV: 10,000 results per page
- All Others: 100 results per page
As you can see, most endpoints are limited to 100 results per page. Historical Data allows for 1,000 results per page because the size of each entry is relatively small. Historical Data is the quickest way to get historical values the data point you need. CSV Downloads allow for 10,000 results per page. However, because CSV Downloads are for human use, they are limited to 1 download per second.
How can you write code to get all of the pages?
Every coding language is different and within each coding languages there are different packages and techniques for pulling in multiple pages. Additionally, different APIs use different methods for paging. Intercom, for example, uses a "scroll parameter" and Stripe uses a "has more" parameter. For Stripe, if has_more = TRUE, there are more pages and the developer can get the next one.
Intrinio provides the total number of pages as part of the initial API call response. This means that you can write a loop that will continue to make API calls until you reach the total number of pages. This API call, for example, lets you know there are 64 pages of data for this API call:
In whatever programming language you use, you can use this initial API call to determine how many pages there are. Then you need to make calls that include ?page_number=, with a page number at the end, until you hit that number:
Usually, making each subsequent API call is handled with a "for" loop. It can also be done with a "while loop". The logic is that "while" the page number is less than or equal to the total number of pages, repeat the API call and increase the page number by 1.
If you add the resulting data from each API call to the data you have, you will end up with each page of data.
An Example of Paging Limits in R
You will need to signup for an account and paste your own API keys into this code for it to work when using R, however the code will run with the free account on the US Public Company Financials Data Feed:
Update 11/30/2017- Feel free to skip ahead to this recently released package that does the hard work for you if you are an R user.