| Nov | DEC | Jan |
| 05 | ||
| 2020 | 2021 | 2022 |
COLLECTED BY
Collection: GDELT Project
requests and APIs: A Match Made in Heaven
requests. With it, you should be able to do most, if not all, of the actions required to consume any public API.
You can install requests by running the following command in your console:
$ python -m pip install requests
To follow the code examples in this tutorial, make sure you’re using Python 3.8.1 and requests 2.24.0 or higher.
https://randomuser.me/api/, and this is the tiniest API call you can make:
>>>>>> import requests
>>> requests.get("https://randomuser.me/api/")
<Response [200]>
In this small example, you import the requests library and then fetch (or get) data from the URL for the Random User Generator API. But you don’t actually see any of the data returned. What you get instead is a Response [200], which in API terms means everything went OK.
If you want to see the actual data, then you can use .text from the returned Response object:
>>>>>> import requests
>>> response = requests.get("https://randomuser.me/api/")
>>> response.text
'{"results":[{"gender":"female",
"name":{"title":"Ms","first":"Isobel","last":"Wang"}...'
That’s it! That’s the very basics of API consumption. You managed to fetch your first random user from the Random User Generator API using Python and the requests library.
Remove ads
api. This is not mandatory, just more of a rule of thumb.
For example, here are the base URLs for a few well-known API players:
●https://api.twitter.com
●https://api.github.com
●https://api.stripe.com
As you can see, all of the above start with https://api and include the remaining official domain, such as .twitter.comor.github.com. There’s no specific standard for how the API base URL should look, but it’s quite common for it to mimic this structure.
If you try opening any of the above links, then you’ll notice that most of them will return an error or ask for credentials. That’s because APIs sometimes require authentication steps before you can use them. You’ll learn more about this a bit later in the tutorial.
TheDogAPI: This API is quite fun but also a really good example of a well-done API with great documentation. With it, you can fetch the different dog breeds and some images, but if you register, you can also cast votes on your favorite dogs.
Next, using the just-introduced TheDogAPI, you’ll try to make a basic request to see how it may differ from the Random User Generator API you tried above:
>>>>>> import requests
>>> response = requests.get("https://api.thedogapi.com/")
>>> response.text
'{"message":"The Dog API"}'
In this case, when calling the base URL, you get this generic message saying The Dog API. This is because you’re calling the base URL, which is typically used for very basic information about an API, not the real data.
Calling the base URL alone isn’t a lot of fun, but that’s where endpoints come in handy. An endpoint is a part of the URL that specifies what resource you want to fetch. Well-documented APIs usually contain an API reference, which is extremely useful for knowing the exact endpoints and resources an API has and how to use them.
You can check the official documentation to learn more about how to use TheDogAPI and what endpoints are available. In there, you’ll find a /breeds endpoint that you can use to fetch all the available breed resources or objects.
If you scroll down, then you’ll find the Send a Test Request section, where you’ll see a form like the following:

>>> response = requests.get("https://api.thedogapi.com/v1/breeds")
>>> response.text
'[{"weight":{"imperial":"6 - 13","metric":"3 - 6"},"height": ...}]'
There you go, your first breed listing using the dog API!
If you’re a cat person, don’t fret. There’s an API for you, too, with the same endpoint but a different base URL:
>>>>>> response = requests.get("https://api.thecatapi.com/v1/breeds")
>>> response.text
'[{..."id":"abys","name":"Abyssinian"}]'
I bet you’re already thinking about different ways you can use these APIs to make some cute side project, and that’s the great thing about APIs. Once you start using them, there’s nothing stopping you from turning a hobby or passion into a fun little project.
Before you move forward, one thing you need to know about endpoints is the difference between http:// and https://. In a nutshell, HTTPS is the encrypted version of HTTP, making all traffic between the client and the server much safer. When consuming public APIs, you should definitely stay away from sending any private or sensitive information to http:// endpoints and use only those APIs that provide a secure https:// base URL.
For more information on why it’s important to stick to HTTPS when online browsing, check out Exploring HTTPS With Python.
In the next section, you’ll dig a bit further into the main components of an API call.
Remove ads
Request and Response objects:
>>>>>> response = requests.get("https://api.thedogapi.com/v1/breeds")
>>> response
<Response [200]>
>>> response.request
<PreparedRequest [GET]>
>>> request = response.request
>>> request.url
'https://api.thedogapi.com/v1/breeds'
>>> request.path_url
'/v1/breeds'
>>> request.method
'GET'
>>> request.headers
{'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate',
'Accept': '*/*', 'Connection': 'keep-alive'}
>>> response
<Response [200]>
>>> response.text
'[{"weight":{"imperial":"6 - 13","metric":"3 - 6"},
"height":{"imperial":"9 - 11.5","metric":"23 - 29"},"id":1,
"name":"Affenpinscher", ...}]'
>>> response.status_code
200
>>> response.headers
{'Cache-Control': 'post-check=0, pre-check=0', 'Content-Encoding': 'gzip',
'Content-Type': 'application/json; charset=utf-8',
'Date': 'Sat, 25 Jul 2020 17:23:53 GMT'...}
The example above shows you a few of the most important attributes available for Request and Response objects.
You’ll learn more about some of these attributes in this tutorial, but if you want to dig even further, then you can check Mozilla’s documentation on HTTP messages for a more in-depth explanation of each attribute.
| Status code | Description |
|---|---|
200 OK |
Your request was successful! |
201 Created |
Your request was accepted and the resource was created. |
400 Bad Request |
Your request is either wrong or missing some information. |
401 Unauthorized |
Your request requires some additional permissions. |
404 Not Found |
The requested resource does not exist. |
405 Method Not Allowed |
The endpoint does not allow for that specific HTTP method. |
500 Internal Server Error |
Your request wasn’t expected and probably broke something on the server side. |
200 OK earlier in the examples you executed, and you might even recognize 404 Not Found from browsing the web.
Fun fact: Companies tend to use 404 error pages for private jokes or pure fun, like these examples below:
●Mantra Labs
●Gymbox
●Pixar
●Slack
In the API world, though, developers have limited space in the response for this kind of fun. But they make up for it in other places, like the HTTP headers. You’ll see some examples soon enough!
You can check the status of a response using .status_code and .reason. The requests library also prints the status code in the representation of the Response object:
>>>>>> response = requests.get("https://api.thedogapi.com/v1/breeds")
>>> response
<Response [200]>
>>> response.status_code
200
>>> response.reason
'OK'
The request above returns 200, so you can consider it a successful request. But now have a look at a failing request triggered when you include a typo in the endpoint /breedz:
>>>>>> response = requests.get("https://api.thedogapi.com/v1/breedz")
>>> response
<Response [404]>
>>> response.status_code
404
>>> response.reason
'Not Found'
As you can see, the /breedz endpoint doesn’t exist, so the API returns a 404 Not Found status code.
You can use these status codes to quickly see if your request needs to be changed or if you should check the documentation again for any typos or missing pieces.
| HTTP Header | Description |
|---|---|
Accept |
What type of content the client can accept |
Content-Type |
What type of content the server will respond with |
User-Agent |
What software the client is using to communicate with the server |
Server |
What software the server is using to communicate with the client |
Authentication |
Who is calling the API and what credentials they have |
response.headers:
>>>>>> response = requests.get("https://api.thedogapi.com/v1/breeds/1")
>>> response.headers
{'Content-Encoding': 'gzip',
'Content-Type': 'application/json; charset=utf-8',
'Date': 'Sat, 25 Jul 2020 19:52:07 GMT'...}
To do the same with the request headers, you can use response.request.headers since request is an attribute of the Response object:
>>>>>> response = requests.get("https://api.thedogapi.com/v1/breeds/1")
>>> response.request.headers
{'User-Agent': 'python-requests/2.24.0',
'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*',
'Connection': 'keep-alive'}
In this case, you don’t define any specific headers when you make the request, so the default headers are returned.
X-, but they’re not required to. API developers typically use custom headers to send or request additional custom information from clients.
Fun fact: A few companies go to extra lengths to be funny and innovative, using HTTP headers in a way they weren’t meant to be used, such as to solicit job applications.
You can use a dictionary to define headers, and you can send them along with your request using the headers parameter of .get().
For example, say you want to send some request ID to the API server, and you know you can do that using X-Request-Id:
>>>>>> headers = {"X-Request-Id": "<my-request-id>"}
>>> response = requests.get("https://example.org", headers=headers)
>>> response.request.headers
{'User-Agent': 'python-requests/2.24.0', 'Accept-Encoding': 'gzip, deflate',
'Accept': '*/*', 'Connection': 'keep-alive',
'X-Request-Id': '<my-request-id>'}
If you go through the request.headers dictionary, then you’ll find X-Request-Id right at the end, among a few other headers that come by default with any API request.
There are many useful headers a response might have, but one of the most important ones is Content-Type, which defines the kind of content returned in the response.
Content-Type
Content-Type header, then you’ll notice how it was defined as application/json:
>>>>>> response = requests.get("https://api.thedogapi.com/v1/breeds/1")
>>> response.headers.get("Content-Type")
'application/json; charset=utf-8'
Apart from the specific type of content (in this case application/json), the header might also return the specified encoding for the response content.
PlaceGOAT API: This is a very silly API that returns images of goats in different sizes that you can use as placeholder images in your website.
If, for example, you try to fetch an image of a goat from the PlaceGOAT API, then you’ll notice that the content type is no longer application/json, but instead it’s defined as image/jpeg:
>>>>>> response = requests.get("http://placegoat.com/200/200")
>>> response
<Response [200]>
>>> response.headers.get("Content-Type")
'image/jpeg'
In this case, the Content-Type header states that the returned content is a JPEG image. You’ll learn how to view this content in the next section.
The Content-Type header is very important for you to know how to handle a response and what to do with its content. There are hundreds of other acceptable content types, including audio, video, fonts, and more.
Remove ads
Content-Type header. To properly read the response contents according to the different Content-Type headers, the requests package comes with a couple of different Response attributes you can use to manipulate the response data:
●.text returns the response contents in Unicode format.
●.content returns the response contents in bytes.
You already used the .text attribute above. But for some specific types of data, like images and other nontextual data, using .content is typically a better approach, even if it returns a very similar result to .text:
>>>>>> response = requests.get("https://api.thedogapi.com/v1/breeds/1")
>>> response.headers.get("Content-Type")
'application/json; charset=utf-8'
>>> response.content
b'{"weight":{"imperial":"6 - 13","metric":"3 - 6"}...'
As you can see, there isn’t a big difference between .content and the previously used .text.
However, by looking at the response’sContent-Type header, you can see the content is application/json;, a JSON object. For that kind of content, the requests library includes a specific .json() method that you can use to immediately convert the API bytes response into a Python data structure:
>>>>>> response = requests.get("https://api.thedogapi.com/v1/breeds/1")
>>> response.headers.get("Content-Type")
'application/json; charset=utf-8'
>>> response.json()
{'weight': {'imperial': '6 - 13', 'metric': '3 - 6'},
'height': {'imperial': '9 - 11.5', 'metric': '23 - 29'}
...}
>>> response.json()["name"]
'Affenpinscher'
As you can see, after executing response.json(), you get a dictionary that you’re able to use as you’d use any other dictionary in Python.
Now, looking back at the recent example you ran using the PlaceGOAT API, try to fetch that same goat image and have a look at its content:
>>>>>> response = requests.get("http://placegoat.com/200/200")
>>> response
<Response [200]>
>>> response.headers.get("Content-Type")
'image/jpeg'
>>> response.content
b'\xff\xd8\xff\xe0\x00\x10JFIF\x00\x01\x01\x01\x00H\...'
In this case, because you’re requesting an image, .content isn’t very helpful. In fact, it’s nearly impossible to understand. However, you know this is a JPEG image, so you can try storing it into a file and see what happens:
>>>>>> response = requests.get("http://placegoat.com/200/200")
>>> response
<Response [200]>
>>> response.headers.get("Content-Type")
'image/jpeg'
>>> file = open("goat.jpeg", "wb")
>>> file.write(response.content)
>>> file.close()
Now if you open the folder you’re working from, then you’ll find a goat.jpeg file, which is a random image of a goat that you just fetched using an API. Isn’t that amazing?
GET, and if you wanted to create some data, then you’d use the method POST.
When purely consuming data using APIs, you’ll typically stick to GET requests, but here’s a list of the most common methods and their typical use case:
| HTTP Method | Description | Requests method |
|---|---|---|
POST |
Create a new resource. | requests.post() |
GET |
Read an existing resource. | requests.get() |
PUT |
Update an existing resource. | requests.put() |
DELETE |
Delete an existing resource. | requests.delete() |
PATCH method that’s also associated with CRUD operations, but it’s slightly less common than the four above. It’s used to make partial modifications instead of completely replacing a resource using PUT.
You can read a bit more about the differences between PUT and PATCH to understand their different needs.
If you’re curious about the remaining HTTP methods, or if you just want to learn a bit more about those already mentioned, then have a look through Mozilla’s documentation.
Until now, you’ve only used .get() to fetch data, but you can use the requests package for all the other HTTP methods as well:
>>>>>> requests.post("https://api.thedogapi.com/v1/breeds/1")
>>> requests.get("https://api.thedogapi.com/v1/breeds/1")
>>> requests.put("https://api.thedogapi.com/v1/breeds/1")
>>> requests.delete("https://api.thedogapi.com/v1/breeds/1")
If you try these on your console, then you’ll notice that most of them will return a 405 Method Not Allowed status code. That’s because not all endpoints will allow for POST, PUT, or DELETE methods. Especially when you’re reading data using public APIs, you’ll find that most APIs will only allow GET requests since you’re not allowed to create or change the existing data.
Remove ads
/breeds endpoint, you get a lot of information about a given breed. But in some cases, you might want to extract only certain information about a given breed. That’s where query parameters come in!
You might have seen or used query parameters when browsing online. For example when watching a YouTube video, you have a URL like https://www.youtube.com/watch?v=aL5GK2LVMWI. The v= in the URL is what you call a query parameter. It typically comes after the base URL and endpoint.
To add a query parameter to a given URL, you have to add a question mark (?) before the first query parameter. If you want to have multiple query parameters in your request, then you can split them with an ampersand (&).
The same YouTube URL above with multiple query parameters would look like this: https://www.youtube.com/watch?v=aL5GK2LVMWI&t=75.
In the API world, query parameters are used as filters you can send with your API request to further narrow down the responses. For example, going back to the Random User Generator API, you know how to generate a random user:
>>>>>> requests.get("https://randomuser.me/api/").json()
{'results': [{'gender': 'male', 'name':
{'title': 'Mr', 'first': 'Silvijn', 'last': 'Van Bekkum'},
'location': {'street': {'number': 2480, 'name': 'Hooijengastrjitte'},
'city': 'Terherne', 'state': 'Drenthe',
'country': 'Netherlands', 'postcode': 59904...}
However, let’s say you specifically want to generate only random female users. According to the documentation, you can use the query parameter gender= for that:
>>>>>> requests.get("https://randomuser.me/api/?gender=female").json()
{'results': [{'gender': 'female', 'name':
{'title': 'Mrs', 'first': 'Marjoleine', 'last': 'Van Huffelen'},
'location': {'street': {'number': 8993, 'name': 'De Teebus'},
'city': 'West-Terschelling', 'state': 'Limburg',
'country': 'Netherlands', 'postcode': 24241...}
That’s great! Now let’s say you want to generate only female users from Germany. Again, looking through the documentation, you find a section on nationality, and you can use the query parameter nat= for that:
>>>>>> requests.get("https://randomuser.me/api/?gender=female&nat=de").json()
{'results': [{'gender': 'female', 'name':
{'title': 'Ms', 'first': 'Marita', 'last': 'Hertwig'},
'location': {'street': {'number': 1430, 'name': 'Waldstraße'},
'city': 'Velden', 'state': 'Rheinland-Pfalz',
'country': 'Germany', 'postcode': 30737...}
Using query parameters, you can start fetching more specific data from an API, making the whole experience a bit more tailored to your needs.
To avoid having to rebuild the URL over and over again, you can use the params attribute to send in a dictionary of all query parameters to append to a URL:
>>>>>> query_params = {"gender": "female", "nat": "de"}
>>> requests.get("https://randomuser.me/api/", params=query_params).json()
{'results': [{'gender': 'female', 'name':
{'title': 'Ms', 'first': 'Janet', 'last': 'Weyer'},
'location': {'street': {'number': 2582, 'name': 'Meisenweg'},
'city': 'Garding', 'state': 'Mecklenburg-Vorpommern',
'country': 'Germany', 'postcode': 56953...}
You can apply the above to any other API you like. If you go back to TheDogAPI, the documentation has a way for you to filter the breeds endpoint to return only the breeds that match a specific name. For example, if you wanted to look for the Labradoodle breed, then you could do that with the query parameter q:
>>>>>> query_params = {"q": "labradoodle"}
>>> endpoint = "https://api.thedogapi.com/v1/breeds/search"
>>> requests.get(endpoint, params=query_params).json()
[{'weight': {'imperial': '45 - 100', 'metric': '20 - 45'},
'height': {'imperial': '14 - 24', 'metric': '36 - 61'},
'id': 148, 'name': 'Labradoodle', 'breed_group': 'Mixed'...}]
There you have it! By sending the query parameter qwith the value labradoodle, you’re able to filter all breeds that match that specific value.
Tip: When you’re reusing the same endpoint, it’s a best practice to define it as a variable at the top of your code. This will make your life easier when interacting with an API over and over again.
With the help of query parameters, you’re able to further narrow your requests and specify exactly what you’re looking for. Most APIs you’ll find online will have some sort of query parameters that you can use to filter data. Remember to look through the documentation and API reference to find them.
401 Unauthorizedor403 Forbidden status code.
DEMO_KEY API key that NASA provides by default. Otherwise, you can quickly generate your own by going to NASA’smain API page and clicking Get Started.
You can add the API key to your request by appending the api_key= query parameter:
>>>>>> endpoint = "https://api.nasa.gov/mars-photos/api/v1/rovers/curiosity/photos"
>>> # Replace DEMO_KEY below with your own key if you generated one.
>>> api_key = "DEMO_KEY"
>>> query_params = {"api_key": api_key, "earth_date": "2020-07-01"}
>>> response = requests.get(endpoint, params=query_params)
>>> response
<Response [200]>
So far, so good. You managed to make an authenticated request to NASA’s API and to get back a 200 OK response.
Now have a look at the Response object and try to extract some pictures from it:
>>>>>> response.json()
{'photos': [{'id': 754118,
'sol': 2809,
'camera': {'id': 20,
'name': 'FHAZ',
'rover_id': 5,
'full_name': 'Front Hazard Avoidance Camera'},
'img_src': 'https://mars.nasa.gov/msl-raw-images/...JPG',
'earth_date': '2020-07-01',
'rover': {'id': 5,
'name': 'Curiosity',
'landing_date': '2012-08-06',
'launch_date': '2011-11-26',
'status': 'active'}},
...
}
>>> photos = response.json()["photos"]
>>> print(f"Found {len(photos)} photos")
Found 12 photos
>>> photos[4]["img_src"]
'https://mars.nasa.gov/msl-raw-images/proj/msl/redops/ods/surface/sol/02809/opgs/edr/rcam/RRB_646869036EDR_F0810628RHAZ00337M_.JPG'
Using .json() to convert the response to a Python dictionary and then fetching the photos field from the response, you’re able to iterate through all Photo objects and even fetch a specific photo’s image URL. If you open that URL in your browser, then you’ll see the following picture of Mars taken by one of the Mars rovers:

earth_date (2020-07-01) and then a specific photo from the response dictionary (4). Before moving forward, try changing the date or fetching pictures from a different camera to see how it changes the end result.

client_id) and a URL (redirect_uri) to redirect the user after success or error.
You’ll be redirected to the Facebook website and asked to log in with your credentials. The Spotify app won’t see or have access to these credentials. This is the most important benefit of OAuth.
Facebook will show you all the data the Spotify app is requesting from your profile and ask you to accept or reject sharing that data.
If you accept giving Spotify access to your data, then you’ll be redirected back to the Spotify app, already logged in.
When going through step 4, Facebook will provide Spotify with a special credential (access_token) that can be used repeatedly to fetch your information. This specific Facebook login token is valid for sixty days, but other apps might have different expiration periods. If you’re curious, then Facebook has a settings page that you can check to see which apps have been given your Facebook access token.
Now, from a more technical standpoint, here are the things you need to know when consuming APIs using OAuth:
●You need to create an application that will have an ID (app_idorclient_id) and a secret (app_secretorclient_secret).
●You need to have a redirect URL (redirect_uri), which the API will use to send information to you.
●You’ll get a code as the result of the authentication, which you need to exchange for an access token.
There are a few variations to the above, but generally speaking, most OAuth flows will have steps similar to these.
Tip: When you’re just testing things out and you need some sort of redirect URL to get a code, you can use a service called httpbin.
More specifically, you can use https://httpbin.org/anything as a redirect URL, as it’ll simply output whatever it gets as an input. You can test it yourself by navigating to that URL.
Next, you’ll dive into an example using the GitHub API!
https://httpbin.org/anything URL mentioned above for the Authorization callback URL field.
GitHub API: You can use the GitHub API for a lot of different use cases, such as getting a list of repositories you’re a part of, getting a list of followers you have, and much more.
Once you’ve created your app, copy and paste the Client_ID and Client_Secret, together with your selected redirect URL, into a Python file called github.py:
import requests
# REPLACE the following variables with your Client ID and Client Secret
CLIENT_ID = "<REPLACE_WITH_CLIENT_ID>"
CLIENT_SECRET = "<REPLACE_WITH_CLIENT_SECRET>"
# REPLACE the following variable with what you added in the
# "Authorization callback URL" field
REDIRECT_URI = "<REPLACE_WITH_REDIRECT_URI>"
Now that you have all the important variables in place, you need to be able to create a link to redirect the user to their GitHub account, as explained in the GitHub documentation:
def create_oauth_link():
params = {
"client_id": CLIENT_ID,
"redirect_uri": REDIRECT_URI,
"scope": "user",
"response_type": "code",
}
endpoint = "https://github.com/login/oauth/authorize"
response = requests.get(endpoint, params=params)
url = response.url
return url
In this piece of code, you first define the required parameters that the API expects and then call the API using the requests package and .get().
When you make the request to the /login/oauth/authorize endpoint, the API will automatically redirect you to the GitHub website. In that case, you want to fetch the url parameter from the response. This parameter contains the exact URL that GitHub is redirecting you to.
The next step in the authorization flow is to exchange the code you get for an access token. Again, following the steps in GitHub’s documentation, you can make a method for it:
def exchange_code_for_access_token(code=None):
params = {
"client_id": CLIENT_ID,
"client_secret": CLIENT_SECRET,
"redirect_uri": REDIRECT_URI,
"code": code,
}
headers = {"Accept": "application/json"}
endpoint = "https://github.com/login/oauth/access_token"
response = requests.post(endpoint, params=params, headers=headers).json()
return response["access_token"]
Here, you make a POST request to exchange the code for an access token. In this request, you have to send your CLIENT_SECRET and code so that GitHub can validate that this specific code was initially generated by your application. Only then will the GitHub API generate a valid access token and return it to you.
Now you can add the following to your file and try running it:
link = create_oauth_link()
print(f"Follow the link to start the authentication with GitHub: {link}")
code = input("GitHub code: ")
access_token = exchange_code_for_access_token(code)
print(f"Exchanged code {code} with access token: {access_token}")
If everything goes according to plan, then you should be rewarded with a valid access token that you can use to make calls to the GitHub API, impersonating the authenticated user.
Now try adding the following code to fetch your user profile using the User API and to print your name, username, and number of private repositories:
def print_user_info(access_token=None):
headers = {"Authorization": f"token {access_token}"}
endpoint = "https://api.github.com/user"
response = requests.get(endpoint, headers=headers).json()
name = response["name"]
username = response["login"]
private_repos_count = response["total_private_repos"]
print(
f"{name} ({username}) | private repositories: {private_repos_count}"
)
Now that you have a valid access token, you need to send it on all your API requests using the Authorization header. The response to your request will be a Python dictionary containing all the user information. From that dictionary, you want to fetch the fields name, login, and total_private_repos. You can also print the response variable to see what other fields are available.
Alright, that should be it! The only thing left to do is to put it all together and try it out:
1import requests
2
3# REPLACE the following variables with your Client ID and Client Secret
4CLIENT_ID = "<REPLACE_WITH_CLIENT_ID>"
5CLIENT_SECRET = "<REPLACE_WITH_CLIENT_SECRET>"
6
7# REPLACE the following variable with what you added in
8# the "Authorization callback URL" field
9REDIRECT_URI = "<REPLACE_WITH_REDIRECT_URI>"
10
11def create_oauth_link():
12 params = {
13 "client_id": CLIENT_ID,
14 "redirect_uri": REDIRECT_URI,
15 "scope": "user",
16 "response_type": "code",
17 }
18 endpoint = "https://github.com/login/oauth/authorize"
19 response = requests.get(endpoint, params=params)
20 url = response.url
21 return url
22
23def exchange_code_for_access_token(code=None):
24 params = {
25 "client_id": CLIENT_ID,
26 "client_secret": CLIENT_SECRET,
27 "redirect_uri": REDIRECT_URI,
28 "code": code,
29 }
30 headers = {"Accept": "application/json"}
31 endpoint = "https://github.com/login/oauth/access_token"
32 response = requests.post(endpoint, params=params, headers=headers).json()
33 return response["access_token"]
34
35def print_user_info(access_token=None):
36 headers = {"Authorization": f"token {access_token}"}
37 endpoint = "https://api.github.com/user"
38 response = requests.get(endpoint, headers=headers).json()
39 name = response["name"]
40 username = response["login"]
41 private_repos_count = response["total_private_repos"]
42 print(
43 f"{name} ({username}) | private repositories: {private_repos_count}"
44 )
45
46link = create_oauth_link()
47print(f"Follow the link to start the authentication with GitHub: {link}")
48code = input("GitHub code: ")
49access_token = exchange_code_for_access_token(code)
50print(f"Exchanged code {code} with access token: {access_token}")
51print_user_info(access_token=access_token)
Here’s what happens when you run the code above:
(一)A link is generated asking you to go to a GitHub page for authentication.
After following that link and logging in with your GitHub credentials, you’re redirected to your defined callback URL with a code field in the query parameters:

$ John Doe (johndoe) | number of private repositories: 42
There are quite a few steps to take here, but it’s important that you take the time to really understand each one. Most APIs using OAuth will share a lot of the same behavior, so knowing this process well will unlock a lot of potential when you’re reading data from APIs.
Feel free to improve this example and add more functionality, such as getting your public and starred repositories or iterating through your followers to identify the most popular ones.
There are plenty of great resources online about OAuth, and if consuming APIs behind OAuth is what you really need, then I’d advise you to do a bit more research on that topic specifically. Here are a few good places to start:
●What the Heck is OAuth?
●OAuth 2 Simplified
●OAuth 2.0 Authorization Framework
From an API consumption perspective, knowing OAuth will definitely come very in handy when you’re interacting with public APIs. Most APIs have adopted OAuth as their authentication standard, and with good reason.
Remove ads

page attribute that defines which page you’re currently requesting
(二)Asize attribute that defines the size of each page
The specific query parameter names might vary a lot depending on the API developers, but the concept is the same. A few API players might also use HTTP headers or the JSON response to return current pagination filters in place.
Using the GitHub API again, you can find an events endpoint in the documentation that contains pagination query parameters. The parameter per_page= defines the number of items to return, and page= allows you to paginate through multiple results. Here’s how to use these parameters:
>>>>>> response = requests.get("https://api.github.com/events?per_page=1&page=0")
>>> response.json()[0]["id"]
'14345572615'
>>> response = requests.get("https://api.github.com/events?per_page=1&page=1")
>>> response.json()[0]["id"]
'14345572808'
>>> response = requests.get("https://api.github.com/events?per_page=1&page=2")
>>> response.json()[0]["id"]
'14345572100'
With the first URL, you’re only able to fetch one event. But using the page= query parameter, you can keep paginating through results, making sure that you’re able to fetch all of the events without overloading the API.
/events endpoint. According to its documentation, GitHub allows about sixty unauthenticated requests per hour. If you go above that, then you’ll get a 403 status code and won’t be able to make any more API calls for quite some time.
Warning: Running the next piece of code will really block you from calling GitHub for some time, so make sure you don’t need access to GitHub’s API for a bit before you run it.
For the sake of demonstration, you’ll purposefully try to exceed GitHub’s rate limit to see what happens. In the code below, you’ll request data until you get a status code other than 200 OK:
>>>>>> endpoint = "https://api.github.com/events"
>>> for i in range(100):
>>> response = requests.get(endpoint)
>>> print(f"{i} - {response.status_code}")
>>> if response.status_code != 200:
>>> break
0 - 200
1 - 200
2 - 200
3 - 200
4 - 200
5 - 200
...
55 - 200
56 - 200
57 - 403
>>> response
<Response [403]>
>>> response.json()
{'message': "API rate limit exceeded for <ip-address>.",
'documentation_url': 'https://developer.github.com/v3/#rate-limiting'}
There you have it: After about sixty requests, the API stopped returning 200 OK responses and returned a 403 Forbidden response instead, informing you that you exceeded the API rate limit.
Some APIs, like GitHub’s, might even include additional information in the headers regarding your current rate limit and how many requests you have remaining. These are very helpful for you to avoid going over the defined limit. Have a look at the latest response.headers to see if you can find those specific rate limiting headers.





API_KEY variable below, you can start consuming the GIPHY API:
1import requests
2
3# Replace the following with the API key generated.
4API_KEY = "API_KEY"
5endpoint = "https://api.giphy.com/v1/gifs/trending"
6
7params = {"api_key": API_KEY, "limit": 3, "rating": "g"}
8response = requests.get(ENDPOINT, params=params).json()
9for gif in response["data"]:
10 title = gif["title"]
11 trending_date = gif["trending_datetime"]
12 url = gif["url"]
13 print(f"{title} | {trending_date} | {url}")
At the top of the file, on lines 4 and 5, you define your API_KEY and the GIPHY API endpoint since they won’t change as often as the rest.
On line 7, making use of what you learned in the query parameters section, you define the params and add your own API key. You also include a couple of other filters: limit to get 3results and rating to get only appropriate content.
Finally, after getting a response, you iterate through the results on line 9. For each GIF, you print its title, date, and URL on line 13.
Running this piece of code in the console would output a somewhat structured list of GIFs:
Excited Schitts Creek GIF by CBC | 2020-11-28 20:45:14 | https://giphy.com/gifs/cbc-schittscreek-schitts-creek-SiGg4zSmwmbafTYwpj
Saved By The Bell Shrug GIF by PeacockTV | 2020-11-28 20:30:15 | https://giphy.com/gifs/peacocktv-saved-by-the-bell-bayside-high-school-dZRjehRpivtJsNUxW9
Schitts Creek Thank You GIF by CBC | 2020-11-28 20:15:07 | https://giphy.com/gifs/cbc-funny-comedy-26n79l9afmfm1POjC
Now, let’s say you want to make a script that allows you to search for a specific word and fetch the first GIPHY match to that word. A different endpoint and slight variation of the code above can do that quite quickly:
import requests
# Replace the following with the API key generated.
API_KEY = "API_KEY"
endpoint = "https://api.giphy.com/v1/gifs/search"
search_term = "shrug"
params = {"api_key": API_KEY, "limit": 1, "q": search_term, "rating": "g"}
response = requests.get(endpoint, params=params).json()
for gif in response["data"]:
title = gif["title"]
url = gif["url"]
print(f"{title} | {url}")
There you have it! Now you can modify this script to your liking and generate GIFs on demand. Try fetching GIFs from your favorite show or movie, adding a shortcut to your terminal to get the most popular GIFs on demand, or integrating with another API from your favorite messaging system—WhatsApp, Slack, you name it. Then start sending GIFs to your friends and coworkers!
1import requests
2from datetime import date, timedelta
3
4today = date.today()
5yesterday = today - timedelta(days=1)
6country = "germany"
7endpoint = f"https://api.covid19api.com/country/{country}/status/confirmed"
8params = {"from": str(yesterday), "to": str(today)}
9
10response = requests.get(endpoint, params=params).json()
11total_confirmed = 0
12for day in response:
13 cases = day.get("Cases", 0)
14 total_confirmed += cases
15
16print(f"Total Confirmed Covid-19 cases in {country}: {total_confirmed}")
On lines 1 and 2, you import the necessary modules. In this case, you have to import the date and timedelta objects to be able to get today’s and yesterday’s dates.
On lines 6 to 8, you define the country slug you want to use, the endpoint, and the query parameters for the API request.
The response is a list of days, and for each day you have a Cases field that contains the total number of confirmed cases on that date. On line 11, you create a variable to keep the total number of confirmed cases, and then on line 14 you iterate through all the days and sum them up.
Printing the end result will show you the total number of confirmed cases in the selected country:
Total Confirmed Covid-19 cases in germany: 1038649
In this example, you’re looking at total number of confirmed cases for a whole country. However, you could also try looking at the documentation and fetching the data for your specific city instead. And why not make it a bit more thorough and get some other data, such as the number of recovered cases?
Remove ads
moby dick in the whole catalog:
1import requests
2
3endpoint = "https://www.googleapis.com/books/v1/volumes"
4query = "moby dick"
5
6params = {"q": query, "maxResults": 3}
7response = requests.get(endpoint, params=params).json()
8for book in response["items"]:
9 volume = book["volumeInfo"]
10 title = volume["title"]
11 published = volume["publishedDate"]
12 description = volume["description"]
13 print(f"{title} ({published}) | {description}")
This code example is pretty similar to the ones you’ve seen before. You start on lines 3 and 4 by defining important variables, such as the endpoint and, in this case, the query.
After making the API request, on line 8 you start iterating through the results. Then, on line 13, you print the most interesting information for each book that matches your initial query:
Moby-Dick (2016-04-12) | "Call me Ishmael." So begins the famous opening...
Moby Dick (1892) | A literary classic that wasn't recognized for its...
Moby Dick; Or, The Whale (1983-08-16) | The story of Captain Ahab's...
You can print the book variable inside the loop to see what other fields you have available. Here are a few that could be useful for further improving this code:
●industryIdentifiers
●averageRating and ratingsCount
●imageLinks
A fun challenge to do with this API is to use your OAuth knowledge and create your own bookshelf app that keeps records of all the books you read or want to read. You can even connect it to your favorite bookstore or library afterward to quickly find books from your wish list that are available near you. This is just one idea—I’m sure you can come up with more.
About Pedro Pregueiro

Hi! My name is Pedro and I'm a Python developer who loves coding, burgers and playing guitar.
» More about Pedro
Geir Arne
Jon
Jacob
Master Real-World Python Skills With Unlimited Access to Real Python

🔒 No spam. Unsubscribe any time.
All Tutorial Topics advanced api basics best-practices community databases data-science devops django docker flask front-end gamedev gui intermediate machine-learning projects python testing tools web-dev web-scraping Table of Contents ●Getting to Know APIs ●SOAP vs REST vs GraphQL ●requests and APIs: A Match Made in Heaven ●Calling Your First API Using Python ●Endpoints and Resources ●Request and Response ●Status Codes ●HTTP Headers ●Response Content ●HTTP Methods ●Query Parameters ●Learning Advanced API Concepts ●Authentication ●Pagination ●Rate Limiting ●Consuming APIs With Python: Practical Examples ●Searching and Fetching Trending GIFs ●Getting COVID-19 Confirmed Cases Per Country ●Searching Google Books ●Conclusion ●Further Reading Tweet Share Email Almost there! Complete this form and click the button below to gain instant access:
Consuming APIs With Python (Source Code)
Remove ads
© 2012–2021 Real Python ⋅ Newsletter ⋅ Podcast ⋅ YouTube ⋅ Twitter ⋅ Facebook ⋅ Instagram ⋅ Python Tutorials ⋅ Search ⋅ Privacy Policy ⋅ Energy Policy ⋅ Advertise ⋅ Contact
❤️ Happy Pythoning!