Discover how to read data from APIs and convert that data into a Python data structure, such as a Pandas DataFrame, a vital skill for many tech-related careers. This video tutorial will guide you through the process, using the free Rick and Morty API for demonstration purposes.
Key Insights
- The tutorial covers how to utilize the Pandas library in Python to convert data received from APIs into a Pandas DataFrame.
- The Requests Library is necessary for Python to connect online and send data requests to a server.
- The Rick and Morty API is used as a free resource for this tutorial, negating the need for registration or a credit card.
- After obtaining data through the `get` method of the Requests Library, it's assigned to a variable, named `raw` in this case.
- The received data is typically in JSON format, which resembles a dictionary, and specific values can be extracted using dictionary notation.
- The tutorial instructs how to use `pd.json_normalize()` to convert the data into a DataFrame, which can then be utilized in various ways, such as writing it into a text file or a SQL database.
Python Workflow
-
Set Up venv
python -m venv .venv — isolated environment per project.
-
Install Dependencies
pip install -r requirements.txt — pin versions for reproducibility.
-
Write & Test
Write functions; test with pytest as you go.
-
Run & Deploy
python script.py locally, deploy to your platform of choice.
Video Transcription
Hi, my name is Art, and in this video I'm going to show you how to read data from APIs and convert that data into a Python data structure such as a Pandas DataFrame.
To use Pandas, you need to import it: `import pandas as pd`.
You also need to import the Requests Library, which helps Python to get online and send requests to a server for data. Most APIs require registration and sometimes require a credit card. To avoid this, we can use the Rick and Morty API which is free.
We can use the `get` method from the Requests Library to send a request to a server and get data. We will assign this data to a variable called `raw`, since it's a response. We can then parse this data, which most likely will come in the form of JSON (JavaScript Object Notation). JSON looks like a dictionary and we can use `.keys()` to get the options.
We can then grab the values we want such as `name`, `status`, and `species`, and use dictionary notation to get them. Now we can initialize the DataFrame and use `pd.json_normalize()` to convert the data into a DataFrame. We can assign it to a variable called `df` and then do something else with it such as writing it into a text file or a SQL database.
Video Transcription:
Full Video Transcript
Hi, my name is Art. I teach Python. In this video, I'm going to show you how you could read data from APIs and how you could convert that data into some other Python data structure, such as maybe a Pandas DataFrame.
So what we're going to do: if you want to use Pandas, you need to import Pandas as PD. That's the first thing. Then we need to import another library: import requests. So what is requests? Requests is a small Python library, very popular.
Understanding APIs and HTTP Requests
You have to understand what is an API. An API is some data served from a server. So you need to get online. Python cannot get online because Python is a programming language, so Python requires some help such as requests to literally get online and send requests to a server for data.
Now, most APIs require a lot of registration, and sometimes you need to provide credit card information. I don't want to do any of that, so here what we're going to be doing: apparently there is a very popular API, totally free, and that's based on a very popular cartoon, Rick and Morty. If you're fans, hey! They provide all these data about characters. So that would be your URL.
Making Your First API Request
Then what I'm going to do next: I need to use the requests library, and there is a method called get. So this method get, if I pass here the URL and if I call it, what it's going to do is send a request for information to a server. So that's https://rickandmortyapi.com.
Now you have to understand that here we have a computer and we call it a client, and then there is a server. The client and the server exchange messages. In this case, we're asking for data, and they send us this data back.
HTTP Status Codes
In the header of each message, there is a code. In this case, we see the code is 200. What does it mean? You probably have seen before, sometimes you're browsing the internet and you would see an error message: page not found or data not found 404. So 404 error is pretty much that code.
For working with APIs, you don't need to know all of them. You could just Google them. But 200 to 300 means that everything went fine. 300 to 400 means there is some kind of problem with authentication. 400 to 500 means there is some problem with information and data. And 500 to 600 means that there is some kind of issue with the server or with the app itself. But in this case, that's okay.
Parsing JSON Response Data
So I'm going to assign it to a variable named "raw" because that's the response. Now we could parse that data, and 99.9 percent of all APIs send data in the form of JSON. What is JSON? JSON stands for JavaScript Object Notation.
Let me show you how it looks like. So we need to use JSON to parse that data. Data would look like, how you probably guessed it, it looks like a dictionary, right? So what they call JSON, here in Python we call a dictionary. So if you run type on data, you see that that's a dictionary. How cool is that!
And you know what to do with a dictionary if you watch my previous videos. The first thing I would do is run keys, because I want to see what are my options. So first you get some kind of metadata, probably not interesting for us usually, like date and maybe time and maybe something else. We don't need that, so all I care about is results.
Iterating Through API Results
And now you see that's another dictionary. However, that dictionary comes in what? Let's run type, and that would be a list. So with the list, we've got a couple of options. First of all, we could iterate through it. So we could say for i in data results, and print i. So i would be what? Another dictionary, dictionary, dictionary.
If it's dictionary, dictionary, dictionary, so you know what to do again: keys, keys, keys, right? So you could run keys, and suppose I care about name, status, and species, maybe these three. So then what's going to happen: I need to grab these three and I need to use this dictionary notation. I hope you remember how to do it. If not, please watch my previous video.
So then it's going to be i name, i status, i species. Now you see we get Rick Sanchez, alive, human. Morty Smith, alive, human. And so on. Now you could leave it like this and maybe you could do something else with this data. I don't know, maybe write it into a text file if you need it, or maybe write it into SQL or whatever. Or you could convert it to a DataFrame.
Converting JSON to Pandas DataFrame
So how are you going to do it? Then what we need to do is initialize the DataFrame. And before we do that, take PD (that's Pandas that we imported). If you're not sure what to do next, always place that period and press Tab. Remember, that's a shortcut for Pandas.
Now this is a dictionary but also kind of JSON, so there is json_normalize. Not sure what it does? That's okay, you know how to use help. So then let's take a look: it requires some data dictionary, and then what it does is it returns a DataFrame.
So here I'll pass data, and if I run it, oh, I don't like it because I wanted to get it in a different format. So then I'm going to do data results. Now you see that's a DataFrame. So we could assign it to df, and that's our DataFrame here. So you see we've got one column: ID, name, status, species, type, everything else.
Cleaning and Filtering the Data
Now if you want to clean it a bit, you could grab just the columns that you actually want to use, suppose these three. And the point I'm trying to make is: when you fetch data from an API, 99.9 percent of the time it will be coming as JSON. So then you could actually convert that JSON pretty much into any other data format that you want to use.
So now you see I just cleaned the data a bit, and now I have a DataFrame with the same data, and now you could do something else.
Summary and Key Takeaways
So that's what you could do. One more time: all APIs will require a library. And that's not the only library that you could use, because there is the module urllib, there is also the library httpx, but requests is the most popular one. That's why I'm showing it to you.
So requests sends requests to a server, and that's our URL. Make sure it serves an API; usually it says "API" within the URL itself. So then we're fetching data from that server, and that data needs to be parsed as JSON. JSON, apparently in this case, will come as a dictionary. And then we could pass the data into a function called json_normalize that comes from Pandas, and that gives us a DataFrame. Pandas DataFrame. Thank you.
