Performing an HTTP Request in Python
Learn about the basics of HTTP and also about the request library in Python to make different types of requests.
Check out DataCamp's Importing Data in Python (Part 2) course that covers making HTTP requests.
In this tutorial, we will cover how to download an image, pass an argument to a request, and how to perform a 'post' request to post the data to a particular route. Also, you'll learn how to obtain a JSON response to do a more dynamic operation.
- Libraries in Python to make HTTP Request
- Request in Python
- Using GET Request
- Downloading and Saving an image using the Request Module
- Passing Argument in the Request
- Using POST Request
- JSON Response
HTTP stands for the 'HyperText Transfer Protocol,' where communication is possible by request done by the client and the response made by the server.
For example, you can use the client(browser) to search for a 'dog' image on Google. Then that sends an HTTP request to the server, i.e., a place where a dog image is hosted, and the response from the server is the status code with the requested content. This is a process also known as a request-response cycle. You can also look at this article, What is HTTP for a more detailed explanation.
Libraries in Python to make HTTP Request
You'll be using the request library for this tutorial and the command for doing this is below:
# pip install -r requirements.txt
Request in Python
According to Wikipedia, "requests are a Python HTTP library, released under the Apache2 License. The goal of the project is to make HTTP requests simpler and more human-friendly. The current version is 2.22.0"
Using GET Request
GET request is the most common method and is used to obtain the requested data from the specific server.
You need to import the required modules in your development environment using the following commands:
You can retrieve the data from the specific resource by using 'request.get('specific_url')', and 'r' is the response object.
According to Wikipedia, "Status codes are issued by a server in response to a client's request made to the server.". There are lots of other status codes and detailed explanations that can be found here: HTTP Status Code.
However, the two most common status code is explained below:
After running the above code you can see the status code is '200' which is 'OK,' and a request is successful.
You can view the response headers by using '.headers.' where it returns the Python Dictionaries. They return a lot of additional information containing the case insensitive name with resources types along with the server name, version, etc., and are also included with the code shown below.
The vital information obtained in the above code is the Server name as 'Apache', content type, Encoding, etc.
You can see above the type of content of the header by using 'content-type' which is case insensitive, meaning that 'Content-Type' would also give the same result.
You can get the HTML text of a page by using '.text.' where the request can decode any content automatically from the server, and it is in a Unicode form with the code below:
You can get an entire page of HTML and parsing can be done by the help of HTML parser.
'<!DOCTYPE html> <html> <head> <link href="http://www.smbc-comics.com/comiccontrol/defaultstyles.css?=2" rel="stylesheet" type="text/css"> <link rel="shortcut icon" href="http://www.smbc-comics.com/favicon.ico" type="image/x-icon" /> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"/> <link rel="shortcut icon" href="http://www.smbc-comics.com/favicon.ico" type="image/x-icon"> <link rel="icon" href="http://www.smbc-comics.com/favicon.ico"> ...
Downloading and Saving an Image Using Request module
You need to import the module, i.e., using the requests command in your local computer and 'receive' the response object with the 'request.get.' along with the image URL to be download as done below.
import requests receive = requests.get('https://imgs.xkcd.com/comics/making_progress.png')
with open(r'graphics/comic1.png','wb') as f: f.write(receive.content)
You can see the 'with' statement above helps to manage the file stream using the with open function where the required path specifies for doing a specific operation, 'r' which converts the normal string to a raw string. I.e., all of the characters need to remain the same where the location of a local computer contains '' in C:\Users\Dell\Desktop\comics\image5.png' needs to be preserved and should not be escaped. The mode for opening is 'wb' which is writing the files in a binary way, and 'f' is the file object that has to write a function to write the appropriate content, i.e., downloading the required image.
The final code for downloading and saving an image using the request module is the following:
import requests receive = requests.get('https://imgs.xkcd.com/comics/making_progress.png') with open(r'graphics/comic1.png','wb') as f: f.write(receive.content)