Overview of the Requests Library
requests is a prominent library in Python used to handle HTTP requests and responses. It is utilized for web scraping, API (Application Programming Interface) requests, and managing various web requests and responses.
Note: API refers to an interface defining the communication protocols between a server and client.
Features of requests
-
Simple to use: A straightforward API makes the handling of HTTP requests and responses easy. -
Support for various HTTP methods: Offers support for different HTTP methods like GET, POST, PUT, DELETE, etc. -
Convenient response handling: Enables simple handling of response data such as status codes, text, and JSON through response objects.
GET Request
To request data from a webpage or an API, a GET request is sent using the requests.get() function.
import requests # Send a GET request to a webpage and receive a response response = requests.get('https://example.com') print(response.text) # Output the HTML received in response
POST Request
When sending data to a server or requesting certain operations, a POST request is sent using the requests.post() function.
# Send a POST request to the server payload = {'key1': 'value1', 'key2': 'value2'} response = requests.post('https://httpbin.org/post', data=payload) res = response.text # Save the response into the res variable print(res)
Handling Response Objects
-
response.status_code: Returns the HTTP status code (e.g., 200, 404). -
response.headers: Returns the response headers as a dictionary. -
response.json(): Converts the JSON response to a Python object.
Practice
Press the Run Code button on the right side of the screen to see the crawling results or modify the code!
Lessons in this chapter · Understanding APIs, JSON, and CSV for Web Scraping
Lecture
AI Tutor
Design
Upload
Notes
Favorites
Help