Mastering Web Scraping with Scrapy: A Step-by-Step Guide to Web Scraping

Posted on January 23, 2025January 23, 2025 by Spo_oky

Configuration

Basic Project Setup

Creating a new Scrapy project

Before diving into the intricacies of web scraping with Scrapy, the first step is to set up a new Scrapy project. This process is straightforward, thanks to Scrapy’s command-line tools. Open your terminal or command prompt and navigate to the directory where you want to create your project. Then, use the following command:

scrapy startproject myproject

Replace myproject with the desired name for your project. This command generates a directory with the name you specified, containing the initial Scrapy project structure. This structure is the foundation upon which you’ll build your web spider.

Project Structure Overview

After creating your project, it’s essential to understand the structure of the generated directory. Scrapy sets up a well-organized file system for your data extraction needs. Let’s take a quick tour:

scrapy.cfg: This is the deployment configuration file. You typically won’t need to modify this file for basic projects.
myproject/: This directory contains your project’s python scraping code.
- __init__.py: An empty file that tells python that this directory should be considered a python package.
- items.py: This file defines the data containers (Items) that you’ll use to store scraped data.
- middlewares.py: Contains classes for request and response processing.
- pipelines.py: Defines how to process scraped items (e.g., storing them in a database).
- settings.py: This is the most important file for configuring your Scrapy project. It controls various settings, such as user agents, request delays, and data mining pipelines.
- spiders/: This directory is where you’ll define your spiders, which are the classes responsible for crawling and scraping websites.
  - __init__.py: Same as above, makes the spiders/ directory a python package.

Understanding this structure is crucial because it dictates where you’ll place your code and how you’ll configure your scraping project. Now, let’s delve deeper into the settings.py file.

Understanding Settings.py

The settings.py file is the central nervous system of your Scrapy project. It allows you to configure various aspects of your web crawling operation, from the user agent to the pipelines that process your scraped data. Let’s explore some of the core settings you’ll encounter in this file.

Core settings

USER_AGENT

The USER_AGENT setting is a string that identifies your Scrapy bot to the websites you’re scraping. It’s essential to set this to a realistic value to avoid being blocked. Many websites block requests with default or generic user agents. A good practice is to use a user agent string from a common web browser. For example:

USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'

ROBOTSTXT_OBEY

The ROBOTSTXT_OBEY setting tells Scrapy whether to respect the robots.txt file of the websites you’re scraping. This file specifies which parts of the site should not be accessed by bots. By default, this setting is set to True, meaning Scrapy will obey the robots.txt rules. If you set it to False, Scrapy will ignore these rules, but be aware that this could be seen as unethical or even illegal. It’s generally a good idea to respect robots.txt unless you have a very good reason not to.

ROBOTSTXT_OBEY = True

CONCURRENT_REQUESTS

The CONCURRENT_REQUESTS setting controls the maximum number of concurrent (i.e., simultaneous) requests that Scrapy will make to a website. Increasing this value can speed up your data scraping, but it can also put more load on the target server and increase the risk of being blocked. It’s a trade-off between speed and politeness. The default value is usually a reasonable starting point, but you may need to adjust it depending on the website’s capacity and your scraping needs.

CONCURRENT_REQUESTS = 16

Configuring Pipelines

Scrapy pipelines are components that process items after they have been scraped by a spider. Pipelines are typically used for cleaning, validating, and storing the scraped data. You can define multiple pipelines in your settings.py file, and Scrapy will process items through them in the order they are defined.

To enable a pipeline, you need to add it to the ITEM_PIPELINES setting in settings.py. This setting is a dictionary where the keys are the pipeline class names and the values are integers that represent the order in which the pipelines should be executed. Lower values mean the pipeline will be executed earlier.

Here’s an example of how to configure pipelines:

ITEM_PIPELINES = {
 'myproject.pipelines.MyPipeline': 300,
 'myproject.pipelines.AnotherPipeline': 400,
}

In this example, MyPipeline will be executed before AnotherPipeline. The numbers 300 and 400 are arbitrary but must be unique for each pipeline.

Configuring Middlewares

Scrapy middlewares are hooks into Scrapy’s request and response processing. They allow you to modify requests before they are sent to the server and responses before they are processed by the spider. Middlewares can be used for various tasks, such as adding headers to requests, handling cookies, and retrying failed requests.

To enable a middleware, you need to add it to the SPIDER_MIDDLEWARES or DOWNLOADER_MIDDLEWARES settings in settings.py, depending on whether the middleware should process requests and responses at the spider level or the downloader level. Similar to pipelines, these settings are dictionaries where the keys are the middleware class names and the values are integers that represent the order in which the middlewares should be executed.

Here’s an example of how to configure middlewares:

DOWNLOADER_MIDDLEWARES = {
 'myproject.middlewares.MyMiddleware': 543,
}

In this example, MyMiddleware will be enabled as a downloader middleware with an order of 543.

Core Components

Creating a Spider

At the heart of every Scrapy project lies the spider. Spiders are classes that define how to crawl a website and extract data. To create a spider, you’ll typically inherit from the scrapy.Spider class and define a few essential attributes and methods. Let’s walk through the process:

Create a new python file inside the spiders/ directory of your Scrapy project. For example, you might create a file named myspider.py.
Import the scrapy library and define your spider class:

import scrapy

class MySpider(scrapy.Spider):
  name = 'myspider'
  allowed_domains = ['example.com']
  start_urls = ['http://www.example.com/']

  def parse(self, response):
    # Your scraping logic goes here
    pass

name: This is the unique name of your spider. It’s used to identify the spider when you run it.
allowed_domains: This is a list of domains that the spider is allowed to crawl. Any links to domains not in this list will be ignored. This helps to prevent your spider from wandering off to unintended websites.
start_urls: This is a list of URLs that the spider will start crawling from. The spider will make initial requests to these URLs, and the responses will be passed to the parse method.
parse(self, response): This is the most important method in your spider. It’s called for each response downloaded from the URLs in start_urls (and any other URLs that you instruct the spider to follow). The response object contains the HTML content of the page.

Understanding start_urls

The start_urls attribute is a list of URLs that your spider will initially crawl. These URLs serve as the entry points for your web crawling process. Scrapy will automatically generate Request objects for each URL in start_urls and schedule them to be downloaded. Once the responses are downloaded, they will be passed to your spider’s parse method for processing.

You can specify multiple URLs in start_urls to start crawling from different sections of a website. For example:

start_urls = [
 'http://www.example.com/category1',
 'http://www.example.com/category2',
 'http://www.example.com/category3',
]

Parsing Responses

The parse method is where you’ll write the code to extract the data you need from the HTML content of the web pages. Scrapy provides powerful tools for selecting and extracting data, including CSS selectors and XPath selectors.

Using CSS Selectors

CSS selectors allow you to target HTML elements based on their CSS classes, IDs, and other attributes. They are a concise and readable way to specify which elements you want to extract data from. To use CSS selectors in Scrapy, you can call the response.css() method, passing in a CSS selector string. For example, to extract the text from all <h1> elements on a page, you would use the following code:

for title in response.css('h1::text').getall():
 print(title)

The ::text selector extracts the text content of the selected elements. The .getall() method returns a list of all matching text strings.

Using XPath Selectors

XPath selectors are another way to target HTML elements. XPath is a more powerful and flexible language than CSS selectors, but it can also be more complex. To use XPath selectors in Scrapy, you can call the response.xpath() method, passing in an XPath expression. For example, to extract the href attribute from all <a> elements on a page, you would use the following code:

for href in response.xpath('//a/@href').getall():
 print(href)

The //a/@href XPath expression selects the href attribute of all <a> elements in the document. The .getall() method returns a list of all matching href values.

Following Links

To create a true web spider, you’ll often need to follow links from one page to another. Scrapy makes it easy to extract links from a page and schedule new requests to those links. To do this, you can use the response.follow() method. This method takes a URL (or a CSS selector or XPath selector that points to a URL) and creates a new Request object for that URL. You can also specify a callback function to be called when the response from the new URL is downloaded.

For example, to extract all links from a page and follow them, you could use the following code:

for link in response.css('a::attr(href)').getall():
 yield response.follow(link, callback=self.parse)

This code extracts the href attribute from all <a> elements on the page and then uses response.follow() to create a new request for each link. The callback=self.parse argument tells Scrapy to call the parse method for each new response. This allows your spider to recursively crawl the website, following links and extracting data from each page.

Item Pipelines

Defining Items

Items are simple containers used to store the scraped data. You define them in the items.py file of your Scrapy project. Each item is a class that inherits from scrapy.Item and defines fields using scrapy.Field. Here’s an example:

import scrapyclass 

Product(scrapy.Item):
name = scrapy.Field()
price = scrapy.Field()
description = scrapy.Field()
image_url = scrapy.Field()

This defines an Item called Product with fields for name, price, description, and image URL. You can then create instances of this class in your spider to store the extracted data.

Creating Item Pipelines

Item pipelines are components that process items after they have been scraped by a spider. Pipelines are typically used for cleaning, validating, and storing the scraped data. To create a pipeline, you define a class that implements the process_item(self, item, spider) method. This method receives the item and the spider that scraped it as arguments. It should return the item (either the original or a modified version) or raise a DropItem exception to discard the item.

from scrapy.exceptions import DropItem

class PricePipeline:
  def process_item(self, item, spider):
    price = item['price']
    if isinstance(price, str):
    price = price.replace('$', '')
    try:
      price = float(price)
    except ValueError:
      raise DropItem("Invalid price format: %s" % price)

    if price < 0:
      raise DropItem("Negative price detected: %s" % price)
    item['price'] = price
    return item

This pipeline converts the price to a float and ensures it’s not negative. If the price is invalid, it raises a DropItem exception to discard the item.

Processing Items

In your spider, after you extract the data, you create an instance of your Item and populate its fields. Then, you yield the item. This sends the item to the item pipelines for processing.

import scrapy
from myproject.items import Product

class MySpider(scrapy.Spider):
  name = 'myspider'
  allowed_domains = ['example.com']
  start_urls = ['http://www.example.com/products']

  def parse(self, response):
    for product in response.css('.product'):
      item = Product()
      item['name'] = product.css('.name::text').get()
      item['price'] = product.css('.price::text').get()
      item['description'] = product.css('.description::text').get()
      item['image_url'] = product.css('img::attr(src)').get()
    yield item

Examples of Pipeline use cases

Item pipelines are powerful tools for post-processing scraped data. Here are some common use cases:

Data Cleaning

Pipelines can be used to clean and normalize data, such as removing whitespace, converting to a consistent format, or correcting errors. For example, you might have a pipeline that removes HTML tags from a description field or converts dates to a standard format.

Data Validation

Pipelines can be used to validate data and ensure it meets certain criteria. For example, you might have a pipeline that checks if a required field is present or if a value is within a valid range. If the data is invalid, the pipeline can raise a DropItem exception to discard the item.

Storing data in a database

Pipelines are commonly used to store scraped data in a database. You can create a pipeline that connects to a database (e.g., MySQL, PostgreSQL, MongoDB) and inserts the scraped data into the appropriate tables or collections. This allows you to easily analyze and use the data in other applications.

Scrapy Shell

Starting the Scrapy Shell

The Scrapy Shell is an interactive console that allows you to test your scraping code and explore web pages. To start the Scrapy Shell, open your terminal or command prompt and navigate to your Scrapy project directory. Then, use the following command:

scrapy shell 'http://www.example.com'

Replace 'http://www.example.com' with the URL of the web page you want to explore. Scrapy will download the page and make it available in the shell.

Inspecting Responses

Once the Scrapy Shell is running, you can access the response object using the response variable. The response object contains the HTML content of the page, as well as other information such as the HTTP headers and status code. You can use CSS selectors and XPath selectors to extract data from the response object, just like you would in a spider.

response.css('h1::text').getall()
response.xpath('//a/@href').getall()

Testing Selectors

The Scrapy Shell is a great tool for testing your CSS selectors and XPath selectors. You can quickly try out different selectors and see what data they extract. This can save you a lot of time and effort when developing your spiders.

Advanced Scrapy

Using Proxies

Configuring Proxies in Scrapy

When engaging in extensive web scraping, it’s crucial to consider using proxies to avoid IP bans and ensure the longevity of your data extraction efforts. Scrapy provides several ways to configure proxies, allowing you to rotate through different IP addresses. One common method is to use a proxy middleware.

Using Proxy Middleware

To use a proxy middleware, you first need to find a reliable proxy provider and obtain a list of proxy servers. Then, you can create a custom middleware that intercepts requests and assigns a proxy to them. Here’s an example:

import base64
from scrapy.exceptions import NotConfigured

class ProxyMiddleware:
  def __init__(self, proxies):
    self.proxies = proxies
    if not self.proxies:
      raise NotConfigured

  @classmethod
  def from_crawler(cls, crawler):
    return cls(crawler.settings.getlist('PROXIES'))

  def process_request(self, request, spider):
    proxy = self.proxies.pop()

    if 'user:password' in proxy:
      b64_auth_string = base64.b64encode(proxy.split('@')[0].encode()).decode()
      request.headers['Proxy-Authorization'] = 'Basic ' + b64_auth_string
      request.meta['proxy'] = proxy.split('@')[1] if '@' in proxy else proxy
    else:
      request.meta['proxy'] = proxy
      self.proxies.insert(0, proxy)

In this example, the ProxyMiddleware class takes a list of proxy servers as input. The process_request method is called for each request, and it assigns a proxy from the list to the request’s meta dictionary. If the proxy requires authentication, the middleware also sets the Proxy-Authorization header.

To enable this middleware, you need to add it to the DOWNLOADER_MIDDLEWARES setting in your settings.py file:

DOWNLOADER_MIDDLEWARES = {
 'myproject.middlewares.ProxyMiddleware': 750,
}

PROXIES = [
 'user:password@host1:port',
 'host2:port',
 'host3:port',
]

Replace 'myproject.middlewares.ProxyMiddleware' with the actual path to your middleware class. Adjust the priority (750 in this example) as needed. Also, replace the example proxy servers with your own list of proxy servers.

Handling Cookies

Configuring Cookie Handling

Many websites use cookies to track user sessions and personalize content. When web scraping, it’s often necessary to handle cookies to maintain session state and access certain parts of the site. Scrapy provides built-in support for cookie handling, which is enabled by default.

Dealing with Login Forms

If you need to scrape a website that requires authentication, you’ll need to simulate the login process. This typically involves submitting a form with a username and password. Scrapy provides tools to help you with this. You can use the FormRequest.from_response() method to create a request that automatically populates the form data from a response. Here’s an example:

import scrapy

class LoginSpider(scrapy.Spider):
  name = 'login_spider'
  start_urls = ['http://www.example.com/login']

  def parse(self, response):
    return scrapy.FormRequest.from_response(
      response,
      formdata={'username': 'your_username', 'password': 'your_password'},
      callback=self.after_login
    )

  def after_login(self, response):
    if 'Welcome, your_username' in response.text:
      self.log('Login successful!')
      # Continue scraping
    else:
      self.log('Login failed!')

In this example, the parse method creates a FormRequest from the login page response. It populates the username and password fields with your credentials and specifies the after_login method as the callback function. The after_login method checks if the login was successful and then continues with the scraping process.

Working with APIs

Fetching data from APIs

Many websites and services provide APIs (Application Programming Interfaces) that allow you to access data in a structured format, typically JSON. Scraping data from APIs is often easier and more reliable than scraping HTML pages. To fetch data from an API, you can simply make a request to the API endpoint and parse the JSON response.

Parsing JSON Responses

Scrapy provides built-in support for parsing JSON responses. When you make a request to an API endpoint, Scrapy automatically decodes the JSON response and makes it available as a python dictionary or list. You can then access the data using standard python dictionary or list syntax. Here’s an example:

import scrapy
import json

class ApiSpider(scrapy.Spider):
  name = 'api_spider'
  start_urls = ['http://www.example.com/api/data']

  def parse(self, response):
    data = json.loads(response.text)
    for item in data:
      yield item

In this example, the parse method uses the json.loads() function to decode the JSON response. It then iterates over the data and yields each item as a Scrapy item.

Using Scrapy with Databases

Connecting to Databases (MySQL, PostgreSQL, MongoDB)

Once you’ve scraped the data, you’ll often want to store it in a database for further analysis or use in other applications. Scrapy can be easily integrated with various databases, such as MySQL, PostgreSQL, and MongoDB. To connect to a database, you’ll typically use a python library that provides a database connector.

Storing Scraped Data

To store scraped data in a database, you can create an item pipeline that connects to the database and inserts the data. Here’s an example of a pipeline that stores data in a MongoDB database:

import pymongo

class MongoPipeline:
  def __init__(self, mongo_uri, mongo_db):
    self.mongo_uri = mongo_uri
    self.mongo_db = mongo_db

  @classmethod
  def from_crawler(cls, crawler):
    return cls(
      mongo_uri=crawler.settings.get('MONGO_URI'),
      mongo_db=crawler.settings.get('MONGO_DATABASE')
    )

  def open_spider(self, spider):
    self.client = pymongo.MongoClient(self.mongo_uri)
    self.db = self.client[self.mongo_db]

  def close_spider(self, spider):
    self.client.close()

  def process_item(self, item, spider):
    self.db['items'].insert_one(dict(item))
    return item

In this example, the MongoPipeline class connects to a MongoDB database when the spider starts and closes the connection when the spider finishes. The process_item method inserts each item into the items collection. To enable this pipeline, you need to add it to the ITEM_PIPELINES setting in your settings.py file and configure the MONGO_URI and MONGO_DATABASE settings.

Real-World Use Cases

Web Scraping for E-commerce

Extracting product information

Scrapy is invaluable for e-commerce businesses aiming to gather product details from competitors or suppliers. Imagine you’re building a price comparison website or need to update your product catalog automatically. With Scrapy, you can efficiently extract product names, descriptions, prices, images, and specifications from various e-commerce sites. This data scraping process allows you to maintain a competitive edge by understanding market trends and pricing strategies.

To achieve this, you would define a Scrapy spider that targets specific e-commerce websites. Using CSS or XPath selectors, you pinpoint the HTML elements containing the desired product information. For example:

# Extracting product name
response.css('.product-name::text').get()
# Extracting product price
response.css('.product-price::text').get()

The extracted data can then be stored in a structured format using Scrapy items and processed through item pipelines for cleaning and storage in a database.

Monitoring prices

Beyond extracting static product information, Scrapy is excellent for dynamic price monitoring. In the fast-paced world of e-commerce, prices fluctuate frequently. Using Scrapy, you can build a web crawling solution that regularly checks product prices on competitor websites and alerts you to any changes. This enables you to adjust your pricing strategy in real-time, ensuring you remain competitive and maximize profit margins.

To implement price monitoring, you would schedule your Scrapy spider to run periodically (e.g., every hour or every day). The spider would extract the current price of each product and compare it to the previously recorded price. If a change is detected, you can trigger an alert via email or update your own pricing database automatically.

Data Extraction for Research

Gathering data from academic websites

Researchers often need to collect large datasets from academic websites for analysis. Manually gathering this data can be time-consuming and tedious. Scrapy provides a powerful solution for automating this process, enabling researchers to focus on analysis rather than data mining.

For example, a researcher studying climate change might want to collect data on temperature, rainfall, and sea levels from various meteorological websites. Using Scrapy, they can define a spider that crawls these websites, extracts the relevant data, and stores it in a structured format for analysis. Similarly, a social scientist might use Scrapy to gather data from online forums or social media platforms for sentiment analysis.

Analyzing research trends

Scrapy is also useful for analyzing research trends by extracting data from academic publications and citation databases. By scraping data on publication dates, authors, keywords, and citations, researchers can identify emerging trends, influential papers, and key researchers in their field. This information can be used to guide future research directions and identify potential collaborations.

News Aggregation

Scraping news articles

News aggregators collect news articles from various sources and present them in a unified format. Scrapy is a perfect tool for building such aggregators, allowing you to automatically extract news articles from different websites. This eliminates the need for manual content curation and ensures that your aggregator is always up-to-date.

To scrape news articles, you would define a Scrapy spider that targets specific news websites. The spider would extract the article title, content, publication date, author, and other relevant metadata. You can use CSS or XPath selectors to pinpoint the HTML elements containing this information. For example:

# Extracting article title
response.css('.article-title::text').get()
# Extracting article content
response.css('.article-content::text').getall()

Building a news aggregator

Once you’ve scraped the news articles, you can store them in a database and build a user interface to display the aggregated content. You can also add features such as keyword filtering, topic categorization, and sentiment analysis to enhance the user experience.

By combining Scrapy with other python scraping libraries and web frameworks, you can create a powerful and customizable news aggregator that meets your specific needs.

Scrapy Commands

scrapy startproject

Creating a new project

The scrapy startproject command is used to create a new Scrapy project. This command sets up the basic directory structure and necessary files for your project. To use it, open your terminal or command prompt and navigate to the directory where you want to create the project. Then, run the following command:

scrapy startproject project_name

Replace project_name with the desired name for your project. This command will create a new directory with the specified name, containing the initial Scrapy project structure, facilitating efficient data extraction.

scrapy genspider

Generating a new spider

The scrapy genspider command is used to generate a new spider within your Scrapy project. This command simplifies the process of creating a spider by providing a template with the basic structure. To use it, navigate to your Scrapy project directory in the terminal or command prompt and run the following command:

scrapy genspider spider_name domain.com

spider_name: This is the name of your spider. It should be unique within the project.
domain.com: This is the domain that your spider will crawl. It will be automatically added to the allowed_domains attribute of the spider.

For example:

scrapy genspider my_spider example.com

This command will create a new spider named my_spider in the spiders directory of your project. The spider will be pre-configured to crawl the example.com domain. This command is very usefull for python scraping projects.

scrapy crawl

Running a spider

The scrapy crawl command is used to run a spider within your Scrapy project. This command starts the web crawling process and executes the spider’s logic to extract data from the specified website. To use it, navigate to your Scrapy project directory in the terminal or command prompt and run the following command:

scrapy crawl spider_name

Replace spider_name with the name of the spider you want to run. For example:

scrapy crawl my_spider

This command will start the my_spider spider. Scrapy will then begin crawling the website specified in the spider’s start_urls attribute and extract data according to the spider’s logic. The results of the crawl will be displayed in the terminal or can be saved to a file.

You can also specify output formats using the -o option. For example, to save the output to a JSON file:

scrapy crawl my_spider -o output.json

Other supported formats include CSV, XML, and pickle.

scrapy shell

Interactive shell

The scrapy shell command is used to start an interactive shell that allows you to test your scraping code and explore web pages. This is a useful tool for debugging and experimenting with CSS and XPath selectors. To use it, open your terminal or command prompt and run the following command:

scrapy shell url

Replace url with the URL of the web page you want to explore. For example:

scrapy shell http://www.example.com

This command will download the HTML content of the specified URL and make it available in the shell. You can then use CSS and XPath selectors to extract data from the response object.

Inside the shell, you have access to the response variable, which contains the HTML content of the page. You can use the response.css() and response.xpath() methods to select elements and extract data.

scrapy settings

View settings

The scrapy settings command is used to view the settings for your Scrapy project. This command displays a list of all the settings that are currently in effect, along with their values. This is useful for debugging and understanding how your project is configured. To use it, navigate to your Scrapy project directory in the terminal or command prompt and run the following command:

scrapy settings [options]

Options:

--get <setting>: Print the value for setting.
--shell: Shell settings.

For example, to view the value of the USER_AGENT setting:

scrapy settings --get USER_AGENT

This command will print the value of the USER_AGENT setting to the terminal.

Best Practices

Respecting robots.txt

When engaging in web scraping, it’s crucial to respect the robots.txt file of the websites you’re targeting. This file, located at the root of a domain (e.g., http://www.example.com/robots.txt), specifies which parts of the site should not be accessed by bots. Disregarding robots.txt can lead to your bot being blocked, or worse, legal issues. Always check the robots.txt file before starting a web crawling project and adhere to its directives.

Scrapy, by default, respects robots.txt. The ROBOTSTXT_OBEY setting in settings.py is set to True by default. If you need to disable this for testing purposes, you can set it to False, but be aware of the potential consequences.

ROBOTSTXT_OBEY = True

Handling Dynamic Content

Many modern websites use JavaScript to generate content dynamically. This means that the HTML source code you receive when you make a request may not contain all the data you need. Scrapy, by itself, cannot execute JavaScript. To handle dynamic content, you’ll need to integrate Scrapy with a tool that can render JavaScript.

Using Selenium with Scrapy

Selenium is a popular tool for automating web browsers. You can use Selenium with Scrapy to render JavaScript and extract the resulting HTML. To do this, you’ll need to install Selenium and a browser driver (e.g., ChromeDriver for Chrome, GeckoDriver for Firefox). You can then create a Scrapy middleware that uses Selenium to render the page before passing it to the spider.

First, install Selenium:

pip install selenium

Next, download the appropriate browser driver and place it in a directory that’s in your system’s PATH. Here’s an example of a Scrapy middleware that uses Selenium:

from scrapy import signals
from scrapy.http import HtmlResponse
from selenium import webdriver

class SeleniumMiddleware:
  @classmethod
  def from_crawler(cls, crawler):
    middleware = cls()
    crawler.signals.connect(middleware.spider_opened, signal=signals.spider_opened)
    crawler.signals.connect(middleware.spider_closed, signal=signals.spider_closed)
    return middleware

  def process_request(self, request, spider):
    if spider.name == 'my_dynamic_spider': # Apply only to specific spiders
      driver = webdriver.Chrome() # Or any other browser
      driver.get(request.url)
      html = driver.page_source
      driver.quit()
    return HtmlResponse(url=request.url, body=html.encode('utf-8'), encoding='utf-8', request=request)

  def spider_opened(self, spider):
    pass

  def spider_closed(self, spider):
    pass

Remember to replace 'my_dynamic_spider' with the actual name of your spider.

Enable the middleware in settings.py:

DOWNLOADER_MIDDLEWARES = {
 'myproject.middlewares.SeleniumMiddleware': 543,
}

Using Splash with Scrapy

Splash is a lightweight, headless browser specifically designed for rendering JavaScript in web scraping applications. It provides an HTTP API that you can use to render pages and extract data. Splash is often more efficient than Selenium, especially for complex JavaScript applications.

First, you’ll need to install and run Splash. You can find instructions on how to do this on the Splash website. Once Splash is running, you can create a Scrapy middleware that uses the Splash API to render pages.

Here’s an example of a Scrapy middleware that uses Splash:

import scrapy
from scrapy.http import HtmlResponse

class SplashMiddleware:
  def process_request(self, request, spider):
    if spider.name == 'my_dynamic_spider': # Apply only to specific spiders
      splash_args = {
        'html': 1,
        'png': 0,
        'jpeg': 0,
        'url': request.url,
    }

    url = 'http://localhost:8050/render.html?' + scrapy.http.urlencode(splash_args)
    req = scrapy.Request(url, self.parse_splash_response)
    req.meta['original_request'] = request
    return req

  def parse_splash_response(self, response):
    original_request = response.meta['original_request']
    return HtmlResponse(url=original_request.url, body=response.body, encoding='utf-8', request=original_request)

Enable the middleware in settings.py:

DOWNLOADER_MIDDLEWARES = {
 'myproject.middlewares.SplashMiddleware': 543,
}

You’ll also want to install scrapy-splash:

pip install scrapy-splash

Avoiding Bans

Websites often implement anti-scraping measures to prevent bots from overloading their servers or extracting data without permission. To avoid being banned, you should implement several strategies:

Implementing delay

The DOWNLOAD_DELAY setting in settings.py controls the amount of time Scrapy waits between requests. Increasing this value can reduce the load on the target server and decrease the likelihood of being banned. A good starting point is 1-2 seconds.

DOWNLOAD_DELAY = 2

For more sophisticated control, consider using the AutoThrottle extension, which automatically adjusts the delay based on the server’s response time.

Rotating User-Agent

Websites can identify bots by their user agent string. To avoid this, you should rotate your user agent regularly. You can create a list of user agents and randomly select one for each request. You can achieve this with a middleware.

First, define a list of user agents in your settings.py file:

USER_AGENT_LIST = [
 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.3 Safari/605.1.15',
 # Add more user agents here
]

Then, create a middleware to rotate the user agent:

import random
from scrapy.downloadermiddlewares.useragent import UserAgentMiddleware

class RotateUserAgentMiddleware(UserAgentMiddleware):
  def __init__(self, user_agent_list):
    self.user_agent_list = user_agent_list

  @classmethod
  def from_crawler(cls, crawler):
    return cls(
      user_agent_list=crawler.settings.get('USER_AGENT_LIST')
    )

  def process_request(self, request, spider):
    ua = random.choice(self.user_agent_list)
    request.headers.setdefault('User-Agent', ua)

Enable the middleware in settings.py:

DOWNLOADER_MIDDLEWARES = {
 'myproject.middlewares.RotateUserAgentMiddleware': 400,
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
}

Using proxies

Using proxies is one of the most effective ways to avoid being banned. By routing your requests through different IP addresses, you can hide your real IP address and make it more difficult for websites to track your bot. See the previous chapter for more information about configuring proxies.

Monitoring and Logging

Configuring Logging

Scrapy has a robust logging system that helps you track the progress of your spiders and identify any issues. You can configure the logging level, output format, and destination in the settings.py file.

To configure the logging level, use the LOG_LEVEL setting:

LOG_LEVEL = 'INFO' # Can be DEBUG, INFO, WARNING, ERROR, CRITICAL

To configure the logging output, use the LOG_FILE setting:

LOG_FILE = 'scrapy.log' # Log to a file
LOG_STDOUT = True # Display log messages in the console

Monitoring Scrapy Jobs

For long-running or critical scrapy jobs, it’s important to monitor their progress and performance. Scrapy provides several ways to do this, including:

Scrapy’s built-in stats collection: Scrapy automatically collects various statistics during a crawl, such as the number of requests made, the number of items scraped, and the time taken. You can access these stats through the Scrapy API or by configuring a stats collector extension.
Third-party monitoring tools: Several third-party tools can be used to monitor Scrapy jobs, such as Scrapyd, which provides a web interface for managing and monitoring Scrapy spiders.

By implementing these best practices, you can build robust and reliable web scraping solutions that are less likely to be blocked and easier to maintain.

In conclusion, Scrapy stands out as a robust and versatile framework for web scraping. Throughout this guide, we’ve explored everything from initial installation and project setup to advanced techniques like handling dynamic content and avoiding bans. Understanding Scrapy’s core components, such as spiders and item pipelines, empowers you to efficiently extract and process data. By adhering to best practices and leveraging Scrapy’s powerful commands, you can build scalable and reliable data extraction solutions for various real-world applications. Whether you’re in e-commerce, research, or news aggregation, Scrapy provides the tools you need to unlock the wealth of information available on the web, streamlining your workflow and enabling data-driven decisions.

Configuration

Basic Project Setup

Creating a new Scrapy project

Project Structure Overview

Understanding Settings.py

Core settings

USER_AGENT

ROBOTSTXT_OBEY

CONCURRENT_REQUESTS

Configuring Pipelines

Configuring Middlewares

Core Components

Creating a Spider

Understanding start_urls

Parsing Responses

Using CSS Selectors

Using XPath Selectors

Following Links

Item Pipelines

Defining Items

Creating Item Pipelines

Processing Items

Examples of Pipeline use cases

Data Cleaning

Data Validation

Storing data in a database

Scrapy Shell

Starting the Scrapy Shell

Inspecting Responses

Testing Selectors

Advanced Scrapy

Using Proxies

Configuring Proxies in Scrapy

Using Proxy Middleware

Handling Cookies

Configuring Cookie Handling

Dealing with Login Forms

Working with APIs

Fetching data from APIs

Parsing JSON Responses

Using Scrapy with Databases

Connecting to Databases (MySQL, PostgreSQL, MongoDB)

Storing Scraped Data

Real-World Use Cases

Web Scraping for E-commerce

Extracting product information

Monitoring prices

Data Extraction for Research

Gathering data from academic websites

Analyzing research trends

News Aggregation

Scraping news articles

Building a news aggregator

Scrapy Commands

scrapy startproject

Creating a new project

scrapy genspider

Generating a new spider

scrapy crawl

Running a spider

scrapy shell

Interactive shell

scrapy settings

View settings

Best Practices

Respecting robots.txt

Handling Dynamic Content

Using Selenium with Scrapy

Using Splash with Scrapy

Avoiding Bans

Implementing delay

Rotating User-Agent

Using proxies

Monitoring and Logging

Configuring Logging

Monitoring Scrapy Jobs

Conclusion

Leave a Reply Cancel reply

Recent Posts

Recent Comments