Scrapy Boilerplate

Overview

The Scrapy Boilerplate serves as a robust starting point for new Scrapy projects, allowing developers to streamline their workflow and enhance scalability. This project is still a work in progress, so users can expect to see regular updates and improvements. With its comprehensive configuration options and modern dependencies, this boilerplate can be a real game-changer for developers looking to leverage the power of web scraping effectively.

By integrating essential features like Docker support and RabbitMQ, along with a clear and intuitive file structure, the Scrapy Boilerplate caters to both newcomers and seasoned developers alike. The adoption of popular tools such as Poetry for dependency management and SQLAlchemy for ORM elevates the overall development experience.

Features

• Python 3.11+ Compatibility: Built to work seamlessly with the latest versions of Python, ensuring access to the newest features and improvements.

• Poetry for Dependency Management: Utilize Poetry for smooth package management, making it easier to manage project dependencies and virtual environments.

• SQLAlchemy ORM with Alembic Migrations: Simplifies database interaction and schema changes, allowing for efficient data handling and management.

• Integrated RabbitMQ: Out-of-the-box support for RabbitMQ, facilitating robust message queuing and inter-process communication.

• Environment Configuration: Supports configuration via environment variables or a .env file, allowing for easy and flexible project setup.

• Docker-Ready: Dockerfiles and docker-compose configurations are included, making it straightforward to run your spiders in containers.

• User-Friendly File Structure: A flattened directory structure for ease of navigation, with each class organized into its own file, promoting better code readability and maintainability.

• Proxy Middleware Support: Comes with built-in support for proxy configurations, enabling easy integration of rotating proxies for enhanced scraping capabilities.