Scrapyd Django Template

screenshot of Scrapyd Django Template
django

Scrapyd Django Template

Basic setup to run ScrapyD + Django and save it in Django Models. You can be up and running in just a few minutes

Overview

Scrapyd-Django-Template is a basic setup that allows users to run ScrapyD with Django and save the scraped data in Django Models. With a quick and easy installation process, users can be up and running in just a few minutes. This template includes the basic structure of a Django project, configuration of Scrapy to access Django models, a pipeline to save crawled objects to Django models, a basic spider definition, and a demo from the official Scrapy tutorial that crawls data from http://quotes.toscrape.com.

Features

  • Basic structure of a Django project: Provides a pre-configured Django project structure for easy setup and implementation.
  • Configuration of Scrapy to access Django models: Guides users on how to configure Scrapy to interact with Django models and save scraped data.
  • Basic scrapy pipeline to save crawled objects to Django models: Includes a predefined pipeline that automatically saves the crawled objects into Django models.
  • Basic spider definition: Offers a basic spider definition as a starting point for users to customize and expand upon.
  • Demo from the official tutorial: Includes a demo spider that showcases how to crawl data from http://quotes.toscrape.com, providing users with a practical example to follow.

Installation

To install Scrapyd-Django-Template, follow these steps:

  1. Install the project requirements:
pip install -r requirements.txt
  1. Configure the database in your Django project settings.

  2. Create a superuser to login to the Django admin:

python manage.py createsuperuser

Summary

Scrapyd-Django-Template is a convenient solution for running ScrapyD with Django and saving scraped data in Django Models. It provides a basic project structure, configuration guidelines, and a predefined pipeline and spider. By following the simple installation steps, users can quickly set up their project and start scraping data with ease. Inspired by an article from Ali Oğuzhan Yıldız, this template offers a simplified and efficient way to integrate Scrapy and Django applications.

django
Django

Django is a high-level Python web framework that encourages rapid development and clean, pragmatic design. It follows the model-view-controller (MVC) architectural pattern, providing an extensive set of built-in tools and conventions to streamline the creation of robust and scalable web applications.