Overview:
The Crawler-website is a website crawler tool developed using the Go language. It allows users to crawl websites, extract data, and store it in an Elasticsearch database. The tool follows the MVC pattern and utilizes Docker for easy deployment and management. It also supports concurrent and distributed crawling using singleton, concurrent, and distributed architectures. The website provides a user-friendly interface to view and query the crawled data.
Features:
- Go language: The crawler is developed using the Go programming language, known for its concurrency and performance benefits.
- Docker: The tool can be easily deployed using Docker containers, simplifying the setup and management process.
- Elasticsearch: The crawled data is stored in an Elasticsearch database, enabling efficient indexing and querying of the data.
- MVC pattern: The crawler follows the Model-View-Controller (MVC) architectural pattern, separating the concerns of data storage, presentation, and user interaction.
- Microservices: The distributed crawling capability is achieved using microservices, allowing the tool to scale and handle large volumes of data.
- Singleton -> Concurrent -> Distribute: The crawler supports different architectures to handle crawling tasks, from a singleton setup to concurrent and distributed setups.
- HTML
HTML templates are pre-designed and pre-built web pages that can be customized and used as a basis for building websites. They often include common elements such as headers, footers, menus, and content sections, and can be easily edited using HTML and CSS to fit specific branding and content needs.