Zimfarm

Overview

If you're looking for an innovative way to create ZIM files efficiently, ZIM FarmBuild offers a semi-decentralized software solution that effectively bundles web content into ZIM files. This method not only streamlines the creation process but also ensures that the resulting files can be stored in an online repository, making them easily accessible for users. The combination of a robust backend, user-friendly frontend, and dedicated workers sets ZIM FarmBuild apart as a reliable tool for both developers and content creators.

The platform is designed for versatility and efficiency. By harnessing a structured approach to task assignment and resource management, ZIM FarmBuild minimizes downtime and maximizes output. Whether you are a seasoned developer or just starting out, this tool provides everything needed to manage ZIM file creation seamlessly.

Features

• Centralized Database: The backend hosts a central database and API that manages recipes and tasks, ensuring smooth operation and easy access to metadata needed for ZIM creation.

• User-Friendly Frontend: Available at farm.openzim.org, the frontend allows users to create, clone, and edit recipes, along with tracking task progress in an intuitive interface.

• Efficient Worker Management: Workers are dedicated machines that handle ZIM creation tasks, organized by a dispatcher for optimal performance and resource allocation.

• Task-Worker Functionality: Each task is managed by a task-worker container that efficiently runs specific tasks while monitoring the scraping process.

• Fast Uploader Capability: The uploader component supports SCP and SFTP uploads, featuring a fast, resume-capable mechanism for handling files.

• DNS Caching: With dnscache, a dnsmasq server, the platform ensures reliable DNS usage, providing stability and maintaining task efficiency even during DNS outages.

• Secure Log Handling: The receiver acts as a secured OpenSSH-server, safely managing the reception of logs and ZIM files, ensuring data integrity during transfers.