Nextjs Firecrawl Starter
Nextjs 15 Firecrawl app to scrape doc links for an LLM. Use it as a starter kit to build your Firecrawl app. Turn any developer documentation into a GPT knowledge base. Pre-made Github Action to crawl and commit markdown responses to repo.
Overview
The Next.js Firecrawl Starter is an impressive tool designed to create a seamless web interface for crawling documentation and preparing it for Large Language Model (LLM) use. This application leverages a modern tech stack to deliver essential functionalities that enable users to convert crawled data into markdown, XML, or zip files, making it easy to build knowledge files compatible with various LLM tools. Designed with both usability and flexibility in mind, this tool supports efficient documentation processing, potentially simplifying workflows for developers and researchers alike.
With its serverless architecture, real-time updates, and extensive customization options through GitHub Actions, the Next.js Firecrawl Starter stands out in the landscape of documentation management tools. Whether you're looking to automate web crawling or enhance your LLM projects, this app promises a robust experience.
Features
-
Serverless Architecture: Utilizes Firecrawl API v1 for seamless, on-demand crawling processes without the need for a dedicated server infrastructure.
-
Real-time Crawl Status Updates: Stay informed as the application provides live updates on the progress of your crawling jobs.
-
Modern UI with Dark/Light Mode Support: Enjoy a user-friendly interface that adapts to your visual preferences, enhancing the overall user experience.
-
Crawl History with Local Storage: Easily access and review past crawls, as the application utilizes local storage to save crawl history.
-
GitHub Actions Integration: Define automations for your crawls, including scheduled tasks through GitHub Actions, to streamline your documentation gathering process.
-
OpenAPI Spec & Custom GPT Actions: Create endpoints for your LLM tools, allowing seamless integration and interaction with the Firecrawl API within chat applications.
-
Flexible Output Formats: Generate output in various formats, such as markdown, XML, or zip files, ready for immediate use in knowledge bases or LLM applications.
-
Tech Stack: Powered by Next.js, Tailwind CSS, and a host of modern libraries, ensuring a fast and responsive user experience backed by solid technology.
- Next.js
Next.js is a React-based web framework that enables server-side rendering, static site generation, and other powerful features for building modern web applications.
- React
React is a widely used JavaScript library for building user interfaces and single-page applications. It follows a component-based architecture and uses a virtual DOM to efficiently update and render UI components
- Tailwind
Tailwind CSS is a utility-first CSS framework that provides pre-defined classes for building responsive and customizable user interfaces.
- cmdk
cmdk is a fast, composable command menu component for React. It provides the foundation for building command palettes, search interfaces, and keyboard-navigable menus similar to those found in applications like VS Code, Linear, and Raycast.
- Shadcn UI
Beautifully designed components that you can copy and paste into your apps. Accessible. Customizable. Open Source.
- Documentation
Documentation themes are built specifically for writing technical and product documentation. They are normally written and maintained in Markdown. The often include a navigation menu, search bar, clear headings, semantic document structure and clean typography.
- Eslint
ESLint is a linter for JavaScript that analyzes code to detect and report on potential problems and errors, as well as enforce consistent code style and best practices, helping developers to write cleaner, more maintainable code.
- Framer Motion
Alpine.js is a lightweight JavaScript framework that simplifies the process of creating dynamic, reactive user interfaces on the web. It uses a declarative syntax that offers a higher level of abstraction compared to vanilla JavaScript, while being more performant and easier to use than jQuery.
- Postcss
PostCSS is a popular open-source tool that enables web developers to transform CSS styles with JavaScript plugins. It allows for efficient processing of CSS styles, from applying vendor prefixes to improving browser compatibility, ultimately resulting in cleaner, faster, and more maintainable code.
- React Hook Form
React Hook Form is a performant, flexible, and extensible form library for React with easy validation. It reduces re-renders and improves performance by using uncontrolled components and native HTML validation, making form handling simple and efficient.
- Recharts
Recharts is a powerful and easy-to-use React library for building customizable and interactive charts. Built on D3.js, it offers a wide range of pre-built chart types, such as line, bar, pie, and scatter charts, all of which can be composed with a declarative syntax.
- Typescript
TypeScript is a superset of JavaScript, providing optional static typing, classes, interfaces, and other features that help developers write more maintainable and scalable code. TypeScript's static typing system can catch errors at compile-time, making it easier to build and maintain large applications.
- Zod
Zod is a TypeScript-first schema declaration and validation library. It allows you to define schemas that can validate data at runtime while providing excellent TypeScript inference, making it perfect for API validation, form validation, and type-safe data handling.