📌Overview

Purpose: DevDocs aims to significantly reduce the time spent on understanding and utilizing technical documentation, streamlining the development process for software engineers and teams.

Overview: DevDocs is a powerful documentation retrieval framework designed to facilitate quick access to and understanding of technology documentation. By automating the extraction and organization of content, it enables developers to focus on implementation rather than research, transforming weeks of effort into mere hours of productive work.

Key Features:

Intelligent Crawling: Employs smart depth control for link discovery and categorization, allowing selective extraction of relevant information and automatic mapping of website structures.
Performance & Speed: Offers parallel processing capabilities, smart caching, and lazy loading support, thus optimizing the crawling experience while ensuring respectful adherence to server load limitations.
Content Processing: Delivers clean extraction of content in multiple formats (MD, JSON), organizes it logically, and integrates seamlessly with MCP servers for AI processing.
Enterprise Features: Includes robust error recovery, full operational logging, API access for tool integration, and team management capabilities for effective collaboration.

DevDocs by CyberAGI 🚀

Turn Weeks of Documentation Research into Hours of Productive Development

🎯 Perfect For

🏢 Enterprise Software Developers

Skip weeks of reading documentation and dealing with technical debt. Implement ANY technology faster by letting DevDocs handle the heavy lifting of documentation understanding.

🕸️ Web Scrapers

Pull entire contents of websites with Smart Discovery of Child URLs up to level 5. Perfect for internal and external website documentation with intelligent crawling.

👥 Development Teams

Leverage internal documentation with built-in MCP servers and Claude integration for intelligent data querying. Transform your team's knowledge base into an actionable resource.

🚀 Indie Hackers

DevDocs + VSCode(cline) + Your Idea = Ship products fast with ANY technology. Avoid getting stuck in documentation hell.

✨ Features

🧠 Intelligent Crawling

Smart Depth Control: Crawl depth from 1-5 levels
Automatic Link Discovery: Finds and categorizes related content
Selective Crawling: Extract exactly what you want
Child URL Detection: Maps website structure automatically

⚡ Performance & Speed

Parallel Processing: Crawl multiple pages simultaneously
Smart Caching: Avoid duplicate content processing
Lazy Loading Support: Handles modern web apps
Rate Limiting: Respectful crawling without server overload

🎯 Content Processing

Clean Extraction: Content without fluff
Multiple Formats: Export to MD or JSON for LLM fine-tuning
Structured Output: Logically organized content
MCP Server Integration: Ready for AI processing

🛡️ Enterprise Features

Error Recovery: Auto-retry on failures
Full Logging: Track every operation
API Access: Integrate with your tools
Team Management: Multiple seats and roles

🤔 Why DevDocs?

The Problem

Documentation is everywhere, and LLMs have outdated knowledge. Researching and implementing takes weeks even for senior engineers.

Our Solution

DevDocs brings documentation to you:

Discovers all related pages
Extracts meaningful content without fluff
Organizes information inside an MCP server ready for your LLM
Presents it cleanly in MD or JSON for fine-tuning

We enable building amazing products quickly using cutting-edge LLM technology.

💰 Pricing Comparison

Feature	DevDocs	Firecrawl
Free Tier	Unlimited pages	None
Starting Price	Free Forever	$16/month
Enterprise Plan	Custom	$333/month
Crawl Speed	1000/min	20/min
Depth Levels	Up to 5	Limited
Team Seats	Unlimited	1-5 seats
Export Formats	MD, JSON, MCP	Limited formats
API Access	Coming Soon	Limited
Model Context Protocol Integration	✅	❌
Support	Priority via Discord	Standard only
Self-hosted (free use)	✅	❌

🚀 Getting Started

DevDocs is easy to use with Docker, requiring minimal setup.

Prerequisites

Docker installed (Get Docker)
Git for cloning

Quick Start with Docker (Recommended)

For Mac/Linux users:

git clone https://github.com/cyberagiinc/DevDocs.git
cd DevDocs
cp .env.template .env

# Ensure NEXT_PUBLIC_BACKEND_URL in .env is correct (e.g., http://localhost:24125)

./docker-start.sh

For Windows users (Experimental):

git clone https://github.com/cyberagiinc/DevDocs.git
cd DevDocs
copy .env.template .env

# Ensure NEXT_PUBLIC_BACKEND_URL in .env is correct (e.g., http://localhost:24125)
# Install WSL 2 and Docker Desktop before running

docker-start.bat

Notes for Windows Users

If you encounter permission issues, run scripts as administrator or manually set permissions on logs, storage, and crawl_results folders.

Accessing DevDocs

Frontend UI: http://localhost:3001
Backend API: http://localhost:24125
Crawl4AI Service: http://localhost:11235

Logs and Monitoring

Use Docker commands to view logs:

docker logs devdocs-frontend
docker logs devdocs-backend
docker logs devdocs-mcp
docker logs devdocs-crawl4ai

# Follow logs in real-time
docker logs -f devdocs-backend

Press Ctrl+C to stop all services.

📜 Scripts and Their Purpose

Startup Scripts

start.sh / start.bat / start.ps1: Start all local dev services
docker-start.sh / docker-start.bat: Start all services via Docker

MCP Server Scripts

check_mcp_health.sh: Verify MCP server health
restart_and_test_mcp.sh: Restart MCP containers and test

Crawl4AI Scripts

check_crawl4ai.sh: Check Crawl4AI status
debug_crawl4ai.sh: Run Crawl4AI with verbose logging
test_crawl4ai.py: Run Crawl4AI tests
test_from_container.sh: Test Crawl4AI from inside Docker

Utility Scripts

view_result.sh: Display crawl results
find_empty_folders.sh: Identify empty directories
analyze_empty_folders.sh: Assess risks of empty folders
verify_reorganization.sh: Check code reorganization success

Scripts are organized in directories:

Root: common scripts
scripts/general/
scripts/docker/
scripts/mcp/
scripts/test/

🌍 Built for Developers, by Developers

DevDocs is your documentation companion that:

Saves time: Weeks of research condensed to hours
Improves understanding: Clean, organized docs
Enables innovation: Build faster with any tech
Supports teams: Share knowledge efficiently
LLM READY: Integrates seamlessly with LLMs and Claude App

🛠️ Setting Up the Cline/Roo Cline for Rapid Software Development

Open Modes Interface
In Roo Code, create a new Mode-Specific Prompt.
Name
Give your mode a name, e.g., Research_MCP.

Role Definition Prompt

Expertise and Personality: Expertise in developer documentation retrieval, synthesis, and search. Personality is systematic, detail-oriented. Provide well-structured answers with clear documentation references.

Behavioral Mandate: Always use Table Of Contents and Section Access tools for MCP documentation queries. Maintain clarity, accuracy, traceability.

Mode-Specific Custom Instructions Prompt

1. Table Of Contents Tool: Lists documentation topics.
2. Section Access Tool: Retrieves detailed doc sections.

Process: Parse queries, identify relevant docs, drill down, synthesize content, and respond citing doc references.

Enforcement: Always use Table Of Contents first, then Section Access.

Final answers should be clear, comprehensive with documentation pointers.

🤝 Join Our Community

🌟 Star us on GitHub
👋🏽 Connect with our founder on LinkedIn
💬 Join our Discord community

Links available on our website and main repo.

🏆 Success Stories

"DevDocs turned our 3-week implementation timeline into 2 days. It's not just a crawler, it's a development accelerator."
— Senior Engineer at Fortune 100 Company

"Launched my SaaS in half the time by using DevDocs to understand and implement new technologies quickly."
— Successful Indie Hacker

Made with ❤️ by CyberAGI Inc in 🇺🇸

Make Software Development Better Again