Markdowner is a free, self-hostable tool designed to convert any website into LLM-ready markdown data, addressing limitations in existing proprietary or overly restrictive solutions. It targets AI developers and researchers seeking high-quality, structured data for improved LLM performance, offering a cost-effective and flexible alternative.

How It Works

Markdowner leverages Cloudflare's Workers and Durable Objects to manage browser instances for rendering web content. It then utilizes the Turndown library to convert the rendered HTML into markdown. This approach allows for efficient, scalable, and serverless execution, enabling the processing of diverse web content.

Quick Start & Requirements

• API Usage: curl 'https://md.dhr.wtf/?url=https://example.com'
• Self-Hosting: Requires cloning the repository, running npm i, creating a Cloudflare Workers KV namespace (npx wrangler kv:namespace create md_cache), updating wrangler.toml, and deploying with npm run deploy.
• Prerequisites: Cloudflare Workers paid plan is required for browser rendering and Durable Objects.
• Documentation: https://git.new/memory

Highlighted Details

• Converts any website to markdown.
• Features LLM filtering for data refinement.
• Supports an auto-crawler for subpages (up to 10).
• Offers both text and JSON response formats.

Maintenance & Community

The project is developed by Supermemory.ai. Support is encouraged via GitHub starring.

Licensing & Compatibility

The repository is licensed under the MIT License, permitting commercial use and integration with closed-source projects.

Limitations & Caveats

Self-hosting requires a paid Cloudflare Workers plan. The auto-crawler is limited to 10 subpages. The llmFilter functionality is experimental.