MinerU-HTML is an advanced HTML main content extraction tool leveraging Small Language Models (SLMs) to accurately parse complex web pages. It is designed for applications requiring clean, structured content, such as Deep Research Agents, Retrieval-Augmented Generation (RAG) systems, and training data generation pipelines, offering a significant improvement over traditional methods by intelligently filtering out noise like ads and navigation.

How It Works

MinerU-HTML employs a multi-stage pipeline powered by SLMs. It begins with HTML simplification, followed by prompt construction to guide an LLM in classifying page elements as either "main" content or "other." The LLM's classification results are then parsed, and the identified main content is extracted from the original HTML. Finally, the extracted content can be converted into various formats like Markdown, JSON, or plain text using the integrated MinerU-Webkit. This LLM-centric approach allows for nuanced understanding and extraction of semantically relevant content.

Quick Start & Requirements

• Installation: Recommended: pip install mineru_html[vllm] for VLLM backend. Alternatives: pip install mineru_html (for Transformers or OpenAI API backends). Install from source via git clone https://github.com/opendatalab/MinerU-HTML and pip install .[vllm].
• Prerequisites: The VLLM backend requires a GPU for optimal performance. Transformers backend supports local inference. OpenAI API backend requires compatible API endpoints. Models can be downloaded manually using huggingface-cli download opendatalab/MinerU-HTML-v1.1-hunyuan0.5B-compact.
• Links: Online demo available at Mineru-Extractor.

Highlighted Details

• Achieves a ROUGE-N.f1 score of 0.9001 on the WebMainBench v1.1 benchmark, competitive with leading LLMs.
• Supports extensible output formats including Markdown, JSON, and Txt via MinerU-Webkit.
• Offers multiple inference backend options: VLLM (high performance, GPU-accelerated), Transformers (local, flexible), and OpenAI API (remote).
• Features a fault tolerance mechanism with fallbacks to trafilatura, bypass, or empty results.
• Utilizes a compact model format for faster inference and regex structured output for vLLM v1 compatibility.

Maintenance & Community

The project has seen recent updates, including the release of MinerU-HTML v1.1 and the AICC dataset. Specific community channels (like Discord/Slack) or detailed contributor information are not explicitly detailed in the provided README.

Licensing & Compatibility

This project is licensed under the Apache License, Version 2.0, which is permissive and generally compatible with commercial use and closed-source applications.

Limitations & Caveats

Optimal performance, particularly with the VLLM backend, is heavily reliant on GPU availability. While fallback mechanisms are in place, highly complex or malformed HTML structures may still present extraction challenges.