This library parses complex Wikipedia wikitext into structured data, offering developers a robust solution for extracting information from MediaWiki sites. It benefits users by simplifying access to Wikipedia's vast knowledge base, overcoming the inherent difficulties of wikitext parsing.

How It Works

The core approach directly parses wikitext, meticulously handling its intricate syntax, recursive templates, redirects, and obscure shorthands. It prioritizes generating machine-readable formats like plaintext and JSON, rather than relying on intermediate HTML. This method is advantageous for its comprehensive support of wikitext's quirks, aiming for accuracy where simpler parsers might fail.

Quick Start & Requirements

• Install: npm install wtf_wikipedia
• Prerequisites: Node.js environment. For full dumps, npm install -g dumpster-dive.
• Resource Footprint: Parsing a full English Wikipedia takes approximately 4 hours on a MacBook (around 100 pages/second/thread).

Highlighted Details

• Parses redirects, disambiguation pages, and handles recursive templates/links.
• Resolves dynamic templates (e.g., {{CURRENTMONTH}}, {{CONVERT}}) and converts DMS geo-coordinates to lat/lon.
• Cleans output by eliminating XML, LaTeX, CSS, and table-sorting cruft.
• Supports fetching data from any MediaWiki API, including third-party wikis.
• Extensible via plugins for enhanced functionality (e.g., wtf-plugin-api, wtf-plugin-classify).

Maintenance & Community

The project lists "many contributors" and is licensed under MIT, suggesting an open-source community model. Specific community links or roadmap details were not present in the provided README.

Licensing & Compatibility

• License: MIT.
• Compatibility: The MIT license is permissive, allowing for commercial use and integration into closed-source projects without significant restrictions.

Limitations & Caveats

Wikitext's inherent complexity presents challenges, including partial inline-CSS support, scope issues, and deep recursion. The library does not support external 'transcluded' page data, AST output, smart HTML formatting, per-sentence references, or template/infobox CSS styling. It is not the fastest parser available and may require specific API configurations for third-party wikis.