MindSpider is an AI-driven crawler system designed for public opinion analysis. It automatically identifies trending topics across multiple social media and tech platforms using AI, then performs in-depth content crawling to gather sentiment feedback. This system benefits researchers and analysts by automating the discovery and collection of relevant online discourse.

How It Works

The system operates in two main phases: BroadTopicExtraction uses AI (defaulting to Deepseek) to identify hot topics and keywords from 13 sources, storing them in MySQL. Subsequently, DeepSentimentCrawling leverages these keywords to perform detailed crawling on 7 platforms via Playwright's browser automation, capturing posts, comments, and interactions. This modular approach allows for automated topic discovery and granular data collection.

Quick Start & Requirements

• Requirements: Python 3.9+, MySQL 5.7+, Conda, Playwright. Requires a Deepseek API key.
• Installation: Clone the repository (git clone https://github.com/666ghj/MindSpider.git), create and activate a Conda environment (e.g., pytorch_python11), install Python dependencies (pip install -r requirements.txt), and install Playwright browsers (playwright install).
• Configuration: Edit config.py for database credentials and the Deepseek API key.
• Setup: Initialize database tables with python main.py --setup.
• Running: Execute topic extraction (--broad-topic), deep crawling (--deep-sentiment), or the full