This project provides a real-time monitoring and intelligent analysis tool for Xianyu (a Chinese second-hand marketplace), leveraging Playwright for web scraping and advanced AI models for filtering and analysis. It targets users who want to automate the process of finding specific items on Xianyu, offering a comprehensive web UI for task management and AI-driven insights, thereby saving users time and effort in manual searching and filtering.

How It Works

The tool utilizes Playwright to simulate user interactions and scrape product information from Xianyu. It supports multi-task concurrency and real-time processing of newly discovered items. A key feature is its deep AI analysis, integrating multimodal large language models (like GPT-4o) to analyze product images, text descriptions, and seller profiles against user-defined criteria, enabling precise filtering and recommendations.

Quick Start & Requirements

• Installation: Clone the repository, install dependencies via pip install -r requirements.txt.
• Configuration: Create a .env file from .env.example and populate it with OPENAI_API_KEY, OPENAI_BASE_URL, and OPENAI_MODEL_NAME (must support image analysis). Other optional notification and proxy settings are available.
• Login: Obtain Xianyu login state via the Web UI's "Manual Update" feature or by running python login.py locally.
• Running: Start the web server with python web_server.py. Access the UI at http://127.0.0.1:8000.
• Docker: Use docker-compose up -d for deployment. Login state must be updated via the Web UI after containers start.
• Prerequisites: Python 3.x, Playwright browser binaries (installed via playwright install), and an OpenAI-compatible API key for a multimodal model.

Highlighted Details

• Natural language to AI-driven task creation.
• Visual Web UI for task management, AI prompt editing, and log viewing.
• Real-time notifications via ntfy.sh, WeChat Work, or Bark.
• Support for multiple AI models (OpenAI compatible) and proxy configurations.
• Robust anti-scraping measures with simulated user behavior.

Maintenance & Community

The project acknowledges contributions from superboyyy/xianyu_spider and others. Support is available through GitHub Issues.

Licensing & Compatibility

The project is available under the MIT License, permitting commercial use and linking with closed-source projects.

Limitations & Caveats

Users must adhere to Xianyu's terms of service and robots.txt. Frequent requests may lead to IP bans or account restrictions. The project is intended for learning and research purposes only. Some Windows users may encounter encoding issues or require manual installation of zbar for pyzbar.