google-search  by web-agent-master

Playwright tool for bypassing anti-scraping, executing Google searches

created 5 months ago
289 stars

Top 91.9% on sourcepulse

GitHubView on GitHub
1 Expert Loves This Project
Project Summary

This Node.js tool provides a local, free alternative to paid SERP APIs for executing Google searches and extracting results. It's designed for developers and AI assistants needing programmatic access to Google search data, offering advanced techniques to bypass anti-scraping measures.

How It Works

The tool leverages Playwright for browser automation, employing sophisticated methods to mimic real user behavior. This includes intelligent browser fingerprint management, automatic saving/restoring of browser state to minimize verification prompts, and smart switching between headless and headed modes. It also randomizes device and locale settings to reduce detection risk.

Quick Start & Requirements

  • Install dependencies: npm install (or yarn, pnpm)
  • Build: npm run build (or yarn build, pnpm build)
  • Link globally for MCP: npm link (or yarn link, pnpm link)
  • Requires Node.js (v16+ for MCP server). Playwright browsers are downloaded on first use.
  • Official Docs: https://github.com/web-agent-master/google-search

Highlighted Details

  • Acts as a local SERP API alternative, bypassing paid services.
  • Implements advanced anti-bot detection bypass techniques.
  • Integrates with AI assistants like Claude via the Model Context Protocol (MCP) server.
  • Can retrieve raw HTML of search result pages.

Maintenance & Community

The project is open source with no usage restrictions. Community links or specific maintainer information are not detailed in the README.

Licensing & Compatibility

The project is described as "Completely Open Source and Free" with "no usage restrictions." Specific license details (e.g., MIT, Apache) are not explicitly stated, implying permissive usage.

Limitations & Caveats

The tool is intended for learning and research; users must comply with Google's terms of service. Frequent requests may lead to blocking. Some regions might require a proxy. Administrator privileges may be needed on Windows for Playwright browser installation.

Health Check
Last commit

3 months ago

Responsiveness

Inactive

Pull Requests (30d)
0
Issues (30d)
3
Star History
67 stars in the last 90 days

Explore Similar Projects

Starred by Tobi Lutke Tobi Lutke(Cofounder of Shopify), Chip Huyen Chip Huyen(Author of AI Engineering, Designing Machine Learning Systems), and
7 more.

firecrawl by mendableai

1.9%
44k
API service for turning websites into LLM-ready data
created 1 year ago
updated 1 day ago
Feedback? Help us improve.