CogAgent is an open-source, end-to-end Vision-Language Model (VLM) designed to act as a GUI agent. It enables automated interaction with graphical user interfaces through natural language commands and screen captures, targeting researchers and developers looking to build sophisticated automation tools.

How It Works

CogAgent is built upon the GLM-4V-9B VLM, enhanced through extensive data collection, multi-stage training, and strategic optimizations. This approach significantly improves its GUI perception, reasoning accuracy, action completeness, and task generalization. The model processes screen captures and natural language, outputting specific actions with bounding box coordinates for GUI element interaction.

Quick Start & Requirements

• Install: pip install -r requirements.txt
• Run Inference: python inference/cli_demo.py --model_dir THUDM/cogagent-9b-20241220 --platform "Mac" --max_length 4096 --top_k 1 --output_image_path ./results --format_key status_action_op_sensitive
• Prerequisites: Python 3.10.16+, NVIDIA GPU.
• VRAM: Minimum 29GB for BF16 inference. INT8 requires ~15GB, INT4 ~8GB (with performance loss).
• Fine-tuning: SFT requires ~60GB VRAM per GPU (8x A100), LoRA requires ~70GB VRAM on a single GPU.
• Demo: HuggingFace Space, ModelScope Space

Highlighted Details

• State-of-the-art performance on GUI Agent benchmarks like Screenspot, OmniAct, and CogAgentBench-basic-cn.
• Supports bilingual (Chinese/English) interaction.
• Offers multiple output formats (e.g., Action-Operation, Status-Plan-Action-Operation-Sensitive).
• Can be fine-tuned for custom tasks.

Maintenance & Community

• Developed by Tsinghua University and Zhipu AI.
• Latest model version: cogagent-9b-20241220 (December 2024).
• Official Technical Blog
• Practical Guide (Chinese)

Licensing & Compatibility

• Code: Apache 2.0 License.
• Model Weights: Follows a separate Model License.

Limitations & Caveats

• Limited platform support: Primarily tested on Windows and macOS; effectiveness on other systems may be suboptimal.
• Fine-tuning requires substantial GPU memory, and SFT requires freezing the Vision Encoder.
• Online demos do not support computer control, only inference viewing.