This repository provides code and data for the paper "LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities." It evaluates Large Language Models (LLMs) on KG construction and reasoning tasks, introduces a virtual knowledge extraction framework, and proposes an agent-based system (AutoKG) for automated KG development. The target audience includes researchers and practitioners in knowledge graphs and LLM applications.

How It Works

The project is structured into three main components: Basic Evaluation, Virtual Knowledge Extraction, and Automatic KG. Basic Evaluation assesses LLMs (text-davinci-003, ChatGPT, GPT-4) against state-of-the-art supervised models on KG construction datasets like DuIE2.0 and SciERC. Virtual Knowledge Extraction uses the VINE dataset to probe LLMs' ability to extract knowledge not explicitly present in text. AutoKG leverages a multi-agent system, inspired by CAMEL, for automated KG construction and reasoning, integrating with tools like LangChain and requiring API keys for OpenAI and SerpApi.

Quick Start & Requirements

• KG Construction: Navigate to KG Construction, run *_processor.py and *_prompts.py for each dataset (e.g., DuIE2.0, MAVEN).
• KG Reasoning: Datasets and prompts are provided for link prediction and question answering.
• Virtual Knowledge Extraction: Navigate to Virtual Knowledge Extraction, run VINE_processor.py and VINE_prompts.py.
• AutoKG: Navigate to AutoKG, set OPENAI_API_KEY in Autokg.py and SERPAPI_API_KEY in RE_CAMEL.py. Run python Autokg.py.
• Dependencies: Python, OpenAI API, SerpApi API. Specific dataset preprocessing scripts are included.

Highlighted Details

• Evaluates LLMs against supervised benchmarks for KG construction.
• Introduces the VINE dataset for virtual knowledge extraction.
• AutoKG utilizes a multi-agent system for automated KG construction and reasoning.
• Supports multiple KG construction datasets (DuIE2.0, SciERC, RE-TACRED, MAVEN) and reasoning benchmarks (FB15k-237, ATOMIC2020, FreebaseQA, MetaQA).

Maintenance & Community

The project is associated with the paper "LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities" (arXiv:2305.13168). No specific community channels or active maintenance indicators are provided in the README.

Licensing & Compatibility

The repository's license is not explicitly stated in the README. The provided citation is for an arXiv preprint, suggesting research-oriented use. Compatibility for commercial or closed-source applications would require clarification of the license.

Limitations & Caveats

The AutoKG component requires API keys for OpenAI and SerpApi, incurring potential costs. The project is tied to a specific research paper, and ongoing maintenance or support is not detailed. The "virtual knowledge extraction" concept and its evaluation methodology may require further understanding.