This toolkit addresses the identification and analysis of "slop"—over-represented lexical patterns—in Large Language Model (LLM) outputs. It enables researchers and developers to generate standardized LLM outputs, profile their repetitive word usage, create canonical slop lists, and cluster models based on linguistic similarity.

How It Works

The toolkit operates in four main stages: dataset generation, slop profiling, slop list creation, and phylogenetic tree building. Slop profiling involves counting word and phrase frequencies, filtering common words and numbers, and calculating repetition scores and vocabulary complexity. Slop lists are created by aggregating these profiles across models to identify consistently overused terms. Phylogenetic trees are then generated by treating models as species and slop term usage as genetic traits, using bioinformatics tools like PHYLIP for parsimony analysis or falling back to hierarchical clustering.

Quick Start & Requirements

• Install: pip install -r requirements.txt
• Prerequisites: Python 3.7+, NLTK data (punkt, punkt_tab, stopwords, cmudict), and optionally PHYLIP for phylogenetic analysis.
• Configuration: Copy .env.example to .env and set OPENAI_API_KEY and optionally PHYLIP_PATH and OPENAI_BASE_URL.
• Example Notebook: https://colab.research.google.com/drive/1SQfnHs4wh87yR8FZQpsCOBL5h5MMs8E6?usp=sharing

Highlighted Details

• Generates standardized LLM outputs for comparative analysis.
• Profiles include word, bigram, trigram usage, repetition scores, and vocabulary complexity.
• Creates canonical slop lists for single words, bigrams, trigrams, and multi-word phrases.
• Utilizes bioinformatics tools (PHYLIP) for clustering models based on output similarity.

Maintenance & Community

• Contact: Sam Paech or create an issue on GitHub.
• Citation: Provided in bibtex format.

Licensing & Compatibility

• License: MIT License.
• Compatibility: Permissive for commercial use and integration with closed-source projects.

Limitations & Caveats

The phylogenetic analysis relies on PHYLIP, which may require manual installation and configuration if not available via package managers. The effectiveness of slop analysis is dependent on the quality and diversity of the generated dataset prompts.