This repository provides the evaluation benchmark suite for deep semi-supervised learning (SSL) algorithms, as detailed in the paper "Realistic Evaluation of Deep Semi-Supervised Learning Algorithms." It is intended for researchers and practitioners in machine learning who need to rigorously evaluate and compare SSL methods.

How It Works

The suite automates the download, preprocessing, and splitting of datasets (CIFAR-10, SVHN, ImageNet 32x32) into labeled and unlabeled subsets using configurable label maps. It then facilitates running and evaluating various SSL algorithms, including VAT, by leveraging tmuxp for managing training and evaluation processes, ensuring a structured and reproducible experimental setup.

Quick Start & Requirements

• Install dependencies: pip3 install -r requirements.txt
• Datasets: CIFAR-10 and SVHN are automatically downloaded. ImageNet 32x32 requires manual download and conversion.
• Experiment execution: Use tmuxp load <.yml file> or directly run provided Python scripts.
• Requires TensorFlow and tmuxp.
• Links: Paper

Highlighted Details

• Provides scripts for dataset preparation and label map generation with reproducible random seeds.
• Includes .yml configuration files for running experiments as described in the paper.
• Offers functionality to simulate small validation sets for robust evaluation.
• Supports multiple SSL methods and dataset configurations.

Maintenance & Community

This project is an open-source release associated with a specific research paper. No active community channels or ongoing maintenance efforts are indicated.

Licensing & Compatibility

The repository does not explicitly state a license. It is presented as an open-source release for research purposes. Commercial use or linking with closed-source projects may require clarification.

Limitations & Caveats

Exact reproducibility of paper results may be affected by subtle differences in TensorFlow versions and random seeds. The project is not an official Google product.