A performant and modular runtime for TensorFlow, TFRT (TensorFlow Runtime) aims to provide a unified, extensible infrastructure layer for diverse hardware, focusing on efficient multithreaded CPU utilization and asynchronous programming models. It targets researchers experimenting with new models, developers seeking production serving performance, and hardware makers looking to integrate their devices into TensorFlow. TFRT offers low-level efficiency and aims for best-in-class performance across various domains.

How It Works

TFRT leverages MLIR (Multi-Level Intermediate Representation) for optimizing and compiling TensorFlow graphs into a Binary Executable Format (BEF). This BEF is then executed by the bef_executor. The runtime is designed for fully asynchronous programming and efficient use of host CPUs, providing a low-level, efficient execution layer. Its modular design allows for extensibility, particularly for integrating custom hardware backends.

Quick Start & Requirements

• Primary install/run command: Build using Bazel (version 4.0.0 verified). Example commands: bazel build //tools:bef_executor, bazel build //tools:tfrt_translate. Execution: bazel-bin/tools/tfrt_translate -mlir-to-bef path/to/program.mlir | bazel-bin/tools/bef_executor.
• Prerequisites: Ubuntu-16.04 (future support for MacOS, Windows), Bazel 4.0.0, clang 11.1, libstdc++8+. For GPU backend: NVIDIA CUDA Toolkit v11.2, cuDNN v8.1 (for CUDA 11.2), pip install libclang.
• Links: TensorFlow Dev Summit 2020 presentation, MLIR Open Design Deep Dive presentation.

Highlighted Details

• Performance benchmarks available for small-batch GPU inference on ResNet 50.
• Extensible architecture designed to support custom hardware integration.
• Core components include MLIR-based graph compilation and BEF execution.
• Focus on asynchronous programming and low-level efficiency.

Maintenance & Community

TFRT is currently not open to contributions; workflows for accepting contributions are under development. General discussions can occur via a mailing list, and bugs/feature requests are tracked on GitHub issues.

Licensing & Compatibility

Licensed under the Apache License 2.0, which is generally permissive for commercial use and integration into closed-source projects.

Limitations & Caveats

TFRT is an early-stage project and not yet ready for general use. Current platform support is limited to Ubuntu-16.04, with other platforms planned for future releases. The project is not yet open for external contributions.