TensorZero Gateway

TensorZero Gateway is an AI inference optimization platform that provides model routing, observability, and experimentation capabilities for AI applications.

Features

• Model Routing: Intelligently route requests to different AI models
• Observability: Built-in metrics and monitoring with ClickHouse integration
• Experimentation: A/B testing and variant management for AI models
• Multi-Provider Support: Works with OpenAI, Anthropic, AWS Bedrock, and many other providers

Configuration

The gateway uses a TOML configuration file that defines:

• Model configurations and routing rules
• Function definitions for different AI tasks
• Metrics collection and optimization settings

Required Setup

1. API Keys: Configure API keys for the AI providers you want to use
2. Configuration Files: The service includes a sample configuration with GPT-4o-mini setup. Configure it in Config Editor.

Health Check

The service includes a built-in health check endpoint at /status that monitors the gateway's operational status.

Documentation

For more information, visit the TensorZero documentation.