Low-latency load balancing for inference endpoints
AI inference workloads benefit from Jetpath's low-latency kernel-level processing:
Keep long-running inference sessions on the same backend with source IP or cookie-based persistence.
Distribute traffic across backends with different capacities using weighted round-robin.
Automatically route around unhealthy inference servers with configurable health checks.
Configure Jetpath for an inference cluster:
{
"name": "inference-cluster",
"protocol": "tcp",
"listeners": [{"vip": "10.0.0.100", "port": 8000}],
"lb_policy": {"type": "weighted_round_robin"},
"persistence": {
"enabled": true,
"type": "source",
"timeout_sec": 7200
},
"destinations": [
{"id": "gpu-1", "ip": "10.0.1.10", "port": 8000, "weight": 4},
{"id": "gpu-2", "ip": "10.0.1.11", "port": 8000, "weight": 4},
{"id": "gpu-3", "ip": "10.0.1.12", "port": 8000, "weight": 2}
]
}| Algorithm | Use Case |
|---|---|
| Round Robin | Equal distribution across identical backends |
| Weighted Round Robin | Distribute based on backend capacity |
| Least Connection | Route to backend with fewest active connections |
| Source Persistence | Keep client sessions on same backend |