Inference Platform
For Post-Trained LLMs

Inference PlatformFor Post-Trained LLMs

Your models, running at scale inside your AWS.
Never break, continuously optimized by R1,
the most token-efficient inference orchestration engine.

Cost, $ / hr · Lower is better

R1llm-dSageMaker HyperPodSageMaker Real-Time Endpoint

$0$50$100$150$200$250

State-of-the-Art Token Efficiency

R1 continiusly learns form live traffic and optimizes inference for speed and cost.

Define requirements, not your infrastructure

R1 continiusly learns form live traffic and optimizes inference for speed and cost.

Define requirements, not your infrastructure

Bring your model weights from Hugging Face, S3, or your local store. Set your latency, throughput, and cost targets. Use the defaults, or customize context length and quantization.

40%

higher token efifciency vs llm-d

70%

higher token efifciency vs SageMaker AI

One platform
to manage inference

Deployments

Model storage

Performance

Costs

Access

Dev tools

Deploy any vLLM-compatible model as dedicated OpenAI API endpoint. Manage all deployments at one place.

Rivvr

Overview

Inference

Model Storage

Monitoring

Project Settings

production

ClustersModelsTesting Runs

Runtime

Back to Models

qwen-9b-v3

Ready

Model is ready to serve traffic and accept deployment updates.

Edit DeploymentDelete Model

Runtime Health

Serving, validation, and optimization status.

View monitoringCluster details

Serving

Ready

Ready to serve traffic

Last updated 2 days ago

Benchmark

Passed

Passed validation

Last updated yesterday

View run

Optimization

Enabled

Automatic tuning enabled

Uses benchmark results to tune deployment settings.

Deployment Policy

Serving constraints and optimization limits for this deployment.

Guardrails

SLO mode

Balanced>95% compliance

Cost guardrail

Up to $30.00 / hr

Scale to zero

Off

Request profile

Token profile

11,700 prefill · 300 decode

Max length

12,000 tokens

Performance targets

QPS

Not set

TTFT target

p50 1,100 ms · p90 1,800 ms

Throughput target

p50 50 · p90 50

Model Runtime

Model-specific runtime settings.

Open model artifactOpen LoRA artifact

PrecisionAuto

KV cacheFp8

QuantizationFp8

LoRA1 attached

Model artifact

qwen35-9b-fp8

s3://rivvr-east-project-production-4b59a741/storage/model/hf-69f6bf7d-2acd-433e-a357-6cddf7678577-a1356633-1556-4de7-8d9e-7ee32686e9b7/model_weights/

Verified

LoRA artifact

support-adaptor-v1-1-4support-adaptor-v1-1-4

s3://rivvr-east-project-production-4b59a741/storage/lora/hf-0b7e2f4c-91d3-4a6e-b8f2-5c1d7a9e3f60/

Verified

API Access

Endpoint and quick-start access for this deployment.

Base URL

https://production-cluster.inference.rivvr.ai

PythoncurlAsync

from openai import OpenAI

client = OpenAI(
    api_key="RIVVR_PROJECT_API_KEY",
    base_url="https://production-cluster.inference.rivvr.ai"
)

response = client.chat.completions.create(
    model="support-adaptor-v1-1-4",
    messages=[{"role": "user", "content": "Hello"}]
)

print(response.choices[0].message.content)

Copy

LoRA deployment: use an adapter name as the model value (support-adaptor-v1-1-4). Authentication uses a Rivvr project API key generated in Project Settings.API Keys

R1 Orchestrator:
continuously optimizing
for speed and cost

TTFT p50

≤ 300 ms

TTFT p90

≤ 800 ms

Output

≥ 40 tok/s

Cost

≤ $120/hr

01 · Routing

Routing, designed for LLM applications

Routes requests by cache locality and priority, probes replica health, and manages circuit breaking.

02 · KV Cache

Distributed, tiered KV cache

Coordinates cache across GPU memory, RAM, SSD, and persistent storage, preserving cached state as nodes enter and leave.

03 · Engine

vLLM auto-tuning

Profiles traffic patterns and continuously tunes vLLM to maximize throughput within your SLOs.

04 · GPU

Heterogeneous GPU bin packing

Places models across mixed GPU types and shares compute between them based on demand and workload type (batched or real-time).

05 · Scaling & recovery

Sub-30-second cold starts and auto-recovery

Fast scaling and model swapping, self-healing dead pods within 1 minute.

06 · Cluster

Demand forecasting and SLO control

Forecasts traffic and adjusts serving capacity using latency, throughput, and cost feedback.

07 · AWS Cloud

Capacity discovery and auto-reservation

Monitors regional availability across compatible VMs and automatically reserves AWS capacity.

Continius

improvement

Instant, clean, ready-to-share summaries delivered seconds after your meeting ends.

70%

Summaries that save hours

Instant, clean, ready-to-share summaries delivered seconds after your meeting ends.

70%

Summaries that save hours

Instant, clean, ready-to-share summaries delivered seconds after your meeting ends.

Runs in your cloud.
Stays in your cloud.

VPC deployment

Run inference inside your AWS account and VPC, with network controls you manage.

VPC deployment

Run inference inside your AWS account and VPC, with network controls you manage.

Data privacy

Your prompts, responses, logs, and model weights stay inside your AWS account.

Data privacy

Your prompts, responses, logs, and model weights stay inside your AWS account.

RBAC and access control

Control who can deploy models, change policies, and view cost and performance data.

RBAC and access control

Control who can deploy models, change policies, and view cost and performance data.

Flexible tenancy isolation

Choose dedicated infrastructure for strict isolation or shared GPU capacity for higher utilization.

Flexible tenancy isolation

Choose dedicated infrastructure for strict isolation or shared GPU capacity for higher utilization.

Built by AI infrastructure experts

Built by a team that shipped AI infrastructure and production software for research institutions, hospitals, big pharma companies, insurers, and other Fortune 500 enterprises.

Define requirements, not your infrastructure

Built by a team that shipped AI infrastructure and production software for research institutions, hospitals, big pharma companies, insurers, and other Fortune 500 enterprises.

Define requirements, not your infrastructure

Bring your model weights from Hugging Face, S3, or your local store. Set your latency, throughput, and cost targets. Use the defaults, or customize context length and quantization.

Frequently Asked Questions

How is this different from SageMaker?

SageMaker gives you infrastructure and autoscaling controls. Rivvr removes the need to operate them: set latency and cost targets, and Rivvr continuously runs the infrastructure for you.

How is this different from AIBrix or NVIDIA Dynamo?

AIBrix and Dynamo are infrastructure software your team deploys, configures and maintains. Rivvr is a managed platform inside your AWS account. Instead of configuring profiling, autoscaling and infrastructure parameters, you specify the outcome: SLO + cost target.

Isn't spot capacity risky for something with an SLO?

Spot capacity is interruptible. Rivvr spreads workloads across compatible GPU capacity pools and uses on-demand as fallback, allowing capacity to change without sacrificing the SLO.

What data leaves my AWS account?

Inside your AWS account, behind your VPC. Inference traffic, data, and model weights never leave your environment.

How is Rivvr priced?

You pay AWS directly for infrastructure. Rivvr charges a separate management fee based on GPUs under orchestration.

Do I need to change my application?

Usually not. Rivvr exposes an OpenAI-compatible API, so most teams only change the endpoint.

Does Rivvr modify my model?

No. Model weights remain unchanged. Rivvr optimizes orchestration, placement and infrastructure.

What models are supported?

Any LLM supported by vLLM, currently up to 400B parameters.

What GPUs are supported?

NVIDIA CUDA compute capability 7.0+, including L4, L40S, A10G, T4, A100, H100, H200, B200, B300 and V100.

Inference PlatformFor Post-Trained LLMs