Autoscaling inference capacity

Review inference scaling and warm-up behavior.

Inference scaling

Increasing or decreasing serving capacity in response to prediction or generation demand.

Concurrency

The number of requests being processed at the same time.

Warm-up

The preparation time before new serving capacity can handle requests effectively.

Capacity limit

A configured ceiling that prevents scaling from creating uncontrolled resource use or cost.