Queues

A queue is a named scheduling lane for your work. Instead of every run and action competing for whatever capacity happens to be free, a queue gives the platform a place to apply policy: how many things run at once, how deep the backlog can get, how this work is prioritized relative to everyone else’s, and which cluster it lands on.

Targeting a queue is a single parameter: you don’t change your task code, you just say where it should be scheduled. Anything you don’t explicitly route goes to the default queue.

Routing work to a queue

The queue parameter can be set at three levels: on the flyte.TaskEnvironment, on the @env.task decorator, and at invocation time via task.override(). The more specific level always wins, exactly like the other task settings.

        
    
import flyte

# Default every task in this environment to a queue...
env = flyte.TaskEnvironment(
    name="my_env",
    queue="default-pool",
)

# ...inherit the environment default...
@env.task
def preprocess(data: list) -> list:
    return [x * 2 for x in data]

# ...or override per task.
@env.task(queue="gpu-pool")
def train_model(data: list) -> dict:
    return {"accuracy": 0.95}

A common pattern is to leave the entry-point task on the default routing and pin only the child steps that need capping to a queue, so the run as a whole goes through the default pool and only the targeted workload is constrained.

Per-run and per-trigger routing

You can also choose a queue when you launch a run, without touching the task definition:

        
    
flyte.with_runcontext(queue="my-queue").run(main, count=10)

And a trigger can send its scheduled runs to a specific queue, useful when you want scheduled or automated work to run under different limits than ad-hoc runs:

        
    
import flyte

trigger = flyte.Trigger(
    "nightly",
    flyte.Cron("0 2 * * *"),
    queue="batch",
)

Overriding a queue at runtime

A running workflow can route a specific task invocation to a different queue, for example to push one heavy step onto a dedicated lane:

        
    
@env.task
async def main(queue_name: str):
    # Route this invocation of train_model to a queue chosen at runtime.
    train_model.override(queue=queue_name)(data=[1, 2, 3])

What a queue controls

Queues are configured by your platform admin (see Managing queues for how they are created and managed), but it helps to understand the knobs so you can pick the right queue for a workload:

Action concurrency: the maximum number of tasks routed to the queue that run at the same time. A queue with a cap of 1 serializes its work; a cap of 3 lets at most three run concurrently and holds the rest until a slot frees up.
Run concurrency: the maximum number of runs on the queue that are active at once. Children of an active run aren’t capped by this; only the runs themselves are. Use this for a job that parallelizes well internally but must not overlap with a previous invocation of itself.
Depth: the total number of in-flight plus waiting items the queue will hold. When a queue is full, new submissions are rejected immediately with a RESOURCE_EXHAUSTED error rather than queueing forever. That rejection is a back-pressure signal to your caller: catch it and slow down, rather than retrying in a tight loop.
Priority: when several queues share the same cluster capacity, higher priority work is scheduled ahead of lower priority work. Priority controls ordering, not preemption: a lower-priority task that has already started is not interrupted when higher-priority work arrives.
Resource caps: the most CPU, memory, and GPUs that the queue’s scheduled tasks may request at once, counted across every cluster the queue routes to. GPUs can be capped per device type, and separately for tasks that ask for a GPU without naming a type. A task that would push the queue past a cap waits until enough running work finishes.
Scheduling policy: what the queue does when the next task in line does not fit. A strict_fifo queue waits for it, so nothing behind it starts first. A greedy_capacity queue skips it and starts the tasks behind it that do fit.

Resource caps and scheduling policies are described in Resource caps and scheduling.

When to use queues

Concurrency control for scheduled and automated runs

A run that updates a shared checkpoint or mutates global state must not overlap with itself. Route it to a queue with run concurrency of 1: even if a schedule fires again before the previous run finishes, the new run waits instead of running concurrently. Inside each run, children still fan out freely.

        
    
@env.task(queue="runs-1")  # run-concurrency 1
async def nightly_job(fan_out: int = 50) -> list[str]:
    # Hundreds of children can run in parallel within this run...
    return list(flyte.map(child, range(fan_out)))
    # ...but two nightly_job runs never overlap.

This pairs naturally with triggers: give the trigger a serialized queue and you get self-non-overlapping scheduled jobs for free.

Backfill control

A backfill can produce thousands of actions in a burst and starve everything else sharing the cluster. Send the backfill to a queue with a bounded action concurrency (and optionally a depth limit) so it drains at a controlled rate while leaving headroom for production traffic. The depth limit turns a flood into explicit back-pressure: when the queue is full your submitter gets RESOURCE_EXHAUSTED and can pause instead of piling on.

Multi-cluster routing and prioritizing certain workloads

Because a queue can be bound to a specific cluster (or set of clusters) and given a priority, queues are how you say “GPU training goes to the GPU cluster” or “this customer-facing pipeline is high priority.” Route latency-sensitive or business-critical work to a high-priority queue and bulk or best-effort work to a lower-priority one; when they contend for the same capacity, the important work is scheduled first.

When several teams run on the same clusters, one team’s large job can take all of the GPUs and leave everyone else waiting. Give each team a queue with resource caps and each team gets a ceiling on the CPU, memory, and GPUs its scheduled work can use at once. Work beyond the ceiling is still accepted and waits in the queue. Because the caps count resources, a team can run many small tasks or a few large ones under the same cap.

Which queue you pick also decides how your work waits. On a greedy_capacity queue, small tasks and tasks for a reusable environment that is already running keep starting while a large task waits for room. On a strict_fifo queue, tasks start strictly in the order they were submitted, which suits a job that needs many GPUs at the same moment and must not be overtaken by smaller work.

Requests that can never run

A task can ask for more than its queue allows, for example 64 CPUs on a queue capped at 32. Waiting would not help, so the action fails right away with the error code INFEASIBLE and is not retried. The error message names the request and the cap. Lower the task’s resources, or route it to a queue with a higher cap.

The same check is being extended to the clusters behind a queue: a task that asks for an accelerator none of the queue’s clusters has, or for a container larger than any of their nodes, will fail the same way. See Infeasible requests fail fast.

Queues and timeouts

How long a task is willing to wait in a queue before it gives up is a separate, per-task concern controlled by the timeout settings. If a queue is busy or capped, an action sits in the Queued phase until a slot opens: set max_queued_time to fail fast when capacity isn’t available within your window, and deadline to put an absolute ceiling on total wall-clock including queue wait. See Retries and timeouts for the full picture.

When max_queued_time fires, the parent task receives flyte.errors.MaxQueuedTimeExceededError. Catching it and re-running the task with override(queue=..., resources=...) lets a workflow fall back to another queue that has a different GPU available; see Routing each option to a different queue.