GPU types

Flash provides access to a wide range of NVIDIA GPUs through both pool-based and specific GPU selection. This page lists all available GPU types and explains how to use them.

GPU selection methods

Flash offers two ways to specify GPU hardware:

GPU pools (GpuGroup): Select from predefined pools of similar GPUs grouped by architecture and VRAM.
Specific GPU types (GpuType): Target exact GPU models when you need precise hardware characteristics.

You can use either method or mix both for advanced fallback strategies.

GPU pools

The GpuGroup enum provides access to GPU pools. Each pool contains specific GPU models grouped by architecture and VRAM capacity.

Available GPU pools

GpuGroup	GPUs Included	VRAM	Best For
`GpuGroup.ANY`	Any available GPU	Varies	Fast provisioning, prototyping
`GpuGroup.AMPERE_16`	RTX A4000, RTX 4000 Ada, RTX 2000 Ada	16GB	Small models, basic inference
`GpuGroup.AMPERE_24`	RTX A4500, RTX A5000, RTX 3090	20-24GB	General ML, mid-size models
`GpuGroup.ADA_24`	L4, RTX 4090	24GB	Cost-effective inference
`GpuGroup.ADA_32_PRO`	RTX 5090	32GB	Latest consumer flagship
`GpuGroup.AMPERE_48`	A40, RTX A6000	48GB	Large models, fine-tuning
`GpuGroup.ADA_48_PRO`	L40S, L40, RTX 6000 Ada	48GB	Professional inference
`GpuGroup.AMPERE_80`	A100 80GB PCIe, A100-SXM4-80GB	80GB	XL models, intensive training
`GpuGroup.ADA_80_PRO`	H100 80GB HBM3	80GB	Cutting-edge inference
`GpuGroup.BLACKWELL_96`	RTX PRO 6000 Blackwell (Server, Workstation, Max-Q)	96GB	Professional Blackwell workloads
`GpuGroup.HOPPER_141`	H200	141GB	Largest models, maximum VRAM
`GpuGroup.BLACKWELL_180`	B200	180GB	Maximum VRAM, next-gen training

Using GPU pools

from runpod_flash import Endpoint, GpuGroup

# Single GPU pool
@Endpoint(name="inference", gpu=GpuGroup.AMPERE_80)
async def infer(data: dict) -> dict:
    ...

# Multiple pools for fallback
@Endpoint(
    name="flexible",
    gpu=[GpuGroup.AMPERE_80, GpuGroup.AMPERE_48, GpuGroup.ADA_24]
)
async def flexible_infer(data: dict) -> dict:
    ...

# Any available GPU (fastest provisioning)
@Endpoint(name="development", gpu=GpuGroup.ANY)
async def dev_infer(data: dict) -> dict:
    ...

Specific GPU types

The GpuType enum provides access to specific GPU models. Use these when you need exact hardware characteristics. A single GpuType requests that exact card — this is the only form guaranteed to pin workers to a specific GPU model. If you pass a list of GpuType values, placement is advisory at the pool/VRAM-tier level instead. See GPU selection behavior for details.

Available GPU types

GpuType	GPU Model	VRAM	Architecture
`GpuType.ANY`	Any available GPU	Varies	Any
`GpuType.NVIDIA_RTX_A4000`	NVIDIA RTX A4000	16GB	Ampere
`GpuType.NVIDIA_RTX_A4500`	NVIDIA RTX A4500	20GB	Ampere
`GpuType.NVIDIA_RTX_4000_ADA_GENERATION`	NVIDIA RTX 4000 Ada	16GB	Ada Lovelace
`GpuType.NVIDIA_RTX_2000_ADA_GENERATION`	NVIDIA RTX 2000 Ada	16GB	Ada Lovelace
`GpuType.NVIDIA_RTX_A5000`	NVIDIA RTX A5000	24GB	Ampere
`GpuType.NVIDIA_L4`	NVIDIA L4	24GB	Ada Lovelace
`GpuType.NVIDIA_GEFORCE_RTX_3090`	NVIDIA GeForce RTX 3090	24GB	Ampere
`GpuType.NVIDIA_GEFORCE_RTX_4090`	NVIDIA GeForce RTX 4090	24GB	Ada Lovelace
`GpuType.NVIDIA_GEFORCE_RTX_5090`	NVIDIA GeForce RTX 5090	32GB	Blackwell
`GpuType.NVIDIA_A40`	NVIDIA A40	48GB	Ampere
`GpuType.NVIDIA_RTX_A6000`	NVIDIA RTX A6000	48GB	Ampere
`GpuType.NVIDIA_RTX_6000_ADA_GENERATION`	NVIDIA RTX 6000 Ada	48GB	Ada Lovelace
`GpuType.NVIDIA_A100_80GB_PCIe`	NVIDIA A100 80GB PCIe	80GB	Ampere
`GpuType.NVIDIA_A100_SXM4_80GB`	NVIDIA A100-SXM4-80GB	80GB	Ampere
`GpuType.NVIDIA_H100_80GB_HBM3`	NVIDIA H100 80GB HBM3	80GB	Hopper
`GpuType.NVIDIA_RTX_PRO_6000_BLACKWELL_SERVER_EDITION`	NVIDIA RTX PRO 6000 Blackwell Server Edition	96GB	Blackwell
`GpuType.NVIDIA_RTX_PRO_6000_BLACKWELL_WORKSTATION_EDITION`	NVIDIA RTX PRO 6000 Blackwell Workstation Edition	96GB	Blackwell
`GpuType.NVIDIA_RTX_PRO_6000_BLACKWELL_MAX_Q_WORKSTATION_EDITION`	NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition	96GB	Blackwell
`GpuType.NVIDIA_H200`	NVIDIA H200	141GB	Hopper
`GpuType.NVIDIA_B200`	NVIDIA B200	180GB	Blackwell

Using specific GPU types

from runpod_flash import Endpoint, GpuType

# Single specific GPU (pins the exact card)
@Endpoint(name="inference", gpu=GpuType.NVIDIA_A100_80GB_PCIe)
async def infer(data: dict) -> dict:
    ...

# Multiple specific GPUs (requests any card in this set)
@Endpoint(
    name="flexible",
    gpu=[
        GpuType.NVIDIA_A100_80GB_PCIe,
        GpuType.NVIDIA_A100_SXM4_80GB,
        GpuType.NVIDIA_A40
    ]
)
async def flexible_infer(data: dict) -> dict:
    ...

When you pass a list, the Flash SDK converts each GpuType to its GPU pool (GpuGroup) when it creates the endpoint, excluding any pool members you didn’t list. Placement within the selected pools is advisory: workers may be scheduled on a different card with equivalent VRAM rather than one of the exact models you listed, and list order is not preserved. Pass a single GpuType when your workload requires an exact GPU model.

Advanced fallback strategies

Combine GpuGroup and GpuType for robust availability:

from runpod_flash import Endpoint, GpuGroup, GpuType

@Endpoint(
    name="hybrid-selection",
    gpu=[
        GpuType.NVIDIA_A100_80GB_PCIe,  # Specific GPU model
        GpuGroup.AMPERE_48,             # Any card in the 48GB Ampere pool
        GpuGroup.ANY                    # Any GPU: expands to all pools for maximum availability
    ]
)
async def infer(data: dict) -> dict:
    ...

GPU selection behavior

Single GPU type: Passing a single GpuType requests that exact GPU model. This is the only form guaranteed to pin workers to a specific card. Flash waits for that model to become available, and jobs stay in queue until capacity is free.

gpu=GpuType.NVIDIA_GEFORCE_RTX_4090  # Only the RTX 4090

Single GPU pool: Passing a single GpuGroup requests any card in that pool.

gpu=GpuGroup.AMPERE_80  # Any A100 80GB card

A list of GPU types or pools: A list requests any one of the listed options. It is not an ordered preference: the SDK does not preserve list order, and placement within the selected pools is advisory — workers may be scheduled on a different card with equivalent VRAM rather than one of the exact models you listed. Be aware of this when benchmarking or when your workload depends on a specific GPU model.

gpu=[GpuGroup.AMPERE_80, GpuGroup.AMPERE_48, GpuGroup.ADA_24]
# Any card from the 80GB, 48GB, or 24GB pools

GpuGroup.ANY: Flash selects the first available GPU based on current capacity.

gpu=GpuGroup.ANY  # Fastest provisioning, unpredictable GPU type

There is currently no API field that reports which GPU model a worker was actually placed on. The endpoint configuration reflects what you requested, not where workers ran, so GPU substitution is invisible in the API. To verify placement, query the GPU from inside the worker (for example, with nvidia-smi).

For production: Pass a single GpuType when you need a specific card for predictable cost and performance. For development: Use GpuGroup.ANY for fastest iteration.

Multi-GPU workers

Request multiple GPUs per worker using gpu_count:

@Endpoint(
    name="multi-gpu-training",
    gpu=GpuGroup.AMPERE_80,
    gpu_count=4,  # Each worker gets 4 GPUs
    workers=2     # Maximum 2 workers = 8 GPUs total
)
async def train(data: dict) -> dict:
    ...

Handling unavailability

If requested GPUs are unavailable, jobs stay in queue:

Initial job status: IN_QUEUE
[Waiting for capacity...]

Solutions:

Add fallback options: Use multiple GPU types.

gpu=[GpuGroup.AMPERE_80, GpuGroup.AMPERE_48, GpuGroup.ADA_24]

Use broader selection: Switch to GpuGroup.ANY.
gpu=GpuGroup.ANY
Contact support: For capacity guarantees, contact Runpod support.

Get started

Flash

Serverless

Pods

Storage

Public Endpoints

Instant Clusters

Accounts and billing

Integrations

Hub

Reference

GPU selection methods

GPU pools

Available GPU pools

Using GPU pools

Specific GPU types

Available GPU types

Using specific GPU types

Advanced fallback strategies

GPU selection behavior

Multi-GPU workers

Handling unavailability

​GPU selection methods

​GPU pools

​Available GPU pools

​Using GPU pools

​Specific GPU types

​Available GPU types

​Using specific GPU types

​Advanced fallback strategies

​GPU selection behavior

​Multi-GPU workers

​Handling unavailability

GPU selection methods

GPU pools

Available GPU pools

Using GPU pools

Specific GPU types

Available GPU types

Using specific GPU types

Advanced fallback strategies

GPU selection behavior

Multi-GPU workers

Handling unavailability