OpenAI

Use Promptfoo to compare OpenAI models, test prompts, and check your application's outputs. Start with the Responses API for new text and image-input evals, following OpenAI's recommendation. Use Chat Completions when that is the API your application calls.

Quickstart

Set OPENAI_API_KEY in your shell or secret manager. You can create a key in the OpenAI dashboard.
```
export OPENAI_API_KEY=your_api_key_here
```

Save this configuration as promptfooconfig.yaml:

promptfooconfig.yaml
prompts:
  - |-
    Classify this support ticket as billing or technical.
    Reply with only the label.
    Ticket: {{ticket}}

providers:
  - id: openai:responses:gpt-5.6-luna
    config:
      reasoning:
        effort: low
      max_output_tokens: 2048

tests:
  - vars:
      ticket: I was charged twice for my subscription.
    assert:
      - type: equals
        value: billing
  - vars:
      ticket: The app crashes when I try to sign in.
    assert:
      - type: equals
        value: technical

Run the eval from the directory containing the configuration:
```
npx promptfoo@latest eval --no-cache -o results.json
```

The expected outputs are billing and technical. Check the pass/fail results and any provider errors in results.json. To compare models, add another entry under providers.

If you keep your key in a local .env file, add --env-file .env to the command. Keep that file out of version control.

Models

Use an explicit endpoint in each provider ID. This makes the request format predictable, including for newly released models.

Task	Provider ID	Guide
Text, image inputs, and built-in tools	`openai:responses:<model>`	Responses API
Chat Completions	`openai:chat:<model>`	Parameters
Embeddings	`openai:embedding:<model>`	Embedding dimensions
Moderation	`openai:moderation:omni-moderation-latest`	Moderation assertions
Image generation	`openai:image:<model>`	Images
Audio input and output	`openai:chat:gpt-audio-1.5`	Audio
Text to speech	`openai:tts:gpt-4o-mini-tts`	Text to speech
Conversational Realtime	`openai:realtime:gpt-realtime-2.1`	Realtime
Full-duplex voice	`openai:live:gpt-live-1`	GPT-Live

For file transcription, see audio transcription. For Agents SDK, ChatKit, and Codex workflows, see agent providers.

Choose a model you can access, then test it with representative inputs. OpenAI's model catalog lists current availability, capabilities, and limits. The main text-model choices are:

Model	Starting point for
`gpt-5.6-luna`	Simple tasks and high-volume evals
`gpt-5.6-terra`	Balancing capability and cost
`gpt-5.6-sol`	Complex tasks
`gpt-6-astra`	The most demanding reasoning and coding tasks

Check OpenAI pricing before a large run. Model access and API billing belong to your OpenAI account.

Aliases, snapshots, and default models

Bare openai:<model> IDs default to Responses for GPT-5.6 and newer GPT models, including named variants and dated snapshots. For example, openai:gpt-5.6, openai:gpt-5.6-luna, and openai:gpt-6-astra all use Responses. Older recognized models keep their model-specific routing; other unknown names fall back to Chat Completions.

Use openai:chat:<model> or openai:responses:<model> to select the endpoint explicitly, including for a compatible gateway. Existing bare GPT-5.6 configurations with Chat-specific options should either select openai:chat:gpt-5.6 or switch to Responses options such as reasoning.effort and max_output_tokens.

Bare openai:chat and openai:responses select gpt-5.6-terra. Built-in grading uses gpt-5.6-sol; suggestions and web search use gpt-5.6-terra. Specify a model ID to override these defaults. When a model has dated snapshots, use one to hold the model version constant across runs. A fixed snapshot does not guarantee identical outputs.

openai:embedding and openai:embeddings default to text-embedding-3-large; both prefixes accept an explicit model. openai:speech: is an alias for openai:tts:.

GPT-5.6

The gpt-5.6 model alias selects Sol. Sol, Terra, and Luna support Chat Completions and Responses. In Responses, use reasoning.effort to set the reasoning budget and reasoning.mode: pro for Pro mode:

providers:
  - id: openai:responses:gpt-5.6-sol
    config:
      reasoning:
        effort: high
        mode: pro
      max_output_tokens: 8192

Accepted reasoning efforts vary by model. See the model catalog before changing them. Codex's ultra setting is not a Responses API reasoning effort.

GPT-6 Astra

Use openai:responses:gpt-6-astra for Astra evals with tools. Explicit openai:chat:gpt-6-astra supports text generation, but Astra tool calling requires Responses.

Astra accepts low, medium, high, xhigh, and max reasoning effort. It does not accept none or minimal. Promptfoo removes unsupported sampling and log-probability parameters for Astra. See the Astra model guide.

Fine-tuned models

Use the full fine-tuned model ID with its supported endpoint:

providers:
  - openai:chat:ft:gpt-4.1-mini-2025-04-14:company-name:ticket-classifier:MODEL_ID

Replace the example ID with your model's ID. Inference availability follows the base model's lifecycle; training access has separate restrictions. See OpenAI's fine-tuning lifecycle.

Chat messages

A plain text prompt becomes a user message. For system instructions, conversation history, or multimodal inputs, use a JSON message array in a prompt file. See chat threads.

Chat Completions and Responses use different multimodal content blocks. Use the formats in image inputs and audio inputs for your endpoint.

Parameters

Put model options under the provider's config. Match the options to the endpoint and model:

Setting	Chat Completions	Responses
Output limit for reasoning models	`max_completion_tokens`	`max_output_tokens`
Reasoning effort	`reasoning_effort: low`	`reasoning: { effort: low }`
Verbosity on supported models	`verbosity: low`	`verbosity: low` (sent as `text.verbosity`)
Structured output	`response_format`	`response_format` (sent as `text.format`)
System instructions	System or developer message in the prompt	`instructions` or messages in the prompt

providers:
  - id: openai:chat:gpt-5.6-luna
    config:
      reasoning_effort: low
      max_completion_tokens: 2048
  - id: openai:responses:gpt-5.6-luna
    config:
      reasoning:
        effort: low
      max_output_tokens: 2048

Reasoning tokens count toward the output limit and billing, even though they are not the visible answer. Leave enough room for both reasoning and the final output.

Promptfoo omits temperature for models it recognizes as reasoning models, including GPT-5, Astra, and o-series models. For a non-reasoning model such as gpt-4.1-mini, you can set temperature: 0 and, on Chat Completions, max_tokens. Check the selected model's API documentation before using other sampling options.

Defaults and additional request options

For non-reasoning requests, Promptfoo defaults to temperature: 0 and an output limit of 1,024 tokens. For reasoning requests, Promptfoo leaves the output limit unset unless you configure it or set an applicable environment variable. Set the limit explicitly when comparing models.

omitDefaults: true omits Promptfoo's default temperature and output limit. Explicit configuration and environment values still apply.

Option	Use
`tools`, `tool_choice`	Declare tools and control tool selection. See tool calling.
`functionToolCallbacks`	Map function names to local callbacks. See callbacks.
`passthrough`	Add fields directly to the request body, or override generated fields. Model-specific validation still applies. Supported by Chat, Responses, embeddings, and speech.
`prompt_cache_key`, `prompt_cache_options`	Configure OpenAI prompt caching.
`service_tier`	Request a service tier supported by your model and account.
`maxRetries`	Retry count for HTTP requests; defaults to 4. Set to 0 to disable retries. Hard quota failures are not retried.

For endpoint-specific fields, see the Chat Completions reference and Responses reference. Promptfoo's configuration types describe the named provider options. An API field without a named option may need passthrough.

Connection settings

The default base URL is https://api.openai.com/v1. Set apiBaseUrl for an OpenAI-compatible gateway and apiKeyEnvar to select its credential:

providers:
  - id: openai:chat:your-model
    config:
      apiBaseUrl: https://gateway.example.com/v1
      apiKeyEnvar: GATEWAY_API_KEY
      omitDefaults: true

Use the model name and endpoint supported by your gateway. apiBaseUrl includes the API prefix, such as /v1, but not /chat/completions or /responses. Promptfoo appends the endpoint path and preserves base URL query parameters.

Option	Use
`apiKeyEnvar`	Read a key from the named environment variable. A missing variable does not fall back to `OPENAI_API_KEY`.
`apiKey`	Set a key directly; takes precedence over environment variables. Prefer a secret-backed value.
`apiKeyRequired`	Set to `false` only for endpoints that do not require an API key.
`useDefaultApiKey`	Set to `false` to disable fallback to `OPENAI_API_KEY`. Explicit `apiKey` and `apiKeyEnvar` still work. Pair with `apiKeyRequired: false` for an unauthenticated compatible server.
`headers`	Add request headers, such as `OpenAI-Project`.
`organization`	Set the OpenAI organization ID.

Provider env overrides take precedence over the corresponding process environment variables. For Azure OpenAI, use the Azure provider and its deployment-specific configuration.

For a runnable starting point, see the openai-compatible-gateway example. vLLM, Llamafile, and LiteLLM document setups for those servers.

Base URL precedence and attribution headers

Promptfoo checks config.apiHost, then config.apiBaseUrl, then the endpoint environment variables listed below. apiHost constructs https://<host>/v1; use apiBaseUrl when you need a protocol, port, or custom path.

Built-in OpenAI API requests include X-OpenAI-Originator: promptfoo. Override that value with config.headers if your integration needs a different originator. Custom headers also override the configured organization header.

Cost estimates

Promptfoo uses returned token usage and its model pricing catalog to estimate costs. Estimates can be incomplete for new models, tools, or gateways that omit usage. Check OpenAI's usage dashboard for billed usage.

Current standard rates in USD per million tokens, for requests with up to 272,000 input tokens:

Model	Input	Cached input	Cache writes	Output
GPT-5.6 Luna	$0.20	$0.02	$0.25	$1.20
GPT-5.6 Terra	$2	$0.20	$2.50	$12
GPT-5.6 Sol (`gpt-5.6` alias)	$4	$0.40	$5	$20
GPT-6 Astra	$10	$1	$12.50	$50

Above 272,000 input tokens, input, cached-input, and cache-write rates double; output rates increase by 50%. Batch and Flex cost half the standard rates. Fast mode (fast or priority) costs twice the standard rates. Regional processing adds 10%; Astra Fast mode is unavailable with EU data residency. Sol's promotional pricing runs at least through November 21, 2026. Rates verified September 9, 2026; see OpenAI pricing.

For Chat Completions and Responses, set inputCost and outputCost to override rates in dollars per token, not per million tokens. For audio, use audioInputCost and audioOutputCost. The older cost and audioCost options are shared input/output fallbacks. These settings affect Promptfoo's estimates, not API billing.

Generating multiple responses

For Chat Completions models that support n, pass it through to the API:

providers:
  - id: openai:chat:gpt-4.1-mini
    config:
      passthrough:
        n: 3

Promptfoo's primary output contains the first choice. The provider response's metadata.choices contains all choices. The Responses API does not use n.

Reducing embedding dimensions

Set dimensions through passthrough for a text-embedding-3 model:

providers:
  - id: openai:embedding:text-embedding-3-large
    config:
      passthrough:
        dimensions: 1024

When grading generated text with embeddings, configure the embedding provider on the similarity assertion. See the Embeddings API reference for model limits.

Responses API

Use openai:responses:<model> for text, image and file inputs, built-in tools, and response state. A basic configuration is:

providers:
  - id: openai:responses:gpt-5.6-luna
    config:
      instructions: Answer support questions using the supplied policy.
      reasoning:
        effort: low
      max_output_tokens: 2048
      store: false

store: false controls storage at OpenAI. It does not disable Promptfoo's local response cache. Use --no-cache when you need fresh API requests. OpenAI's data controls describe retention and account-level restrictions.

Response state and streaming

State, tools, storage, and streaming

Option	Behavior
`instructions`	Set system instructions for this request.
`previous_response_id`	Continue a stored response by ID. It does not automatically connect separate test cases.
`store`	Control storage for later retrieval. OpenAI's default is `true`; account data controls can restrict it.
`include`	Request extra fields in the raw response, such as `web_search_call.results` or `reasoning.encrypted_content`.
`max_tool_calls`	Limit built-in tool calls in one response.
`parallel_tool_calls`	Allow parallel tool calls where supported.
`metadata`	Attach string key/value metadata.
`truncation`	Use `disabled` to fail on excess context, or `auto` to let OpenAI truncate it.
`background`	Create a background response; Promptfoo polls until it completes or times out.
`stream`	Request streaming; Promptfoo collects the stream into the eval result.

The provider response's raw field contains the Responses object, including id and output items. Its metadata includes extracted annotations and HTTP metadata. Use these fields when you need to inspect tool results or continue a conversation.

Prompt caching

OpenAI prompt caching reuses a shared input prefix while still generating a new response. Promptfoo's local cache reuses the response itself. --no-cache bypasses Promptfoo's cache; it does not disable OpenAI prompt caching.

For GPT-5.6 and Astra, configure prompt_cache_options:

providers:
  - id: openai:responses:gpt-5.6-luna
    config:
      prompt_cache_key: support-policy
      prompt_cache_options:
        mode: implicit
        ttl: 30m

implicit lets OpenAI place cache breakpoints. With explicit, add prompt_cache_breakpoint: { mode: explicit } to eligible structured content blocks; without a breakpoint, explicit mode performs no cache reads or writes. See OpenAI's prompt caching guide for eligible inputs and billing.

Earlier models and background requests

Earlier models use prompt_cache_retention where supported. GPT-5.5 Responses requires extended retention; in_memory is invalid for that model. GPT-5.6 and later deprecate this field in favor of prompt_cache_options.ttl.

Authenticated background jobs are persisted for resumption only when a non-secret project or tenant header, such as OpenAI-Project or X-Tenant-Id, isolates the request. OpenAI-Organization alone does not isolate projects. A persisted job may be shared by eval processes, so stopping one subscriber does not cancel it for the others. Use --no-cache for a run whose upstream background job should be cancelled when the eval stops.

Structured output

Use a JSON schema when assertions need specific fields. Promptfoo accepts response_format in both Chat Completions and Responses configurations and translates it to the selected API's format.

promptfooconfig.yaml
prompts:
  - 'Classify this support ticket: {{ticket}}'

providers:
  - id: openai:responses:gpt-5.6-luna
    config:
      reasoning:
        effort: low
      max_output_tokens: 2048
      response_format:
        type: json_schema
        json_schema:
          name: ticket_category
          strict: true
          schema:
            type: object
            properties:
              category:
                type: string
                enum: [billing, technical]
            required: [category]
            additionalProperties: false

tests:
  - vars:
      ticket: I was charged twice for my subscription.
    assert:
      - type: javascript
        value: output.category === 'billing'

Promptfoo parses valid JSON schema output into an object, so the assertion can read output.category directly. Refusals, incomplete responses, or invalid JSON may still produce a different output; check errors and failed assertions. For JSON mode without a schema, use type: json_object and explicitly ask for JSON in the prompt.

External file references

For either endpoint, response_format can reference a JSON or YAML file containing the entire format configuration:

config:
  response_format: file://./response-format.json

Use the nested json_schema shape above for Chat Completions or a shared configuration. Responses also accepts the flattened shape below and always sends JSON schemas with strict: true:

response-format.json
{
  "type": "json_schema",
  "name": "ticket_category",
  "schema": {
    "type": "object",
    "properties": {
      "category": { "type": "string", "enum": ["billing", "technical"] }
    },
    "required": ["category"],
    "additionalProperties": false
  }
}

The schema itself can be a nested file:// reference. File paths support Nunjucks variables, such as file://./schemas/{{ schema_name }}.json.

Prompt-level and per-test formats

A prompt's config.response_format overrides the provider setting. For a different schema per test, set tests[].options.response_format. See the per-test schema example.

For complete configurations, see the structured output example and Responses external format example.

Tool calling

Use tools to test which function the model selects and which arguments it produces. A tool definition alone does not execute your application code.

Using tools

Chat Completions nests each function definition under function. This eval forces an order lookup and validates both the schema and the requested order ID:

promptfooconfig.yaml
prompts:
  - 'Look up order {{order_id}}.'

providers:
  - id: openai:chat:gpt-5.6-luna
    config:
      tools:
        - type: function
          function:
            name: get_order_status
            description: Get the status of an order by ID.
            strict: true
            parameters:
              type: object
              properties:
                order_id:
                  type: string
              required: [order_id]
              additionalProperties: false
      tool_choice:
        type: function
        function:
          name: get_order_status

tests:
  - vars:
      order_id: ORD-123
    assert:
      - type: is-valid-openai-tools-call
      - type: javascript
        value: |-
          const calls = Array.isArray(output) ? output : output.tool_calls;
          return calls.length === 1 &&
            calls[0].function.name === 'get_order_status' &&
            JSON.parse(calls[0].function.arguments).order_id === context.vars.order_id;

The is-valid-openai-tools-call assertion checks Chat-style tool calls against the configured schema. The JavaScript assertion checks the intended behavior.

Responses tool definitions and results

Responses uses top-level function fields:

config:
  tools:
    - type: function
      name: get_order_status
      description: Get the status of an order by ID.
      strict: true
      parameters:
        type: object
        properties:
          order_id:
            type: string
        required: [order_id]
        additionalProperties: false
  tool_choice:
    type: function
    name: get_order_status

Promptfoo also converts nested Chat-style definitions to the Responses shape. Responses function-call results are available in raw.output as items with type: function_call, name, arguments, and call_id. Inspect those items when asserting on native Responses tool calls; they are not Chat-style output[0].function objects.

Loading tools from a file

Set config.tools to a file reference. Static files contain an array of tool definitions:

config:
  tools: file://./tools.yaml

For dynamic definitions, export a function that returns the array and include its name: file://./tools.ts:getTools, file://./tools.js:getTools, or file://./tools.py:get_tools. Both synchronous and asynchronous functions are supported.

Inline tool definitions and file-reference paths can use test variables. Promptfoo does not render placeholders inside the loaded file or returned tool definitions; supply those values in the file or function itself.

Run tool callbacks

For Chat Completions and Responses, functionToolCallbacks maps tool names to local functions. A callback receives the arguments as a JSON string and should return a string or Promise<string>.

These providers return callback results as eval output; they do not run a general model-to-tool loop that sends every result back to the model. To evaluate a complete agent loop, use the Agents SDK provider or a custom provider.

Use a local callback in a YAML configuration

Add a callback to a provider that defines get_order_status:

config:
  functionToolCallbacks:
    get_order_status: file://./callbacks.mjs:getOrderStatus

For a deterministic test, this callback returns a fixed fixture:

callbacks.mjs
export function getOrderStatus(args) {
  const { order_id } = JSON.parse(args);
  return JSON.stringify({ order_id, status: 'shipped' });
}

Keep callback files inside the configuration's base directory. Promptfoo rejects callback paths that escape it. Only run configurations and callbacks you trust.

Web search

Add OpenAI's web_search tool to a Responses provider:

providers:
  - id: openai:responses:gpt-5.6-luna
    config:
      tools:
        - type: web_search
          search_context_size: low
          filters:
            allowed_domains: [developers.openai.com]
      include:
        - web_search_call.results

This example limits searches to OpenAI's developer documentation. Tool options are forwarded to OpenAI, including search_context_size, filters, user_location, external_web_access, and return_token_budget. See the web search guide for supported values and model restrictions.

Location, live access, and search budgets

For location-sensitive searches, add an approximate location to the tool:

tools:
  - type: web_search
    search_context_size: medium
    user_location:
      type: approximate
      country: US
      city: San Francisco
      region: California
      timezone: America/Los_Angeles
    external_web_access: true

search_context_size accepts low, medium, or high. Set external_web_access: false to use cached or indexed results without fetching live pages. For longer research with GPT-5+ reasoning models, return_token_budget: unlimited removes the standard search-result token cap; default keeps it. Removing the cap can increase latency and cost. The budget option applies to web_search, not web_search_preview.

Inspect citations in the provider response's metadata.annotations and search items in raw.output. Use --no-cache for fresh searches. Web search can incur tool charges in addition to token usage; see OpenAI pricing.

The search-rubric assertion uses a search-enabled grading model to verify an output against current information. Configuring the target's search tools and configuring a search-based grader are separate choices.

MCP tools

Choose the integration based on what you want to test:

Task	Configuration
Let OpenAI call a remote MCP server	Responses with `config.tools` containing `type: mcp`, as below
Connect Promptfoo to a local or remote MCP server for model tool calls	Explicit `openai:chat:<model>` with `config.mcp`
Evaluate an MCP server's tools directly	The MCP provider, without an OpenAI model

With config.mcp, Promptfoo connects to the server and executes the model's tool calls. This works with servers on your machine or private network. With a Responses type: mcp tool, OpenAI connects to the server, so it must be reachable from OpenAI.

Connect Promptfoo to an MCP server

Use an explicit Chat provider, even for models whose bare IDs default to Responses:

providers:
  - id: openai:chat:gpt-5.6-luna
    config:
      mcp:
        enabled: true
        server:
          url: http://localhost:8000/mcp

Start your MCP server at the configured URL before running the eval. To launch a local server process instead, use server.command and server.args. See the MCP integration guide for authentication and multiple servers.

The Chat provider returns executed tool results as eval output. For a full agent loop that sends results back to the model, use the Agents SDK provider or a custom provider.

For a remote MCP server, add a tool with type: mcp. OpenAI connects to that server. This example limits access to one public documentation tool and skips approval only for that tool:

providers:
  - id: openai:responses:gpt-5.6-luna
    config:
      tools:
        - type: mcp
          server_label: deepwiki
          server_url: https://mcp.deepwiki.com/mcp
          allowed_tools: [ask_question]
          require_approval:
            never:
              tool_names: [ask_question]

Use headers inside the MCP tool for authentication, with secret values supplied through environment variables. Approval requests appear in the output; this provider does not interactively approve them. Configure approvals deliberately for automated evals. See OpenAI's MCP guide and the Promptfoo MCP example.

Images

Sending images in prompts

For Responses, use input_text and input_image blocks. Save this as image-prompt.json, reference it with prompts: [file://image-prompt.json], and supply question and image_url test variables:

image-prompt.json
[
  {
    "role": "user",
    "content": [
      { "type": "input_text", "text": "{{question}}" },
      { "type": "input_image", "image_url": "{{image_url}}" }
    ]
  }
]

Use a publicly accessible image URL or a base64 data URL. For file inputs, Responses accepts input_file blocks with a file ID or supported file data; see the OpenAI file input guide.

Chat Completions image format

Chat Completions uses text and image_url, with the URL nested inside an object:

chat-image-prompt.json
[
  {
    "role": "user",
    "content": [
      { "type": "text", "text": "{{question}}" },
      { "type": "image_url", "image_url": { "url": "{{image_url}}" } }
    ]
  }
]

See the OpenAI vision example.

Generating images

openai:image:gpt-image-2.5-flare calls /v1/images/generations for text-to-image evals. Use gpt-image-2.5-sunburst to compare Sunburst on the same prompts; both aliases and their 2026-09-08 snapshots are supported.

promptfooconfig.yaml
prompts:
  - 'A product photo of {{product}} on a plain white background.'

providers:
  - id: openai:image:gpt-image-2.5-flare
    config:
      size: 1024x1024
      quality: low
      output_format: webp

tests:
  - vars:
      product: a blue ceramic mug

GPT Image 2.5 also accepts quality: xhigh and quality: max. For transparent output, use background: transparent with PNG or WebP. Cost comes from the response's token usage; it is left unset when usage is missing because older models' per-image estimates do not apply. See the Image API guide.

This provider supports generation only. Image editing, masks, reference images, variations, and streaming are not implemented.

GPT Image 2 options

Option	Values
`size`	`auto`, standard sizes such as `1024x1024`, or valid custom dimensions
`quality`	`low`, `medium`, `high`, `auto`
`background`	`opaque`, `auto`; transparency is unsupported for this model
`output_format`	`png`, `jpeg`, `webp`
`output_compression`	0 to 100; only with `jpeg` or `webp`
`moderation`	`auto`, `low`
`n`	1 to 10 images

Custom dimensions must be multiples of 16, with a maximum edge of 3,840 pixels, an aspect ratio no greater than 3:1, and 655,360 to 8,294,400 total pixels. Promptfoo validates these constraints before sending the request.

Cost estimates may be absent for quality: auto or custom sizes. Returned usage remains available for inspection. See the image example and OpenAI image guide.

Audio

Choose the route for your task: openai:chat:gpt-audio-1.5 for audio input or output in a chat request, text to speech for reading supplied text aloud, or Realtime for conversational sessions. The Responses provider does not support this audio-chat format.

Using audio inputs

Chat audio inputs use base64-encoded WAV or MP3 data:

audio-input.json
[
  {
    "role": "user",
    "content": [
      { "type": "text", "text": "Summarize the customer's request." },
      {
        "type": "input_audio",
        "input_audio": { "data": "{{audio_file}}", "format": "mp3" }
      }
    ]
  }
]

Supply your own audio fixture; a file:// test variable loads its base64 content:

prompts:
  - file://audio-input.json

providers:
  - id: openai:chat:gpt-audio-1.5
    config:
      modalities: [text]

tests:
  - vars:
      audio_file: file://assets/customer-request.mp3

For spoken responses, use modalities: [text, audio] and configure audio.voice and audio.format:

config:
  modalities: [text, audio]
  audio:
    voice: alloy
    format: wav

The web viewer displays audio outputs with a player and transcript. See the audio example and OpenAI audio guide for supported voices and output formats.

Text-to-speech

Use openai:tts:gpt-4o-mini-tts to turn the prompt into speech:

promptfooconfig.yaml
prompts:
  - Your order has shipped and will arrive tomorrow.

providers:
  - id: openai:tts:gpt-4o-mini-tts
    config:
      voice: coral
      instructions: Speak warmly and clearly.
      response_format: wav
      speed: 1.0

response_format supports mp3, opus, aac, flac, wav, and pcm. speed ranges from 0.25 to 4.0. Speech input is limited to 4,096 characters. See the OpenAI text-to-speech guide.

Speech formats, custom voices, and caching

format is an alias for response_format; the explicit response_format wins if both are set. Use passthrough for speech request fields without a dedicated provider option.

Custom voices use voice: { id: voice_123 } and require access to that voice in your OpenAI project. To cache a custom-voice response, set a non-secret project or tenant header such as OpenAI-Project. OpenAI-Organization alone does not isolate projects.

On api.openai.com, secret authentication headers are excluded from the cache key; non-secret project or tenant headers keep cached results separate. Rotating an authentication secret alone does not invalidate that cache. Authenticated custom endpoints bypass persistent caching, even with a tenant header. Caching also skips requests with detected secrets in the body, URL path, or non-authentication header values.

The binary GPT-4o mini TTS response does not provide token usage, so Promptfoo leaves its cost unset.

Audio transcription

Use openai:transcription:gpt-transcribe for recorded audio. The prompt is the path to an audio file. Supply expected languages and literal terms as arrays:

providers:
  - id: openai:transcription:gpt-transcribe
    config:
      languages: [en, fr]
      keywords: [AC-42, premium plan]
      prompt: A customer support call.

gpt-transcribe uses languages instead of language. Keywords must be non-empty, single-line strings without < or >. Detected languages appear in metadata.languages; cost uses the API's duration when available. See the complete transcription example.

Keep gpt-4o-transcribe-diarize for speaker labels and whisper-1 for word timestamps. gpt-live-transcribe uses a dedicated Realtime transcription session, which this file-upload provider does not implement. See the OpenAI transcription guide and deprecation schedule for migration details.

Realtime

Use openai:realtime:gpt-realtime-2.1 for a conversational WebSocket session. Choose a text-only output mode when your eval does not need generated audio:

promptfooconfig.yaml
prompts:
  - Explain how to reset a password in one sentence.

providers:
  - id: openai:realtime:gpt-realtime-2.1
    config:
      modalities: [text]
      websocketTimeout: 60000

For audio output, set modalities: [text, audio] and a top-level voice, such as marin. Promptfoo sends the current Realtime API schema; if the requested modalities include audio, it selects audio output with a transcript.

The legacy gpt-4o-mini-realtime-preview-2024-12-17 selector still routes to Realtime. Check OpenAI's lifecycle notices and its model card before using this preview model.

The result includes audio for playback and a transcript for text assertions. To grade tone, pacing, or pronunciation, select an audio-capable Chat Completions grader:

defaultTest:
  assert:
    - type: llm-rubric
      value: The speaker sounds calm and speaks at a steady pace.
      provider:
        id: openai:chat:gpt-audio-1.5
        config:
          modalities: [text]

Promptfoo sends the generated audio to this grader and requests a text grade. Text-only graders continue to evaluate the transcript. Audio must be inline base64 WAV or MP3, up to 20 MiB. Keep the Realtime provider's default output_audio_format: pcm16; Promptfoo converts it to WAV for both single requests and persistent conversations. G.711 output requires conversion before audio grading. See audio grading for limits and transformed outputs.

Session settings

Session, audio, and tool settings

Option	Behavior
`instructions`	System instructions for the session
`voice`	Audio output voice; defaults to `alloy`
`input_audio_format`, `output_audio_format`	`pcm16`, `g711_ulaw`, or `g711_alaw`
`input_audio_transcription`	Input transcription settings for a model supported by the Realtime API
`turn_detection`	`server_vad`, `semantic_vad`, or `null`
`reasoning`	Model-specific reasoning settings
`max_response_output_tokens`	Integer from 1 to 4,096, or `'inf'` for the model maximum; invalid values fall back to `'inf'`
`websocketTimeout`	Timeout in milliseconds; defaults to 30,000
`tools`, `tool_choice`	Native Realtime function tool definitions and selection
`functionCallHandler`	JavaScript handler receiving a tool name and JSON argument string; returns a `Promise<string>`
`toolCallTimeout`	Per-tool timeout; falls back to `websocketTimeout`, then 30,000 milliseconds
`maxToolIterations`	Maximum tool follow-up rounds in one turn; defaults to 8, allowed range 1 to 64

Realtime function definitions have top-level name, description, and parameters. Promptfoo also accepts nested Chat-style definitions and converts them. A functionCallHandler can return results to the model; validate the tool name and arguments before any side effects.

Structured user messages use input_text, input_audio, or input_image blocks. Multi-turn evals use test.metadata.conversationId to identify the conversation. See the Realtime example for message formats, session management, and a function handler.

This provider creates conversational sessions. Dedicated Realtime transcription and translation sessions require their own integrations.

Custom endpoints and proxies (Realtime)

Set apiBaseUrl as for other OpenAI providers. Promptfoo converts https:// to wss:// and http:// to ws://, then appends /realtime. For example, https://gateway.example.com/v1 becomes wss://gateway.example.com/v1/realtime.

Environment variables

Prefer provider configuration when comparing different settings in the same eval.

Credentials, endpoints, and request defaults

Variable	Behavior
`OPENAI_API_KEY`	Default API key
`OPENAI_ORGANIZATION`	Organization ID
`OPENAI_API_HOST`	Constructs `https://<host>/v1`; checked before base URL environment variables
`OPENAI_API_BASE_URL`	Full base URL; preferred over `OPENAI_BASE_URL` at the same environment level
`OPENAI_BASE_URL`	Alternate full base URL
`OPENAI_TEMPERATURE`	Temperature for supported non-reasoning requests; defaults to 0
`OPENAI_MAX_TOKENS`	Output limit for non-reasoning requests; also a fallback for reasoning Responses requests
`OPENAI_MAX_COMPLETION_TOKENS`	Output limit for reasoning Chat requests; preferred environment fallback for reasoning Responses requests. No built-in default.
`PROMPTFOO_EVAL_TIMEOUT_MS`	Overall eval-call timeout, including Responses background polling
`REQUEST_TIMEOUT_MS`	Standard request timeout, except requests with a longer model-specific timeout
`PROMPTFOO_REQUEST_BACKOFF_MS`	Retry backoff base in milliseconds; defaults to 5,000
`PROMPTFOO_RETRY_5XX`	Set to `true` to retry server errors
`PROMPTFOO_DELAY_MS`	Delay between calls in milliseconds; defaults to 0

Within endpoint environment settings, OPENAI_API_HOST is checked first. Provider env base URL overrides are checked before process base URL values. Explicit provider connection settings take precedence over these environment variables.

Troubleshooting

Symptom	Check
Authentication failure	Confirm the selected key variable, project, and endpoint. With `apiKeyEnvar`, a missing named key does not fall back to the default key.
Model not found	Check your account's access and the model lifecycle. Use an explicit endpoint prefix.
Unsupported parameter	Match the option to the model and API. For reasoning models, check the output limit and reasoning setting first.
Empty or incomplete answer	Check the raw response and token usage. Reasoning may exhaust the output limit before producing a visible answer.
Unexpectedly reused output	Run with `--no-cache` to bypass Promptfoo's local response cache.

Rate limits

Promptfoo retries transient rate limits and adapts concurrency. For manual control, use --max-concurrency 1, add a delay such as --delay 3000, or adjust PROMPTFOO_REQUEST_BACKOFF_MS. Hard quota errors require resolving the account's quota or billing issue. See rate limits.

Server errors

Set PROMPTFOO_RETRY_5XX=true to retry HTTP server errors. Check the error and endpoint before increasing timeouts or retries.

Timeouts

Responses requests with background: true and GPT-5 Pro variants use a 10-minute timeout unless PROMPTFOO_EVAL_TIMEOUT_MS is set. Regular requests use the standard request timeout, normally 5 minutes. Set an overall limit for a longer run:

PROMPTFOO_EVAL_TIMEOUT_MS=1200000 npx promptfoo@latest eval --no-cache

For these long-running Responses requests, REQUEST_TIMEOUT_MS does not override the automatic 10-minute timeout. See background caching and cancellation before relying on an interrupted run to cancel upstream work.

Migrating older configurations

Use OpenAI's deprecation schedule as the source for shutdown dates and replacements. These migrations require more than changing a model name:

Existing configuration	Migration
`openai:assistant:<id>`	The native Assistants API shut down on August 26, 2026. Move instructions, tools, and state to Responses; assistant IDs are not response IDs. See the migration guide.
`openai:completion:*`	Native Babbage, Davinci, and GPT-3.5 Turbo Instruct models retire on September 28, 2026. Use Chat Completions or Responses with compatible prompts and options.
`openai:video:*`	The native Videos API and Sora 2 models retire on September 24, 2026. OpenAI lists no replacement API.
`functions` and `function_call`	Replace them with `tools` and `tool_choice`, using the selected endpoint's schema.
Retired deep-research, Codex, or chat snapshots	Select an available model and re-run representative evals. Built-in research tools require the Responses endpoint.

OpenAI-compatible services have their own lifecycle and API contracts. A provider implementation remaining in Promptfoo does not mean its model is still available from OpenAI.

Agent providers

Choose a provider that matches the application you are testing:

Application	Provider guide
Managed Codex sessions and hosted sandboxes	OpenAI Agents API
TypeScript Agents SDK tools, handoffs, and sessions	OpenAI Agents SDK
Python Agents SDK application	Agents SDK Python guide
ChatKit integration	OpenAI ChatKit
Coding workflow with working-directory access	Codex SDK
Codex Security scan or finding validation	Codex Security SDK
App-server events, approvals, and thread lifecycle	Codex app-server

Quickstart​

Models​

GPT-5.6​

GPT-6 Astra​

Fine-tuned models​

Chat messages​

Parameters​

Connection settings​

Cost estimates​

Generating multiple responses​

Reducing embedding dimensions​

Responses API​

Response state and streaming​

Prompt caching​

Structured output​

External file references​

Tool calling​

Using tools​

Loading tools from a file​

Run tool callbacks​

Web search​

MCP tools​

Images​

Sending images in prompts​

Generating images​

Audio​

Using audio inputs​

Text-to-speech​

Audio transcription​

Realtime​

Session settings​

Custom endpoints and proxies (Realtime)​

Environment variables​

Troubleshooting​

Rate limits​

Server errors​

Timeouts​

Migrating older configurations​

Agent providers​