Custom Javascript

To create a custom API provider, implement the ApiProvider interface in a separate module. Here is the interface:

export interface CallApiContextParams {
  vars: Record<string, string | object>;
}

export interface CallApiOptionsParams {
  // Whether to include logprobs in API response (used with OpenAI providers)
  includeLogProbs?: boolean;

  // Used when provider is overridden on the test case.
  originalProvider?: ApiProvider;
}

export interface ApiProvider {
  constructor(options: { id?: string; config: Record<string, any> });

  // Unique identifier for the provider
  id: () => string;

  // Text generation function
  callApi: (
    prompt: string,
    context?: CallApiContextParams,
    options?: CallApiOptionsParams,
  ) => Promise<ProviderResponse>;

  // Embedding function
  callEmbeddingApi?: (prompt: string) => Promise<ProviderEmbeddingResponse>;

  // Classification function
  callClassificationApi?: (prompt: string) => Promise<ProviderClassificationResponse>;

  // Shown on output UI
  label?: ProviderLabel;

  // Applied by the evaluator on provider response
  transform?: string;

  // Custom delay for the provider.
  delay?: number;
}

Example

Here's an example of a custom API provider that returns a predefined output along with token usage:

// customApiProvider.js
import fetch from 'node-fetch';

class CustomApiProvider {
  constructor(options) {
    // Provider ID can be overridden by the config file (e.g. when using multiple of the same provider)
    this.providerId = options.id || 'custom provider';

    // options.config contains any custom options passed to the provider
    this.config = options.config;
  }

  id() {
    return this.providerId;
  }

  async callApi(prompt, context) {
    // Add your custom API logic here
    // Use options like: `this.config.temperature`, `this.config.max_tokens`, etc.

    console.log('Vars for this test case:', JSON.stringify(context.vars));

    return {
      // Required
      output: 'Model output',

      // Optional
      tokenUsage: {
        total: 10,
        prompt: 5,
        completion: 5,
      },
    };
  }
}

module.exports = CustomApiProvider;

Using the provider

Include the custom provider in promptfoo config:

providers: ['file://relative/path/to/customApiProvider.js']

Alternatively, you can pass the path to the custom API provider directly in the CLI:

promptfoo eval -p prompt1.txt prompt2.txt -o results.csv  -v vars.csv -r ./customApiProvider.js

This command will evaluate the prompts using the custom API provider and save the results to the specified CSV file.

A full working example is available in the examples directory.

Multiple instances of the same provider

You can instantiate multiple providers of the same type with distinct IDs. In this example, we pass a different temperature config to the provider:

providers:
  - id: file:///absolute/path/to/customProvider.js
    label: custom-provider-hightemp
    config:
      temperature: 1.0
  - id: file:///absolute/path/to/customProvider.js
    label: custom-provider-lowtemp
    config:
      temperature: 0

ES modules

ES modules are supported, but must have a .mjs file extension. Alternatively, if you are transpiling Javascript or Typescript, we recommend pointing promptfoo to the transpiled plain Javascript output.

Custom Javascript

Example​

Using the provider​

Multiple instances of the same provider​

ES modules​

Example

Using the provider

Multiple instances of the same provider

ES modules