To use the OpenAI API, set the OPENAIAPIKEY environment variable, specify via apiKey field in the configuration file or pass the API key as an argument to the constructor.
The anthropic provider supports the following models:
The azureopenai provider is an interface to OpenAI through Azure. It behaves the same as the OpenAI provider.
The llama provider is compatible with the HTTP server bundled with llama.cpp.
The ollama provider is compatible with Ollama,
📄️ Google Vertex
The vertex provider is compatible with Google's Vertex AI offering, which offers access to models such as Gemini and Bison.
📄️ Google AI Studio
The google provider is compatible with Google AI Studio (formerly known as PaLM), which offers access to Gemini models.
📄️ Generic webhook
The webhook provider can be useful for triggering more complex flows or prompt chains end to end in your app.
📄️ Custom API Provider
To create a custom API provider, implement the ApiProvider interface in a separate module. Here is the interface:
📄️ Custom scripts
You may use any shell command as an API provider. This is particularly useful when you want to use a language or framework that is not directly supported by promptfoo.
📄️ Python script
promptfoo includes support for the HuggingFace Inference API, specifically text generation and feature extraction tasks.
📄️ LocalAI (Llama, Alpaca, GPT4All, ...)
LocalAI is an API wrapper for open-source LLMs that is compatible with OpenAI. You can run LocalAI for compatibility with Llama, Alpaca, Vicuna, GPT4All, RedPajama, and many other models compatible with the ggml format.
Replicate is an API for machine learning models. It currently hosts models like Llama v2, Gemma, and Mistral/Mixtral.
The bedrock lets you use Amazon Bedrock in your evals. This is a common way to access Anthropic's Claude and other models.
The cohere provider is an interface to Cohere AI's chat inference API.
📄️ Mistral AI
The Mistral AI API offers access to Mistral models such as mistral-tiny, mistral-small, and mistral-medium.
To use OpenLLM with promptfoo, we take advantage of OpenLLM's support for OpenAI-compatible endpoint.
OpenRouter provides a unified interface for LLM APIs.
The Perplexity API (pplx-api) offers access to Perplexity, Mistral, Llama, and other models.
promptfoo can run evals on oobabooga's text-generation-webui-hosted models through the OpenAPI API extension.
📄️ Together AI
The Together.AI API offers access to numerous models such as Mistral/Mixtral, Llama, and others.
vllm's OpenAI-compatible server offers access to many supported models for local inference from Huggingface Transformers.