📄️ OpenAI
To use the OpenAI API, set the OPENAIAPIKEY environment variable, specify via apiKey field in the configuration file or pass the API key as an argument to the constructor.
📄️ Anthropic
This provider supports the Anthropic Claude series of models.
📄️ Azure
The azureopenai provider is an interface to OpenAI through Azure. It behaves the same as the OpenAI provider.
📄️ Llama.cpp
The llama provider is compatible with the HTTP server bundled with llama.cpp.
📄️ Ollama
The ollama provider is compatible with Ollama, which enables access to Llama, Mixtral, Mistral, and more.
📄️ Google Vertex
The vertex provider is compatible with Google's Vertex AI offering, which offers access to models such as Gemini and Bison.
📄️ IBM BAM
The bam provider integrates with IBM's BAM API, allowing access to various models like meta-llama/llama-2-70b-chat and ibm/granite-13b-chat-v2.
📄️ Google AI Studio
The google provider is compatible with Google AI Studio (formerly known as PaLM), which offers access to Gemini models.
📄️ Generic webhook
The webhook provider can be useful for triggering more complex flows or prompt chains end to end in your app.
📄️ Custom Javascript
To create a custom API provider, implement the ApiProvider interface in a separate module. Here is the interface:
📄️ Custom Python
The python provider allows you to use a Python script as an API provider for evaluating prompts. This is useful when you have custom logic or models implemented in Python that you want to integrate with your test suite.
📄️ Custom scripts
You may use any shell command as an API provider. This is particularly useful when you want to use a language or framework that is not directly supported by promptfoo.
📄️ HuggingFace
promptfoo includes support for the HuggingFace Inference API, specifically text generation and feature extraction tasks.
📄️ LocalAI (Llama, Alpaca, GPT4All, ...)
LocalAI is an API wrapper for open-source LLMs that is compatible with OpenAI. You can run LocalAI for compatibility with Llama, Alpaca, Vicuna, GPT4All, RedPajama, and many other models compatible with the ggml format.
📄️ Replicate
Replicate is an API for machine learning models. It currently hosts models like Llama v2, Gemma, and Mistral/Mixtral.
📄️ Bedrock
The bedrock lets you use Amazon Bedrock in your evals. This is a common way to access Anthropic's Claude and other models.
📄️ Cohere
The cohere provider is an interface to Cohere AI's chat inference API, with models such as Command R that are optimized for RAG and tool usage.
📄️ Groq
The Groq API supports the OpenAI format, which makes it easy to integrate with promptfoo as a variation of the openai provider.
📄️ Mistral AI
The Mistral AI API offers access to Mistral models such as mistral-tiny, mistral-small, and mistral-medium.
📄️ OpenLLM
To use OpenLLM with promptfoo, we take advantage of OpenLLM's support for OpenAI-compatible endpoint.
📄️ OpenRouter
OpenRouter provides a unified interface for LLM APIs.
📄️ Perplexity
The Perplexity API (pplx-api) offers access to Perplexity, Mistral, Llama, and other models.
📄️ text-generation-webui
promptfoo can run evals on oobabooga's text-generation-webui-hosted models through the OpenAPI API extension.
📄️ Together AI
The Together.AI API offers access to numerous models such as Mistral/Mixtral, Llama, and others.
📄️ vllm
vllm's OpenAI-compatible server offers access to many supported models for local inference from Huggingface Transformers.