Anthropic
This provider supports the Anthropic Claude series of models.
API Key
To use Anthropic, you need to set the ANTHROPIC_API_KEY
environment variable or specify the apiKey
in the provider configuration.
Create Anthropic API keys here.
Example of setting the environment variable:
export ANTHROPIC_API_KEY=your_api_key_here
Latest API (Messages)
The messages API supports all the latest Anthropic models.
The anthropic
provider supports the following models via the messages API:
anthropic:messages:claude-instant-1.2
anthropic:messages:claude-2.0
anthropic:messages:claude-2.1
anthropic:messages:claude-3-haiku-20240307
anthropic:messages:claude-3-sonnet-20240229
anthropic:messages:claude-3-opus-20240229
Prompt Template
To allow for compatibility with the OpenAI prompt template, the following format is supported.
Example: prompt.json
[
{
"role": "system",
"content": "{{ system_message }}"
},
{
"role": "user",
"content": "{{ question }}"
}
]
If the role system
is specified, then it will be automatically added to the API request.
All user
or assistant
roles will automatically converted into the right format for the API request.
Currently, only type text
is supported.
The system_message
and question
are example variables that can be set with the var
directive.
Options
The Anthropic provider supports several options to customize the behavior of the model. These include:
temperature
: Controls the randomness of the output.max_tokens
: The maximum length of the generated text.top_p
: Controls nucleus sampling, affecting the randomness of the output.top_k
: Only sample from the top K options for each subsequent token.
Example configuration with options and prompts:
providers:
- id: anthropic:messages:claude-3-opus-20240229
config:
temperature: 0.0
max_tokens: 512
prompts: [prompt.json]
Images / Vision
You can include images in the prompts in Claude 3 models.
See the Claude vision example.
One important note: The Claude API only supports base64 representations of images. This is different from how OpenAI's vision works, in that it supports grabbing images from a URL. As a result, if you are trying to compare Claude 3 and OpenAI vision capabilities, you will need to have separate prompts for each.
See the OpenAI vision example to see where there are differences here.
Additional Capabilities
- Caching: Caches previous LLM requests by default.
- Token Usage Tracking: Provides detailed information on the number of tokens used in each request, aiding in usage monitoring and optimization.
- Cost Calculation: Calculates the cost of each request based on the number of tokens generated and the specific model used.
Deprecated API (Completions)
The completions API is deprecated, see the migration guide here.
The anthropic
provider supports the following models:
anthropic:completion:claude-1
anthropic:completion:claude-1-100k
anthropic:completion:claude-instant-1
anthropic:completion:claude-instant-1-100k
anthropic:completion:<insert any other supported model name here>
Supported environment variables:
ANTHROPIC_API_KEY
- requiredANTHROPIC_STOP
- stopwords, must be a valid JSON stringANTHROPIC_MAX_TOKENS
- maximum number of tokens to sample, defaults to 1024ANTHROPIC_TEMPERATURE
- temperature
Config parameters may also be passed like so:
providers:
- id: anthropic:completion:claude-1
prompts: chat_prompt
config:
temperature: 0
Model-graded tests
Model-graded assertions such as factuality
or llm-rubric
use OpenAI by default and expect OPENAI_API_KEY
as an environment variable. If you are using Anthropic, you may override the grader to point a different provider.
Because of how model-graded evals are implemented, the model must support chat-formatted prompts (except for embedding or classification models).
The easiest way to do this for all your test cases is to add the defaultTest
property to your config:
defaultTest:
options:
provider:
id: anthropic:messages:claude-3-opus-20240229
config:
# Provider config options
However, you can also do this for individual assertions:
# ...
assert:
- type: llm-rubric
value: Do not mention that you are an AI or chat assistant
provider:
id: anthropic:messages:claude-3-opus-20240229
config:
# Provider config options
Or individual tests:
# ...
tests:
- vars:
# ...
options:
provider:
id: anthropic:messages:claude-3-opus-20240229
config:
# Provider config options
assert:
- type: llm-rubric
value: Do not mention that you are an AI or chat assistant