Replicate

Replicate is an API for machine learning models. It currently hosts models like Llama v2, Gemma, and Mistral/Mixtral.

To run a model, specify the Replicate model name and version, like so:

replicate:replicate/llama70b-v2-chat:e951f18578850b652510200860fc4ea62b3b16fac280f83ff32282f87bbd2e48

Examples

Here's an example of using Llama on Replicate. In the case of Llama, the version hash and everything under config is optional:

providers:
  - id: replicate:meta/llama-2-7b-chat
    config:
      temperature: 0.01
      max_length: 1024
      prompt:
        prefix: '[INST] '
        suffix: ' [/INST]'

Here's an example of using Gemma on Replicate. Note that unlike Llama, it does not have a default version, so we specify the model version:

providers:
  - id: replicate:google-deepmind/gemma-7b-it:2790a695e5dcae15506138cc4718d1106d0d475e6dca4b1d43f42414647993d5
    config:
      temperature: 0.01
      max_new_tokens: 1024
      prompt:
        prefix: "<start_of_turn>user\n"
        suffix: "<end_of_turn>\n<start_of_turn>model"

Configuration

The Replicate provider supports several configuration options that can be used to customize the behavior of the models, like so:

Parameter	Description
temperature	Controls randomness in the generation process.
max_length	Specifies the maximum length of the generated text.
max_new_tokens	Limits the number of new tokens to generate.
top_p	Nucleus sampling: a float between 0 and 1.
top_k	Top-k sampling: number of highest probability tokens to keep.
repetition_penalty	Penalizes repetition of words in the generated text.
system_prompt	Sets a system-level prompt for all requests.
stop_sequences	Specifies stopping sequences that halt the generation.
seed	Sets a seed for reproducible results.

Not every model supports every completion parameter. Be sure to review the API provided by the model beforehand.

These parameters are supported for all models:

Parameter	Description
apiKey	The API key for authentication with Replicate.
prompt.prefix	String added before each prompt. Useful for instruction/chat formatting.
prompt.suffix	String added after each prompt. Useful for instruction/chat formatting.

Supported environment variables:

• REPLICATE_API_TOKEN - Your Replicate API key.
• REPLICATE_API_KEY - An alternative to REPLICATE_API_TOKEN for your API key.
• REPLICATE_MAX_LENGTH - Specifies the maximum length of the generated text.
• REPLICATE_TEMPERATURE - Controls randomness in the generation process.
• REPLICATE_REPETITION_PENALTY - Penalizes repetition of words in the generated text.
• REPLICATE_TOP_P - Controls the nucleus sampling: a float between 0 and 1.
• REPLICATE_TOP_K - Controls the top-k sampling: the number of highest probability vocabulary tokens to keep for top-k-filtering.
• REPLICATE_SEED - Sets a seed for reproducible results.
• REPLICATE_STOP_SEQUENCES - Specifies stopping sequences that halt the generation.
• REPLICATE_SYSTEM_PROMPT - Sets a system-level prompt for all requests.