Skip to main content

HuggingFace

promptfoo includes support for the HuggingFace Inference API, specifically text generation and feature extraction tasks.

To run a model, specify the task type and model name. Supported models include:

  • huggingface:text-generation:<model name>
  • huggingface:text-classification:<model name>
  • huggingface:feature-extraction:<model name>
  • huggingface:sentence-similarity:<model name>

For example, autocomplete with GPT-2:

huggingface:text-generation:gpt2

Generate text with Mistral:

huggingface:text-generation:mistralai/Mistral-7B-v0.1

Embeddings similarity with sentence-transformers:

# Model supports the sentence similarity API
huggingface:sentence-similarity:sentence-transformers/all-MiniLM-L6-v2

# Model supports the feature extraction API
huggingface:feature-extraction:sentence-transformers/paraphrase-xlm-r-multilingual-v1

Supported environment variables:

  • HF_API_TOKEN - your HuggingFace API key

The provider can pass through configuration parameters to the API. See text generation parameters and feature extraction parameters.

Here's an example of how this provider might appear in your promptfoo config:

providers:
- id: huggingface:text-generation:mistralai/Mistral-7B-v0.1
config:
temperature: 0.1
max_length: 1024

Local inference

If you're running the Huggingface Text Generation Inference server locally, override the apiEndpoint:

providers:
- id: huggingface:text-generation:my-local-model
config:
apiEndpoint: http://127.0.0.1:8080/generate