Skip to main content

text-generation-webui

promptfoo can run evals on oobabooga's text-generation-webui-hosted models through the OpenAPI API extension.

The text-gen-webui extension can be activated from with the UI or via command line. Here is an example of command line usage.

python server.py --loader <LOADER-NAME> --model <MODEL-NAME> --api
# Replace `python server.py` with ./start_linux if simple installer is used

Usage is compatible with the OpenAI API.

In promptfoo we can address the API as follows.

providers:
- openai:chat:<MODEL-NAME>:
id: <MODEL-ID>
config:
apiKey: placeholder
apiBaseUrl: http://localhost:5000/v1
temperature: 0.8
max_tokens: 1024
passthrough: # These config values are passed directly to the API
mode: instruct
instruction_template: LLama-v2

If desired, you can instead use the OPENAI_API_BASE_URL and OPENAI_API_KEY environment variables instead of the apiBaseUrl and apiKey configs.