The YAML configuration format runs each prompt through a series of example inputs (aka "test case") and checks if they meet requirements (aka "assert").
Here is the main structure of the promptfoo configuration file:
📄️ Input and output files
🗃️ Assertions & metrics
📄️ Dataset generation
Your dataset is the heart of your LLM eval. To the extent possible, it should closely represent true inputs into your LLM app.
The scenarios configuration lets you group a set of data along with a set of tests that should be run on that data.
promptfoo caches the results of API calls to LLM providers. This helps save time and cost.
promptfoo collects basic anonymous telemetry by default. This telemetry helps us decide how to spend time on development.