promptfoo

CLI and library for evaluating, testing, and red-teaming LLM apps. Side-by-side prompt comparisons in your CI.

⭐ 7,000 stars🍴 0 forksTypeScripttestingclired-team

Start hereRead the setup guide →Install steps and usage, straight from the docs.Source codeDownload ZIPLatest commit on default branch.SourceView on GitHubgithub.com/promptfoo/promptfoo

Project site: promptfoo.dev

About promptfoo

promptfoo is a TypeScript open-source llm evals project by promptfoo. CLI and library for evaluating, testing, and red-teaming LLM apps. Side-by-side prompt comparisons in your CI. With 7,000 GitHub stars and 0forks, it's one of the LLM Evals tools worth knowing. You can download the source, browse it on GitHub, or read the full setup guide in the README below.

README

Promptfoo: LLM evals & red teaming

promptfoo is a CLI and library for evaluating and red-teaming LLM apps. Stop the trial-and-error approach - start shipping secure, reliable AI apps.

Website · Getting Started · Red Teaming · Documentation · Discord

Promptfoo is now part of OpenAI. Promptfoo remains open source and MIT licensed. Read the company update.

Quick Start

Requires Node.js >=22.22.0 for npm and npx usage. Node.js 24 LTS is recommended; see the runtime support guide.

npm install -g promptfoo
promptfoo init --example getting-started

Also available via brew install promptfoo and pip install promptfoo. You can also use npx promptfoo@latest to run any command without installing.

Most LLM providers require an API key. Set yours as an environment variable:

export OPENAI_API_KEY=sk-abc123

Once you're in the example directory, run an eval and view results:

cd getting-started
promptfoo eval
promptfoo view

See Getting Started (evals) or Red Teaming (vulnerability scanning) for more.

What can you do with Promptfoo?

Test your prompts and models with automated evaluations
Secure your LLM apps with red teaming and vulnerability scanning
Compare models side-by-side (OpenAI, Anthropic, Azure, Bedrock, Ollama, and more)
Automate checks in CI/CD
Review pull requests for LLM-related security and compliance issues with code scanning
Share results with your team

Here's what it looks like in action:

It works on the command line too:

It also can generate security vulnerability reports:

Why Promptfoo?

Developer-first: Fast, with features like live reload and caching
Private: LLM evals run 100% locally - your prompts never leave your machine
Flexible: Works with any LLM API or programming language
Battle-tested: Powers LLM apps serving 10M+ users in production
Data-driven: Make decisions based on metrics, not gut feel
Open source: MIT licensed, with an active community

Learn More

Contributing

We welcome contributions! Check out our contributing guide to get started.

Join our Discord community for help and discussion.

promptfoo — FAQ

What is promptfoo?

CLI and library for evaluating, testing, and red-teaming LLM apps. Side-by-side prompt comparisons in your CI.

Is promptfoo free to use?

Yes. promptfoo is open-source and free. You can browse the source on GitHub or download it here at no cost — no signup required.

How do I install and run promptfoo?

Download the latest source as a ZIP from GitHub (or clone the main branch), then follow the setup steps in the README below.

What language is promptfoo written in?

promptfoo is primarily written in TypeScript.

How popular is promptfoo?

promptfoo has 7,000 stars and 0 forks on GitHub — a useful signal of how actively it's maintained and trusted by developers.

More LLM Evals skills

See all LLM Evals →

OpenAI Evals⭐ 16,000

OpenAI's framework for benchmarking LLMs and an open-source registry of evals. Industry-standard test harness.

Langfuse⭐ 11,000

Open-source LLM engineering platform — tracing, prompt management, evaluations, datasets, playground.

Arize Phoenix⭐ 6,000

Open-source LLM observability — traces, evaluation, datasets, retrieval debugging. OpenTelemetry-native.