Whisper
by openaiOpenAI's open-source multilingual speech recognition model. State-of-the-art transcription, runs locally with the right hardware.
About Whisper
Whisper is a Python open-source voice & speech ai project by openai. OpenAI's open-source multilingual speech recognition model. State-of-the-art transcription, runs locally with the right hardware. With 75,000 GitHub stars and 0forks, it's one of the Voice & Speech AI tools worth knowing. You can download the source, browse it on GitHub, or read the full setup guide in the README below.
From the project README
Whisper
[[Blog]](https://openai.com/blog/whisper) [[Paper]](https://arxiv.org/abs/2212.04356) [[Model card]](https://github.com/openai/whisper/blob/main/model-card.md) [[Colab example]](https://colab.research.google.com/github/openai/whisper/blob/master/notebooks/LibriSpeech.ipynb)
Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech recognition, speech translation, and language identification.
Approach
[image: Approach]
A Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identification, and voice activity detection. These tasks are jointly represented as a sequence of tokens to be predicted by the decoder, allowing a single model to replace many stages of a traditional speech-processing pipeline. The multitask training format uses a set of special tokens that serve as task specifiers or classification targets.
Setup
We used Python 3.9.9 and PyTorch 1.10.1 to train and test our models, but the codebase is expected to be compatible with Python 3.8-3.11 and recent PyTorch versions. The codebase also depends on a few Python packages, most notably OpenAI's tiktoken for their fast tokenizer implementation. You can download and install (or update to) the latest release of Whisper with the following command:
This is an excerpt from the project's own README, quoted here for reference. Read the full README on GitHub →
Whisper — FAQ
What is Whisper?
OpenAI's open-source multilingual speech recognition model. State-of-the-art transcription, runs locally with the right hardware.
Is Whisper free to use?
Yes. Whisper is open-source and free. You can browse the source on GitHub or download it here at no cost — no signup required.
How do I install and run Whisper?
Download the latest source as a ZIP from GitHub (or clone the main branch), then follow the setup steps in the README below.
What language is Whisper written in?
Whisper is primarily written in Python.
How popular is Whisper?
Whisper has 75,000 stars and 0 forks on GitHub — a useful signal of how actively it's maintained and trusted by developers.
More Voice & Speech AI skills
See all Voice & Speech AI →High-performance C/C++ port of OpenAI Whisper. Real-time transcription on a Mac, no GPU required.
Deep-learning toolkit for text-to-speech, including 1100+ languages, voice cloning from 6-second samples, and many pretrained models.
Build real-time voice and video AI agents. Plug in any LLM + TTS + STT — LiveKit handles the WebRTC plumbing.