My Skill Lab
Artificial IntelligenceIntermediate

Speech AI: Transcription and Voice Generation

Whisper, real-time speech-to-text and natural text-to-speech.

Taught by Rohan Maharjan · ML Engineer

What you'll learn

  • Explain how modern speech recognition works and measure its accuracy with word error rate
  • Transcribe audio files locally with Whisper and faster-whisper, with timestamps and voice activity filtering
  • Choose between cloud speech-to-text APIs on accuracy, latency, language support and cost
  • Build a real-time streaming transcription client with interim results and endpointing
  • Add speaker diarisation and generate structured meeting notes from a transcript
  • Generate natural speech with text-to-speech APIs and assemble a low-latency voice assistant

About this course

Speech AI: Transcription and Voice Generation is an intermediate, code-first course for developers who want to add speech to their products: transcribing calls and meetings, streaming speech-to-text in real time, generating natural voices, and assembling a complete voice assistant. Voice agents and meeting tools are driving demand for these skills, and the tooling has matured quickly. You will run open-source Whisper locally with faster-whisper, compare cloud speech-to-text APIs from OpenAI, Deepgram and others, build a streaming transcription client over WebSockets, add speaker diarisation with pyannote and turn transcripts into meeting notes, handle multilingual audio and translation, generate speech with modern text-to-speech APIs and open-source models, and finish by wiring speech-to-text, a language model and text-to-speech into a low-latency voice assistant. You should be comfortable with Python and asynchronous code; no prior audio or signal-processing knowledge is needed.

Course content

8 lessons
  1. 1How Speech AI Works: From Waveform to WordsAudio as numbers, how encoder-decoder models turn sound into text, and how to measure transcription accuracy.Preview
  2. 2Batch Transcription With WhisperRun Whisper locally with faster-whisper, choose a model size, get timestamps, and prepare audio for the best results.
  3. 3Cloud Speech-to-Text APIs and Choosing OneCompare hosted transcription services on accuracy, features, languages and cost, and call one from Python.
  4. 4Real-Time Streaming TranscriptionStream audio over WebSockets, handle interim and final results, and tune endpointing for live captions and voice agents.
  5. 5Speaker Diarisation and Meeting NotesIdentify who spoke when with pyannote, align speakers with transcripts, and generate structured meeting summaries.
  6. 6Multilingual Speech and TranslationDetect languages, transcribe non-English and code-switched audio, translate speech, and handle low-resource languages honestly.
  7. 7Text-to-Speech: Natural Voices and Streaming AudioGenerate expressive speech with modern TTS APIs and open-source models, control style, and stream audio for low latency.
  8. 8Building a Voice Assistant End to EndWire streaming speech-to-text, a language model and text-to-speech into a responsive assistant, and know when speech-to-speech models fit.