Speech, Audio Synthesis and Generative Video Pipelines
You can snap word times onto a 25 fps grid and write two WebVTT cues, convert an SRT comma to a WebVTT period, apply a loudness gain toward -16 LUFS, and stop a voice render when the written consent file fails.
- Level
- Navigator
- Time
- 24 h
- Prerequisites
- R06-101

Jump to
After this course you can
The student can build an automated audio/video processing pipeline that transcribes speech to frame-accurate WebVTT captions, normalises audio loudness, verifies signed voice-consent manifests, and exports compliant video containers.
Snap word times onto 40 ms frames
This lesson's video clock runs at 25 frames per second, so a frame boundary falls every 40 ms. Frame k covers the half-open stretch from 40k ms up to 40(k+1) ms. The speech-to-text step returns each word with a start and an end in whole milliseconds. Snap the start down and the end up: the snapped start is floor(start/40) times 40, and the snapped end is ceil(end/40) times 40. An end that already divides by 40 stays where it is. The cue then owns frames from snapped-start/40 inclusive up to snapped-end/40 exclusive.
The worked words arrive in order. the sits at 120–280, kettle at 300–740, and boils at 800–1180. the snaps to 120–280. 120/40 = 3 and 280/40 = 7, so the word owns frames 3, 4, 5, and 6, four frames. kettle snaps to 280–760. floor(300/40) = 7, and 7 times 40 = 280. 740/40 = 18.5, so the end climbs to 19 times 40 = 760. Those frames run from 7 inclusive to 19 exclusive, twelve frames. boils snaps to 800–1200. 1180/40 = 29.5, so the end climbs to 30 times 40 = 1200, and the frames run from 20 inclusive to 30 exclusive, ten frames.
Merge two neighboring snapped words when the gap, next start minus previous end, is 0 or below. The merged cue keeps the earlier start, the later end, and the two words joined by one space. A gap above 0 stays split. the ends at 280 and kettle starts at 280, so the gap is 0 and the cue becomes 120–760 with the text the kettle. That cue owns frames 3 through 18, and 18 - 3 + 1 = 16 frames. boils starts at 800. The gap is 800 - 760 = 40, so boils stays a second cue, 800–1200, ten frames. WebVTT writes a time as two-digit hours, two-digit minutes, two-digit seconds, a period, and three millisecond digits. The timing line puts one space, then -->, then one space, between the times. The worked timing lines are 00:00:00.120 --> 00:00:00.760 and 00:00:00.800 --> 00:00:01.200. The second end is 1200 ms, which is 1 second and 200 ms. The file opens with the line WEBVTT, then a blank line, then cue id 1, its timing line, its text, a blank line, cue id 2, its timing line, and its text. The millisecond fields 120, 760, 800, and 200 all divide by 40. A time of 00:00:00.100 fails that grid test, because 100/40 = 2.5.
SRT commas, order, and overlap
SRT uses the same clock and puts a comma before the three millisecond digits. The worked SRT timing lines are 00:00:00,120 --> 00:00:00,760 and 00:00:00,800 --> 00:00:01,200. Conversion keeps the cue ids and the cue text, changes each of those commas to a period, and writes WEBVTT plus a blank line at the top. The cue count stays 2. A header that already says WEBVTT is left as the single header. The grid test from the frame section still applies after the comma becomes a period.
The order check has three parts, and the file passes only when all three pass. Each cue end sits strictly above its own start. The next cue's start sits at or after the previous cue's end. Every millisecond field divides by 40. Cue ends are exclusive on the frame grid, so a following cue may start on the same instant the previous cue ends. Cue 1 runs from 120 to 760. A later cue that starts at 760 touches that end and passes the overlap part. A cue that starts at 720 collides with it: 760 - 720 = 40 ms of overlap, so the file fails. The worked second start is 800, and 800 sits after 760, so the worked pair passes. A file whose first line is a cue, with WEBVTT missing, fails before the times are read.
Loudness gain toward -16 LUFS
The export target is -16.0 LUFS, and the pass band is -17.000 through -15.000 inclusive. Integrated loudness follows ITU-R BS.1770. A K-weighting filter runs first. Blocks are 400 ms long and hop forward by 100 ms. For one mono block the block loudness is -0.691 + 10 times log10 of the mean square. An absolute gate drops every block below -70 LUFS. The relative gate sits 10 LU under the loudness of the blocks that survived the absolute gate, and it drops blocks below that line. When the surviving blocks share one loudness, the integrated loudness equals that shared number. Four blocks at -19.0 and one block at -72 keep the four and drop the -72. The integrated loudness printed by the fixture is therefore -19.0. You take that printed number and choose the gain.
The delta in dB is -16.0 minus the measured loudness. The linear gain is 10 raised to (delta/20), then rounded half away from zero to 4 decimals. The shift you record is 20 times log10 of that rounded gain, rounded half away from zero to 3 decimals. This lesson's delta table is the lookup. Delta +6.0 gives gain 1.9953 and shift +6.000. Delta +3.0 gives gain 1.4125 and shift +3.000. Delta +2.0 gives gain 1.2589 and shift +2.000. Delta +0.4 gives gain 1.0471 and shift +0.400. Delta -1.8 gives gain 0.8128 and shift -1.800. Two further gains appear only after a peak cap: 1.2500 shifts by +1.938, and 1.1111 shifts by +0.915.
The sample peak is the largest absolute sample in the file. Multiply it by the table gain. A product of at most 1 keeps that gain. A product above 1 replaces the gain with 1/peak cut after 4 decimals, with the leftover digits dropped rather than rounded up. Then read the shift from the matching capped row and add it to the measured loudness. Measured -19.0 and peak 0.50 give delta +3.0, gain 1.4125, and shift +3.000. The recorded loudness is -19.0 + 3.000 = -16.000, inside the band. The product is 0.50 times 1.4125 = 0.70625, so the gain stays 1.4125. Measured -22.0 and peak 0.90 give delta +6.0 and table gain 1.9953. The product is 0.90 times 1.9953 = 1.79577, above 1. Then 1/0.90 = 1.111111..., cut to 1.1111, and 0.90 times 1.1111 = 0.99999, which sits at or below 1. The shift is +0.915. The recorded loudness is -22.0 + 0.915 = -21.085. That sits outside -17.000 through -15.000, so the export stops and the record reads peak_limited.
Consent, planner frames, and the MP4 gate
A voice render runs only after a written consent manifest passes. The file carries speaker_id, voice_id, consent, signed_date, expires, and signature. The consent field has to be the string written. The job's speaker and voice have to match the file. The signature has length 64, and every character is a digit 0 through 9 or a lowercase letter a through f. Dates compare as eight-digit integers YYYYMMDD. The run date has to fall on signed_date or later, and on expires or earlier. Expires is the last valid day. This lesson's run date is 2026-04-15, written 20260415, and the job asks for speaker s1 and voice v3. Manifest m1 has consent written, speaker s1, voice v3, signed 20251101, expires 20261101, and a 64-character signature in that hex alphabet. 20251101 is on or before 20260415, and 20260415 is on or before 20261101, so m1 passes. A missing file stops on the consent step: 1 call, 0 text-to-speech calls, export blocked, reason missing. Manifest m3 matches m1 except consent is verbal, so m3 stops on that same step with reason consent. Manifest m4 matches m1 except expires is 20260414. 20260415 falls after 20260414, so m4 stops with reason date.
The text-to-speech planner estimates length before any audio exists. Count the words, count the letters, and skip the spaces. Milliseconds equal 350 times the words plus 30 times the letters. The frame count is the smallest whole number of 40 ms frames that covers that length. The script the kettle boils has 3 words and 14 letters. 350 times 3 + 30 times 14 = 1050 + 420 = 1470 ms. Thirty-six frames end at 36 times 40 = 1440 ms, and 1470 is 30 ms past 1440, so the plan takes 37 frames and reserves 37 times 40 = 1480 ms. A requested timeline of 50 frames is longer. The pad is 50 - 37 = 13 frames of silence after the reserved speech. The measured cues end at 1200 ms, inside both the 1480 ms reserve and the 50-frame request. Each frame at 48000 Hz holds 1920 samples per channel, because 40 ms times 48 samples per ms = 1920. Fifty frames hold 50 times 1920 = 96000 samples per channel, and the export uses 2 channels.
Lip-sync offset is the cue's first frame minus the mouth's first frame, and only a cue that carries a mouth mark is tested. The gate passes when the absolute offset is at most 2. The worked mouth for the kettle opens on frame 3, the cue starts on frame 3, and the offset is 0, so the gate passes. The mux writes MP4, video codec H.264, profile High, level 4.0, pixel format yuv420p, 1280 by 720, 25 fps, and 50 frames. Level 4.0 allows at most 8192 macroblocks in one frame and at most 245760 macroblocks in one second. A macroblock is 16 by 16 pixels. The worked frame holds (1280/16) times (720/16) = 80 times 45 = 3600 macroblocks, and 3600 times 25 = 90000 per second. Both sit under the level limits. Audio is AAC at 48000 Hz, 2 channels, with that PCM length of 96000 samples per channel. A text-to-video request of 1000 by 700 misses the 16-pixel grid, because 1000/16 = 62.5 and 700/16 = 43.75. Choose the multiple of 16 with the smallest distance, and on a tie choose the lower multiple. 1000 - 992 = 8 and 1008 - 1000 = 8, so the width is 992. 704 - 700 = 4 and 700 - 688 = 12, so the height is 704. Image-to-video steps run in this order: consent, grid, text-to-speech, speech-to-text, loudness, lip-sync, mux. A full pass is 7 calls. Text-to-video inserts one still-generation call after consent, so a full pass is 8 calls. The still-generation call returns the conditioning image, and the video step then emits the timeline's frame count at 25 fps. A failed step is the last call. The mux runs only after every step listed before it has passed.
One image-to-video job, from consent to mux
The still is 1280 by 720. The request is 50 frames. The script is the kettle boils. The measured words are the 120–280, kettle 300–740, and boils 800–1180. The measured loudness is -19.0 LUFS and the sample peak is 0.50. The mouth for the first cue opens on frame 3. The manifest is m1. The run date is 2026-04-15. The path is image-to-video, so there is no still-generation call.
Consent on m1 passes, and the run continues. The grid holds 80 by 45 macroblocks, 3600 in the frame and 90000 per second, inside level 4.0. The planner reserves 37 frames from 1470 ms, the pad to 50 is 13 frames, and the PCM length is 96000 samples per channel. Speech-to-text writes two cues, 00:00:00.120 --> 00:00:00.760 with the text the kettle, and 00:00:00.800 --> 00:00:01.200 with the text boils. The SRT form of the first timing line is 00:00:00,120 --> 00:00:00,760. The order check sees 800 after 760, and every millisecond field divides by 40. Loudness uses gain 1.4125, product 0.70625, and recorded loudness -16.000. The lip-sync offset is 0. The mux writes the MP4 line from the container section.
The record you can tick before you trust the file is: consent pass, cues 2, frames 50, pcm 96000, gain 1.4125, loudness -16.000, offset 0, macroblocks 3600, calls 7, export allowed. Those 7 calls are consent, grid, text-to-speech, speech-to-text, loudness, lip-sync, and mux.
Mark a text-to-video timeline on paper
The script is hot water. Speech-to-text returns hot at 200–360 and water at 400–900. The mouth for hot opens on frame 5, and the mouth for water opens on frame 14. The job is text-to-video at a requested 1000 by 700, held for 40 frames. Measured loudness is -18.0 LUFS and the sample peak is 0.80. The manifest matches m1, and the run date is 2026-04-15. Write the snapped cues and both timing lines. Write the planner milliseconds, the reserved frames, the pad, and the PCM length per channel. Write the corrected width and height, the macroblocks in one frame, and the macroblocks per second. Write the gain path and the recorded loudness. Write each lip-sync offset. Write the call count and say whether the mux runs.
hot snaps to 200–360 and water snaps to 400–920. The gap is 400 - 360 = 40, so the cues stay split. The timing lines are 00:00:00.200 --> 00:00:00.360 and 00:00:00.400 --> 00:00:00.920. hot owns frames 5 through 8, four frames. water owns frames 10 through 22, thirteen frames, because 920/40 = 23 and the end is exclusive. The planner uses 2 words and 8 letters: 350 times 2 + 30 times 8 = 700 + 240 = 940 ms. Twenty-three frames end at 920 ms, and 940 is 20 ms past that, so the reserve is 24 frames and 960 ms. The pad to 40 frames is 40 - 24 = 16. The PCM length is 40 times 1920 = 76800 samples per channel. The corrected size is 992 by 704. Macroblocks are 62 times 44 = 2728 in the frame, and 2728 times 25 = 68200 per second. Both sit under 8192 and 245760. Delta is -16.0 - (-18.0) = +2.0, so the table gain is 1.2589. The product is 0.80 times 1.2589 = 1.00712, above 1. The cap is 1/0.80 = 1.2500, the shift is +1.938, and the recorded loudness is -18.0 + 1.938 = -16.062, inside the band. The hot offset is 5 - 5 = 0 and passes. The water offset is 10 - 14 = -4, and the absolute value 4 fails the gate of 2. The run stops on lip-sync. The calls are consent, still, grid, text-to-speech, speech-to-text, loudness, and lip-sync, which is 7. The mux stays out. Export blocked, reason lip_sync.
Check questions
What are the two worked WebVTT timing lines for the kettle boils, and how many frames does the first cue cover?
00:00:00.120 --> 00:00:00.760 and 00:00:00.800 --> 00:00:01.200. The first cue covers frames 3 through 18, which is 16 frames.
Measured loudness -19.0 LUFS and sample peak 0.50. What gain is multiplied, what recorded loudness follows, and what is the peak product?
The gain is 1.4125. The recorded loudness is -16.000. The product is 0.70625, so the gain stays uncapped.
Measured loudness -22.0 LUFS and sample peak 0.90. What capped gain is used, what recorded loudness follows, and does the file export?
The capped gain is 1.1111. The recorded loudness is -21.085. The export stops, and the record reads peak_limited.
On run date 2026-04-15, what does a missing consent file do to the call count, and why does manifest m4 fail when expires is 2026-04-14?
The pipeline stops on the consent step: 1 call, 0 text-to-speech calls, export blocked, reason missing. m4 fails because 20260415 falls after 20260414.
Tick the record before the mux: written consent passed, cues on the 40 ms grid with no overlap, recorded loudness from -17.000 through -15.000, lip-sync offset at most 2 frames, and an H.264 frame inside the level 4.0 macroblock limits.
Final task
Submit a media pipeline script tested against frozen audio/video fixtures; automated ffprobe and parser tests verify -16 LUFS (±1 dB) loudness, valid WebVTT sync timestamps, blocked execution when a voice-consent manifest is missing, and exact MP4/H.264 output specs. (Optional Portfolio item: 60-second explainer film).
Terms in this course
This course has no terms of its own in the glossary yet. Terms from earlier courses are used here too.
The first lesson is open to read. The rest of the course is being written, and tests and diplomas come later. Join the waitlist