The Natural Statistics of Audiovisual Speech

When you watch someone speak, something is already happening before your brain does any work. The mouth and the voice are not two separate streams waiting to be stitched together. They are, mathematically, already locked to each other. That's not an assumption — it's what Chandrasekaran, Ghazanfar, and their colleagues set out to measure, and what they found is striking. The theoretical setup is this: brains are efficient because they exploit patterns already present in the world. If the environment reliably produces certain correlations, a sensory system that capitalizes on those correlations needs to do less work. We've known this for natural visual scenes and for sound. What nobody had done was characterize those statistics for audiovisual speech — the most important multisensory signal most humans encounter every day. The question Chandrasekaran and colleagues were really asking is: how much of speech perception is solved by the signal itself, before the brain has to actively bind anything? The first answer is the most direct. Mouth opening area and the acoustic envelope — the slow rise and fall of sound energy over time — are robustly correlated across speakers, languages, and contexts. This isn't a subtle effect. For a single sentence from the English GRID corpus, a database of 20 speakers, the correlation between mouth area and the wideband auditory envelope hit an R value of 0.74. Averaged across all 20 GRID speakers, the mean correlation was 0.45, with individual speakers ranging from 0.31 to 0.65. In the Wisconsin x-ray microbeam database, which contains longer prose passages from 15 subjects, the mean intact correlation was 0.28. Shuffled controls, where the temporal pairing between mouth and voice was randomized, landed near zero. The difference was highly significant, and crucially, it replicated in a spontaneous French corpus as well, with correlations of 0.22 and 0.29 for two speakers. Different languages, different speaking styles, same result: the mouth tracks the voice. To understand just how specific that coupling is, the team went beyond overall amplitude and looked at how mouth opening relates to particular frequency bands in the acoustic signal. This is where the finding gets sharper. Formants — the resonant frequencies of the vocal tract that give vowels their identity — are not uniformly predicted by mouth shape. Two spectral bands stand out. The first falls below one kilohertz, in the region of the first formant, around 300 to 800 hertz. The second, larger peak falls between roughly one and three-and-a-half kilohertz, overlapping the second and third formant region. In the GRID data, those peaks centered at 648 hertz and 2,900 hertz. The Wisconsin data showed peaks at 720 hertz and 2.44 kilohertz; the French data at 530 hertz and 2.62 kilohertz. Above 5 kilohertz, correlations essentially vanished. What this means is that the visible geometry of the mouth is not just tracking loudness. It's tracking the specific frequency structure that carries speech identity. Chandrasekaran and colleagues describe this using the phrase "amodal speech gesture" — the idea that mouth shape is a visual shadow of vocal tract acoustics. The same articulatory event that shapes the formants also shapes the mouth opening. The signal itself carries that redundancy. The brain doesn't have to construct it. Now add a third dimension: time. Both mouth movements and the acoustic envelope are rhythmically modulated, and they share the same dominant tempo. That tempo is between 2 and 7 hertz. If you think about how fast syllables come at you in normal speech — roughly three to five per second — you're already in that range. What's remarkable is that this 2 to 7 hertz band is also the territory of theta and delta cortical oscillations, the neural rhythms that the auditory cortex uses to track and segment speech. The jaw and the brain are literally keeping the same beat. The team discusses how this overlap enables neural entrainment — the idea that slow modulations in the speech signal lock or entrain cortical oscillations so the brain can carve continuous speech into processable chunks. They point to evidence that intelligibility degrades sharply when local time-reversed speech segments exceed about 130 milliseconds, and that auditory cortical tracking breaks down for speech rates faster than roughly 8 hertz. There's also a hierarchy of coupling: gamma-band neural activity — high-frequency bursts around 25 to 50 hertz — rides on the phase of theta oscillations, and theta amplitude in turn rides on the phase of slower delta oscillations around 1 to 2 hertz. The upshot is a nested system. The 2 to 7 hertz rhythm in the audiovisual speech signal feeds directly into the entry point of that hierarchy. The fourth finding is about timing asymmetry, and it adds a directional logic to everything else. Mouth movements don't just correlate with the voice. They come first. Chandrasekaran and colleagues measured what they call the time to voice — the interval between the first visible lip movement and the onset of vocalization — and found it consistently falls between 100 and 300 milliseconds across speakers and consonant types. They used the Wisconsin x-ray database for precision, with lip markers sampled at 146 frames per second. For word-initial bilabial consonants — sounds like p, m, and b, where the lips come together — the means were 195 milliseconds for p, 137 for m, 205 for b, and 240 for the labiodental f. In vowel-consonant-vowel contexts, those numbers shrank but stayed within the window: 127 to 188 milliseconds depending on consonant. This matters because 100 to 300 milliseconds is also the range within which human listeners tolerate audiovisual asynchrony without noticing it. The McGurk illusion — where watching someone mouth one syllable while hearing another changes what you perceive — stays robust with vision leading by up to about 240 milliseconds. Artificially constructed stimuli reveal asynchrony at differences as small as 20 milliseconds. Natural speech sits comfortably inside the tolerance zone, and it does so because the lip movement arrives early enough to be useful but not so early as to seem decoupled. The neural mechanism proposed is phase resetting. The visual head start could reset the phase of low-frequency oscillations in the auditory cortex so that when the acoustic signal arrives, it lands on a high-excitability peak and gets amplified. The brain, in this picture, is pre-tuned by the face before the voice even starts. Pull all four findings together and a coherent picture emerges. Mouth opening and the acoustic envelope are robustly correlated. Those correlations are formant-specific, concentrated in the first formant and second-to-third formant bands. Both mouth motion and voice envelope oscillate at a frequency between 2 and 7 hertz, matching cortical rhythms for syllabic processing. And visible lip motion consistently precedes vocal onset by 100 to 300 milliseconds, within perceptual tolerance and positioned to pre-tune auditory processing. Chandrasekaran and colleagues frame this with the phrase "reciprocally coupled." The outputs of the speaker are matched to the neural processes of the receiver. That matching is not accidental — it may reflect evolutionary and developmental pressure toward a system where the face and the voice are co-tuned for the brain that has to decode them. The visual signal doesn't supplement the acoustic one. It partially duplicates it, arrives earlier, and rides the same rhythmic structure. That redundancy is what lets you understand someone speaking in a noisy room when you can see their face, and why speech perception degrades so sharply when the visual signal is removed in difficult listening conditions. The natural statistics of audiovisual speech turn out to be tightly constrained. The coupling is real, it's spectrally specific, it's temporally rhythmic, and it has a consistent directional timing. The brain exploiting that structure isn't doing something clever. It's doing something obvious — given what the signal already contains. This lecture was created by ennepō. Go to https://ennepo.ai to Discover, Create and Follow the latest research in your field. Read when you can. Listen when you want to.