No one should be shut out of a conversation because the room was too loud.

VisionScribe Research (VS R) is a research project building visual speech recognition (VSR) technology, artificial intelligence (AI) that decodes what someone is saying by reading the movement of their lips, mouth, and jaw, without needing to hear their voice at all. The model is trained on real volunteers reading everyday words and phrases in front of a webcam, and it's designed to work as an accuracy layer for existing captioning tools, especially in loud or low-audio environments where speech recognition alone breaks down.

Visual speech recognition is the process of converting the visible movement of a person's mouth into text, using computer vision and machine learning instead of, or alongside, audio. It's sometimes called automated lip reading, and it works by tracking how the lips, teeth, jaw, and tongue move frame by frame and matching those movement patterns to the words most likely producing them.

Anyone who needs the words on screen, not just in their ears. In loud environments, a bar, an airport, a concert, it carries the sentence when audio can't be heard, and in quiet environments, a library, public transportation, or wherever others are sleeping, it carries the sentence when audio can't be turned on. It also helps people who read a language more easily than they can follow it spoken, including non-native speakers and people learning to read, in a first language or a new one: a national study found 66% of English as a second language (ESL) students rated closed captions "very" or "extremely" helpful. Captioning apps, video conferencing platforms, and hearing-accessibility products are the intended buyers, since they can license the model as an added accuracy layer for the tools people already use.

Source: Linder, K. (2016). Student Uses and Perceptions of Closed Captions and Transcripts: Results from a National Study. Corvallis, OR: Oregon State University Ecampus Research Unit.

AI lip reading works by training a machine learning model on video of many different people saying the same words and phrases, so it learns the visual patterns, called visemes, that correspond to different sounds. Once trained, the model can take a new video of someone's mouth moving and predict the words being spoken, the same underlying approach behind VisionScribe Research's model.

Yes. Visual speech recognition, a category of technology known as a silent speech interface, generates captions directly from video of a person's mouth movements, with no audio input required. This matters because standard speech-to-text tools rely entirely on audio, so background noise, crowd sound, music, or overlapping speakers drown out or distort the target voice before the software can isolate it. Reading mouth movement instead of sound means captioning keeps working in exactly the conditions where audio-only systems break down.

No. Volunteers only need a standard laptop or webcam to participate, there's no headset, specialized camera, or capture rig required. Participants sit in front of their own camera and read a short list of publicly available words and phrases, which is the same real-world camera setup the model is ultimately built to work with.

Participants review and electronically sign a consent agreement before any recording begins, and they can request deletion of their recordings at any time afterward. Full detail on how recordings are stored, used, and protected is covered in VisionScribe Research's privacy policy and consent agreement, which every volunteer agrees to before contributing data.