# Teaching Whisper to Forget Language *2026-05-19 ยท Barathan* > A small research note on speech therapy, IPA transcription, weak labels, and the strange idea of making an ASR model listen before it reads. > The whole idea can be compressed into one sentence: make Whisper forget about languages. For the last fifteen months, I have been thinking about a model that listens to human speech and transcribes it, but not into English, Turkish, Japanese, or any other writing system. Instead, I want it to transcribe speech into the International Phonetic Alphabet. Normal transcription asks: what words did this person say? Phonetic transcription asks a slightly stranger question: what sounds did this person actually make? That difference is small enough to explain in one sentence, but large enough to become a research problem, and an exceptionally hard one. ## The Clinical Origin The idea started from speech therapy. My sister is a speech therapist. At some point she told me about a test where a person reads a carefully chosen set of words and sentences out loud. The therapist listens, writes down what the person actually said in phonetic form, and uses that transcription to understand what might be going wrong in the person's speech. From there, they can decide what kind of therapy would help. This is important work, but the testing step is slow, tiring, and hard to master. Learning the phonetic alphabet is already difficult. Applying it consistently while listening to real people, in real time, is much harder. It is also hard to meet in the middle: even when two trained people listen to the same recording, they may not write exactly the same phonetic transcription. The task is supposed to be objective. In theory, the sound is the sound. But human speech does not arrive as neat symbols. It arrives with accents, timing, coarticulation, missing edges, habits, noise, and all the little accidents of a mouth trying to move through language. So the gold standard is often not a perfect truth. It is another human's best attempt, accepted as the target. That does not make the task useless. It makes it interesting. ## Why Whisper Felt Like the Right Place to Start When I first looked at the problem, the obvious model to think about was Whisper. Whisper is very good at speech recognition. It can listen to audio in many languages, infer the language, recover the words, and print those words in the right writing system. It does not only hear sound. Somewhere inside the process it climbs from sound into meaning, from meaning into language, and from language into text. For IPA transcription, I wanted to interrupt that climb. I did not want a model that hears a Turkish sentence, decides it is Turkish, and writes Turkish. I wanted a model that hears speech and stays closer to the sound. If two people in unrelated languages produce the same sound, the model should not care that the alphabets, words, or grammar around that sound are different. From a phonetic perspective, the languages are not the point. This is where the phrase in my head became simple: > make Whisper forget about language differences. Not forget speech. Not forget the acoustics. Not forget timing, voice, or pronunciation. Only forget the need to route every sound back into a language-specific writing system. ## The Data Problem The main obstacle is not mysterious. There is not enough good data. High-quality IPA transcription is manual, slow, and specialized. Medium-quality data is also scarce. Low-quality data is everywhere, but it usually comes from the same small family of tools: grapheme-to-phoneme libraries, text-to-speech pipelines, speech-to-text systems, and text-to-text conversion scripts. Those tools can generate a lot of labels, but they often repeat the same mistakes in the same way. That sameness matters. If a large noisy dataset contains many kinds of noise, a model sometimes has room to learn the stable pattern underneath. But if every weak label is wrong in the same direction, scale can become a way of teaching the error more confidently. This is why Whisper's own history is tempting but not directly transferable. Whisper was trained through large-scale weak supervision. The labels were imperfect, but the scale and diversity were enormous. For phonetic transcription, the available high-quality data is much smaller, and the available weak labels are less diverse. So the question becomes: can a model become better than the phonetic labels it was trained on, the way Whisper became better than the messy transcription data behind it? I do not know yet. But I still think the question is worth chasing. ## What I Tried Up to now, my work has been the most direct version of the idea: take Whisper as it is, keep the part that listens, and try to change what it writes. The results are not a finished product. I would not hand this to a clinic and say, here, this solves the testing problem. My trials so far are not successful enough to be used as a general tool. But they did keep the original intuition alive for me. Whisper already listens well. It already carries some understanding of speech across many languages. So maybe the problem is not teaching a model to hear from zero. Maybe the problem is teaching a model that already hears to answer in a different alphabet. ## The Part I Have Not Tried Yet The next idea is less supervised. If phonetic transcription is supposed to sit below language, then maybe languages can be used against themselves. Two utterances can belong to completely different languages and still contain some of the same sounds. Orthographic transcripts would be unrelated. Phonetic structure would overlap. That suggests a different way of thinking about data. Instead of only asking for clean audio-to-IPA pairs, maybe we can use the relationship between audio, ordinary transcription, generated phonetic labels, and cross-lingual sound similarity. Not because any one of these signals is perfect, but because disagreement between them may reveal where the weak labels are lying. I do not have the labeling system for this yet. I do not have the training recipe. I do not even know if this path will work. But this is the shape of the bet: Maybe the right abstraction is not language. Maybe it is sound. ## Why It Matters Beyond Speech Therapy The speech-therapy use case is enough reason to care. If a model could make the testing phase easier, faster, or more consistent, that would matter. But I do not think the idea stops there. IPA is interesting because it can behave like an abstraction layer between speech and ordinary text. It would not preserve everything in a voice. It would not magically store emotion, intention, or the full texture of a person speaking. But it can preserve something ordinary writing systems often throw away: how the words were actually pronounced. That layer could be useful outside speech therapy too. A text-to-speech system, for example, should not have to guess pronunciation from every writing system separately if a more sound-like representation already exists. An accent tool, a language-learning tool, or another speech model could use the same layer for different reasons. That is what I mean when I say I want to act as if languages do not exist for a moment. Not because languages are unimportant, but because there may be a useful layer underneath them where some spelling differences stop mattering. Most speech models land on written language. IPA is still text, but it points to a lower layer: the sounds underneath the spelling. --- References, implementation details, and the model checkpoints used in this work can be found here: - [`phonetic-whisper-mlx`](https://github.com/barathanaslan/phonetic-whisper-mlx) - [`phonetic-whisper-mlx-broad-multi`](https://huggingface.co/barathanasln/phonetic-whisper-mlx-broad-multi) - [`phonetic-whisper-mlx-narrow-en`](https://huggingface.co/barathanasln/phonetic-whisper-mlx-narrow-en)