End-to-end versus cascading AI interpreting

· Machine Interpreting End-to-end

Until recently, end-to-end speech translation still felt like a distant research topic. The idea was elegant: instead of moving from speech recognition to translation to speech synthesis, as in the current architecture, why not build a system that listens in one language and speaks in another directly?

We all remember the pioneering work of Meta's Seamless project and Google's Translatotron work. Today, however, this technology is no longer a laboratory curiosity, a paper or a demo. It is entering the real world, being used in web-conferencing platforms such as Google Meet and in applications such as Interpreter24.

What matters is not the fact that these new systems translate speech in real time. Cascaded systems have done that for years. The important shift lies in the architecture and the consequences it brings.

The traditional cascading approach consists of at least three steps: speech recognition, translation using neural machine translation or, more recently, large language models, and text-to-speech synthesis. Each component can be optimized, replaced, instructed and monitored. That is powerful, especially in commercial contexts where terminology, language coverage, workflow control and auditability matter. But cascades are difficult to orchestrate. Segmentation, timing, reformulation and speech synthesis have to be coordinated with great care. Many commercial applications simply put them together, one after another, without any fine-tuning, and the results are at best questionable.

Excellent individual components do not automatically produce an excellent interpreting experience.

End-to-end models attack the problem differently. They learn the relationship between incoming speech and outgoing translated speech more directly. In doing so, they can also learn something that is central to simultaneous interpreting: when to speak.

That question is often underestimated. Simultaneity is not simply a matter of speed. A system must decide when to wait, when to begin, when to accelerate, and how much it can safely anticipate. In cascading models, these rules are usually defined explicitly, from basic punctuation-based segmentation (translate at the end of a sentence) to syntactic or semantic analysis (translate after a particular construction). They tend to be rather rigid. They work, but not always elegantly. And elegance in language processing always translates into better results. This is where end-to-end approaches become genuinely interesting. They do not merely shorten the pipeline by replacing three steps with one; this also reduces latency, one of the main factors affecting a system’s usability. They change the nature of the timing problem by letting the model learn and decide autonomously.

Beyond reducing latency and supporting true simultaneity, end-to-end models do something else of immense importance: they use the speaker’s prosody to infer meaning and, at least in principle, adapt their rendition accordingly. This information is lost in cascading models. If a speaker sounds hurried or angry, for example, this can carry over into the translated speech. These models can even reproduce the speaker’s vocal features, something that is very complex to orchestrate in cascading models, where simply matching the original speaker’s pace is already a challenge.

Let's listen to an example of AI interpreting:

In the video, the same speech is translated with two different architectures: a cascading pipeline that I fine-tuned for simultaneity, and an end-to-end pipeline. Listening to and comparing these systems is quite interesting. The end-to-end pipeline is smoother. The strongest impression produced by it does not always come from perfect wording. It comes from rhythm, spontaneity and vocal naturalness. A translated voice that contains small irregularities, changes of pace and traces of hesitation can feel more convincing than a perfectly polished synthetic reading. In speech translation, perceived naturalness is not a decorative layer. It changes how the translation is received. Latency is also very low, about 2 seconds (the value varies because the model makes the timing decisions). The end-to-end pipeline emerges as a clear winner here.

Will cascaded systems become obsolete? I don't think so, at least not in the immediate future. First of all, this level of naturalness and latency can also be achieved with cascading models through better fine-tuning and engineering. However, this requires substantial work and people who can think outside the box when fine-tuning these systems. As far as I can see, not many commercial players are investing time in improving their cascading pipelines at the moment. Secondly, for many use cases, cascades remain more transparent, more controllable and more adaptable. They can cover more language combinations, integrate terminology preferences more easily through glossaries, use contextual material, including visual information, and allow different parts of the system to be tuned independently.

The gap is narrowing, but it has not yet closed.

With end-to-end models performing so well, how do they compare with human experts?

The gap is narrowing, but it has not yet closed. Depending on which aspects we consider, closing it may still take a long time. And there is also a chance that improvements will hit a wall at some point. Furthermore, we still lack a framework that defines good performance and how to measure it. Without it, comparisons between cascading and direct models, and likewise between artificial and human expertise, will remain purely anecdotal.

A final prediction based on current trends: speech translation is improving fast. If technological development does not hit a hard wall (something that might happen, though we do not know when), it is plausible to anticipate that by around 2030, the performance of automated interpreting systems will be virtually indistinguishable from that of experts. Not accepting this very tangible prediction based on current trends is technological myopia. At the same time, human expertise is not condemned to oblivion even in such a scenario. As I argue in Interpreting without Intelligence, human experts will remain essential where accountability, relational trust, ethical judgment and participation in the communicative event are central. Their role might even be revived, since it is also a matter of deciding, as a society, which tasks we want to automate and which we prefer to leave to humans, even when AI can perform them (see also Bill Gates on the choices we face about AI).

The future will not be decided by one architecture defeating another, or by machines simply replacing humans.

I believe strongly enough in this coexistence of machines and human experts performing the same task—interpreting—at the highest level that I am still investing a considerable amount of effort in InterpretBank, my other software project alongside Interpreter24, precisely because the future will not consist solely of fully automated interpreting.

The future will not be decided by one architecture defeating another, or by machines simply replacing humans. It will be shaped by choices: when to use cascaded systems, when to use end-to-end systems, when to keep humans in the loop, and when the human role is not optional.