Tontaube V1

Overview

Text-to-speech systems generally trade natural prosody against inference cost: higher perceptual quality tends to come with more GPU memory, a higher cost per hour of audio, and longer waits for the first sample. TontaubeV1 targets the lower-cost end of that range while remaining competitive on prosody in our audiobook-reading evaluation.

Speech is encoded by the hierarchical DualCodec representation at 12.5 Hz, which separates a semantic stream from successive acoustic refinements. The design assumes prosodic structure is largely established once the semantic stream exists, and allocates capacity accordingly: a Qwen3-1.7B-derived transformer predicts that stream, and thereby the utterance duration, while three progressively smaller Qwen3-0.6B-derived transformers add the first three acoustic refinements.

Text is tokenized per character rather than by subword. Paired text and audio markers at shared positions support long-form generation with bounded context, and overlapping DualCodec reconstructions are mapped into the VibeVoice acoustic latent space and decoded causally, enabling streaming despite DualCodec's noncausal decoder. The model accepts up to one minute of reference audio for voice conditioning and is designed primarily for English and German, with additional multilingual support.

Full detail is in the technical report. This page summarizes it.

Audio Samples

Generations spanning the registers covered by the release.

Architecture

Four autoregressive predictors, run coarse to fine, over a 12.5 Hz discrete audio representation.

Speech representation

TontaubeV1 operates on the 12hz_v1 configuration of DualCodec, which encodes 24 kHz audio into eight residual vector-quantized streams at 12.5 Hz. DualCodec quantizes its first layer from self-supervised w2v-BERT-2.0 features, which carry phonetic and linguistic information; the remaining layers quantize acoustic residuals. We call the first the semantic stream and the others the acoustic streams. The semantic codebook holds 16,384 entries and each acoustic codebook holds 4,096.

The model retains the semantic stream and the first three acoustic streams, yielding 625 bit/s against 1,225 bit/s for the full stack. The four finest refinements are omitted because informal listening indicated diminishing returns from the later streams. At 12.5 Hz one minute of audio occupies 750 tokens, which lets the model retain substantial text and audio context.

Four-stage generation

Each stage gets its own network, sized independently, and the stages run in order without a later stage revising an earlier stream. CB0 reads the text and its reference-prompt stream and produces the semantic stream, stopping when it emits a boundary token; the number of tokens it produced fixes the utterance length. CB1, CB2, and CB3 then each produce exactly that many tokens for their own stream, conditioned on the text, their available prompt streams, and every stream already completed below them. The four streams share one timeline, fixed once by CB0.

Most published systems either interleave the codebooks into a single output stream with a delay pattern, or assign the residuals to one small shared per-frame module. Assigning a separate, independently sized model to each codebook costs four checkpoints and sequential execution within a chunk. In return the acoustic stages carry no state across chunk boundaries and can be scheduled separately at serving time. Early experiments showed better convergence under this factorization.

Stored parameter counts, summing every tensor element in the released safetensors checkpoints. The checkpoints keep Qwen's full embedding table; the documented input grammar can only reach a fraction of those IDs, so an implementation that enforces it can drop the unreachable rows — 681,479,168 parameters, roughly 1.4 GB at bf16 — for 2.19B effective parameters. The rows are kept in the release so later tuning can widen the control-tag vocabulary. DualCodec, VibeVoice, and the optional verbalizer are excluded.

Text handling

Spoken text is tokenized one character at a time, reusing IDs the inherited Qwen tokenizer already produces, so ordinary content occupies exactly one token per character and no merge crosses a character boundary. Two things follow: chunk sizes, lookahead windows, and text positions are measured in a unit that does not depend on neighbouring words; and pronunciation is learned over a few hundred character IDs rather than tens of thousands of sparsely observed subword types. Subword tokenization is confined to control syntax. Case is preserved.

The model expects text already in spoken form. Digits, dates, currencies, and abbreviations are not reliably pronounced from their written shape: 1984 must be supplied as nineteen eighty-four or one thousand nine hundred eighty-four depending on the intended meaning, and the model cannot reliably make that choice from the characters alone. A separately released English verbalizer is available. Keeping it separate means its output can be inspected and corrected before synthesis, and callers who need exact control can supply spoken-form text directly. Text in the other supported languages must be supplied already verbalized.

Training

All four predictors were trained exclusively with supervised fine-tuning on approximately 200,000 hours of paired speech and text across seven languages, predominantly from public-domain audiobook recordings and openly released speech corpora.

Long-Form Layout and Streaming

Positions

Positions are assigned to match when a token occurs rather than where it was serialized. The tenth semantic frame and the tenth acoustic frame describe the same moment of speech but can sit hundreds of token indices apart once flattened; giving them the same coordinate sets their relative offset under RoPE to zero whatever the layout, since RoPE depends only on coordinate differences. Text shares that timeline, so character positions stay close to those of the corresponding audio frames. The prompt streams overlay one another rather than running end to end, so four streams of P frames occupy P coordinates, not 4P.

Bounded context

TontaubeV1 treats prosodic context as predominantly local, so CB0 keeps one preceding chunk rather than the whole passage. For an internal chunk its text window holds the previous chunk, the current chunk, and a short lookahead into the next, while its audio context holds only the previous chunk's semantic tokens. Once a chunk completes, the oldest text–audio pair is discarded and the window moves on, so the transformer's input context stays bounded however long the passage grows. Paired <|text_split|> and <|audio_split|> markers keep the two clocks from drifting each time the window moves. The released splitter caps chunks at 350 characters, cutting after the last period, exclamation mark, question mark, semicolon, colon, or comma within the cap, and falling back to whitespace where none occurs.

Streaming reconstruction

DualCodec's decoder is not causal, so waveform samples near a cut depend on codec frames beyond it, and independently decoding chunks would introduce an audible seam at each boundary. TontaubeV1 joins the reconstructed windows in a latent space instead: overlapping DualCodec reconstructions are re-encoded by the VibeVoice acoustic tokenizer, the context-padded central frames of each window are kept, and a single causal VibeVoice decoder cache is carried across the joined sequence. Only VibeVoice's acoustic encoder and decoder are used, not its language model or diffusion head.

Streaming applies the same construction incrementally. By default the system first generates a 40-frame semantic prefix; five DualCodec frames are withheld at each unstable boundary, leaving 35 frames — 2.8 seconds — initially eligible for emission. As more frames arrive, the accumulated prefix is decoded and re-encoded, revising the unstable latent boundary region, and newly stable frames are committed at two-second boundaries. Audio therefore begins playing before generation of the utterance is complete, without a separately decoded waveform boundary at any generation boundary. More generally, this gives streaming from a noncausal codec without retraining it, at the cost of a second codec in the inference path.

Serving Performance

Measured on one NVIDIA GeForce RTX 5090 with weights resident and the process warmed; startup and model loading are excluded. The inference repository ships vLLM adapters for the four predictors.

• Time to first audio: approximately 200 ms for a single input text, on the streaming path.
• Single-stream throughput: end-to-end real-time factor of 0.08, about 12.5× faster than playback.
• Concurrent throughput: with eight texts generated concurrently, aggregate real-time factor of approximately 0.02, about 50× real time.

The 200 ms figure is measured on the streaming path; the real-time factors are separate non-streaming measurements.

Benchmark Results

Audiobook reading, pairwise LLM-as-a-judge

The frozen corpus contains 400 English book passages of 250–500 characters sampled from the PG-19 test split. TontaubeV1 renders them in audiobook mode at semantic sampling temperature 0.55 and acoustic temperature zero. For each passage, TontaubeV1 and the comparator synthesize the same reference text, and each waveform is independently normalized to −20 dBFS before judging.

Gemini 3.1 Pro Preview receives the reference text and both waveforms and judges two dimensions independently: prosody (rhythm, intonation, emphasis, pacing, naturalness) and word-by-word correctness. The prompt explicitly directs it to ignore voice identity and timbre as well as recording artifacts, codec artifacts, and overall sound quality. Each pair is judged exactly twice, once in each presentation order. A TontaubeV1 preference, tie, or comparator preference scores 1, ½, or 0, and the reported preference score is the mean over all 800 judgments — so 50% denotes parity. Uncertainty comes from bootstrap-resampling the 400 passages as paired clusters, so the order-swapped calls are not treated as independent.

TontaubeV1 preference score over 800 order-balanced judgments. The dashed line marks 50% parity. parity tags a comparison whose 95% bootstrap interval includes 50%.

Within this English audiobook-reading benchmark, prosody is therefore comparable to ElevenLabs Flash v2.5 and ahead of the other three systems. On correctness the interval includes parity against ElevenLabs and Fish Audio, while the scores against Gradium and Cartesia favor TontaubeV1. Note the high tie rates on correctness: on read prose the judge returns a tie on most pairs, so that axis separates these systems far less than prosody does.

The Fish Audio comparison uses the same frozen cloning reference for both systems. In the other three, TontaubeV1 uses that reference while the comparators use fixed provider voices. Voice identity and timbre are excluded from the judging rubric, but prosody is not fully separable from a cloned reference, so those three comparisons may favor TontaubeV1, which can inherit aspects of the reference recording's reading style.

Seed-TTS WER

On the 1,088 English zero-shot examples of the Seed-TTS evaluation set, at semantic sampling temperature 0.6, TontaubeV1 obtains 1.66% mean utterance-level WER with Whisper large-v3 transcription. For context, the original Seed-TTS paper reports the human ground-truth recordings on this set at 2.14% WER. Any system in this range is at the ceiling of what the benchmark can distinguish; residual error reflects ASR noise on natural accents and prosody more than synthesis mistakes.

Validity

This protocol follows the audio-language-model-as-judge approach of EmergentTTS-Eval, which reports a Spearman correlation of 0.905 between aggregate human and model-judge system rankings in its study. We adapt it to order-balanced pairwise comparisons on a fixed reading set, which gives a repeatable and scalable alternative to commissioning a listener panel for each comparison. We do not claim that this exact Gemini 3.1 protocol has been independently validated against human judgments, or that model judges are superior to human raters.

The benchmark measures English reading prosody and word-level correctness. It does not establish voice similarity, general sound quality, German or broader multilingual performance, long-form continuity, or streaming quality.

Languages and Styles

The control block accepts a language and a style label. Labels outside the released sets are not supported.

Accepted labels specify the input contract; they do not by themselves establish equal quality or complete coverage. In informal listening, German intonation is strong but phoneme realization is sometimes inaccurate. The remaining languages have not been checked by native speakers, so we report no conclusions about them.

User-supplied reference audio is optional and requires no transcript; the inference server uses the bundled Miles voice by default. Up to roughly 60 seconds are encoded by DualCodec; before serialization the prompt streams are jointly truncated to 750, 300, 150, or 100 frames for CB0 through CB3 — 60, 24, 12, and 8 seconds at 12.5 Hz.

Limitations and Safety

Technical limitations

• Autoregressive semantic generation can omit, repeat, or alter text, and can terminate too early or too late.
• Reference conditioning may transfer identity imperfectly or reproduce incidental recording properties.
• Long-form chunking can introduce discontinuities, and the serial four-stage factorization adds latency.
• The optional generative verbalizer can normalize incorrectly or alter wording.
• Training was weighted toward audiobook speech, so audiobook generation may be more reliable than conversational or agentic generation.

The system should be evaluated on the intended domain, language, speakers, text lengths, and deployment hardware before use.

Safety

Voice cloning can enable impersonation, fraud, nonconsensual synthesis, and misleading media. The release includes synthetic reference voices, generated by the model rather than recorded from speakers. Users remain responsible for consent on references they supply. Deployers should obtain permission for reference voices, authenticate callers, rate-limit and log access, disclose that generated audio is synthetic where appropriate, and maintain abuse-response procedures.

License and Release Boundary

The TTS weights are distributed under the Tontaube Community Model License 1.0, which is not an open-source license. They are free to use for research and qualifying commercial use subject to its revenue and service restrictions. Readers should consult the license for the terms that apply.

The optional verbalizer and the inference implementation are both distributed separately under the Apache License 2.0. Users must also comply with the notices and licenses applicable to Qwen3, DualCodec, VibeVoice, vLLM, and other third-party components used by the implementation.

Citation

Cremer, F. and Cremer, J. TontaubeV1: Streaming Text-to-Speech with Hierarchical Codec Modeling and Bounded Context. Tontaube, August 2026.

@techreport{cremer2026tontaubev1,
  title       = {TontaubeV1: Streaming Text-to-Speech with Hierarchical
                 Codec Modeling and Bounded Context},
  author      = {Cremer, Fritz and Cremer, Jonathan},
  institution = {Tontaube},
  year        = {2026},
  month       = {8},
  url         = {https://tontaube.ai/papers/tontaube-v1-technical-report.pdf}
}

Previous Release

TontaubeV0 was developed concurrently with Qwen3-TTS and released through an API rather than as weights. It served as the prototype for the design developed further here. Read the V0 model card →