feat(tts): text-to-speech via the OuteTTS + WavTokenizer pipeline

[vaiju1981]

Adds TextToSpeech — an AutoCloseable native type that synthesizes audio from text over llama.cpp's two-model OuteTTS pipeline (OuteTTS text-to-codes → WavTokenizer codes-to-speech vocoder), returning a 24 kHz mono 16-bit WAV byte stream.

Approach: the OuteTTS helpers are DERIVED from upstream, never hand-copied

cmake/generate-tts-upstream.cmake reads the pinned upstream tools/tts/tts.cpp at configure time, drops main(), gives the called helpers external linkage, extracts the two default-speaker literals, and writes build/tts_generated/tts_upstream_gen.cpp (never committed, regenerated on every configure). tts_engine.{h,cpp} is only the orchestration; tts_wav.hpp is our in-memory WAV writer.

What's included

• Native: generator + tts_upstream.h (declarations) + tts_engine.{h,cpp} (orchestration) + tts_wav.hpp, compiled into libjllama; 3 new JNI methods.
• Java: TextToSpeech with synthesize(text) and an explicit-sampling overload; GPU-offload constructor.
• Tests: test_tts_wav.cpp (the WAV writer) + TtsIntegrationTest (self-skips without GGUFs). The derived DSP is upstream's, covered end-to-end by the integration test.
• CI: both test GGUFs wired into the Linux x86_64 job so TtsIntegrationTest runs in CI.
• Docs: README "Text-to-Speech" section + system properties; CLAUDE.md "OuteTTS build-time extraction" section + architecture/test entries.

English number words are now expanded for speech (e.g. 3 → "three", via upstream's real process_text); non-English text is not romanized. Synthesis uses the built-in default speaker profile.

Verified end-to-end locally

OuteTTS-0.2-500M-Q4 + WavTokenizer on CPU: TtsIntegrationTest PASS; manual synthesis → valid 24 kHz mono 16-bit WAV, 2.91 s, genuine modulated speech (afinfo-validated). Builds + passes Java tests on every platform (Linux aarch64, Windows VS+Ninja, macOS Metal/no-Metal, Android ±OpenCL, manylinux CUDA).

---

Responding to the review feedback

1. DRY / no copy of tts.cpp (+ MIT attribution). Done via the build-time extraction above — tts_dsp.hpp and all copied helpers are gone; divergence is impossible (derived from the pinned source each build). The derived TU carries the upstream MIT/the llama.cpp authors banner.

Why not "add tts.cpp to target_sources" / the wrap-main() patch. Verified against b9739: even with main() guarded, every helper is static (internal linkage → uncallable cross-TU) and the pipeline + default-speaker literals live inside main() (compiled out by the guard) — so direct inclusion links but exports nothing usable, and speaker_from_file / audio_*_from_speaker aren't reachable either (also static). Hence the extraction route (the reviewer's own suggested fallback). It is read-only, so no patches/ entry is required.

3. ArchUnit. Fixed (12/12). The failure was pre-existing on main (the #266 LlamaModelBackend → value edge), not TextToSpeech, which sits in the root layer and needs no rule change.

Also fixed (pre-existing on main, inherited via rebase): SpotBugs findings in ContentPart/OpenAiRequestMapper (#266/#267) suppressed with the established rationale; PIT mutation gate restored to 100% by covering the ContentPart audio paths (#267) that lacked tests.

Deliberate follow-ups (not in this PR)

• Companion upstream llama.cpp PR to split tts.cpp into a library TU + thin CLI (so the generator could shrink/drop later) — the "ideal long-term" route; left as a follow-up since the extraction is self-contained.
• Upstream SPDX copyright line on committed files — intentionally not added, because no committed file contains upstream code (the derived TU does, and carries the banner). Provenance is referenced in the committed headers' comments.

Pre-existing, not addressable here

• FOSSA "License Compliance" flags the dependency tree; this PR changes no dependencies, so its findings are identical to main.

[bernardladenthin]

Thank you very much for the idea and the effort — TTS support via the OuteTTS pipeline is genuinely exciting and a great direction for this project.

However, before this can be merged, a cleanup phase is needed. Here are the issues to address:

1. Copyright / DRY violation: do not copy tools/tts/tts.cpp

A line-by-line comparison of tts_dsp.hpp and tts_engine.cpp against llama.cpp/tools/tts/tts.cpp shows that virtually every function — fill_hann_window, twiddle, irfft, fold, embd_to_audio, process_text, prompt_add, prompt_init, prepare_guide_tokens, the default speaker profile strings, and the codec token range 151672..155772 — is copied nearly verbatim. Only cosmetic changes were made (static → inline, the WAV writer refactored to return bytes).

This violates the DRY principle and creates a maintenance burden: every llama.cpp upgrade risks silent divergence between the copied code and the upstream implementation. It also raises a MIT attribution concern: the MIT licence requires preserving the original copyright notice, which is absent from the new files.

The right approach is to compile tools/tts/tts.cpp directly, exactly as server-context.cpp, server-queue.cpp etc. are already wired in via CMakeLists.txt. The functions needed (fill_hann_window, irfft, fold, embd_to_audio, …) live in the upstream file — include it in the build rather than duplicating it.

2. For anything that cannot be used directly: patches/ + upstream PR

If parts of tools/tts/tts.cpp need adaptation for the JNI embedding (e.g. making the WAV writer return bytes instead of writing a file, or extracting the pipeline into a callable function rather than main()), the correct workflow is:

1. Add a minimal *.patch file under patches/ (the existing cmake/apply-llama-patches.cmake machinery picks it up automatically).
2. Open a PR on the llama.cpp side proposing the clean API surface (e.g. a tts_synthesize() function with an output-buffer parameter) so the patch can eventually be dropped.

This keeps all modifications auditable and keeps the upgrade path clean.

3. ArchUnit tests are currently failing

Please check and fix the ArchUnit layering violations before requesting re-review. The TextToSpeech class likely needs to be placed in (or explicitly permitted by) the architecture rules.

---

Summary of what a revised PR should look like:

• CMakeLists.txt adds ${llama.cpp_SOURCE_DIR}/tools/tts/tts.cpp to the jllama target (or links the object, same pattern as the server files).
• tts_dsp.hpp and the duplicated functions in tts_engine.cpp are removed entirely; the JNI bridge calls into the upstream functions directly.
• Any necessary surface changes to tts.cpp (in-memory WAV output, callable API) are delivered as a patches/0002-tts-jni-api.patch with a companion llama.cpp PR.
• Copyright headers on new files reference the upstream origin.
• ArchUnit violations are fixed.

Happy to help design the patch or the upstream PR if that would be useful. Looking forward to a revised version!

---

Generated by Claude Code

[bernardladenthin]

One more thought on the fallback strategy: the strong preference is to simply add tools/tts/tts.cpp to the CMake compile phase — the same way server-context.cpp, server-queue.cpp etc. are already pulled in from the FetchContent tree. If the functions are in a linkable translation unit, there is nothing to copy at all.

If it turns out that tts.cpp cannot be included directly (e.g. because it defines a main() or has other link conflicts that make it unsuitable as a library TU), the next-best option before resorting to a manual copy would be a build-time extraction script: a small CMake configure_file step or a Python/shell script that runs during the configure or generate phase, reads the specific functions out of the upstream tts.cpp source file (which is already on disk via FetchContent), and writes a derived .cpp into the build tree. That way the "copy" is always derived mechanically from the exact upstream file at the pinned version, divergence is impossible, and upgrading llama.cpp automatically picks up any fixes. It is more complex than a direct include, but still infinitely better than a hand-maintained 1:1 duplicate.

That said — given that the existing server files (server-context.cpp etc.) are already compiled into libjllama without a main() conflict, it is very likely that the TTS functions can be extracted into a small helper header/source by the upstream PR mentioned above, making even the extraction script unnecessary. That route is worth trying first.

---

Generated by Claude Code

[bernardladenthin]

Following up on the earlier comment about the DRY principle and avoiding the hard copy of tools/tts/tts.cpp — here is a concrete proposal for how to include the upstream file directly.

Why direct inclusion is blocked

tools/tts/tts.cpp contains int main(int argc, char ** argv). Adding it to target_sources(jllama ...) would cause a multiple-definition linker error for main when building the shared library. This is the same reason tools/server/server.cpp is excluded from this project's CMakeLists.txt while server-context.cpp, server-queue.cpp, server-task.cpp etc. are included — those library TUs have no main().

Proposed solution: patch via patches/ mechanism

The project already has a patch mechanism (patches/ + cmake/apply-llama-patches.cmake) that applies git apply-compatible unified diffs to the llama.cpp source before it compiles. A small patch wrapping main() in a preprocessor guard solves the problem cleanly:

patches/0002-tts-wrap-main.patch

--- a/tools/tts/tts.cpp
+++ b/tools/tts/tts.cpp
@@ -312,6 +312,7 @@ static void prepare_guide_tokens(llama_model * model, const std::string & text,
 }
 
+#ifndef JLLAMA_SKIP_TTS_MAIN
 int main(int argc, char ** argv) {
     common_params params;
 
@@ -420,3 +421,4 @@ int main(int argc, char ** argv) {
 
     return 0;
 }
+#endif // JLLAMA_SKIP_TTS_MAIN

Then in CMakeLists.txt, replace the hand-copied tts_engine.cpp with:

target_sources(jllama PRIVATE
    ${llama.cpp_SOURCE_DIR}/tools/tts/tts.cpp
)
target_compile_definitions(jllama PRIVATE JLLAMA_SKIP_TTS_MAIN)

The JLLAMA_SKIP_TTS_MAIN compile definition is scoped to the jllama target only. If the upstream standalone tts executable is also in the CMake tree, it compiles normally (no define → main() present). The standalone tool is completely unaffected.

This removes tts_dsp.hpp, tts_engine.h, and tts_engine.cpp from the project entirely, replacing them with the single upstream source. The richer upstream API (speaker_from_file, audio_text_from_speaker, audio_data_from_speaker for JSON-based speaker profiles) also becomes available without any extra effort.

A companion PR to llama.cpp to properly split tts.cpp into a library TU + CLI would be ideal long-term (so the patch can eventually be dropped), but the patch is a complete, self-contained solution on its own.

The exact @@ -NNN @@ line numbers in the patch will need to match the actual file at the pinned b9739 tag — please adjust them accordingly when creating the patch file.

---

Generated by Claude Code

[vaiju1981]

@bernardladenthin , i will take a look at patch, that sounds more realistic, the other is looking at llama-box as alternate option. I will work on this first thing tomorrow after my work day.

[vaiju1981]

@bernardladenthin the PR is ready to re-review again.

[bernardladenthin]

ty, I'll check later