Streaming + JSON Schema produces invalid JSON

[Xnyle]

LocalAI version:
localai/localai:latest-gpu-nvidia-cuda-13

Environment, CPU architecture, OS, and Version:
K8S
Model from https://huggingface.co/unsloth/Qwen3.5-9B-GGUF (but I don't think it's model related)

Given the v1/completions request below, chunks start with the following deltas. (two {{ at beginning).
This is not according to the schema (not even valid JSON) and then trips my application logic that can't handle broken JSON.
When setting stream to false this does not happen, response is valid JSON in this case.
BTW I don't know if all chunks are supposed to have the same id.

data: {"created":1776253697,"object":"chat.completion.chunk","id":"1763134f-d0b0-4cae-9900-9840c49de2bc","model":"Qwen3.5-9B-GGUF","choices":[{"index":0,"finish_reason":null,"delta":{"role":"assistant","content":null}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

data: {"created":1776253697,"object":"chat.completion.chunk","id":"1763134f-d0b0-4cae-9900-9840c49de2bc","model":"Qwen3.5-9B-GGUF","choices":[{"index":0,"finish_reason":null,"delta":{"content":"{"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

data: {"created":1776253697,"object":"chat.completion.chunk","id":"1763134f-d0b0-4cae-9900-9840c49de2bc","model":"Qwen3.5-9B-GGUF","choices":[{"index":0,"finish_reason":null,"delta":{"content":"{"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

data: {"created":1776253697,"object":"chat.completion.chunk","id":"1763134f-d0b0-4cae-9900-9840c49de2bc","model":"Qwen3.5-9B-GGUF","choices":[{"index":0,"finish_reason":null,"delta":{"content":" \""}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

data: {"created":1776253697,"object":"chat.completion.chunk","id":"1763134f-d0b0-4cae-9900-9840c49de2bc","model":"Qwen3.5-9B-GGUF","choices":[{"index":0,"finish_reason":null,"delta":{"content":"m"}}],"usage":{"prompt_tokens":0,"completion_tokens":0,"total_tokens":0}}

{
  "model": "Qwen3.5-9B-GGUF",
  "messages": [
    {
      "id": "fc92663c-ad66-4834-b0a8-94d017816552",
      "created_at": -1863054981,
      "completed_at": -1863054980,
      "role": "system",
      "content": "say hello",
      "metadata": {}
    },
    {
      "id": "3dad2b20-bb46-466e-ad39-2240a6c4f974",
      "created_at": -1863054977,
      "completed_at": -1863054847,
      "role": "user",
      "content": "hi?",
      "metadata": {}
    }
  ],
  "metadata": {
    "enable_thinking": "false"
  },
  "logprobs": false,
  "temperature": 0.7,
  "response_format": {
    "type": "json_schema",
    "json_schema": {
      "name": "result",
      "schema": {
        "$schema": "http://json-schema.org/draft-04/schema#",
        "type": "object",
        "properties": {
          "markdownAnswer": {
            "type": "string"
          },
          "references": {
            "type": "array",
            "items": {
              "type": "object",
              "properties": {
                "originIdentifier": {
                  "type": "string",
                  "pattern": "^[0-9]{1,4}-[0-9]{1,2}$"
                },
                "originMediaType": {
                  "type": "string"
                }
              },
              "required": [
                "originIdentifier",
                "originMediaType"
              ]
            },
            "minItems": 1
          }
        },
        "required": [
          "markdownAnswer",
          "references"
        ]
      }
    }
  },
  "stream": true,
  "reasoning": {
    "effort": "none"
  },
  "stream_options": {
    "include_usage": true
  },
  "reasoning_effort": "none"
}

[mudler]

that's fixed already in #9299, but not yet in a release (we are having CI issues right now, after that will cut a new release)

[flueckiger]

I notice the same behavior without a JSON schema: the first token gets multiplied in streaming mode, usually doubled but sometimes up to 5x. I've never seen this when streaming is off.

When reasoning is on, the duplication happens in the reasoning part. When reasoning is off, it happens in the normal output.

I tested the same model as @Xnyle and other models from Unsloth's Qwen3.5 series. I used LocalAI version 833b7e8 (Docker image), which should include #9299.

Could this be model-related? Or is there a different bug here?

[mudler]

I notice the same behavior without a JSON schema: the first token gets multiplied in streaming mode, usually doubled but sometimes up to 5x. I've never seen this when streaming is off.

When reasoning is on, the duplication happens in the reasoning part. When reasoning is off, it happens in the normal output.

I tested the same model as @Xnyle and other models from Unsloth's Qwen3.5 series. I used LocalAI version 833b7e8 (Docker image), which should include #9299.

Could this be model-related? Or is there a different bug here?

Did you upgrade the llama.cpp backend to development? That fix is present only in development backend images until a new version is cut

[flueckiger]

I notice the same behavior without a JSON schema: the first token gets multiplied in streaming mode, usually doubled but sometimes up to 5x. I've never seen this when streaming is off.
When reasoning is on, the duplication happens in the reasoning part. When reasoning is off, it happens in the normal output.
I tested the same model as @Xnyle and other models from Unsloth's Qwen3.5 series. I used LocalAI version 833b7e8 (Docker image), which should include #9299.
Could this be model-related? Or is there a different bug here?

Did you upgrade the llama.cpp backend to development? That fix is present only in development backend images until a new version is cut

Yes, I have upgraded the llama.cpp backend. /backends/cuda13-llama-cpp/metadata.json shows the digest:

sha256:c1cfab4d9ceb24423ba2c74b88469e592c1e51ea42d8b4d4a50a814375ea0780

Is that the correct version?

[flueckiger]

I notice the same behavior without a JSON schema: the first token gets multiplied in streaming mode, usually doubled but sometimes up to 5x. I've never seen this when streaming is off.
When reasoning is on, the duplication happens in the reasoning part. When reasoning is off, it happens in the normal output.
I tested the same model as @Xnyle and other models from Unsloth's Qwen3.5 series. I used LocalAI version 833b7e8 (Docker image), which should include #9299.
Could this be model-related? Or is there a different bug here?

Did you upgrade the llama.cpp backend to development? That fix is present only in development backend images until a new version is cut

Yes, I have upgraded the llama.cpp backend. /backends/cuda13-llama-cpp/metadata.json shows the digest:

sha256:c1cfab4d9ceb24423ba2c74b88469e592c1e51ea42d8b4d4a50a814375ea0780

Is that the correct version?

I apparently only upgraded the production version of the llama.cpp backend. Switching to the backend image quay.io/go-skynet/local-ai-backends:sha-833b7e8-gpu-nvidia-cuda-13-llama-cpp resolved the issue.

Thanks for the clarification!