Skip to content

When using follow=True option to read pod logs, the response stream sometimes returns many log lines at once #2426

Description

@gjohnston9

What happened (please include outputs or screenshots):
I'm using read_namespaced_pod_log, setting follow=True and _preload_content=False:

response = client.read_namespaced_pod_log(follow=True, ...)
for log in response.stream():
  ...

We expect each line to be valid JSON. However something changed recently, so that sometimes, a single item returned from the stream contains multiple log lines, so that we fail to parse it. When I run the exact same command but without setting follow=True, and call splitlines() on the whole response, each line is separated correctly.

With Follow=True: (removing a lot of details from these logs, keeping the structure and newlines)

> log_lines = get_logs_for_pods()
(Pdb) !l=next(log_lines)
(Pdb) p l
b'{"level":"info","ts":...,"caller":"...","msg":"...","rel":"...","tr":{"tid":"...","sid":"...","op":"..."}}\n{"level":"info","ts":...,"caller":"...","msg":"...","rel":"...","tr":{"tid":"...","sid":"...","op":"..."}}\n{"level":"info","ts":...,"caller":"...","msg":"...","rel":"snip","tr":{"tid":"...","sid":"...","op":"..."}}\n'

Without:

{"level":"info","ts":...,"caller":"...","msg":"...","rel":"...","tr":{"tid":"...","sid":"...","op":"..."}}

What you expected to happen:
Each item from the stream would be a single log line.

I am not 100% sure if it's expected for the SDK to behave this way or not, based on the documentation. I can split the lines myself, but it would be a lot more intuitive if each item were a single log line.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kubernetes version (kubectl version):
Client Version: v1.30.2
Server Version: v1.32.4-gke.1698000
  • OS (e.g., MacOS 10.13.6): Sequoia 15.5
  • Python version (python --version): python 3.8.13
  • Python client version (pip list | grep kubernetes): 20.13.0

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    help wantedDenotes an issue that needs help from a contributor. Must meet "help wanted" guidelines.kind/bugCategorizes issue or PR as related to a bug.

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions