Skip to content

Python: [Feature]: Option in create_harness_agent to return only the agent's answer from a looping run (result.text currently includes the loop's prompts) #10209

Description

Problem

When create_harness_agent is given loop_should_continue, a non-streaming run returns the loop's aggregated response: every iteration's messages plus the injected next_message prompts as user messages. That is documented on AgentLoopMiddleware. But AgentResponse.text is "the concatenated text of all messages", so the usual way to read an answer returns the loop's own prompts mixed into the agent's replies:

Answer 2.Progress so far:

  • Answer 2.You still have 1 open todo item(s) that must be addressed before you can finish:
  • Fix the inverter

Continue working through them now. Mark each todo complete as you finish it, and only stop once every todo item is complete.Answer 3.

#10067 and its fix #10093 add a newline between message texts. That solves the run-together text, but not this: the prompts are user messages with text, so they stay in .text. With the #10093 join, the same run reads:
Answer 2.
Progress so far:

  • Answer 2.
    You still have 1 open todo item(s) that must be addressed before you can finish:
  • Fix the inverter

Continue working through them now. Mark each todo complete as you finish it, and only stop once every todo item is complete.
Answer 3.

AgentLoopMiddleware already has the switch for this (return_final_only=True), but create_harness_agent builds the loop without it and does not expose it. A hand-built loop passed through middleware= is not an equivalent substitute: the factory deliberately places its loop outermost, ahead of ToolApprovalMiddleware, while middleware= entries are appended after it.

Use Case and Evidence

Night Shift, an open-source operations agent built on the harness (https://github.com/mcekikj/nightshift-ms-harness), runs its evaluation non-streaming with the todo loop on. Saving response.text as the agent's final answer captured the loop prompts as well, so the evaluation now reads the last assistant message instead (https://github.com/mcekikj/nightshift-ms-harness/blob/main/evaluate.py#L68-L69): "a looped run's response also carries earlier iterations and injected nudges". Any application that stores, shows or forwards result.text from a looping harness agent gets the same mix: chat replies, ticket comments, evaluation records.

Reproduced offline with a mock transport on agent-framework-core 1.24.0 and 1.25.0: result.messages holds assistant, user ("Progress so far: ..."), user ("You still have 1 open todo item(s) ..."), assistant, and result.text joins all four.

pip install agent-framework-core agent-framework-openai httpx (no credentials needed: a mock transport plays the Responses API)

import asyncio
import inspect
import json

import httpx
from openai import AsyncOpenAI

from agent_framework import AgentModeProvider, TodoProvider, create_harness_agent, todos_remaining, todos_remaining_message
from agent_framework.openai import OpenAIChatClient

replies = 0

def response(item: dict) -> dict:
return {"id": f"resp_{replies}", "object": "response", "created_at": 0, "status": "completed", "model": "mock",
"output": [item], "parallel_tool_calls": True, "tool_choice": "auto", "tools": [],
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2,
"input_tokens_details": {"cached_tokens": 0}, "output_tokens_details": {"reasoning_tokens": 0}}}

def handler(request: httpx.Request) -> httpx.Response:
global replies
replies += 1
if replies == 1: # the model plans one todo
item = {"type": "function_call", "id": "fc_1", "call_id": "call_1", "status": "completed",
"name": "todos_add", "arguments": json.dumps({"todos": [{"title": "Fix the inverter"}]})}
else: # then answers without completing it
item = {"type": "message", "id": f"msg_{replies}", "role": "assistant", "status": "completed",
"content": [{"type": "output_text", "text": f"Answer {replies}.", "annotations": []}]}
return httpx.Response(200, json=response(item))

async def main() -> None:
openai_client = AsyncOpenAI(base_url="http://mock/v1/", api_key="unused", max_retries=0,
http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)))
todo = TodoProvider()
agent = create_harness_agent(
client=OpenAIChatClient(model="mock", async_client=openai_client),
agent_instructions="Operations agent.",
todo_provider=todo,
mode_provider=AgentModeProvider(default_mode="execute"),
loop_should_continue=todos_remaining(looping_modes=["execute"]),
loop_next_message=todos_remaining_message,
loop_max_iterations=2,
disable_file_memory=True,
disable_web_search=True,
default_options={"store": False},
)
session = agent.create_session()
result = await agent.run("Fix the inverter.", session=session)

print("result.text:", repr(result.text))
print("result.messages:")
for m in result.messages:
    if m.text:
        print(f"  {m.role}: {m.text[:72]!r}")
items = todo.store.load_items(session, source_id=todo.source_id)
items = await items if inspect.isawaitable(items) else items
print("finish_reason:", result.finish_reason, "| open todos:", sum(not i.is_complete for i in items))

asyncio.run(main())


Output on agent-framework-core 1.25.0:
result.text: 'Answer 2.Progress so far:\n- Answer 2.You still have 1 open todo item(s) that must be addressed before you can finish:\n- Fix the inverter\n\nContinue working through them now. Mark each todo complete as you finish it, and only stop once every todo item is complete.Answer 3.'
result.messages:
assistant: 'Answer 2.'
user: 'Progress so far:\n- Answer 2.'
user: 'You still have 1 open todo item(s) that must be addressed before you can'
assistant: 'Answer 3.'
finish_reason: stop | open todos: 1

Desired Outcome

A looping harness agent can be configured from create_harness_agent to return only its answer, so result.text is the agent's reply rather than a mix of replies and the loop's internal prompts.

Alternatives Considered

  • Reading the last assistant message instead of result.text, which is what Night Shift does. It works, but every caller has to know about it, and it drops answers from earlier iterations.
  • Passing AgentLoopMiddleware(..., return_final_only=True) through middleware=. Not equivalent: that middleware sits inside ToolApprovalMiddleware, while the factory's loop sits outermost so that each iteration is a full run including tool approval and the loop's approval escape hatch returns pending approvals to the caller.
  • Streaming needs nothing: the prompts arrive as user updates, as documented, and stream consumers can filter by role (Night Shift's console does: https://github.com/mcekikj/nightshift-ms-harness/blob/main/nightshift/console.py#L66-L69). The non-streaming result.text gives no such signal.

Proposed Direction

  1. A loop_return_final_only: bool = False parameter on create_harness_agent, passed through to AgentLoopMiddleware(return_final_only=...). Keeping False as the default preserves today's behavior.
  2. Separately, worth deciding together with Python: Separate message texts with a newline in AgentResponse.text #10093: whether AgentResponse.text should include the loop's injected user prompts at all, since they can already be told apart by role.

Scope and Non-Goals

Language

Python

Contribution Intent

I can help with implementation or testing

Acknowledgements

  • I searched existing issues and did not find a duplicate.
  • I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.

Activity

  1. added
    pythonUsage: [Issues, PRs], Target: Python
    triageUsage: [Issues], Target: All issues that still need to be triaged
    on Nov 9, 2026
  2. changed the title [-][Feature]: Option in create_harness_agent to return only the agent's answer from a looping run (result.text currently includes the loop's prompts)[/-] [+]Python: [Feature]: Option in create_harness_agent to return only the agent's answer from a looping run (result.text currently includes the loop's prompts)[/+] on Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

harness[Issues, PRs], Target: harness-level itemspythonUsage: [Issues, PRs], Target: PythontriageUsage: [Issues], Target: All issues that still need to be triaged

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions