Repository navigation
Description
Problem
When create_harness_agent is given loop_should_continue, a non-streaming run returns the loop's aggregated response: every iteration's messages plus the injected next_message prompts as user messages. That is documented on AgentLoopMiddleware. But AgentResponse.text is "the concatenated text of all messages", so the usual way to read an answer returns the loop's own prompts mixed into the agent's replies:
Answer 2.Progress so far:
- Answer 2.You still have 1 open todo item(s) that must be addressed before you can finish:
- Fix the inverter
Continue working through them now. Mark each todo complete as you finish it, and only stop once every todo item is complete.Answer 3.
#10067 and its fix #10093 add a newline between message texts. That solves the run-together text, but not this: the prompts are user messages with text, so they stay in .text. With the #10093 join, the same run reads:
Answer 2.
Progress so far:
- Answer 2.
You still have 1 open todo item(s) that must be addressed before you can finish: - Fix the inverter
Continue working through them now. Mark each todo complete as you finish it, and only stop once every todo item is complete.
Answer 3.
AgentLoopMiddleware already has the switch for this (return_final_only=True), but create_harness_agent builds the loop without it and does not expose it. A hand-built loop passed through middleware= is not an equivalent substitute: the factory deliberately places its loop outermost, ahead of ToolApprovalMiddleware, while middleware= entries are appended after it.
Use Case and Evidence
Night Shift, an open-source operations agent built on the harness (https://github.com/mcekikj/nightshift-ms-harness), runs its evaluation non-streaming with the todo loop on. Saving response.text as the agent's final answer captured the loop prompts as well, so the evaluation now reads the last assistant message instead (https://github.com/mcekikj/nightshift-ms-harness/blob/main/evaluate.py#L68-L69): "a looped run's response also carries earlier iterations and injected nudges". Any application that stores, shows or forwards result.text from a looping harness agent gets the same mix: chat replies, ticket comments, evaluation records.
Reproduced offline with a mock transport on agent-framework-core 1.24.0 and 1.25.0: result.messages holds assistant, user ("Progress so far: ..."), user ("You still have 1 open todo item(s) ..."), assistant, and result.text joins all four.
pip install agent-framework-core agent-framework-openai httpx (no credentials needed: a mock transport plays the Responses API)
import asyncio
import inspect
import json
import httpx
from openai import AsyncOpenAI
from agent_framework import AgentModeProvider, TodoProvider, create_harness_agent, todos_remaining, todos_remaining_message
from agent_framework.openai import OpenAIChatClient
replies = 0
def response(item: dict) -> dict:
return {"id": f"resp_{replies}", "object": "response", "created_at": 0, "status": "completed", "model": "mock",
"output": [item], "parallel_tool_calls": True, "tool_choice": "auto", "tools": [],
"usage": {"input_tokens": 1, "output_tokens": 1, "total_tokens": 2,
"input_tokens_details": {"cached_tokens": 0}, "output_tokens_details": {"reasoning_tokens": 0}}}
def handler(request: httpx.Request) -> httpx.Response:
global replies
replies += 1
if replies == 1: # the model plans one todo
item = {"type": "function_call", "id": "fc_1", "call_id": "call_1", "status": "completed",
"name": "todos_add", "arguments": json.dumps({"todos": [{"title": "Fix the inverter"}]})}
else: # then answers without completing it
item = {"type": "message", "id": f"msg_{replies}", "role": "assistant", "status": "completed",
"content": [{"type": "output_text", "text": f"Answer {replies}.", "annotations": []}]}
return httpx.Response(200, json=response(item))
async def main() -> None:
openai_client = AsyncOpenAI(base_url="http://mock/v1/", api_key="unused", max_retries=0,
http_client=httpx.AsyncClient(transport=httpx.MockTransport(handler)))
todo = TodoProvider()
agent = create_harness_agent(
client=OpenAIChatClient(model="mock", async_client=openai_client),
agent_instructions="Operations agent.",
todo_provider=todo,
mode_provider=AgentModeProvider(default_mode="execute"),
loop_should_continue=todos_remaining(looping_modes=["execute"]),
loop_next_message=todos_remaining_message,
loop_max_iterations=2,
disable_file_memory=True,
disable_web_search=True,
default_options={"store": False},
)
session = agent.create_session()
result = await agent.run("Fix the inverter.", session=session)
print("result.text:", repr(result.text))
print("result.messages:")
for m in result.messages:
if m.text:
print(f" {m.role}: {m.text[:72]!r}")
items = todo.store.load_items(session, source_id=todo.source_id)
items = await items if inspect.isawaitable(items) else items
print("finish_reason:", result.finish_reason, "| open todos:", sum(not i.is_complete for i in items))
asyncio.run(main())
Output on agent-framework-core 1.25.0:
result.text: 'Answer 2.Progress so far:\n- Answer 2.You still have 1 open todo item(s) that must be addressed before you can finish:\n- Fix the inverter\n\nContinue working through them now. Mark each todo complete as you finish it, and only stop once every todo item is complete.Answer 3.'
result.messages:
assistant: 'Answer 2.'
user: 'Progress so far:\n- Answer 2.'
user: 'You still have 1 open todo item(s) that must be addressed before you can'
assistant: 'Answer 3.'
finish_reason: stop | open todos: 1
Desired Outcome
A looping harness agent can be configured from create_harness_agent to return only its answer, so result.text is the agent's reply rather than a mix of replies and the loop's internal prompts.
Alternatives Considered
- Reading the last assistant message instead of result.text, which is what Night Shift does. It works, but every caller has to know about it, and it drops answers from earlier iterations.
- Passing AgentLoopMiddleware(..., return_final_only=True) through middleware=. Not equivalent: that middleware sits inside ToolApprovalMiddleware, while the factory's loop sits outermost so that each iteration is a full run including tool approval and the loop's approval escape hatch returns pending approvals to the caller.
- Streaming needs nothing: the prompts arrive as user updates, as documented, and stream consumers can filter by role (Night Shift's console does: https://github.com/mcekikj/nightshift-ms-harness/blob/main/nightshift/console.py#L66-L69). The non-streaming result.text gives no such signal.
Proposed Direction
- A loop_return_final_only: bool = False parameter on create_harness_agent, passed through to AgentLoopMiddleware(return_final_only=...). Keeping False as the default preserves today's behavior.
- Separately, worth deciding together with Python: Separate message texts with a newline in AgentResponse.text #10093: whether AgentResponse.text should include the loop's injected user prompts at all, since they can already be told apart by role.
Scope and Non-Goals
- Separators between message texts are handled by .NET: Python: [Bug]: AgentResponse.text joins message texts without a separator #10067 and Python: Separate message texts with a newline in AgentResponse.text #10093.
- No change to streaming, which already behaves as documented.
- No change to the default of AgentLoopMiddleware itself.
- Telling the caller why the loop stopped (finished, or loop_max_iterations reached) is tracked in Add signal-based completion detection and typed exit reasons to AgentLoopMiddleware #8860. In the repro above, the capped run reports finish_reason: stop with one todo still open.
Language
Python
Contribution Intent
I can help with implementation or testing
Acknowledgements
- I searched existing issues and did not find a duplicate.
- I will wait for explicit maintainer agreement before starting implementation of a non-trivial change.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
[-][Feature]: Option in create_harness_agent to return only the agent's answer from a looping run (result.text currently includes the loop's prompts)[/-][+]Python: [Feature]: Option in create_harness_agent to return only the agent's answer from a looping run (result.text currently includes the loop's prompts)[/+]