Summary
In the foundry_hosting non-workflow response handler (ResponsesHostServer._handle_inner_agent), the three pre-model storage reads run as strictly-sequential awaits before the agent/model call:
session_storage.get(...) — loads the agent sessioncontext.get_input_items() — loads the caller's input itemscontext.get_history() — loads prior conversation history (AgentServer history mode)
These are independent storage round-trips with no data dependency between them — they are only combined afterward into [*history_messages, *input_messages] and session=…. Running them serially adds avoidable latency to the request critical path on warm / multi-turn requests.
Impact
A latency waterfall of the warm baseline showed the pre-model reads as a chain of serialized private-endpoint round-trips (~100-260ms each). On warm / multi-turn requests (where session + history actually hit storage), overlapping the independent reads can remove ~0.8-1s of serial wait in principle. Stateless single-turn requests are unaffected (history + session loads short-circuit without a round-trip).
Proposed fix
Load the input-items and history reads via asyncio.gather and kick them off (ensure_future) so they overlap the session load, while preserving result ordering and existing error semantics (read task cancelled if session prep fails).
Notes / measured result
End-to-end benchmark (multi-turn, 500 cycles @ conc 50, turn-2 = the request that loads session + history) on a hosted Foundry agent (uksouth) showed the change is within measurement noise at this metric (p50 2,799ms vs 2,803ms baseline; p99 4,902ms vs 5,305ms). The whole-request TTLB is dominated by model generation + per-turn persistence, so the read-path saving is not visible at p50 — but the change removes serial waits with no downside and is a correctness-preserving cleanup.
Fix PR
Addressed by #8363.
Summary
In the foundry_hosting non-workflow response handler (
ResponsesHostServer._handle_inner_agent), the three pre-model storage reads run as strictly-sequentialawaits before the agent/model call:session_storage.get(...)— loads the agent sessioncontext.get_input_items()— loads the caller's input itemscontext.get_history()— loads prior conversation history (AgentServer history mode)These are independent storage round-trips with no data dependency between them — they are only combined afterward into
[*history_messages, *input_messages]andsession=…. Running them serially adds avoidable latency to the request critical path on warm / multi-turn requests.Impact
A latency waterfall of the warm baseline showed the pre-model reads as a chain of serialized private-endpoint round-trips (~100-260ms each). On warm / multi-turn requests (where session + history actually hit storage), overlapping the independent reads can remove ~0.8-1s of serial wait in principle. Stateless single-turn requests are unaffected (history + session loads short-circuit without a round-trip).
Proposed fix
Load the input-items and history reads via
asyncio.gatherand kick them off (ensure_future) so they overlap the session load, while preserving result ordering and existing error semantics (read task cancelled if session prep fails).Notes / measured result
End-to-end benchmark (multi-turn, 500 cycles @ conc 50, turn-2 = the request that loads session + history) on a hosted Foundry agent (uksouth) showed the change is within measurement noise at this metric (p50 2,799ms vs 2,803ms baseline; p99 4,902ms vs 5,305ms). The whole-request TTLB is dominated by model generation + per-turn persistence, so the read-path saving is not visible at p50 — but the change removes serial waits with no downside and is a correctness-preserving cleanup.
Fix PR
Addressed by #8363.