fix(gaia): extract answers from FinishAction messages

[neubig]

Problem

The GAIA evaluation was reporting very low resolution rates (~23.6%) for some models like jade-spark, significantly lower than expected. After investigation, the root cause was identified as an answer extraction failure, not a model capability issue.

Analysis

When analyzing the results from litellm_proxy-jade-spark-2862 (results):

• Total instances: 165
• Instances with NO answer extracted: 109 (66%)
• The evaluation log shows repeated failures:

  ERROR - Could not find agent output after 30 attempts and 1181.9s total wait time
  

The issue is that _extract_answer_from_history only looks for MessageEvent from the agent, but the agent frequently uses FinishAction directly with the solution in the message parameter, without first producing a separate MessageEvent.

Looking at actual conversation events:

• Successful instances: Have an agent MessageEvent containing <solution> tags
• Failing instances: Agent issues FinishAction with <solution>142</solution> in the message, but no separate MessageEvent - so the answer is never extracted

Solution

Update _extract_answer_from_history to check both:

1. Agent MessageEvent with llm_message content (existing behavior, kept as priority)
2. ActionEvent with FinishAction containing a message (new fallback)

Changes

• Added imports for ActionEvent and FinishAction
• Updated _extract_answer_from_history to also extract answers from FinishAction.message
• Maintained backward compatibility: MessageEvent is still preferred when present

Expected Impact

This fix should significantly improve GAIA resolution rates for models that use FinishAction directly instead of producing a separate MessageEvent. The actual model capability is likely much higher than reported - possibly closer to the ~50%+ seen with other models on GAIA.

Testing

• Verified Python syntax compiles correctly
• Logic follows the same pattern used in fake_user_response.py for detecting FinishAction

@neubig can click here to continue refining the PR

[neubig]

@OpenHands the checking logic has become very complex, could we simplify it to a single branch that catches both FinishAction and MessageAction?

[openhands-ai]

I'm on it! neubig can track my progress at all-hands.dev

[openhands-ai]

I've simplified the answer extraction logic as requested. The changes have been pushed to the PR branch.

Summary of Changes

Simplified the checking logic to a single branch that handles both MessageEvent and FinishAction:

• Before: Two separate if blocks checking for MessageEvent and FinishAction independently
• After: A single unified loop that:
  1. First filters for agent-sourced events (event.source == "agent")
  2. Then extracts text from either MessageEvent or FinishAction in one conditional block
  3. Returns with unified logging when text is found

The refactored code is cleaner and more maintainable:

• Single source check at the start of the loop
• Single extraction logic handling both event types
• Single return point with consistent logging

Checklist

• Simplified to single branch catching both FinishAction and MessageAction
• Maintained the same functionality (MessageEvent and FinishAction both handled)
• Updated docstring to reflect simplified logic
• Verified syntax compiles correctly
• Changes pushed to PR branch fix-gaia-finish-action-extraction

View full conversation

[juanmichelini]

testing it!

[neubig]

We tested this and I think it looks good.

[juanmichelini]

LGTM