fix(swebench-multimodal): create output.report.json for consistency

[juanmichelini]

Problem

The swebench-multimodal evaluation was only creating report.json, while other benchmarks (swebench, commit0, gaia, etc.) create output.report.json. This inconsistency caused the push-to-index workflow to fail when trying to find output.report.json.

Evidence: GitHub Actions run 21077636459 failed with:

[push-to-index] Failed to find report.json: No output.report.json found in /tmp/tmpyebg1v7b/extracted/eval_outputs

Solution

After the multimodal evaluation completes, copy report.json to output.report.json to match the behavior of other benchmarks.

This follows the same pattern used in:

• benchmarks/swebench/eval_infer.py (lines 258-267)
• Other benchmark evaluation scripts

Testing

The fix ensures that the archive structure matches what the push-to-index script expects:

• ✅ Creates output.report.json alongside output.jsonl
• ✅ Maintains backward compatibility (original report.json still exists)
• ✅ Follows established patterns from other benchmarks

[openhands-ai]

Looks like there are a few issues preventing this PR from being merged!

• GitHub Actions are failing:
  • Pre-commit checks

If you'd like me to help, just leave a comment, like

@OpenHands please fix the failing actions on PR #331 at branch `fix-swebench-multimodal-report-filename`

Feel free to include any additional details that might help me get this PR into a better state.

_{^{You can manage your notification settings}}

[juanmichelini]

@OpenHands please fix the failing actions on PR #331 at branch fix-swebench-multimodal-report-filename

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary

I fixed the failing pre-commit check on PR #331 by adding the missing shutil import.

Issue

The pre-commit check was failing because the code used shutil.copy() on line 288 but the shutil module was never imported.

Fix

Added import shutil to the imports section of benchmarks/swebenchmultimodal/eval_infer.py.

Changes Made

• Added import shutil to the imports (line 14)
• Minor formatting adjustments applied by ruff formatter (trailing whitespace and line wrapping)

Verification

• ✅ ruff check passes
• ✅ ruff format passes
• ✅ pyright type checking passes
• ✅ Changes pushed to fix-swebench-multimodal-report-filename branch

The PR should now pass the pre-commit checks when CI runs again.

View full conversation

[simonrosenberg]

@OpenHands no defensive programming here: remove the if else

[openhands-ai]

I'm on it! simonrosenberg can track my progress at all-hands.dev

[openhands-ai]

Done! I've addressed the PR comment by removing the defensive if-else check for the report.json copy operation.

Changes made:

• Removed the if report_path.exists(): check and the else branch with the warning
• The code now directly calls shutil.copy() without checking if the file exists first

Final diff shows:

• Added shutil import
• Added 6 lines that copy report.json to output.report.json without defensive checks

The changes have been committed and pushed to the fix-swebench-multimodal-report-filename branch, updating PR #331.

View full conversation

[simonrosenberg]

@juanmichelini removed the defensive if:else so now the code fails if the report.json is not where it was supposed to be. I think that's OK?