Add PDF-aware preliminary investigation to arXiv research by Copilot · Pull Request #55644 · github/gh-aw

Copilot · 2026-08-25T04:42:55Z

The daily arXiv workflow previously screened papers from abstracts only and redacted arXiv links in generated discussions. This adds a best-effort PDF evidence pass before screening.

PDF investigation
- Imports the existing Kreuzberg MCP server.
- Adds preliminary-investigator to download and extract paper PDFs.
- Enriches screening and ranking with concise methodology/results notes.
- Continues with abstract-only processing if retrieval or extraction fails.

{"title":"...","abstract":"...","preliminary_note":"Technique and results found in the PDF"}

Network and output policy
- Allows arxiv.org and export.arxiv.org for PDF retrieval.
- Allows arXiv URLs in safe-output discussions.
Kreuzberg shared component
- Converts usage documentation from hidden HTML-comment content to normal Markdown.

Fixes [q] Add PDF-aware preliminary investigation sub-agent + allow arXiv URLs in reports #55400

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions · 2026-08-25T05:06:28Z

Hey @copilot-swe-agent 👋 — thanks for implementing the PDF-aware preliminary investigation enhancement to the daily arXiv researcher! This is a solid improvement that directly addresses the request in #55400.

What looks good:

The PDF investigation pass is well-integrated with graceful fallback if extraction fails — it won't break the pipeline.
Network policy is clearly defined (arxiv.org and export.arxiv.org allowed).
Kreuzberg MCP server integration leverages existing, vetted infrastructure.
The move of kreuzberg usage docs from HTML comments to Markdown is a security-positive change (blocks hidden-content scanner flags).
Clear, structured PR description with all implementation details.
Focused scope: three related files only.

This looks ready for review by maintainers. The changes are aligned with the project's agentic workflow patterns and directly implement a feature request from the core team discussion.

Generated by ✅ Contribution Check · copilot · auto · 56.2 AIC · ⌖ 6.18 AIC · ⊞ 9.3K · ◷

pelikhan · 2026-08-25T05:06:34Z

      # Excluded feature-flag-gated operations:
      # - "extract_structured"  # Requires liter-llm feature flag at build time
 ---
-<!--


@copilot keep the comments, and only expose a tiny amount of prompting.

Addressed in f328a86: retained the component documentation as comments and reduced the exposed prompt to the extract_file instruction.

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot

Pull request overview

Adds PDF-based evidence gathering to the daily arXiv research workflow before screening and ranking.

Changes:

Adds a preliminary PDF investigator using Kreuzberg.
Allows arXiv network access and discussion links.
Exposes Kreuzberg usage documentation and updates generated output.

Show a summary per file

File	Description
`.github/workflows/daily-arxiv-researcher.md`	Adds PDF investigation and enriched screening.
`.github/workflows/shared/mcp/kreuzberg.md`	Makes MCP documentation visible.
`.github/workflows/daily-arxiv-researcher.lock.yml`	Updates compiled workflow metadata and network policy.

Review details

Files reviewed: 3/3 changed files
Comments generated: 4
Review effort level: Balanced

 # Resolved workflow manifest:
 #   Imports:
 #     - shared/crush.md
+#     - shared/mcp/kreuzberg.md



+## Step 1b: Investigate Paper PDFs
+
+For each paper in `papers`, invoke the `preliminary-investigator` sub-agent with its full paper object. Merge the returned `pdf_fetched` and `preliminary_note` fields into that paper object.


+
+For each paper in `papers`, invoke the `preliminary-investigator` sub-agent with its full paper object. Merge the returned `pdf_fetched` and `preliminary_note` fields into that paper object.
+
+PDF extraction failures are non-fatal: continue screening and ranking with an empty `preliminary_note` when `pdf_fetched` is `false`.


+1. Download `https://arxiv.org/pdf/<id>` to `/tmp/gh-aw/agent/arxiv/<id>.pdf` with `curl`. If the download fails, return `{"pdf_fetched": false, "preliminary_note": ""}`.
+2. Use the `kreuzberg` MCP server's `extract_bytes` tool with the PDF's base64 content to extract its text. If extraction fails or produces no text, return `{"pdf_fetched": false, "preliminary_note": ""}`.


Initial plan

43f03ae

Copilot AI assigned Copilot and pelikhan Aug 25, 2026

Copilot started work on behalf of pelikhan August 25, 2026 04:43 View session

Copilot AI linked an issue Aug 25, 2026 that may be closed by this pull request

[q] Add PDF-aware preliminary investigation sub-agent + allow arXiv URLs in reports #55400

Closed

Add PDF-aware arXiv research screening

55c7630

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

Copilot AI changed the title ~~[WIP] Add PDF-aware preliminary investigation sub-agent~~ Add PDF-aware preliminary investigation to arXiv research Aug 25, 2026

Copilot finished work on behalf of pelikhan August 25, 2026 04:55

Copilot AI requested a review from pelikhan August 25, 2026 04:55

github-actions Bot mentioned this pull request Aug 25, 2026

[Contribution Check Report] Contribution Check — 2026-08-25 #55566

Open

pelikhan reviewed Aug 25, 2026

View reviewed changes

Copilot started work on behalf of pelikhan August 25, 2026 05:06 View session

pelikhan marked this pull request as ready for review August 25, 2026 05:06

Copilot AI balanced review requested due to automatic review settings August 25, 2026 05:06

Copilot started reviewing on behalf of pelikhan August 25, 2026 05:07 View session

Keep Kreuzberg prompting concise

f328a86

Co-authored-by: pelikhan <4175913+pelikhan@users.noreply.github.com>

github-actions Bot mentioned this pull request Aug 25, 2026

[pr-sous-chef] PR Sous Chef run report #55546

Open

Copilot AI reviewed Aug 25, 2026

View reviewed changes

Copilot finished work on behalf of pelikhan August 25, 2026 05:18

Copilot AI requested a review from pelikhan August 25, 2026 05:18

Merge branch 'main' into copilot/add-pdf-aware-preliminary-investigator

1c98355

pelikhan merged commit c901870 into main Aug 25, 2026

pelikhan deleted the copilot/add-pdf-aware-preliminary-investigator branch August 25, 2026 05:22

Navigation Menu

Uh oh!

Add PDF-aware preliminary investigation to arXiv research - #55644

Add PDF-aware preliminary investigation to arXiv research#55644
pelikhan merged 4 commits into
mainfrom
copilot/add-pdf-aware-preliminary-investigator

Copilot AI commented Aug 25, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

pelikhan Aug 25, 2026

Uh oh!

Copilot AI Aug 25, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		## Step 1b: Investigate Paper PDFs

		For each paper in `papers`, invoke the `preliminary-investigator` sub-agent with its full paper object. Merge the returned `pdf_fetched` and `preliminary_note` fields into that paper object.


		For each paper in `papers`, invoke the `preliminary-investigator` sub-agent with its full paper object. Merge the returned `pdf_fetched` and `preliminary_note` fields into that paper object.

		PDF extraction failures are non-fatal: continue screening and ranking with an empty `preliminary_note` when `pdf_fetched` is `false`.

		1. Download `https://arxiv.org/pdf/<id>` to `/tmp/gh-aw/agent/arxiv/<id>.pdf` with `curl`. If the download fails, return `{"pdf_fetched": false, "preliminary_note": ""}`.
		2. Use the `kreuzberg` MCP server's `extract_bytes` tool with the PDF's base64 content to extract its text. If extraction fails or produces no text, return `{"pdf_fetched": false, "preliminary_note": ""}`.

Uh oh!

Conversation

Copilot AI commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 25, 2026

Uh oh!

pelikhan Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI Aug 25, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Review details

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Copilot AI commented Aug 25, 2026 •

edited

Loading