fix(ai-gemini, ai-openai): align image-input URL handling — passthrough where possible, error-by-default where an upload is required#908
Merged
Conversation
…miniImage URL sources in multimodal image-generation prompts now pass through as fileData.fileUri (Gemini fetches them server-side), matching the chat adapter, instead of being fetched locally and inlined as base64 — which double-buffered the image and could OOM on memory-constrained runtimes such as Cloudflare Workers. Veo keeps its fetch: the predict API's Image type only accepts imageBytes or gcsUri, so there is no URI passthrough to align with. Fixes #907 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
🚀 Changeset Version Preview2 package(s) bumped directly, 0 bumped as dependents. 🟨 Minor bumps
|
Contributor
📝 WalkthroughWalkthroughThe PR updates Gemini image generation to pass URL inputs through to Gemini instead of fetching them locally, adds an opt-in fetch flag for Gemini video and OpenAI image/video flows, and updates tests, docs, and changesets to match the new URL handling. ChangesGemini image URL passthrough fix
Gemini video URL fetch opt-in
OpenAI URL fetch opt-in
Estimated code review effort: 4 (Complex) | ~45 minutes Possibly related issues
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
View your CI Pipeline Execution ↗ for commit d3661bb
☁️ Nx Cloud last updated this comment at |
@tanstack/ai
@tanstack/ai-acp
@tanstack/ai-angular
@tanstack/ai-anthropic
@tanstack/ai-bedrock
@tanstack/ai-claude-code
@tanstack/ai-client
@tanstack/ai-code-mode
@tanstack/ai-code-mode-skills
@tanstack/ai-codex
@tanstack/ai-devtools-core
@tanstack/ai-elevenlabs
@tanstack/ai-event-client
@tanstack/ai-fal
@tanstack/ai-gemini
@tanstack/ai-grok
@tanstack/ai-grok-build
@tanstack/ai-groq
@tanstack/ai-isolate-cloudflare
@tanstack/ai-isolate-node
@tanstack/ai-isolate-quickjs
@tanstack/ai-mcp
@tanstack/ai-mistral
@tanstack/ai-ollama
@tanstack/ai-openai
@tanstack/ai-opencode
@tanstack/ai-openrouter
@tanstack/ai-preact
@tanstack/ai-react
@tanstack/ai-react-ui
@tanstack/ai-sandbox
@tanstack/ai-sandbox-cloudflare
@tanstack/ai-sandbox-daytona
@tanstack/ai-sandbox-docker
@tanstack/ai-sandbox-local-process
@tanstack/ai-sandbox-sprites
@tanstack/ai-sandbox-vercel
@tanstack/ai-solid
@tanstack/ai-solid-ui
@tanstack/ai-svelte
@tanstack/ai-utils
@tanstack/ai-vue
@tanstack/ai-vue-ui
@tanstack/openai-base
@tanstack/preact-ai-devtools
@tanstack/react-ai-devtools
@tanstack/solid-ai-devtools
commit: |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (1)
packages/ai-gemini/tests/image-adapter.test.ts (1)
802-835: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAssert that URL passthrough does not call
fetch.This test verifies
fileData, but the regression objective is also “without fetching.” Add a localfetchspy that throws or assert it was not called so future changes cannot reintroduce local buffering while preserving the same payload shape.🧪 Proposed test hardening
it('passes arbitrary HTTPS URL sources through as fileData without fetching (`#907`)', async () => { + const fetchSpy = vi.spyOn(globalThis, 'fetch') + // Matches the chat adapter: Gemini fetches URL inputs server-side. // Fetching locally and inlining as base64 OOMs on memory-constrained // runtimes (e.g. Cloudflare Workers). const { adapter, mockGenerateContent } = mockedNativeAdapter() @@ expect(args.contents).toEqual([ @@ ]) + expect(fetchSpy).not.toHaveBeenCalled() })🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/ai-gemini/tests/image-adapter.test.ts` around lines 802 - 835, The `generateImage` URL passthrough test only asserts the `fileData` payload and does not verify that no local fetching occurs. Update the `passes arbitrary HTTPS URL sources through as fileData without fetching` case in `image-adapter.test.ts` to spy on `fetch` (or mock it to fail) and assert it is never called while keeping the existing `mockedNativeAdapter`/`mockGenerateContent` payload expectations intact. This will harden the regression coverage around the no-fetch behavior for HTTPS image sources.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@packages/ai-gemini/tests/image-adapter.test.ts`:
- Around line 802-835: The `generateImage` URL passthrough test only asserts the
`fileData` payload and does not verify that no local fetching occurs. Update the
`passes arbitrary HTTPS URL sources through as fileData without fetching` case
in `image-adapter.test.ts` to spy on `fetch` (or mock it to fail) and assert it
is never called while keeping the existing
`mockedNativeAdapter`/`mockGenerateContent` payload expectations intact. This
will harden the regression coverage around the no-fetch behavior for HTTPS image
sources.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 19d5fce4-c23e-4ce7-b9a5-6715be28a3e8
📒 Files selected for processing (6)
.changeset/gemini-image-url-passthrough.mddocs/config.jsondocs/media/image-generation.mdpackages/ai-gemini/src/adapters/image.tspackages/ai-gemini/src/adapters/video.tspackages/ai-gemini/tests/image-adapter.test.ts
…uire upload Veo (createGeminiVideo), OpenAI image edits (createOpenaiImage), and OpenAI Sora input_reference (createOpenaiVideo) have no URL passthrough — the provider only accepts uploaded bytes (or gs:// for Veo). Previously an HTTP(S) URL image input was silently fetched and buffered in memory, which can OOM constrained runtimes (e.g. Cloudflare Workers). These paths now throw by default on an HTTP(S) URL image input, with an error pointing to the alternatives. data: URIs (and gs:// for Veo) still work with no flag. Opt back into fetching + buffering via allowUrlFetch: true on the adapter config. Threaded through GeminiVideoConfig, OpenAIImageConfig, and OpenAIVideoConfig. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
tombeckenham
changed the title
AlemTuzlak
approved these changes
Jul 7, 2026
AlemTuzlak
deleted the
907-fixai-gemini-align-creategeminiimage-url-handling-with-chat-stop-fetching-arbitrary-https-urls
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🎯 Changes
Aligns how the Gemini and OpenAI media adapters handle
type: 'url'image inputs, so none of them silently fetch + base64-buffer arbitrary HTTPS URLs (an OOM hazard on Workers/edge). Two complementary parts:1. Gemini native image — passthrough (fixes #907)
createGeminiImage()previously fetched arbitraryhttps://image inputs locally and re-encoded them as base64inlineData, unlikecreateGeminiChat().imagePartToGeminiPartnow passes all URL sources (public HTTPS, Files API URIs,gs://) through asfileData.fileUri— Gemini fetches server-side, nothing is buffered locally.datasources still map toinlineData.2. Paths that can't passthrough — error by default +
allowUrlFetchopt-inThree paths have no URL passthrough because the provider requires uploaded bytes: Gemini Veo (
imagePartToVeoImage), OpenAI image edits (imagePartToFile), and OpenAI Sorainput_reference. For an HTTP(S) URL these must download + buffer the image in memory. They now throw by default with an actionable message instead of silently buffering.data:URIs (andgs://for Veo) still work with no flag. Opt back in withallowUrlFetch: true:Threaded through
GeminiVideoConfig,OpenAIImageConfig,OpenAIVideoConfig. (Bedrock already errors on URL image sources — this brings these three onto the same stance.)Also updated
stubGlobal('fetch')hack); error-by-default +allowUrlFetchopt-in +data:-URI-still-works cases for Veo, OpenAI edits, and Sora.media/image-generation.mdand themedia-generationagent skill document passthrough vs.allowUrlFetch.Gemini's
generateContentimage mapping stays unit-test covered (aimock doesn't mock that endpoint); image E2E specs pass unchanged.✅ Checklist
pnpm run test:pr.🚀 Release Impact
🤖 Generated with Claude Code
Summary by CodeRabbit
allowUrlFetchoption to enable remote URL fetching/buffering when desired (Gemini video, and OpenAI image/video).