Skip to content

perf(files): resolve name references through an index instead of listing the workspace - #8908

Open
waleedlatif1 wants to merge 4 commits into
stagingfrom
perf/flat-query-latency
Open

waleedlatif1 wants to merge 4 commits into
stagingfrom
perf/flat-query-latency

Conversation

@waleedlatif1

@waleedlatif1 waleedlatif1 commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • A file reference that missed the exact-name lookup fell back to loading every live file in the workspace and matching in memory. A new-file existence check always misses, so each file a workflow wrote paid a full workspace read.
  • Index the displayed name of live workspace files (displaySegmentKey, the SQL twin of normalizeDisplaySegment) and read only the rows the matcher can pick; resolution results are unchanged.
  • Index is built CONCURRENTLY on a partial (live workspace files) predicate; no locks block reads or writes.
  • Whitespace class shared from @sim/utils/string so the TS and SQL normalizers can't drift.

Type of Change

  • Performance improvement

Testing

  • Old vs new resolver compared on Postgres 17 over 11k references (real-world unnormalized names, folder and soft-deleted twins, ids, misses): identical results
  • Planner: candidate lookup goes from a scan of every file in the workspace (tens of ms per lookup on a large workspace, warm cache) to BitmapOr(pkey, display-name index) at ~0.1 ms
  • New integration cases: SQL vs TS normalization parity, resolution of unnormalized stored names, EXPLAIN uses the new index
  • check:migrations, drizzle-kit generate (no drift), biome on changed files

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing (new tests pass the test-audit authoring gate)
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

…ing the workspace

A reference that misses the exact-name lookup fell back to loading every
live file in the workspace and matching in memory. A workflow's new-file
existence check always misses, so each file it wrote paid a full read.

Index the displayed name (the SQL twin of normalizeDisplaySegment) and
read only the rows the matcher can pick, keeping its results unchanged.
@vercel

vercel Bot commented Oct 10, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
docs Skipped Skipped Oct 10, 2026 11:15pm UTC

Request Review

@greptile-apps

greptile-apps Bot commented Oct 10, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerConfidence Score: 5/5

[Critical impact] The changes since the previous review appear safe to merge.

Summary

The PR replaces full-workspace file reads with an indexed lookup of possible matches.

  • Workspace file references query only rows that could match.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[File reference] --> B[Existing ID and exact-name lookup]
  B --> C{Found?}
  C -- Yes --> D[Return file]
  C -- No --> E[Read candidates by ID or indexed display name]
  E --> F[Build folder paths]
  F --> G[Apply existing matcher]
  G --> H[Return file or no match]
Loading

Reviews (2) · Last reviewed commit: "test(files): explain the resolver's real..." · Reviewed by Greptile

Comment thread apps/sim/lib/uploads/contexts/workspace/__integration__/file-names.integration.ts Outdated
…ures

Mock displaySegmentKey in the central schema mock, store size_bytes on the
raw fixture rows, and assert the plan of the full candidate query (id arm,
display-name arm, order) rather than a reduced one.
@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@greptile

@waleedlatif1

Copy link
Copy Markdown
Collaborator Author

@cubic-dev-ai review this PR

@cubic-dev-ai

cubic-dev-ai Bot commented Oct 10, 2026

Copy link
Copy Markdown
Contributor

@cubic-dev-ai review this PR

@waleedlatif1 I have started the AI code review. It will take a few minutes to complete.

@cubic-dev-ai cubic-dev-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues found across 9 files

Confidence score: 5/5

  • Automated review surfaced no issues in the provided summaries.
  • No files require special attention.

Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.

Turn on auto-fix | Re-trigger cubic

This branch was previously deployed

1 inactive deployment
Preview — 9911ba93 Deployed Oct 10, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant