Skip to content

Conversation

@waleedlatif1
Copy link
Collaborator

@waleedlatif1 waleedlatif1 commented Dec 31, 2025

Summary

  • removed zustand cache syncing in kb, added chunk text tokenizer
  • removed dead code
  • added context menu for chunks, kbs, and documents with varying options

Type of Change

  • Bug fix
  • New feature

Testing

Tested manually

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

@vercel
Copy link

vercel bot commented Dec 31, 2025

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Review Updated (UTC)
docs Skipped Skipped Dec 31, 2025 4:12am

@greptile-apps
Copy link
Contributor

greptile-apps bot commented Dec 31, 2025

Greptile Summary

Refactored knowledge base state management from Zustand to React Query, removing 923 lines of complex cache synchronization code. Added interactive chunk tokenizer visualization using js-tiktoken for accurate token counting. Implemented context menus for knowledge bases, documents, and chunks with right-click actions.

Key Changes:

  • Migrated from dual state management (Zustand + React Query) to React Query as single source of truth
  • Removed fuse.js dependency - client-side fuzzy search replaced with server-side search
  • Added tokenization utilities (apps/sim/lib/tokenization/estimators.ts) with accurate token counting via tiktoken
  • Enhanced chunk editor with toggleable tokenizer view showing color-coded tokens
  • Created reusable context menu components for KB, documents, and chunks
  • Moved type definitions from store to apps/sim/lib/knowledge/types.ts
  • Simplified data flow - removed manual cache invalidation and sync logic

Architecture Impact:

  • Eliminated state synchronization bugs between Zustand store and React Query cache
  • Improved data consistency by removing duplicate caching layers
  • Better TypeScript type safety with centralized type definitions
  • Reduced bundle size by removing unused dependencies

Confidence Score: 5/5

  • Safe to merge - well-architected refactor that simplifies state management and adds useful features
  • Major architectural improvement with net deletion of ~800 lines. Migration from Zustand to React Query eliminates dual-cache complexity. All references to deleted store successfully migrated. New tokenizer integration properly implemented with appropriate caching and error handling. Context menus follow established patterns. No breaking changes to external APIs.
  • No files require special attention

Important Files Changed

Filename Overview
apps/sim/stores/knowledge/store.ts Removed 923-line Zustand store; migrated to React Query for state management
apps/sim/hooks/use-knowledge.ts Refactored to use React Query; removed Zustand store sync and Fuse.js client-side search
apps/sim/lib/tokenization/estimators.ts Added new tokenization utility with tiktoken integration for accurate token counting
apps/sim/app/workspace/[workspaceId]/knowledge/[id]/[documentId]/components/edit-chunk-modal/edit-chunk-modal.tsx Added interactive tokenizer visualization with color-coded tokens; refactored to use React Query
apps/sim/app/workspace/[workspaceId]/knowledge/[id]/[documentId]/document.tsx Replaced Zustand store calls with React Query hooks; added context menu support for chunks
apps/sim/hooks/queries/knowledge.ts Added useDocumentQuery hook; removed unused mutation; enhanced query key structure
apps/sim/lib/knowledge/types.ts Moved type definitions from Zustand store; cleaned up and consolidated interfaces

Sequence Diagram

sequenceDiagram
    participant User
    participant Component
    participant ReactQuery
    participant API
    participant Database

    Note over User,Database: Knowledge Base Data Flow (Before: Zustand Store)
    
    User->>Component: View knowledge base
    Component->>ReactQuery: useKnowledgeBaseQuery(id)
    ReactQuery->>API: GET /api/knowledge/:id
    API->>Database: Query knowledge base
    Database-->>API: Knowledge base data
    API-->>ReactQuery: Response
    ReactQuery-->>Component: Cached data
    Component-->>User: Display knowledge base
    
    Note over User,Database: Chunk Editing with Tokenizer
    
    User->>Component: Edit chunk content
    Component->>Component: Toggle tokenizer ON
    Component->>Component: getTokenStrings(text)
    Component->>Component: Render colored tokens
    User->>Component: Save changes
    Component->>API: PATCH /api/knowledge/:kbId/documents/:docId/chunks/:chunkId
    API->>Database: Update chunk
    Database-->>API: Success
    API-->>Component: Updated chunk
    Component->>ReactQuery: invalidateQueries(knowledgeKeys)
    ReactQuery->>API: Refetch data
    API->>Database: Query fresh data
    Database-->>API: Updated data
    API-->>ReactQuery: Fresh data
    ReactQuery-->>Component: Updated cache
    Component-->>User: Display updated chunk
    
    Note over User,Database: Context Menu Actions
    
    User->>Component: Right-click on chunk
    Component->>Component: Show ChunkContextMenu
    User->>Component: Select action (Edit/Delete/Copy)
    Component->>API: Execute action API call
    API->>Database: Perform operation
    Database-->>API: Success
    API-->>Component: Result
    Component->>ReactQuery: invalidateQueries(knowledgeKeys)
    ReactQuery-->>Component: Trigger refetch
    Component-->>User: Updated UI
Loading

@waleedlatif1
Copy link
Collaborator Author

@greptile

@waleedlatif1 waleedlatif1 merged commit 4787909 into staging Dec 31, 2025
10 checks passed
@waleedlatif1 waleedlatif1 deleted the improvement/kb branch December 31, 2025 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants