feat(langsmith): add run update, get run, and feedback tools

[waleedlatif1]

Summary

• Audited the LangSmith integration against the live API — the block only supported ingesting runs (create + batch create), with no way to update a run once it started, read one back, or attach evaluation feedback
• Added langsmith_update_run (PATCH /runs/{id}) to complete the standard create-then-patch tracing lifecycle
• Added langsmith_get_run (GET /runs/{id}) to read a run's status/outputs back into a workflow
• Added langsmith_create_feedback (POST /feedback) to attach scores, values, corrections, and comments to a run
• Wired all three into the LangSmith block, sharing existing subBlocks (name, outputs, tags, status, error, etc.) across the create/update operations via widened conditions instead of duplicating fields
• Fixed the "LangSmith feedback capture" BlockMeta template, which previously faked feedback by tagging a new run — it now uses the real feedback API
• Verified every new endpoint and field against LangSmith's live OpenAPI spec, including a required/optional field diff
• Switched manual Object.fromEntries(...).filter(...) payload filtering to the shared filterUndefined helper across all langsmith tools

Type of Change

• New feature

Testing

Tested manually — verified endpoint existence and schemas against LangSmith's live OpenAPI spec (api.smith.langchain.com/openapi.json) and live HTTP probing (401-vs-404 to confirm routes exist). Ran bun run lint and tsc --noEmit clean for all changed files.

Checklist

• Code follows project style guidelines
• Self-reviewed my changes
• Tests added/updated and passing
• No new warnings introduced
• I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[greptile-apps]

Greptile Summary

This PR extends the LangSmith block from a create-only integration to a full tracing lifecycle by adding langsmith_update_run (PATCH), langsmith_get_run (GET), and langsmith_create_feedback (POST). It also fixes the "LangSmith feedback capture" BlockMeta template and migrates create_run.ts to the shared filterUndefined helper.

• Three new tools (update_run, get_run, create_feedback) are wired into the block with shared subBlocks (name, outputs, tags, status, error) widened to cover create/update operations instead of duplicating fields.
• Block output schema is extended to cover all new response fields including inputs/outputs from get_run and id/score/key from create_feedback; all three new tool files include response.ok guards before parsing the response body.
• BlockMeta templates are updated so the feedback capture prompt describes real feedback submission rather than the previous workaround of tagging a synthetic run.

Confidence Score: 5/5

Safe to merge — all three new tools have proper HTTP error guards and the block wiring is consistent.

The new tools are well-structured, previously flagged issues (hardcoded success, ignored response body, missing response.ok guards, NaN score coercion, boolean score type) have all been addressed in earlier commits. The remaining observation is a minor empty-string inconsistency for optional feedback fields that does not cause data corruption or workflow failures.

No files require special attention.

Important Files Changed

Filename	Overview
apps/sim/blocks/blocks/langsmith.ts	Block config extended with three new operations; subBlocks are shared across create/update via array conditions; output schema includes all new fields; emptyToUndefined and parseScore helpers are clean. One minor inconsistency: value and comment in the create_feedback path are not passed through emptyToUndefined, unlike every string field in update_run.
apps/sim/tools/langsmith/update_run.ts	New tool for PATCH /runs/{id}; checks response.ok before parsing, attempts JSON parse with graceful fallback for empty bodies, and enforces at least-one-field validation in both the tool body and block config. Previously flagged issues (hardcoded success, ignored response) have been resolved.
apps/sim/tools/langsmith/get_run.ts	New tool for GET /runs/{id}; response.ok guard is present, response fields are correctly mapped including inputs/outputs, and both id and runId aliases are returned for consistency with other operations.
apps/sim/tools/langsmith/create_feedback.ts	New tool for POST /feedback; uses generateId() (UUID v4) for client-side ID, response.ok guard in place, filterUndefined strips absent optional fields. Empty strings for value and comment are not guarded, creating a minor inconsistency with update_run's empty-string handling.
apps/sim/tools/langsmith/types.ts	New TypeScript interfaces for update, get, and feedback operations are well-typed. LangsmithResponse union is extended. Minor: LangsmithCreateFeedbackResponse.output.value is typed string
apps/sim/tools/langsmith/create_run.ts	Minor refactor: manual Object.fromEntries/filter replaced with shared filterUndefined helper. No behavioral change.
apps/sim/tools/registry.ts	Three new tools registered with correct keys matching tool id fields and block access list.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant Block as LangSmith Block
    participant Tool as Tool Config (params fn)
    participant API as api.smith.langchain.com

    Note over Block,API: Create Run (existing)
    Block->>Tool: "operation=langsmith_create_run"
    Tool->>API: "POST /runs {id, name, run_type, inputs, ...}"
    API-->>Tool: 202 Accepted
    Tool-->>Block: "{accepted, runId, message}"

    Note over Block,API: Update Run (new)
    Block->>Tool: "operation=langsmith_update_run, runId"
    Tool->>API: "PATCH /runs/{runId} {outputs, status, end_time, ...}"
    API-->>Tool: 200/204 + optional body
    Tool-->>Block: "{accepted, runId, message}"

    Note over Block,API: Get Run (new)
    Block->>Tool: "operation=langsmith_get_run, runId"
    Tool->>API: "GET /runs/{runId}"
    API-->>Tool: "200 {id, name, run_type, status, inputs, outputs, ...}"
    Tool-->>Block: "{id, runId, name, runType, status, inputs, outputs, ...}"

    Note over Block,API: Create Feedback (new)
    Block->>Tool: "operation=langsmith_create_feedback, runId, key, score, ..."
    Tool->>API: "POST /feedback {id, run_id, key, score, value, comment, ...}"
    API-->>Tool: "200 {id, key, run_id, score, ...}"
    Tool-->>Block: "{id, key, runId, score, value, comment, createdAt}"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant Block as LangSmith Block
    participant Tool as Tool Config (params fn)
    participant API as api.smith.langchain.com

    Note over Block,API: Create Run (existing)
    Block->>Tool: "operation=langsmith_create_run"
    Tool->>API: "POST /runs {id, name, run_type, inputs, ...}"
    API-->>Tool: 202 Accepted
    Tool-->>Block: "{accepted, runId, message}"

    Note over Block,API: Update Run (new)
    Block->>Tool: "operation=langsmith_update_run, runId"
    Tool->>API: "PATCH /runs/{runId} {outputs, status, end_time, ...}"
    API-->>Tool: 200/204 + optional body
    Tool-->>Block: "{accepted, runId, message}"

    Note over Block,API: Get Run (new)
    Block->>Tool: "operation=langsmith_get_run, runId"
    Tool->>API: "GET /runs/{runId}"
    API-->>Tool: "200 {id, name, run_type, status, inputs, outputs, ...}"
    Tool-->>Block: "{id, runId, name, runType, status, inputs, outputs, ...}"

    Note over Block,API: Create Feedback (new)
    Block->>Tool: "operation=langsmith_create_feedback, runId, key, score, ..."
    Tool->>API: "POST /feedback {id, run_id, key, score, value, comment, ...}"
    API-->>Tool: "200 {id, key, run_id, score, ...}"
    Tool-->>Block: "{id, key, runId, score, value, comment, createdAt}"

Loading

_{Reviews (6): Last reviewed commit: "fix(langsmith): normalize empty-string p..." | Re-trigger Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[greptile-apps]

Greptile Summary

Extends the LangSmith block with three new operations — update_run (PATCH), get_run (GET), and create_feedback (POST /feedback) — completing the standard create-then-patch tracing lifecycle and adding evaluation feedback support. The PR also fixes the LangSmith feedback capture BlockMeta template to use the real feedback API instead of a tagged run, and migrates manual Object.fromEntries filtering to the shared filterUndefined helper.

• langsmith_update_run sends a PATCH to /runs/{id}; all optional fields are forwarded through filterUndefined, but transformResponse ignores the HTTP response body entirely and always returns accepted: true.
• langsmith_get_run retrieves a run by ID and maps all major fields, but the block-level outputs schema omits inputs and outputs (the run's actual payload fields), so downstream blocks cannot reference them.
• langsmith_create_feedback posts scored/categorical feedback; generateId() correctly produces a UUID v4 for the required id field, and filterUndefined correctly preserves a score of 0.

Confidence Score: 3/5

Mostly safe to merge, but Get Run is missing two of its most useful output fields from the block schema, limiting its practical usefulness in workflows.

The block omits inputs and outputs from the Get Run output schema — the fields users would most commonly retrieve a run to read. Until those are added, Get Run silently drops the run payload that downstream blocks need. The Update Run tool also ignores the HTTP response body entirely, unlike every other tool in the module, so API-level errors that come back in a 2xx body are invisible.

apps/sim/blocks/blocks/langsmith.ts (missing outputs) and apps/sim/tools/langsmith/update_run.ts (response body ignored)

Important Files Changed

Filename	Overview
apps/sim/blocks/blocks/langsmith.ts	Wires three new operations (update_run, get_run, create_feedback) into the block; block outputs schema is missing inputs and outputs fields returned by get_run, making those fields inaccessible to downstream blocks.
apps/sim/tools/langsmith/update_run.ts	New PATCH tool; correctly uses filterUndefined for the body, but transformResponse ignores the HTTP response entirely and always returns accepted: true — inconsistent with other LangSmith tools that at least read the body.
apps/sim/tools/langsmith/get_run.ts	New GET tool; straightforward implementation that reads and maps all run fields from the API response correctly.
apps/sim/tools/langsmith/create_feedback.ts	New POST feedback tool; generates a UUID for the feedback id, uses filterUndefined correctly; score type allows boolean in the interface but only numeric values are reachable through the block or parseScore.
apps/sim/tools/langsmith/types.ts	Adds well-typed interfaces for update_run, get_run, and create_feedback; LangsmithResponse union extended correctly.
apps/sim/tools/langsmith/create_run.ts	Minor cleanup: manual Object.fromEntries filter replaced with filterUndefined helper, no behavior change.
apps/sim/tools/langsmith/index.ts	Exports three new tools; all entries correctly ordered.
apps/sim/tools/registry.ts	Registers the three new tools in the global registry under the expected key names.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant W as Workflow Block
    participant LS as LangSmith API

    Note over W,LS: Create Run (existing)
    W->>LS: "POST /runs {id, name, run_type, inputs…}"
    LS-->>W: "200 {runId, message}"

    Note over W,LS: Update Run (new)
    W->>LS: "PATCH /runs/{runId} {outputs, status, end_time…}"
    LS-->>W: 200 (body ignored → accepted: true)

    Note over W,LS: Get Run (new)
    W->>LS: "GET /runs/{runId}"
    LS-->>W: "200 {id, name, status, inputs, outputs, …}"
    Note right of W: inputs/outputs missing from block outputs schema

    Note over W,LS: Create Feedback (new)
    W->>LS: "POST /feedback {id(UUID), run_id, key, score…}"
    LS-->>W: "200 {id, key, run_id, score, createdAt}"

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant W as Workflow Block
    participant LS as LangSmith API

    Note over W,LS: Create Run (existing)
    W->>LS: "POST /runs {id, name, run_type, inputs…}"
    LS-->>W: "200 {runId, message}"

    Note over W,LS: Update Run (new)
    W->>LS: "PATCH /runs/{runId} {outputs, status, end_time…}"
    LS-->>W: 200 (body ignored → accepted: true)

    Note over W,LS: Get Run (new)
    W->>LS: "GET /runs/{runId}"
    LS-->>W: "200 {id, name, status, inputs, outputs, …}"
    Note right of W: inputs/outputs missing from block outputs schema

    Note over W,LS: Create Feedback (new)
    W->>LS: "POST /feedback {id(UUID), run_id, key, score…}"
    LS-->>W: "200 {id, key, run_id, score, createdAt}"

Loading

Comments Outside Diff (1)

1. apps/sim/blocks/blocks/langsmith.ts, line 424-451 (link)

   get_run outputs missing inputs and outputs from block schema

   The get_run tool's transformResponse returns both inputs (run inputs payload) and outputs (run outputs payload) as JSON fields, and the tool's own outputs descriptor declares them too. However, the block-level outputs schema (lines 424–451) does not include either field.

   Because the block schema drives downstream block autocompletion and LLM tool awareness, consumers of the "Get Run" operation have no way to reference the retrieved run's actual inputs and outputs in subsequent steps — which is arguably the primary reason for calling Get Run. All other get_run fields (status, startTime, endTime, error, etc.) are correctly forwarded; only these two json-typed fields are missing.

_{Reviews (2): Last reviewed commit: "fix(langsmith): harden error handling an..." | Re-trigger Greptile}

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[waleedlatif1]

@greptile

[waleedlatif1]

@cursor review

[cursor]

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 51322c8. Configure here.}