Replace timer-based span buffering with streaming-onEnd micro-batching

[Copilot]

Problem

Timer-based flush logic (grace period, max-age sweeps) drops late-ending spans when trace state is deleted. Export latency is unpredictable and memory usage scales with buffered span count.

Changes

Streaming export on span end

• Spans export immediately via setImmediate micro-batching (default batch size: 64)
• Traces persist after flush to handle late-ending child spans
• No timer coordination or sweep intervals

LRU eviction replaces trace dropping

• When maxBufferedTraces reached, evict least-recently-active entry
• Flush pending spans before eviction to prevent data loss
• Late spans for evicted traces export with fallback context + warning

Data structure changes

// Before: buffered all ended spans until timer fired
type TraceBuffer = {
  spans: ReadableSpan[];
  openCount: number;
  rootEnded: boolean;
  startedAtMs: number;
  rootEndedAtMs?: number;
  // ...
};

// After: small queue for micro-batching
type TraceEntry = {
  queue: ReadableSpan[];           // bounded by maxBatchSize
  flushScheduled: boolean;         // prevents duplicate flushes
  lastActivityMs: number;          // LRU tracking
  rootCtx?: Context;               // token access
  fallbackCtx?: Context;
  exportedCount: number;
  // ...
};

Removed

• ensureSweepTimer(), stopSweepTimerIfIdle(), sweep() methods
• flushGraceMs, maxTraceAgeMs usage (parameters retained for compatibility)
• sweepTimer, isSweeping fields

Added

• scheduleMicroBatchFlush(): next-tick export via setImmediate
• evictLeastRecentlyUsed(): capacity management with linear scan (O(n), acceptable for default limit of 1000)
• A365_PER_REQUEST_MAX_BATCH_SIZE environment variable

Token correctness preserved

• Export occurs under context.with(rootCtx ?? fallbackCtx) as before
• Exporter reads token from active context via ALS

Memory footprint

• Small queues (≤64 spans) vs. unbounded span arrays
• Trace entries persist but remain lightweight (~1-2KB each)
• Expected usage: 1-2MB at default maxBufferedTraces=1000

Original prompt

Summary
Switch PerRequestSpanProcessor to a streaming-onEnd span export approach with minor micro-batching to eliminate timeout-induced drops of late-ending spans while preserving token correctness by exporting under the saved request Context (rootCtx). Remove sweeper and time-based flushes, retain only per-trace Context and a tiny queue, and keep concurrency guardrails. Add idle/LRU eviction for contexts to bound memory while avoiding per-trace timeouts.

Background and current behavior
The existing PerRequestSpanProcessor buffers ended spans per trace and flushes them based on:

• trace_completed: root span ended and no open spans remain
• root_ended_grace: grace period after root ended to flush if children never end
• max_trace_age: safety timeout to flush long-lived traces

On flush, the trace entry is deleted. Any spans that end afterward (common for long-running children) will find no buffer and are effectively dropped because onEnd returns early when buf is missing.

Snippet showing current export call (from the original file):

              this.exporter.export(spans, (result) => {

Problem

• Long-running spans can end long after the root or after time-based sweeps. Because the trace state is deleted on flush, these late spans are dropped.
• Export latency is variable and depends on timers.
• Memory usage can be elevated due to buffering ended spans until a flush trigger fires.

Goals

• Avoid dropping late-ending spans due to time-based deletion of trace state.
• Preserve token correctness by exporting under the saved rootCtx.
• Reduce memory footprint by avoiding large per-trace span buffers.
• Keep export throughput controlled via maxConcurrentExports and minor batching to limit call volume.

Proposed approach: Streaming on onEnd with micro-batching (Method 1)

• Capture rootCtx (request Context with token) in onStart when the root span begins; capture a fallback first-seen ctx for cases where rootCtx is missing.
• Do NOT buffer spans long-term. On onEnd, enqueue the finished span into a small per-trace queue and schedule a micro-batch flush using setImmediate, or flush immediately when the queue hits a small size threshold (configurable).
• Export under context.with(savedCtx) so the exporter can read the token via context.active().
• Remove sweeper, flushGraceMs, and maxTraceAge logic entirely; no per-trace timeouts. Keep concurrency guardrail (maxConcurrentExports).
• Bound retained state by limiting how many trace contexts we keep (A365_PER_REQUEST_MAX_TRACES); when at capacity, evict the least-recently-active context (LRU). If spans arrive for an evicted trace, export without saved context and log a warning.
• Optional guardrail: A365_PER_REQUEST_MAX_SPANS_PER_TRACE applies to (exportedCount + queued) to drop spans only under extreme pressure.
• Optional new env: A365_PER_REQUEST_MAX_BATCH_SIZE controls micro-batch size.

Detailed changes

1. Replace TraceBuffer state with TraceEntry holding only Contexts and a tiny queue:

• rootCtx?: Context // request Context (token via ALS)
• fallbackCtx?: Context
• queue: ReadableSpan[]
• flushScheduled: boolean
• lastActivityMs: number
• exportedCount: number
• droppedSpans: number

2. Remove timer-based sweep:

• Delete ensureSweepTimer, stopSweepTimerIfIdle, sweep, and the isSweeping flag.
• Remove usages of flushGraceMs and maxTraceAgeMs; constructor can keep parameters for compatibility but they are unused in the new mode.

3. Modify lifecycle methods:

• onStart(span, ctx): create/update TraceEntry, capture rootCtx when root span starts; set fallbackCtx otherwise; update lastActivityMs.
• onEnd(span): enqueue span; schedule micro-batch flush via setImmediate unless queue reaches A365_PER_REQUEST_MAX_BATCH_SIZE, then flush immediately.
• flushTrace(traceId): splice the queue, select exportCtx = rootCtx ?? fallbackCtx, then exportBatch(spans, exportCtx).

4. Concurrency and backpressure:

• Keep acquireExportSlot/releaseExportSlot and maxConcurrentExports guardrail; retain exportWaiters queue.

5. Eviction policy:

• When traces map size reaches A365_PER_REQUEST_MAX_TRACES, evict the least recently active entry (LRU by lastActivityMs). Log the evicted traceId. If later spans for that trace end, they export without saved context (warning logged).

6. Logging:

• Maintain informative lifecycle logs for start/end, batch flushes, export success/failure, and warnings when exporting without saved context.

7. Env variables (backward-compatible names):

• A365_PER_REQUEST_MAX_TRACES: maximum retained trace contexts
• A365_PER_REQUEST_MAX_SPANS_PER_TRACE: guardrail; set <= 0 to disable
• A365_PER_REQUEST_MAX_CONCURRENT_EXPORTS: max concurrent exporter calls
• A365_PER_REQUEST_MAX_BATCH_SIZE: new; micro-...

This pull request was created from Copilot chat.

---

💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.