diff --git a/CHANGES.md b/CHANGES.md index 1525b6ac..b32d8060 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -50,6 +50,9 @@ Release Notes. * Keep a trace that the caller has already sampled. An entry span that starts a tracing context checked for the upstream `sw8` reference before attaching it, so the local sampler ran even when the request carried a valid `sw8` header: with `agent.sampler` below 1, a downstream service dropped already-sampled traces, sent no `sw8` header onward, and its own downstream services could start unrelated traces. The sampler also received an empty operation name, and the `sw_go_created_tracing_context_counter{created_by="propagated"}` self-observability counter never increased. A `sw8` header now counts as valid only when every required field is set, as in the Java agent, so a header such as `1---0----` no longer creates a segment with an empty trace ID. * Apply the ignore rules (`agent.ignore_suffix` and `agent.trace_ignore_path`) and the disconnected-reporter check to every request that starts a trace. A goroutine keeps its tracing context after a trace ends, and the checks ran only on a goroutine without one. On a goroutine that serves several requests, such as the goroutine of an HTTP/1 keep-alive connection, every request after the first was therefore traced even when it should be ignored. Also, the spans that an ignored request creates in its own goroutine, such as its database calls, are now ignored, as they already were under a request that the sampler drops, instead of each one starting its own trace. When several goroutines continue one captured context, such as the send and receive goroutines of a gRPC stream, each now gets its own noop span for an ignored or sampled-out request, instead of sharing one span's nesting count. * Fix the gin and goframe plugins with a negative `header_length_threshold`. Collecting the request headers panicked after the entry span was created, so the span was never ended and stayed active on its goroutine. A value of 0 or less now means no limit, as for `redis.max_args_bytes`; before, 0 sent an empty value, while `collect_request_headers` already decides whether headers are collected at all. The plugins also pass the span to the after-hook before collecting the headers. +* End the spans that a plugin leaves active on a goroutine, so they cannot ignore or capture the later requests on it. The agent binds a span to the interceptor that stores it with `invocation.SetContext`. When a later span starts on the goroutine and the active span belongs to an interceptor that has finished, for example because its after-hook did not end the span or panicked first, the agent ends that span and counts it in `sw_go_possible_leaked_context_counter`. When `BeforeInvoke` returns an error or panics after it created spans and before it stored one with `SetContext`, the agent ends the spans it created; a context that the hook continued, such as the snapshot of a stream, is kept. Before, such a span stayed active: under the noop span of an ignored, sampled-out, or disconnected-reporter request, every later request on the goroutine, such as an HTTP/1 keep-alive connection, was ignored, and under a traced span they joined the leaked trace. Spans that no interceptor stores, such as the spans of the toolkit API or of gorm callbacks, are not ended by the agent. +* Keep a reused entry span active until the outer plugin ends it, when the nested plugin that reused it is an interceptor that stores it with `invocation.SetContext`, such as gin mounted on a net/http `ServeMux`. Before, the nested End made the parent active while the outer plugin was still serving the request, so the spans created afterwards started new traces. A nested reuser that only creates and ends the span, such as a kratos middleware, a pull consumer reading messages inside a request, or the toolkit API, still makes the parent active, so its next span is created as before. An exit span reused by a nested plugin, such as the sql driver inside gorm, is also still popped. +* Keep the names of all Go builtin functions, such as `close`, `copy`, `min`, and `max`, when the agent copies the agent core packages into an instrumented package. Before, only some builtins were known, and the others were renamed like functions of the copied package, so the instrumented build failed. #### Unsupported Library Versions diff --git a/docs/en/development-and-contribution/development-guide.md b/docs/en/development-and-contribution/development-guide.md index f20bc2d3..b9458ede 100644 --- a/docs/en/development-and-contribution/development-guide.md +++ b/docs/en/development-and-contribution/development-guide.md @@ -238,6 +238,16 @@ rather than creating a new instance each time a method is intercepted. Therefore, do not declare objects in the interceptor, and instead use `Invocation.Context` to pass data. +##### Span lifecycle + +A span that an interceptor creates must be ended by the same interception, usually in `AfterInvoke`. Store it with `invocation.SetContext` right after creating it, before anything that can fail, so that `AfterInvoke` gets it. + +The agent protects the goroutine from a span that is not ended: +- A span stored with `SetContext` belongs to that interception. If it is still active when a later span starts on the goroutine after the interception has finished, the agent ends it and counts it in `sw_go_possible_leaked_context_counter`. Only the span itself is bound: a value that wraps the span, such as a struct stored with `SetContext`, is not. +- If `BeforeInvoke` returns an error or panics after creating spans and before storing anything with `SetContext`, the agent ends the spans it created and counts them the same way. A context that the hook continued with `ContinueContext` is kept. Nothing is ended when the hook stored any value with `SetContext`, because `AfterInvoke` still runs and ends it, or when the hook changed the active span in another way, for example by ending a span that was active before it. + +A span that must stay active after its interception has finished, such as a span that a later call ends, must not be stored directly with `SetContext`: store it in a wrapper value. A span ended with `PrepareAsync` is not active anymore once `End` is called in the interception, and is finished later with `AsyncFinish`. + ##### Package Path If the method you want to intercept is not located in the root directory of the framework, diff --git a/plugins/core/operator/intercepter_test_base.go b/plugins/core/operator/intercepter_test_base.go index dc1fff77..32a9e934 100644 --- a/plugins/core/operator/intercepter_test_base.go +++ b/plugins/core/operator/intercepter_test_base.go @@ -31,3 +31,47 @@ func NewInvocation(receiver interface{}, args ...interface{}) Invocation { }, } } + +// InvokeIntercepted runs the interceptor around call in the same order as the +// code that the agent generates: method_intercept_before.tmpl, +// method_inserts.tmpl and method_intercept_after.tmpl. A panic in an +// interceptor is recovered, as in the generated code. It returns false when +// the before-hook skipped the call. +func InvokeIntercepted(interceptor Interceptor, invocation Invocation, call func() []interface{}) (called bool) { + inv, ok := invocation.(*realInvocation) + if !ok { + panic("InvokeIntercepted needs an invocation from NewInvocation") + } + skip := func() (skip bool) { + var mark interface{} + depth := -1 + defer func() { + if r := recover(); r != nil { + rollbackActiveSpan(inv, mark, depth) + } + }() + mark, depth = markActiveSpan() + if err := interceptor.BeforeInvoke(inv); err != nil { + rollbackActiveSpan(inv, mark, depth) + return false + } + if inv.isContinue { + inv.markExited() + return true + } + return false + }() + if skip { + return false + } + var results []interface{} + defer func() { + defer inv.markExited() + defer func() { + _ = recover() + }() + _ = interceptor.AfterInvoke(inv, results...) + }() + results = call() + return true +} diff --git a/plugins/core/operator/invocation.go b/plugins/core/operator/invocation.go index 83fee062..37e52e5b 100644 --- a/plugins/core/operator/invocation.go +++ b/plugins/core/operator/invocation.go @@ -27,6 +27,10 @@ type realInvocation struct { returnValues []interface{} context interface{} + // owner is closed when the interceptor has finished, so the span it + // stored with SetContext can be ended if it is still active. It is a + // channel because this copied file cannot import sync/atomic. + owner chan struct{} // self obs interTimeCost int64 @@ -56,6 +60,68 @@ func (i *realInvocation) DefineReturnValues(values ...interface{}) { func (i *realInvocation) SetContext(context interface{}) { i.context = context + if span, ok := context.(spanOwnerBinder); ok { + if i.owner == nil { + i.owner = make(chan struct{}) + } + span.BindOwner(i.owner) + } +} + +// spanOwnerBinder is a span that can be bound to the invocation that ends it. +// Only builtin types are used, because every instrumented package has its own +// copy of this package. +type spanOwnerBinder interface { + BindOwner(owner chan struct{}) +} + +type activeSpanMarker interface { + ActiveSpanMark() (interface{}, int) + RollbackActiveSpan(mark interface{}, depth int) +} + +// markExited records that the interceptor of the invocation has finished, so +// the span it stored with SetContext must no longer be active. +func (i *realInvocation) markExited() { + if i != nil && i.owner != nil { + close(i.owner) + i.owner = nil + } +} + +func spanMarker() activeSpanMarker { + op := GetOperator() + if op == nil { + return nil + } + marker, _ := op.Tracing().(activeSpanMarker) + return marker +} + +// markActiveSpan returns the active span before BeforeInvoke runs. +func markActiveSpan() (mark interface{}, depth int) { + if marker := spanMarker(); marker != nil { + return marker.ActiveSpanMark() + } + return nil, 0 +} + +// rollbackActiveSpan ends the spans that a failed BeforeInvoke created and did +// not store with SetContext, so nothing else would end them. +func rollbackActiveSpan(i *realInvocation, mark interface{}, depth int) { + // It runs in the recover of the before-hook too, where a panic would + // reach the application. + defer func() { + _ = recover() + }() + if i == nil || i.context != nil || depth < 0 { + // the after-hook still runs and ends the span it got, or the active + // span was not marked + return + } + if marker := spanMarker(); marker != nil { + marker.RollbackActiveSpan(mark, depth) + } } func (i *realInvocation) GetContext() interface{} { diff --git a/plugins/core/operator/invocation_test.go b/plugins/core/operator/invocation_test.go new file mode 100644 index 00000000..f20f3a4f --- /dev/null +++ b/plugins/core/operator/invocation_test.go @@ -0,0 +1,137 @@ +// Licensed to Apache Software Foundation (ASF) under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Apache Software Foundation (ASF) licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package operator + +import "testing" + +type ownedSpan struct { + owners []chan struct{} +} + +func closed(owner chan struct{}) bool { + select { + case <-owner: + return true + default: + return false + } +} + +func (s *ownedSpan) BindOwner(owner chan struct{}) { + s.owners = append(s.owners, owner) +} + +// fakeTracing records the calls of the template helpers. +type fakeTracing struct { + mark interface{} + depth int + rollbacks []int +} + +func (f *fakeTracing) ActiveSpanMark() (mark interface{}, depth int) { + return f.mark, f.depth +} + +func (f *fakeTracing) RollbackActiveSpan(_ interface{}, depth int) { + f.rollbacks = append(f.rollbacks, depth) +} + +type fakeOperator struct { + Operator + tracing interface{} +} + +func (f *fakeOperator) Tracing() interface{} { + return f.tracing +} + +func withOperator(t *testing.T, op Operator) { + t.Helper() + old := GetOperator + GetOperator = func() Operator { return op } + t.Cleanup(func() { GetOperator = old }) +} + +func TestSetContextBindsSpanToInvocation(t *testing.T) { + inv := &realInvocation{} + span := &ownedSpan{} + inv.SetContext(span) + inv.SetContext(span) + if len(span.owners) != 2 || span.owners[0] == nil || span.owners[0] != span.owners[1] { + t.Fatalf("every SetContext should bind the same owner of the invocation, got %v", span.owners) + } + if closed(span.owners[0]) { + t.Fatal("the owner should not be finished before the interceptor finishes") + } + inv.markExited() + if !closed(span.owners[0]) { + t.Fatal("markExited should finish the owner") + } + // the after-hook and the skip path may both finish the invocation + inv.markExited() +} + +func TestSetContextWithoutSpanHasNoOwner(t *testing.T) { + inv := &realInvocation{} + inv.SetContext("not a span") + inv.SetContext(nil) + inv.markExited() + if inv.owner != nil { + t.Fatal("a context that is not a span should not create an owner") + } + if inv.GetContext() != nil { + t.Fatal("SetContext should still store the context") + } + var nilInvocation *realInvocation + nilInvocation.markExited() +} + +func TestMarkActiveSpanWithoutTracer(t *testing.T) { + withOperator(t, nil) + if mark, depth := markActiveSpan(); mark != nil || depth != 0 { + t.Fatalf("no tracer should mark nothing, got %v %d", mark, depth) + } + withOperator(t, &fakeOperator{tracing: struct{}{}}) + if mark, depth := markActiveSpan(); mark != nil || depth != 0 { + t.Fatalf("a tracer without marks should mark nothing, got %v %d", mark, depth) + } + rollbackActiveSpan(&realInvocation{}, nil, 0) +} + +func TestRollbackActiveSpan(t *testing.T) { + tracer := &fakeTracing{mark: "active", depth: 2} + withOperator(t, &fakeOperator{tracing: tracer}) + mark, depth := markActiveSpan() + if mark != "active" || depth != 2 { + t.Fatalf("markActiveSpan should return the tracer mark, got %v %d", mark, depth) + } + + rollbackActiveSpan(nil, mark, depth) + rollbackActiveSpan(&realInvocation{}, mark, -1) + withContext := &realInvocation{} + withContext.SetContext("context") + rollbackActiveSpan(withContext, mark, depth) + if len(tracer.rollbacks) != 0 { + t.Fatalf("no rollback without an invocation, without a mark, or with a context, got %v", tracer.rollbacks) + } + + rollbackActiveSpan(&realInvocation{}, mark, depth) + if len(tracer.rollbacks) != 1 || tracer.rollbacks[0] != 2 { + t.Fatalf("a failed hook without a context should roll back to the mark, got %v", tracer.rollbacks) + } +} diff --git a/plugins/core/span_default.go b/plugins/core/span_default.go index b5f7c8d6..cf46a5f4 100644 --- a/plugins/core/span_default.go +++ b/plugins/core/span_default.go @@ -56,6 +56,9 @@ type DefaultSpan struct { // into SegmentSpanImpl/SnapshotSpan, and an embedded sync.Mutex value would // trip the go vet copylocks check. opLock *sync.Mutex + // owner is closed when the interceptor that stored the span with + // SetContext has finished (see BindOwner). Guarded by opLock. + owner chan struct{} // ended is set by endAndFreeze right before the span is handed over to the // segment collector. Once set, every late mutation is dropped, so the span // data is frozen and the reporting path may read it without locking @@ -65,11 +68,12 @@ type DefaultSpan struct { // span leaked across goroutines would otherwise flood the log with one // line per dropped write. droppedLogged bool - // reuseCount counts the extra logical owners that the span reuse rule in - // CreateEntrySpan/CreateExitSpan hands this span to. Every owner calls End - // exactly once and only the LAST End freezes and reports the span (see - // enterReuse and endSyncAndFreeze). Guarded by opLock. - reuseCount int + // reuses has one entry per extra owner that the span reuse rule in + // CreateEntrySpan/CreateExitSpan hands this span to, innermost last; only + // the LAST End freezes and reports the span. An entry is true when that + // owner is an interceptor that stored the span with SetContext. Guarded + // by opLock. + reuses []bool } func NewDefaultSpan(tracer *Tracer, parent TracingSpan) *DefaultSpan { @@ -280,7 +284,7 @@ func (ds *DefaultSpan) enterReuse() { ds.opLock.Lock() defer ds.opLock.Unlock() if !ds.ended { - ds.reuseCount++ + ds.reuses = append(ds.reuses, false) } } @@ -293,22 +297,30 @@ func (ds *DefaultSpan) enterReuse() { // PrepareAsync/End pair free of data races). func (ds *DefaultSpan) endSyncAndFreeze() bool { ds.opLock.Lock() - if ds.reuseCount > 0 { + if n := len(ds.reuses); n > 0 { // a nested owner of a reused span (see enterReuse) finished: this is // not the last End, so keep the span open - the outer owner still // writes to it (e.g. gorm tags db.statement after the sql driver's - // End) and will freeze it with its own End. Restoring the active span - // is kept here because the nested plugin expects its End to pop the - // span, exactly like before the freeze mechanism. - ds.reuseCount-- + // End) and will freeze it with its own End. + stored := ds.reuses[n-1] + ds.reuses = ds.reuses[:n-1] ds.opLock.Unlock() - if ctx := getTracingContext(); ctx != nil { - ctx.SaveActiveSpan(ds.Parent) + // An entry span that a nested interceptor stored, such as gin inside + // the net/http server, stays active for the outer owner's request. + // Otherwise it is popped, so the next call of the outer owner, such as + // a gorm hook query or a pull consumer's next message, gets its own span. + if ds.SpanType != SpanTypeEntry || !stored { + if ctx := getTracingContext(); ctx != nil { + ctx.SaveActiveSpan(ds.Parent) + } } return false } if !ds.EndTime.IsZero() { // already ended ds.opLock.Unlock() + // A reused entry span stays active after its nested End, and + // AsyncFinish may have ended it since: the last End still pops it. + ds.popIfActive() return false } ds.EndTime = time.Now() @@ -328,6 +340,17 @@ func (ds *DefaultSpan) endSyncAndFreeze() bool { return frozen } +// popIfActive makes the parent active when this span is the active span. +func (ds *DefaultSpan) popIfActive() { + ctx := getTracingContext() + if ctx == nil { + return + } + if active, ok := ctx.ActiveSpan().(SegmentSpan); ok && active.GetDefaultSpan() == ds { + ctx.SaveActiveSpan(ds.Parent) + } +} + func (ds *DefaultSpan) IsEntry() bool { return ds.SpanType == SpanTypeEntry } diff --git a/plugins/core/span_noop.go b/plugins/core/span_noop.go index dc6e4769..b22d0bfd 100644 --- a/plugins/core/span_noop.go +++ b/plugins/core/span_noop.go @@ -26,12 +26,17 @@ const noopContextValue = "N/A" type NoopSpan struct { stackCount int tracer *Tracer + // owner is closed when the interceptor that stored the span has finished + owner chan struct{} + // snapshot is set on the span that stands for a continued context + snapshot bool } func newSnapshotNoopSpan() *NoopSpan { // snapshot noop span is not a real span return &NoopSpan{ stackCount: 0, + snapshot: true, } } @@ -99,6 +104,12 @@ func (*NoopSpan) ErrorOccured() { } func (n *NoopSpan) enterNoSpan() { + if n.owner == nil && n.stackCount > 0 { + // the creator holds the span but did not store it, so it is not an + // interceptor. A continued snapshot span starts at 0 without a holder, + // so its first nested span may own it. + n.owner = noOwner + } n.stackCount++ } diff --git a/plugins/core/span_owner.go b/plugins/core/span_owner.go new file mode 100644 index 00000000..2ea0ad23 --- /dev/null +++ b/plugins/core/span_owner.go @@ -0,0 +1,238 @@ +// Licensed to Apache Software Foundation (ASF) under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Apache Software Foundation (ASF) licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package core + +// maxLeakedSpans bounds the loops that end leaked spans, in case a span does +// not leave the active position when it ends. +const maxLeakedSpans = 64 + +// noOwner is the owner of a noop span that its creator nested before storing +// it with SetContext: it is never closed. +var noOwner = make(chan struct{}) + +// BindOwner binds the span to the invocation that stores it with SetContext. +// While the span is reused, the nested owner that stores it marks its reuse +// as stored instead (see endSyncAndFreeze), so only the creator binds. +func (ds *DefaultSpan) BindOwner(owner chan struct{}) { + ds.opLock.Lock() + defer ds.opLock.Unlock() + if n := len(ds.reuses); n > 0 { + ds.reuses[n-1] = true + return + } + if ds.owner == nil { + ds.owner = owner + } +} + +// BindOwner does nothing: a snapshot span is owned by another goroutine. +func (s *SnapshotSpan) BindOwner(chan struct{}) { +} + +// BindOwner binds the noop span to the invocation that stores it with SetContext. +func (n *NoopSpan) BindOwner(owner chan struct{}) { + if n.owner == nil { + n.owner = owner + } +} + +// isClosed reports whether the owner of a span has finished. A nil channel, +// a span without an owner, is never closed. +func isClosed(owner chan struct{}) bool { + select { + case <-owner: + return true + default: + return false + } +} + +// spanOwnerFinished reports whether the interceptor that stored the span has +// finished, so nothing will end the span anymore. A span that no interceptor +// stored, such as a span of the toolkit API or of a gorm callback, has no owner. +func spanOwnerFinished(span TracingSpan) bool { + switch s := span.(type) { + case *NoopSpan: + return isClosed(s.owner) + case *SnapshotSpan: + return false + case SegmentSpan: + ds := s.GetDefaultSpan() + ds.opLock.Lock() + owner := ds.owner + ds.opLock.Unlock() + return isClosed(owner) + } + return false +} + +// spanDepth is how many times the span was handed out again while it was +// active: the nested spans of a noop span, or the owners of a reused span. +func spanDepth(span TracingSpan) int { + switch s := span.(type) { + case *NoopSpan: + return s.stackCount + case *SnapshotSpan: + return 0 + case SegmentSpan: + ds := s.GetDefaultSpan() + ds.opLock.Lock() + defer ds.opLock.Unlock() + return len(ds.reuses) + } + return 0 +} + +// restoreDepth undoes the nesting that was added after the depth was taken. +func restoreDepth(span TracingSpan, depth int) { + switch s := span.(type) { + case *NoopSpan: + if s.stackCount > depth { + s.stackCount = depth + } + case *SnapshotSpan: + case SegmentSpan: + ds := s.GetDefaultSpan() + ds.opLock.Lock() + if len(ds.reuses) > depth { + ds.reuses = ds.reuses[:depth] + } + ds.opLock.Unlock() + } +} + +// endLeakedSpan ends the active span of ctx for a plugin that cannot end it +// anymore, and makes its parent the active span. +func (t *Tracer) endLeakedSpan(ctx *TracingContext, span TracingSpan) { + switch s := span.(type) { + case *NoopSpan: + GetSo11y(t).MeasureLeakedTracingContext(true) + // the nested spans that never ended are ended together with it + s.stackCount = 1 + s.End() + case *SnapshotSpan: + // owned by another goroutine, and its End panics: only leave it + case SegmentSpan: + GetSo11y(t).MeasureLeakedTracingContext(false) + ds := s.GetDefaultSpan() + // the owners that never ended are ended together with it, so this + // End reports the span and its segment is not held back + ds.opLock.Lock() + ds.reuses = nil + name := ds.OperationName + ds.opLock.Unlock() + if t.Log != nil { + t.Log.Warnf("ending the span %q that its plugin did not end", name) + } + span.End() + } + if ctx.ActiveSpan() == span { + ctx.SaveActiveSpan(span.ParentSpan()) + } +} + +// endOrphanSpans ends the active spans whose interceptor has finished without +// ending them, and returns the active span that is left. Such a span would +// otherwise stay active on the goroutine, and every later request on it would +// become its child, or be ignored under a noop span. +func (t *Tracer) endOrphanSpans(ctx *TracingContext) TracingSpan { + for i := 0; i < maxLeakedSpans; i++ { + span := ctx.ActiveSpan() + if span == nil || !spanOwnerFinished(span) { + return span + } + t.endLeakedSpan(ctx, span) + } + return ctx.ActiveSpan() +} + +// ActiveSpanMark returns the active span and its depth before a before-hook +// runs, for RollbackActiveSpan. +func (t *Tracer) ActiveSpanMark() (mark interface{}, depth int) { + ctx := getTracingContext() + if ctx == nil { + return nil, 0 + } + // the spans that the before-hook creates replace the orphans first, so + // the mark must not be an orphan + span := t.endOrphanSpans(ctx) + if span == nil { + return nil, 0 + } + return span, spanDepth(span) +} + +// RollbackActiveSpan ends the spans that a failed before-hook created and did +// not store with SetContext, back to the active span of ActiveSpanMark. A +// context that the before-hook continued, such as the snapshot of a stream, +// is kept, and only the spans created on it are ended. It changes nothing when +// neither is an ancestor of the active span, because then the before-hook did +// more than create spans. +func (t *Tracer) RollbackActiveSpan(mark interface{}, depth int) { + ctx := getTracingContext() + if ctx == nil { + return + } + markSpan, _ := mark.(TracingSpan) + if !reachesRollbackBoundary(markSpan, ctx.ActiveSpan()) { + return + } + for i := 0; i < maxLeakedSpans; i++ { + span := ctx.ActiveSpan() + if span == nil { + return + } + if span == markSpan { + restoreDepth(span, depth) + return + } + if isContinuedSnapshot(span) { + return + } + t.endLeakedSpan(ctx, span) + } +} + +// isContinuedSnapshot reports whether the span stands for a context captured +// on another goroutine: a snapshot span, or the noop span of an ignored or +// sampled-out snapshot. +func isContinuedSnapshot(span TracingSpan) bool { + switch s := span.(type) { + case *SnapshotSpan: + return true + case *NoopSpan: + return s.snapshot + } + return false +} + +// reachesRollbackBoundary reports whether the mark, or a continued snapshot, +// is the span or one of its parents. A nil mark is the parent of every root +// span. +func reachesRollbackBoundary(mark, span TracingSpan) bool { + for i := 0; i < maxLeakedSpans; i++ { + if span == mark || (span != nil && isContinuedSnapshot(span)) { + return true + } + if span == nil { + return false + } + span = span.ParentSpan() + } + return false +} diff --git a/plugins/core/span_owner_test.go b/plugins/core/span_owner_test.go new file mode 100644 index 00000000..dd1c8387 --- /dev/null +++ b/plugins/core/span_owner_test.go @@ -0,0 +1,708 @@ +// Licensed to Apache Software Foundation (ASF) under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Apache Software Foundation (ASF) licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package core + +import ( + "errors" + "fmt" + "testing" + + "github.com/stretchr/testify/assert" + + "github.com/apache/skywalking-go/plugins/core/operator" + "github.com/apache/skywalking-go/plugins/core/reporter" + "github.com/apache/skywalking-go/plugins/core/tracing" +) + +const ( + tracedPage = "GET:/page" + ignoredPage = "GET:/page.html" +) + +type failMode int + +const ( + failNone failMode = iota + failError + failPanic +) + +// spanInterceptor creates an entry or exit span in BeforeInvoke and ends it in +// AfterInvoke, with the defects found in plugins. +type spanInterceptor struct { + operation string + exit bool + // fail is how BeforeInvoke fails after it creates the span and before SetContext + fail failMode + // skipEnd makes AfterInvoke return without ending the span + skipEnd bool + // panicInAfter makes AfterInvoke panic before ending the span + panicInAfter bool + // skipCall defines the return values, so the method and AfterInvoke don't run + skipCall bool + // failBeforeCreate makes BeforeInvoke return an error before it creates the span + failBeforeCreate bool + // async makes AfterInvoke end the span in async mode; asyncSpan keeps it + async bool + asyncSpan tracing.Span +} + +func (h *spanInterceptor) BeforeInvoke(invocation operator.Invocation) error { + if h.failBeforeCreate { + return errors.New("the plugin failed before creating the span") + } + var s tracing.Span + var err error + if h.exit { + s, err = tracing.CreateExitSpan(h.operation, "localhost:8080", func(string, string) error { return nil }) + } else { + s, err = tracing.CreateEntrySpan(h.operation, noHeader) + } + if err != nil { + return err + } + switch h.fail { + case failError: + return errors.New("the plugin failed after creating the span") + case failPanic: + panic("the plugin panicked after creating the span") + } + invocation.SetContext(s) + if h.skipCall { + invocation.DefineReturnValues() + } + return nil +} + +func (h *spanInterceptor) AfterInvoke(invocation operator.Invocation, _ ...interface{}) error { + s, ok := invocation.GetContext().(tracing.Span) + if !ok { + return nil + } + if h.panicInAfter { + panic("the plugin panicked before ending the span") + } + if h.async { + s.PrepareAsync() + h.asyncSpan = s + } + if !h.skipEnd { + s.End() + } + return nil +} + +// intercept runs the interceptor around call on the current goroutine, as the +// generated code does. +func intercept(h *spanInterceptor, call func()) { + operator.InvokeIntercepted(h, operator.NewInvocation(nil), func() []interface{} { + if call != nil { + call() + } + return nil + }) +} + +func activeTraceID() string { + if s := tracing.ActiveSpan(); s != nil { + return s.TraceID() + } + return "" +} + +// activeTracingSpan returns the active span of the current goroutine. +func activeTracingSpan() TracingSpan { + s, _ := Tracing.ActiveSpan().(TracingSpan) + return s +} + +// assertNewTrace checks that the span starts a new trace: it is traced, and it +// is a root span, not the leaked span or one of its children. Trace IDs cannot +// tell traces apart in these tests: two contexts created in the same +// millisecond on the shared test goroutine get the same IDs. +func assertNewTrace(t *testing.T, span, leaked TracingSpan, msgAndArgs ...interface{}) { + t.Helper() + if !assert.NotNil(t, span, msgAndArgs...) { + return + } + _, isNoop := span.(*NoopSpan) + assert.False(t, isNoop, msgAndArgs...) + assert.Nil(t, span.ParentSpan(), msgAndArgs...) + if leaked != nil { + assert.NotSame(t, leaked, span, msgAndArgs...) + } +} + +// assertNextRequestTraced checks that the goroutine is clean: a new request +// starts its own trace. +func assertNextRequestTraced(t *testing.T) { + t.Helper() + assert.Nil(t, Tracing.ActiveSpan(), "no span should be active between requests") + var next TracingSpan + intercept(&spanInterceptor{operation: "GET:/next"}, func() { next = activeTracingSpan() }) + assertNewTrace(t, next, nil, "the next request should start a new trace") + assert.Nil(t, Tracing.ActiveSpan()) +} + +func TestFailedBeforeInvokeEndsItsSpan(t *testing.T) { + for _, fail := range []failMode{failError, failPanic} { + for _, ignored := range []bool{true, false} { + ResetTracingContext() + Tracing.ignoreSuffix = []string{".html"} + operation := tracedPage + if ignored { + operation = ignoredPage + } + intercept(&spanInterceptor{operation: operation, fail: fail}, nil) + assert.Nilf(t, Tracing.ActiveSpan(), "fail=%d ignored=%v: the span of the failed hook should be ended", fail, ignored) + if !ignored { + waitReportedSpans(t, 1) + } + assertNextRequestTraced(t) + } + } + ResetTracingContext() +} + +func TestFailedNestedBeforeInvokeKeepsTheOuterSpan(t *testing.T) { + cases := []struct { + name string + outer string + inner *spanInterceptor + }{ + {"exit under entry", tracedPage, &spanInterceptor{operation: "GET:/downstream", exit: true, fail: failError}}, + {"reused entry", tracedPage, &spanInterceptor{operation: "GET:/page/inner", fail: failPanic}}, + {"noop under ignored entry", ignoredPage, &spanInterceptor{operation: "GET:/downstream", exit: true, fail: failError}}, + } + for _, c := range cases { + ResetTracingContext() + Tracing.ignoreSuffix = []string{".html"} + var outer, afterInner TracingSpan + var outerDepth, afterInnerDepth int + intercept(&spanInterceptor{operation: c.outer}, func() { + outer = activeTracingSpan() + outerDepth = spanDepth(outer) + intercept(c.inner, nil) + afterInner = activeTracingSpan() + afterInnerDepth = spanDepth(afterInner) + }) + assert.Samef(t, outer, afterInner, "%s: the outer span should stay active", c.name) + assert.Equalf(t, outerDepth, afterInnerDepth, "%s: the nesting of the failed hook should be undone", c.name) + assert.Nilf(t, Tracing.ActiveSpan(), "%s: no span should be active after the request", c.name) + if _, isNoop := outer.(*NoopSpan); !isNoop { + // the outer span is reported: no owner is left on it + waitReportedSpans(t, 1) + } + assertNextRequestTraced(t) + } + ResetTracingContext() +} + +func TestSpanLeftByFinishedInterceptorIsEnded(t *testing.T) { + leaks := []struct { + name string + h *spanInterceptor + }{ + {"after-hook without End", &spanInterceptor{skipEnd: true}}, + {"after-hook panics", &spanInterceptor{panicInAfter: true}}, + {"skipped call", &spanInterceptor{skipCall: true}}, + } + for _, leak := range leaks { + for _, ignored := range []bool{true, false} { + ResetTracingContext() + Tracing.ignoreSuffix = []string{".html"} + leak.h.operation = tracedPage + if ignored { + leak.h.operation = ignoredPage + } + var leaked TracingSpan + intercept(leak.h, func() { leaked = activeTracingSpan() }) + assert.NotNilf(t, Tracing.ActiveSpan(), "%s: the span is left active", leak.name) + + var next TracingSpan + intercept(&spanInterceptor{operation: "GET:/next"}, func() { next = activeTracingSpan() }) + assertNewTrace(t, next, leaked, "%s ignored=%v: the next request should start a new trace", leak.name, ignored) + assert.Nilf(t, Tracing.ActiveSpan(), "%s ignored=%v", leak.name, ignored) + if !ignored { + // the leaked span is reported too + waitReportedSpans(t, 2) + } + } + } + ResetTracingContext() +} + +// A span stays active while the interceptor that created it is running, even +// after nested interceptors finished. +func TestSpanOfRunningInterceptorIsKept(t *testing.T) { + defer ResetTracingContext() + var entryTraceID, exitTraceID string + intercept(&spanInterceptor{operation: tracedPage}, func() { + entryTraceID = activeTraceID() + // a nested interceptor that reuses the entry span, such as gin inside + // the net/http server, leaves it active when it ends + intercept(&spanInterceptor{operation: "GET:/page/inner"}, nil) + intercept(&spanInterceptor{operation: "GET:/first", exit: true}, func() { + intercept(&spanInterceptor{operation: "GET:/nested", exit: true}, nil) + }) + intercept(&spanInterceptor{operation: "GET:/second", exit: true}, func() { exitTraceID = activeTraceID() }) + }) + assert.Equal(t, entryTraceID, exitTraceID, "the exit spans should join the running request") + assert.Nil(t, Tracing.ActiveSpan()) + waitReportedSpans(t, 3) +} + +// A span that no interceptor stored, such as a toolkit span, is never ended by +// the agent. +func TestSpanWithoutOwnerIsKept(t *testing.T) { + defer ResetTracingContext() + local, err := tracing.CreateLocalSpan("toolkit") + assert.NoError(t, err) + intercept(&spanInterceptor{operation: "GET:/downstream", exit: true, fail: failError}, nil) + intercept(&spanInterceptor{operation: "GET:/downstream", exit: true, skipEnd: true}, nil) + var traceID string + intercept(&spanInterceptor{operation: "GET:/downstream", exit: true}, func() { traceID = activeTraceID() }) + assert.Equal(t, local.TraceID(), traceID, "spans should still join the toolkit span") + assert.Equal(t, local.TraceID(), activeTraceID(), "the toolkit span should stay active") + local.End() + assert.Nil(t, Tracing.ActiveSpan()) +} + +// The spans created after a nested plugin ends a reused entry span, while the +// outer plugin still serves the request, belong to the request. +func TestReusedEntrySpanStaysActiveAfterNestedEnd(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + var outerID, afterInnerID, exitID string + intercept(&spanInterceptor{operation: "GET:/raw"}, func() { + outerID = activeTraceID() + intercept(&spanInterceptor{operation: "GET:/renamed"}, nil) + afterInnerID = activeTraceID() + intercept(&spanInterceptor{operation: "GET:/downstream", exit: true}, func() { exitID = activeTraceID() }) + }) + assert.NotEmpty(t, outerID) + assert.Equal(t, outerID, afterInnerID, "the reused entry span should stay active after the nested End") + assert.Equal(t, outerID, exitID, "the exit span should join the request") + assert.Nil(t, Tracing.ActiveSpan()) + spans := waitReportedSpans(t, 2) + assert.Equal(t, 2, len(spans)) + assert.NotNil(t, findReportedSpan(spans, "GET:/renamed"), "the reused span keeps the nested rename") +} + +// The last End of a reused entry span makes its parent active, whichever +// owner ends last. +func TestReusedEntrySpanOuterEndsFirst(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + outer, err := tracing.CreateEntrySpan("GET:/raw", noHeader) + assert.NoError(t, err) + inner, err := tracing.CreateEntrySpan("GET:/renamed", noHeader) + assert.NoError(t, err) + // the inner owner is an interceptor + operator.NewInvocation(nil).SetContext(inner) + outer.End() + assert.Equal(t, outer.TraceID(), activeTraceID(), "the span stays active until its last owner ends") + inner.End() + assert.Nil(t, Tracing.ActiveSpan()) + assert.Equal(t, 1, len(waitReportedSpans(t, 1))) +} + +// gorm creates an exit span that the sql driver reuses. After the driver ends, +// the exit span is no longer active, so a gorm hook query gets its own span. +func TestReusedExitSpanIsPoppedAfterNestedEnd(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + entry, err := tracing.CreateEntrySpan("GET:/users", noHeader) + assert.NoError(t, err) + gormSpan, err := tracing.CreateExitSpan("users/create", "db:3306", func(string, string) error { return nil }) + assert.NoError(t, err) + intercept(&spanInterceptor{operation: "MySQL/Exec", exit: true}, nil) + active := activeTracingSpan() + if assert.NotNil(t, active) { + assert.True(t, active.IsEntry(), "the entry span should be active after the driver ended the reused exit span") + } + intercept(&spanInterceptor{operation: "users/hook", exit: true}, nil) + gormSpan.Tag("db.statement", "INSERT INTO users") + gormSpan.End() + entry.End() + spans := waitReportedSpans(t, 3) + assert.Equal(t, 3, len(spans), "the hook query should have its own span") + if s := findReportedSpan(spans, "users/create"); assert.NotNil(t, s) { + v, ok := reportedTagValue(s, "db.statement") + assert.True(t, ok) + assert.Equal(t, "INSERT INTO users", v) + } +} + +// A span whose creator is not an interceptor, such as a toolkit or gorm +// callback span, is not owned by a nested interceptor that reuses it. +func TestNestedInterceptorDoesNotOwnUnboundSpan(t *testing.T) { + for _, ignored := range []bool{false, true} { + ResetTracingContext() + Tracing.ignoreSuffix = []string{".html"} + operation := "GET:/toolkit" + if ignored { + operation = "GET:/toolkit.html" + } + root, err := tracing.CreateEntrySpan(operation, noHeader) + assert.NoError(t, err) + intercept(&spanInterceptor{operation: "GET:/nested"}, nil) + intercept(&spanInterceptor{operation: "GET:/downstream", exit: true}, nil) + var traceID string + intercept(&spanInterceptor{operation: "GET:/downstream", exit: true}, func() { traceID = activeTraceID() }) + assert.Equalf(t, root.TraceID(), traceID, "ignored=%v: the spans should still join the root span", ignored) + assert.Equalf(t, root.TraceID(), activeTraceID(), "ignored=%v: the root span should stay active", ignored) + root.End() + assert.Nilf(t, Tracing.ActiveSpan(), "ignored=%v", ignored) + } + ResetTracingContext() +} + +// endActiveThenFail ends the active span and then fails, so the active span +// before the hook is no longer an ancestor of the active span. +type endActiveThenFail struct{} + +func (endActiveThenFail) BeforeInvoke(operator.Invocation) error { + if s := tracing.ActiveSpan(); s != nil { + s.End() + } + return errors.New("failed after ending the active span") +} + +func (endActiveThenFail) AfterInvoke(operator.Invocation, ...interface{}) error { + return nil +} + +func TestRollbackStopsWhenMarkedSpanEnded(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + entry, err := tracing.CreateEntrySpan(tracedPage, noHeader) + assert.NoError(t, err) + local, err := tracing.CreateLocalSpan("local") + assert.NoError(t, err) + operator.InvokeIntercepted(endActiveThenFail{}, operator.NewInvocation(nil), func() []interface{} { return nil }) + active := activeTracingSpan() + if assert.NotNil(t, active, "the entry span should not be rolled back") { + assert.True(t, active.IsEntry()) + } + _ = local + entry.End() + assert.Nil(t, Tracing.ActiveSpan()) +} + +func TestFailedBeforeInvokeBeforeCreatingSpanChangesNothing(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + var traceID, afterID string + intercept(&spanInterceptor{operation: tracedPage}, func() { + traceID = activeTraceID() + intercept(&spanInterceptor{operation: "GET:/downstream", exit: true, failBeforeCreate: true}, nil) + afterID = activeTraceID() + }) + assert.Equal(t, traceID, afterID) + assert.Nil(t, Tracing.ActiveSpan()) + assert.Equal(t, 1, len(waitReportedSpans(t, 1))) +} + +// The noop spans of a disconnected reporter and of the sampler are reclaimed +// and rolled back like the noop span of an ignored request. +func TestNoopSpansOfOtherPathsAreReclaimed(t *testing.T) { + setups := map[string]func(){ + "disconnected reporter": func() { ReportConnectionStatus = reporter.ConnectionStatusDisconnect }, + "sampled out": func() { Tracing.Sampler = NewConstSampler(false) }, + } + for name, setup := range setups { + for _, h := range []*spanInterceptor{ + {operation: tracedPage, skipEnd: true}, + {operation: tracedPage, fail: failError}, + {operation: tracedPage, fail: failPanic}, + } { + ResetTracingContext() + setup() + intercept(h, nil) + ReportConnectionStatus = reporter.ConnectionStatusConnected + Tracing.Sampler = NewConstSampler(true) + var traceID string + intercept(&spanInterceptor{operation: "GET:/next"}, func() { traceID = activeTraceID() }) + assert.NotEqualf(t, noopContextValue, traceID, "%s %+v: the next request should be traced", name, *h) + assert.Nilf(t, Tracing.ActiveSpan(), "%s", name) + } + } + ResetTracingContext() +} + +func TestReclaimedSpansAreCountedAsLeaked(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + Tracing.ignoreSuffix = []string{".html"} + so11y := GetSo11y(Tracing) + leaked, leakedIgnored := so11y.leakedContextCounter.Get(), so11y.leakedIgnoreContextCounter.Get() + + intercept(&spanInterceptor{operation: tracedPage, skipEnd: true}, nil) + intercept(&spanInterceptor{operation: "GET:/next"}, nil) + assert.Equal(t, leaked+1, so11y.leakedContextCounter.Get()) + + intercept(&spanInterceptor{operation: ignoredPage, skipEnd: true}, nil) + intercept(&spanInterceptor{operation: "GET:/next"}, nil) + assert.Equal(t, leakedIgnored+1, so11y.leakedIgnoreContextCounter.Get()) + + // a rolled back span is counted too + intercept(&spanInterceptor{operation: tracedPage, fail: failError}, nil) + assert.Equal(t, leaked+2, so11y.leakedContextCounter.Get()) +} + +// A goroutine that continues a captured context keeps the snapshot span: only +// the leaked spans created on it are ended. +func TestContinuedContextKeepsSnapshotSpan(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + entry, err := tracing.CreateEntrySpan(tracedPage, noHeader) + assert.NoError(t, err) + requestCtx := getTracingContext() + snapshot := tracing.CaptureContext() + + SetGLS(nil) + tracing.ContinueContext(snapshot) + intercept(&spanInterceptor{operation: "GET:/leaked", exit: true, skipEnd: true}, nil) + var traceID string + intercept(&spanInterceptor{operation: "GET:/next", exit: true}, func() { traceID = activeTraceID() }) + assert.Equal(t, entry.TraceID(), traceID, "the continued goroutine should stay in the trace") + _, isSnapshot := Tracing.ActiveSpan().(*SnapshotSpan) + assert.True(t, isSnapshot, "the snapshot span should stay active") + + SetGLS(requestCtx) + entry.End() + assert.Nil(t, Tracing.ActiveSpan()) +} + +// A goroutine that continues an ignored request owns its first noop span, so a +// leak there is reclaimed instead of ignoring the goroutine forever. +func TestContinuedNoopSnapshotLeakIsReclaimed(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + Tracing.ignoreSuffix = []string{".html"} + entry, err := tracing.CreateEntrySpan("GET:/ignored.html", noHeader) + assert.NoError(t, err) + requestCtx := getTracingContext() + snapshot := tracing.CaptureContext() + + SetGLS(nil) + tracing.ContinueContext(snapshot) + intercept(&spanInterceptor{operation: "GET:/leaked", exit: true, skipEnd: true}, nil) + var traceID string + intercept(&spanInterceptor{operation: "GET:/next"}, func() { traceID = activeTraceID() }) + assert.NotEqual(t, noopContextValue, traceID, "the next request on the continued goroutine should be traced") + assert.Nil(t, Tracing.ActiveSpan()) + + SetGLS(requestCtx) + entry.End() + assert.Nil(t, Tracing.ActiveSpan()) +} + +func TestAsyncSpanIsNotReclaimed(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + async := &spanInterceptor{operation: "GET:/stream", exit: true, async: true} + var traceID, nextID string + intercept(&spanInterceptor{operation: tracedPage}, func() { + traceID = activeTraceID() + intercept(async, nil) + intercept(&spanInterceptor{operation: "GET:/next", exit: true}, func() { nextID = activeTraceID() }) + }) + assert.Equal(t, traceID, nextID) + if assert.NotNil(t, async.asyncSpan) { + async.asyncSpan.AsyncFinish() + } + assert.Equal(t, 3, len(waitReportedSpans(t, 3))) +} + +// continueThenFail continues a captured context, optionally creates a span on +// it, and then fails, as a stream hook that fails after ContinueContext. +type continueThenFail struct { + snapshot tracing.ContextSnapshot + createSpan bool + panics bool +} + +func (h *continueThenFail) BeforeInvoke(operator.Invocation) error { + tracing.ContinueContext(h.snapshot) + if h.createSpan { + if _, err := tracing.CreateLocalSpan("on-snapshot"); err != nil { + return err + } + } + if h.panics { + panic("the hook panicked after continuing the context") + } + return errors.New("the hook failed after continuing the context") +} + +func (h *continueThenFail) AfterInvoke(operator.Invocation, ...interface{}) error { + return nil +} + +// A failed hook keeps the context it continued: a snapshot span cannot be +// ended on this goroutine, and its End panics. +func TestRollbackKeepsContinuedSnapshot(t *testing.T) { + for _, ignored := range []bool{false, true} { + for _, createSpan := range []bool{false, true} { + for _, panics := range []bool{false, true} { + ResetTracingContext() + Tracing.ignoreSuffix = []string{".html"} + operation := tracedPage + if ignored { + operation = ignoredPage + } + entry, err := tracing.CreateEntrySpan(operation, noHeader) + assert.NoError(t, err) + requestCtx := getTracingContext() + snapshot := tracing.CaptureContext() + so11y := GetSo11y(Tracing) + leaked, leakedIgnored := so11y.leakedContextCounter.Get(), so11y.leakedIgnoreContextCounter.Get() + + SetGLS(nil) + h := &continueThenFail{snapshot: snapshot, createSpan: createSpan, panics: panics} + name := fmt.Sprintf("ignored=%v span=%v panic=%v", ignored, createSpan, panics) + assert.NotPanicsf(t, func() { + operator.InvokeIntercepted(h, operator.NewInvocation(nil), func() []interface{} { return nil }) + }, "%s", name) + active := activeTracingSpan() + assert.Truef(t, active != nil && isContinuedSnapshot(active), "%s: the continued context should stay active, got %T", name, active) + if !createSpan { + assert.Equalf(t, leaked, so11y.leakedContextCounter.Get(), "%s", name) + assert.Equalf(t, leakedIgnored, so11y.leakedIgnoreContextCounter.Get(), "%s", name) + } + + SetGLS(requestCtx) + entry.End() + } + } + } + ResetTracingContext() +} + +// A pull consumer inside a request creates and ends an entry span per message +// without storing it. The first message reuses the request entry span, as +// before, and the next ones get their own span linked to the producer. +func TestPullConsumerInsideRequestKeepsUpstreamLink(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + headers := map[string]string{} + producer, err := tracing.CreateEntrySpan("producer", noHeader) + assert.NoError(t, err) + send, err := tracing.CreateExitSpan("send", "mq:6650", func(k, v string) error { + headers[k] = v + return nil + }) + assert.NoError(t, err) + send.End() + producer.End() + waitReportedSpans(t, 2) + SetGLS(nil) + + assert.NotEmpty(t, headers[Header]) + Tracing.Reporter = &StoreReporter{} + + var request TracingSpan + var consumed []TracingSpan + intercept(&spanInterceptor{operation: "GET:/http"}, func() { + request = activeTracingSpan() + for i := 0; i < 2; i++ { + _, err := tracing.CreateEntrySpan("consume", func(k string) (string, error) { return headers[k], nil }) + assert.NoError(t, err) + consumed = append(consumed, activeTracingSpan()) + tracing.ActiveSpan().End() + } + }) + if assert.Len(t, consumed, 2) { + assert.Same(t, request, consumed[0], "the first message reuses the request entry span") + assert.NotSame(t, request, consumed[1], "the next message should get its own span") + } + assert.Nil(t, Tracing.ActiveSpan()) + // the request span, renamed by the reuse, and the second message's span + // with the producer reference + spans := waitReportedSpans(t, 2) + var linked int + for _, s := range spans { + if s.OperationName() == "consume" && len(s.Refs()) > 0 { + linked++ + } + } + assert.Equal(t, 2, len(spans)) + assert.Equal(t, 1, linked, "the second message should be linked to the producer") +} + +// A failed hook after an orphan rolls back its own span: the mark is taken +// after the orphan is ended. +func TestFailedHookAfterOrphanIsRolledBack(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + so11y := GetSo11y(Tracing) + leaked := so11y.leakedContextCounter.Get() + intercept(&spanInterceptor{operation: tracedPage, skipEnd: true}, nil) + for _, fail := range []failMode{failError, failPanic} { + intercept(&spanInterceptor{operation: "GET:/second", fail: fail}, nil) + assert.Nilf(t, Tracing.ActiveSpan(), "fail=%d: the span of the failed hook should be ended", fail) + } + assert.Equal(t, leaked+3, so11y.leakedContextCounter.Get(), "the orphan and both failed spans are counted") + assertNextRequestTraced(t) +} + +// An End after AsyncFinish ended a reused entry span still makes its parent +// active, so the span is not left active and counted as leaked. +func TestReusedEntrySpanFinishedAsyncIsPopped(t *testing.T) { + defer ResetTracingContext() + ResetTracingContext() + so11y := GetSo11y(Tracing) + leaked := so11y.leakedContextCounter.Get() + async := &spanInterceptor{operation: "GET:/async", async: true} + intercept(&spanInterceptor{operation: "GET:/raw"}, func() { + intercept(async, nil) + if assert.NotNil(t, async.asyncSpan) { + async.asyncSpan.AsyncFinish() + } + }) + assert.Nil(t, Tracing.ActiveSpan(), "the outer End should pop the span") + assertNextRequestTraced(t) + assert.Equal(t, leaked, so11y.leakedContextCounter.Get(), "nothing leaked") +} + +func BenchmarkMarkActiveSpan(b *testing.B) { + defer ResetTracingContext() + marker := operator.GetOperator().Tracing().(interface { + ActiveSpanMark() (interface{}, int) + }) + s, _ := tracing.CreateEntrySpan("GET:/bench", noHeader) + b.ResetTimer() + for i := 0; i < b.N; i++ { + _, _ = marker.ActiveSpanMark() + } + b.StopTimer() + s.End() +} + +func BenchmarkMarkActiveSpanNoContext(b *testing.B) { + defer ResetTracingContext() + SetGLS(nil) + marker := operator.GetOperator().Tracing().(interface { + ActiveSpanMark() (interface{}, int) + }) + for i := 0; i < b.N; i++ { + _, _ = marker.ActiveSpanMark() + } +} diff --git a/plugins/core/tracing.go b/plugins/core/tracing.go index dc0dd691..9793b42d 100644 --- a/plugins/core/tracing.go +++ b/plugins/core/tracing.go @@ -327,7 +327,9 @@ func (t *Tracer) createNoop(operationName string) (*TracingContext, TracingSpan, if ctx != nil { // Under an active span, the new span is a child of it, or a noop when // the active span is a noop. The checks below don't apply to it. - if span := ctx.ActiveSpan(); span != nil { + // A span left active by an interceptor that has finished is ended + // first: it doesn't belong to this request. + if span := t.endOrphanSpans(ctx); span != nil { noop, ok := span.(*NoopSpan) if ok { // increase the stack count for ensure the noop span can be clear in the context diff --git a/plugins/core/tracing/bridge.go b/plugins/core/tracing/bridge.go index 2d7216e2..c0f5e9df 100644 --- a/plugins/core/tracing/bridge.go +++ b/plugins/core/tracing/bridge.go @@ -51,6 +51,13 @@ func newSpanAdapter(s AdaptSpan) Span { return &SpanWrapper{Span: s} } +// BindOwner binds the span to the invocation that stores it with SetContext. +func (s *SpanWrapper) BindOwner(owner chan struct{}) { + if binder, ok := s.Span.(interface{ BindOwner(chan struct{}) }); ok { + binder.BindOwner(owner) + } +} + func (s *SpanWrapper) TraceID() string { return s.Span.GetTraceID() } diff --git a/plugins/core/tracing/bridge_test.go b/plugins/core/tracing/bridge_test.go new file mode 100644 index 00000000..08d77847 --- /dev/null +++ b/plugins/core/tracing/bridge_test.go @@ -0,0 +1,44 @@ +// Licensed to Apache Software Foundation (ASF) under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Apache Software Foundation (ASF) licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package tracing + +import "testing" + +type ownedAdaptSpan struct { + AdaptSpan + owner chan struct{} +} + +func (s *ownedAdaptSpan) BindOwner(owner chan struct{}) { + s.owner = owner +} + +type unownedAdaptSpan struct { + AdaptSpan +} + +func TestSpanWrapperBindOwner(t *testing.T) { + owner := make(chan struct{}) + owned := &ownedAdaptSpan{} + newSpanAdapter(owned).(*SpanWrapper).BindOwner(owner) + if owned.owner != owner { + t.Fatal("the wrapper should bind the owner to the agent span") + } + // a span that cannot be owned is left alone + newSpanAdapter(&unownedAdaptSpan{}).(*SpanWrapper).BindOwner(owner) +} diff --git a/plugins/grpc-v184/client_recvmsg_interceptor.go b/plugins/grpc-v184/client_recvmsg_interceptor.go index 3a2356fd..b0603a81 100644 --- a/plugins/grpc-v184/client_recvmsg_interceptor.go +++ b/plugins/grpc-v184/client_recvmsg_interceptor.go @@ -46,6 +46,14 @@ type clientRecvTrailer struct { data *contextData } +// clientRecvSpan is the invocation context of a RecvMsg call that creates a +// span. The span is wrapped, not stored directly, because it may stay active +// after this call until the trailer read: the agent ends a span stored with +// SetContext that is still active after its interceptor finished. +type clientRecvSpan struct { + span tracing.Span +} + func (h *ClientRecvMsgInterceptor) BeforeInvoke(invocation operator.Invocation) error { cs := invocation.CallerInstance().(*nativeclientStream) method := cs.callHdr.Method @@ -69,7 +77,7 @@ func (h *ClientRecvMsgInterceptor) BeforeInvoke(invocation operator.Invocation) if err != nil { return err } - invocation.SetContext(s) + invocation.SetContext(&clientRecvSpan{span: s}) return nil } @@ -83,7 +91,11 @@ func (h *ClientRecvMsgInterceptor) AfterInvoke(invocation operator.Invocation, r endTrailerRead(cs, trailer.data, err) return nil } - span := invocation.GetContext().(tracing.Span) + recv, ok := invocation.GetContext().(*clientRecvSpan) + if !ok { + return nil + } + span := recv.span if err == nil && cs.desc != nil && !cs.desc.ServerStreams && keepRecvSpanUntilTrailer(invocation.CallerInstance(), span) { return nil } diff --git a/tools/go-agent/instrument/plugins/enhance_method_test.go b/tools/go-agent/instrument/plugins/enhance_method_test.go new file mode 100644 index 00000000..6ed0b0be --- /dev/null +++ b/tools/go-agent/instrument/plugins/enhance_method_test.go @@ -0,0 +1,131 @@ +// Licensed to Apache Software Foundation (ASF) under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Apache Software Foundation (ASF) licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package plugins + +import ( + "bytes" + "go/parser" + "go/token" + "strings" + "testing" + + "github.com/dave/dst" + "github.com/dave/dst/decorator" + + "github.com/apache/skywalking-go/plugins/core/instrument" + "github.com/apache/skywalking-go/tools/go-agent/tools" +) + +// renderDelegator renders the before and after functions that the agent +// generates for a method. +func renderDelegator(t *testing.T) (before, after string) { + t.Helper() + file, err := decorator.Parse(`package gin + +type Engine struct{} + +func (e *Engine) handle(path string) (int, error) { + return 0, nil +} +`) + if err != nil { + t.Fatal(err) + } + var fn *dst.FuncDecl + for _, decl := range file.Decls { + if f, ok := decl.(*dst.FuncDecl); ok { + fn = f + } + } + enhance := NewMethodEnhance(NewTestInstrument("github.com/gin-gonic/gin"), + &instrument.Point{Interceptor: "HandleInterceptor"}, fn, "/gin/engine.go", tools.CreateImportAnalyzer()) + for _, decl := range enhance.BuildForDelegator() { + f, ok := decl.(*dst.FuncDecl) + if !ok { + continue + } + var buf bytes.Buffer + if err := decorator.Fprint(&buf, &dst.File{Name: dst.NewIdent("gin"), Decls: []dst.Decl{f}}); err != nil { + t.Fatal(err) + } + if _, err := parser.ParseFile(token.NewFileSet(), "", buf.String(), 0); err != nil { + t.Fatalf("the generated %s is not valid Go: %v\n%s", f.Name.Name, err, buf.String()) + } + switch f.Name.Name { + case enhance.AdapterPreFuncName: + before = buf.String() + case enhance.AdapterPostFuncName: + after = buf.String() + } + } + if before == "" || after == "" { + t.Fatal("the before or after function was not generated") + } + return before, after +} + +func indexOf(t *testing.T, src, part string) int { + t.Helper() + i := strings.Index(src, part) + if i < 0 { + t.Fatalf("%q is missing from the generated code:\n%s", part, src) + } + return i +} + +// The before function marks the active span before BeforeInvoke, rolls it back +// when BeforeInvoke fails or panics, and finishes the invocation when the +// call is skipped, because the after function is not called then. +func TestBeforeTemplateRollsBackFailedHook(t *testing.T) { + before, _ := renderDelegator(t) + mark := indexOf(t, before, "operator.markActiveSpan()") + invoke := indexOf(t, before, ".BeforeInvoke(invocation)") + if mark > invoke { + t.Fatal("the active span must be marked before BeforeInvoke") + } + if n := strings.Count(before, "operator.rollbackActiveSpan(invocation, _sw_span_mark, _sw_span_depth)"); n != 2 { + t.Fatalf("the rollback should run on a panic and on an error, found %d calls:\n%s", n, before) + } + recoverAt := indexOf(t, before, "recover()") + if recoverAt > mark { + t.Fatal("the panic rollback must be deferred before the mark is taken") + } + indexOf(t, before, "var _sw_span_depth = -1") + skip := indexOf(t, before, "if invocation.isContinue") + exited := indexOf(t, before, "invocation.markExited()") + if exited < skip { + t.Fatal("the skipped call must finish the invocation") + } + // a failed conversion of the return values must not finish it, because + // the call and the after-hook still run then + if conversion := indexOf(t, before, "invocation.returnValues[0]"); exited < conversion { + t.Fatal("the skipped call must be finished after the return values are converted") + } +} + +// The after function finishes the invocation last, even when AfterInvoke panics. +func TestAfterTemplateFinishesInvocation(t *testing.T) { + _, after := renderDelegator(t) + exited := indexOf(t, after, "defer invocation.markExited()") + if recoverAt := indexOf(t, after, "recover()"); exited > recoverAt { + t.Fatal("markExited must be deferred first, so it runs after the recover") + } + if invoke := indexOf(t, after, ".AfterInvoke(invocation"); exited > invoke { + t.Fatal("markExited must be deferred before AfterInvoke runs") + } +} diff --git a/tools/go-agent/instrument/plugins/rewrite/context.go b/tools/go-agent/instrument/plugins/rewrite/context.go index 7f8fb3f4..aafc4302 100644 --- a/tools/go-agent/instrument/plugins/rewrite/context.go +++ b/tools/go-agent/instrument/plugins/rewrite/context.go @@ -342,9 +342,17 @@ func (c *Context) enhanceTypeNameWhenRewrite(fieldType dst.Expr, parent dst.Node return "", "" } +// builtinFuncs are the predeclared functions of Go. They are not declared in +// the copied packages, so they must never be renamed. +var builtinFuncs = map[string]bool{ + "append": true, "cap": true, "clear": true, "close": true, "complex": true, "copy": true, + "delete": true, "imag": true, "len": true, "make": true, "max": true, "min": true, + "new": true, "panic": true, "print": true, "println": true, "real": true, "recover": true, +} + func (c *Context) typeIsBasicTypeValueOrEnhanceName(name string) bool { if strings.HasPrefix(name, OperatePrefix) || strings.HasPrefix(name, GenerateMethodPrefix) || tools.IsBasicDataType(name) || - name == "nil" || name == "true" || name == "false" || name == "append" || name == "panic" || name == "new" || name == "delete" { + name == "nil" || name == "true" || name == "false" || builtinFuncs[name] { return true } if _, valErr := strconv.ParseFloat(name, 64); valErr == nil { @@ -358,8 +366,7 @@ func (c *Context) alreadyGenerated(name string) bool { } func (c *Context) callIsBasicNamesOrEnhanceName(name string) bool { - return strings.HasPrefix(name, OperatePrefix) || strings.HasPrefix(name, GenerateMethodPrefix) || - name == "make" || name == "recover" || name == "len" + return strings.HasPrefix(name, OperatePrefix) || strings.HasPrefix(name, GenerateMethodPrefix) || builtinFuncs[name] } func (r *rewriteImportInfo) generateStaticMethod(name string) *dst.Ident { diff --git a/tools/go-agent/instrument/plugins/rewrite/context_test.go b/tools/go-agent/instrument/plugins/rewrite/context_test.go new file mode 100644 index 00000000..bd928402 --- /dev/null +++ b/tools/go-agent/instrument/plugins/rewrite/context_test.go @@ -0,0 +1,39 @@ +// Licensed to Apache Software Foundation (ASF) under one or more contributor +// license agreements. See the NOTICE file distributed with +// this work for additional information regarding copyright +// ownership. Apache Software Foundation (ASF) licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may +// not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package rewrite + +import "testing" + +// A builtin function called in a copied agent core file, such as close in the +// operator package, must keep its name. +func TestBuiltinFunctionsAreNotRenamed(t *testing.T) { + c := &Context{} + for _, name := range []string{"append", "cap", "clear", "close", "complex", "copy", "delete", "imag", + "len", "make", "max", "min", "new", "panic", "print", "println", "real", "recover"} { + if !c.callIsBasicNamesOrEnhanceName(name) || !c.typeIsBasicTypeValueOrEnhanceName(name) { + t.Errorf("the builtin %s would be renamed", name) + } + info := &rewriteImportInfo{ctx: c, isAgentCore: true, pkgName: "operator"} + if got := info.generateStaticMethod(name).Name; got != name { + t.Errorf("the builtin %s is renamed to %s", name, got) + } + } + if c.callIsBasicNamesOrEnhanceName("markExited") { + t.Error("a function of the copied package must be renamed") + } +} diff --git a/tools/go-agent/instrument/plugins/templates/method_intercept_after.tmpl b/tools/go-agent/instrument/plugins/templates/method_intercept_after.tmpl index 26238e85..e31f7ae5 100644 --- a/tools/go-agent/instrument/plugins/templates/method_intercept_after.tmpl +++ b/tools/go-agent/instrument/plugins/templates/method_intercept_after.tmpl @@ -1,3 +1,5 @@ +// runs last: the interceptor has finished, even when the after-hook panics +defer invocation.markExited() defer func() { if r := recover(); r != nil { operator.ErrorOfPlugin("{{.InstrumentName}}") diff --git a/tools/go-agent/instrument/plugins/templates/method_intercept_before.tmpl b/tools/go-agent/instrument/plugins/templates/method_intercept_before.tmpl index a2c17dd9..5066c262 100644 --- a/tools/go-agent/instrument/plugins/templates/method_intercept_before.tmpl +++ b/tools/go-agent/instrument/plugins/templates/method_intercept_before.tmpl @@ -1,8 +1,12 @@ invocation = &operator.realInvocation{} invocation.interTimeCost = int64(0) invocation.beforeInterStart = operator.NanoTime() +var _sw_span_mark interface{} +// no mark yet, so a rollback does nothing +var _sw_span_depth = -1 defer func() { if err := recover(); err != nil { + operator.rollbackActiveSpan(invocation, _sw_span_mark, _sw_span_depth) operator.ErrorOfPlugin("{{.InstrumentName}}") // log error log.Errorf("execute interceptor before invoke error, instrument name: %s, interceptor name: %s, function ID: %s, error: %v, stack: %s", @@ -32,8 +36,10 @@ invocation.changeArgCallback = func(index int, value interface{}) { {{- range $index, $value := .Results }} var def_res_{{$index}} {{$value.PackagedTypeName}} {{- end}} +_sw_span_mark, _sw_span_depth = operator.markActiveSpan() // real invoke if err := {{.InterceptorVarName}}.BeforeInvoke(invocation); err != nil { + operator.rollbackActiveSpan(invocation, _sw_span_mark, _sw_span_depth) // using go2sky log error log.Warnf("execute interceptor before invoke error, instrument name: %s, interceptor name: %s, function ID: %s, error: %v", "{{.InstrumentName}}", "{{.InterceptorDefineName}}", "{{.FuncID}}", err) @@ -49,6 +55,8 @@ if (invocation.isContinue) { def_res_{{$index}} = (invocation.returnValues[{{$index}}]).({{$value.PackagedTypeName}}) } {{- end}} + // the after-hook is not called, so the interceptor has finished + invocation.markExited() invocation.interTimeCost += operator.NanoTime() - invocation.beforeInterStart return {{ range $index, $value := .Results -}}