Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,9 @@ Release Notes.
* Keep a trace that the caller has already sampled. An entry span that starts a tracing context checked for the upstream `sw8` reference before attaching it, so the local sampler ran even when the request carried a valid `sw8` header: with `agent.sampler` below 1, a downstream service dropped already-sampled traces, sent no `sw8` header onward, and its own downstream services could start unrelated traces. The sampler also received an empty operation name, and the `sw_go_created_tracing_context_counter{created_by="propagated"}` self-observability counter never increased. A `sw8` header now counts as valid only when every required field is set, as in the Java agent, so a header such as `1---0----` no longer creates a segment with an empty trace ID.
* Apply the ignore rules (`agent.ignore_suffix` and `agent.trace_ignore_path`) and the disconnected-reporter check to every request that starts a trace. A goroutine keeps its tracing context after a trace ends, and the checks ran only on a goroutine without one. On a goroutine that serves several requests, such as the goroutine of an HTTP/1 keep-alive connection, every request after the first was therefore traced even when it should be ignored. Also, the spans that an ignored request creates in its own goroutine, such as its database calls, are now ignored, as they already were under a request that the sampler drops, instead of each one starting its own trace. When several goroutines continue one captured context, such as the send and receive goroutines of a gRPC stream, each now gets its own noop span for an ignored or sampled-out request, instead of sharing one span's nesting count.
* Fix the gin and goframe plugins with a negative `header_length_threshold`. Collecting the request headers panicked after the entry span was created, so the span was never ended and stayed active on its goroutine. A value of 0 or less now means no limit, as for `redis.max_args_bytes`; before, 0 sent an empty value, while `collect_request_headers` already decides whether headers are collected at all. The plugins also pass the span to the after-hook before collecting the headers.
* End the spans that a plugin leaves active on a goroutine, so they cannot ignore or capture the later requests on it. The agent binds a span to the interceptor that stores it with `invocation.SetContext`. When a later span starts on the goroutine and the active span belongs to an interceptor that has finished, for example because its after-hook did not end the span or panicked first, the agent ends that span and counts it in `sw_go_possible_leaked_context_counter`. When `BeforeInvoke` returns an error or panics after it created spans and before it stored one with `SetContext`, the agent ends the spans it created; a context that the hook continued, such as the snapshot of a stream, is kept. Before, such a span stayed active: under the noop span of an ignored, sampled-out, or disconnected-reporter request, every later request on the goroutine, such as an HTTP/1 keep-alive connection, was ignored, and under a traced span they joined the leaked trace. Spans that no interceptor stores, such as the spans of the toolkit API or of gorm callbacks, are not ended by the agent.
* Keep a reused entry span active until the outer plugin ends it, when the nested plugin that reused it is an interceptor that stores it with `invocation.SetContext`, such as gin mounted on a net/http `ServeMux`. Before, the nested End made the parent active while the outer plugin was still serving the request, so the spans created afterwards started new traces. A nested reuser that only creates and ends the span, such as a kratos middleware, a pull consumer reading messages inside a request, or the toolkit API, still makes the parent active, so its next span is created as before. An exit span reused by a nested plugin, such as the sql driver inside gorm, is also still popped.
* Keep the names of all Go builtin functions, such as `close`, `copy`, `min`, and `max`, when the agent copies the agent core packages into an instrumented package. Before, only some builtins were known, and the others were renamed like functions of the copied package, so the instrumented build failed.

#### Unsupported Library Versions

Expand Down
10 changes: 10 additions & 0 deletions docs/en/development-and-contribution/development-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,16 @@ rather than creating a new instance each time a method is intercepted.

Therefore, do not declare objects in the interceptor, and instead use `Invocation.Context` to pass data.

##### Span lifecycle

A span that an interceptor creates must be ended by the same interception, usually in `AfterInvoke`. Store it with `invocation.SetContext` right after creating it, before anything that can fail, so that `AfterInvoke` gets it.

The agent protects the goroutine from a span that is not ended:
- A span stored with `SetContext` belongs to that interception. If it is still active when a later span starts on the goroutine after the interception has finished, the agent ends it and counts it in `sw_go_possible_leaked_context_counter`. Only the span itself is bound: a value that wraps the span, such as a struct stored with `SetContext`, is not.
- If `BeforeInvoke` returns an error or panics after creating spans and before storing anything with `SetContext`, the agent ends the spans it created and counts them the same way. A context that the hook continued with `ContinueContext` is kept. Nothing is ended when the hook stored any value with `SetContext`, because `AfterInvoke` still runs and ends it, or when the hook changed the active span in another way, for example by ending a span that was active before it.

A span that must stay active after its interception has finished, such as a span that a later call ends, must not be stored directly with `SetContext`: store it in a wrapper value. A span ended with `PrepareAsync` is not active anymore once `End` is called in the interception, and is finished later with `AsyncFinish`.

##### Package Path

If the method you want to intercept is not located in the root directory of the framework,
Expand Down
44 changes: 44 additions & 0 deletions plugins/core/operator/intercepter_test_base.go
Original file line number Diff line number Diff line change
Expand Up @@ -31,3 +31,47 @@ func NewInvocation(receiver interface{}, args ...interface{}) Invocation {
},
}
}

// InvokeIntercepted runs the interceptor around call in the same order as the
// code that the agent generates: method_intercept_before.tmpl,
// method_inserts.tmpl and method_intercept_after.tmpl. A panic in an
// interceptor is recovered, as in the generated code. It returns false when
// the before-hook skipped the call.
func InvokeIntercepted(interceptor Interceptor, invocation Invocation, call func() []interface{}) (called bool) {
inv, ok := invocation.(*realInvocation)
if !ok {
panic("InvokeIntercepted needs an invocation from NewInvocation")
}
skip := func() (skip bool) {
var mark interface{}
depth := -1
defer func() {
if r := recover(); r != nil {
rollbackActiveSpan(inv, mark, depth)
}
}()
mark, depth = markActiveSpan()
if err := interceptor.BeforeInvoke(inv); err != nil {
rollbackActiveSpan(inv, mark, depth)
return false
}
Comment thread
mrproliu marked this conversation as resolved.
Resolved
if inv.isContinue {
inv.markExited()
return true
}
return false
}()
if skip {
return false
}
var results []interface{}
Comment thread
mrproliu marked this conversation as resolved.
Resolved
defer func() {
defer inv.markExited()
defer func() {
_ = recover()
}()
_ = interceptor.AfterInvoke(inv, results...)
}()
results = call()
return true
}
66 changes: 66 additions & 0 deletions plugins/core/operator/invocation.go
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,10 @@ type realInvocation struct {
returnValues []interface{}

context interface{}
// owner is closed when the interceptor has finished, so the span it
// stored with SetContext can be ended if it is still active. It is a
// channel because this copied file cannot import sync/atomic.
owner chan struct{}

// self obs
interTimeCost int64
Expand Down Expand Up @@ -56,6 +60,68 @@ func (i *realInvocation) DefineReturnValues(values ...interface{}) {

func (i *realInvocation) SetContext(context interface{}) {
i.context = context
if span, ok := context.(spanOwnerBinder); ok {
if i.owner == nil {
i.owner = make(chan struct{})
}
span.BindOwner(i.owner)
}
}

// spanOwnerBinder is a span that can be bound to the invocation that ends it.
// Only builtin types are used, because every instrumented package has its own
// copy of this package.
type spanOwnerBinder interface {
BindOwner(owner chan struct{})
}

type activeSpanMarker interface {
ActiveSpanMark() (interface{}, int)
RollbackActiveSpan(mark interface{}, depth int)
}

// markExited records that the interceptor of the invocation has finished, so
// the span it stored with SetContext must no longer be active.
func (i *realInvocation) markExited() {
if i != nil && i.owner != nil {
close(i.owner)
i.owner = nil
}
}

func spanMarker() activeSpanMarker {
op := GetOperator()
if op == nil {
return nil
}
marker, _ := op.Tracing().(activeSpanMarker)
return marker
}

// markActiveSpan returns the active span before BeforeInvoke runs.
func markActiveSpan() (mark interface{}, depth int) {
if marker := spanMarker(); marker != nil {
return marker.ActiveSpanMark()
}
return nil, 0
}

// rollbackActiveSpan ends the spans that a failed BeforeInvoke created and did
// not store with SetContext, so nothing else would end them.
func rollbackActiveSpan(i *realInvocation, mark interface{}, depth int) {
// It runs in the recover of the before-hook too, where a panic would
// reach the application.
defer func() {
_ = recover()
}()
if i == nil || i.context != nil || depth < 0 {
// the after-hook still runs and ends the span it got, or the active
// span was not marked
return
}
if marker := spanMarker(); marker != nil {
marker.RollbackActiveSpan(mark, depth)
}
}

func (i *realInvocation) GetContext() interface{} {
Expand Down
137 changes: 137 additions & 0 deletions plugins/core/operator/invocation_test.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
// Licensed to Apache Software Foundation (ASF) under one or more contributor
// license agreements. See the NOTICE file distributed with
// this work for additional information regarding copyright
// ownership. Apache Software Foundation (ASF) licenses this file to you under
// the Apache License, Version 2.0 (the "License"); you may
// not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing,
// software distributed under the License is distributed on an
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, either express or implied. See the License for the
// specific language governing permissions and limitations
// under the License.

package operator

import "testing"

type ownedSpan struct {
owners []chan struct{}
}

func closed(owner chan struct{}) bool {
select {
case <-owner:
return true
default:
return false
}
}

func (s *ownedSpan) BindOwner(owner chan struct{}) {
s.owners = append(s.owners, owner)
}

// fakeTracing records the calls of the template helpers.
type fakeTracing struct {
mark interface{}
depth int
rollbacks []int
}

func (f *fakeTracing) ActiveSpanMark() (mark interface{}, depth int) {
return f.mark, f.depth
}

func (f *fakeTracing) RollbackActiveSpan(_ interface{}, depth int) {
f.rollbacks = append(f.rollbacks, depth)
}

type fakeOperator struct {
Operator
tracing interface{}
}

func (f *fakeOperator) Tracing() interface{} {
return f.tracing
}

func withOperator(t *testing.T, op Operator) {
t.Helper()
old := GetOperator
GetOperator = func() Operator { return op }
t.Cleanup(func() { GetOperator = old })
}

func TestSetContextBindsSpanToInvocation(t *testing.T) {
inv := &realInvocation{}
span := &ownedSpan{}
inv.SetContext(span)
inv.SetContext(span)
if len(span.owners) != 2 || span.owners[0] == nil || span.owners[0] != span.owners[1] {
t.Fatalf("every SetContext should bind the same owner of the invocation, got %v", span.owners)
}
if closed(span.owners[0]) {
t.Fatal("the owner should not be finished before the interceptor finishes")
}
inv.markExited()
if !closed(span.owners[0]) {
t.Fatal("markExited should finish the owner")
}
// the after-hook and the skip path may both finish the invocation
inv.markExited()
}

func TestSetContextWithoutSpanHasNoOwner(t *testing.T) {
inv := &realInvocation{}
inv.SetContext("not a span")
inv.SetContext(nil)
inv.markExited()
if inv.owner != nil {
t.Fatal("a context that is not a span should not create an owner")
}
if inv.GetContext() != nil {
t.Fatal("SetContext should still store the context")
}
var nilInvocation *realInvocation
nilInvocation.markExited()
}

func TestMarkActiveSpanWithoutTracer(t *testing.T) {
withOperator(t, nil)
if mark, depth := markActiveSpan(); mark != nil || depth != 0 {
t.Fatalf("no tracer should mark nothing, got %v %d", mark, depth)
}
withOperator(t, &fakeOperator{tracing: struct{}{}})
if mark, depth := markActiveSpan(); mark != nil || depth != 0 {
t.Fatalf("a tracer without marks should mark nothing, got %v %d", mark, depth)
}
rollbackActiveSpan(&realInvocation{}, nil, 0)
}

func TestRollbackActiveSpan(t *testing.T) {
tracer := &fakeTracing{mark: "active", depth: 2}
withOperator(t, &fakeOperator{tracing: tracer})
mark, depth := markActiveSpan()
if mark != "active" || depth != 2 {
t.Fatalf("markActiveSpan should return the tracer mark, got %v %d", mark, depth)
}

rollbackActiveSpan(nil, mark, depth)
rollbackActiveSpan(&realInvocation{}, mark, -1)
withContext := &realInvocation{}
withContext.SetContext("context")
rollbackActiveSpan(withContext, mark, depth)
if len(tracer.rollbacks) != 0 {
t.Fatalf("no rollback without an invocation, without a mark, or with a context, got %v", tracer.rollbacks)
}

rollbackActiveSpan(&realInvocation{}, mark, depth)
if len(tracer.rollbacks) != 1 || tracer.rollbacks[0] != 2 {
t.Fatalf("a failed hook without a context should roll back to the mark, got %v", tracer.rollbacks)
}
}
49 changes: 36 additions & 13 deletions plugins/core/span_default.go
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,9 @@ type DefaultSpan struct {
// into SegmentSpanImpl/SnapshotSpan, and an embedded sync.Mutex value would
// trip the go vet copylocks check.
opLock *sync.Mutex
// owner is closed when the interceptor that stored the span with
// SetContext has finished (see BindOwner). Guarded by opLock.
owner chan struct{}
// ended is set by endAndFreeze right before the span is handed over to the
// segment collector. Once set, every late mutation is dropped, so the span
// data is frozen and the reporting path may read it without locking
Expand All @@ -65,11 +68,12 @@ type DefaultSpan struct {
// span leaked across goroutines would otherwise flood the log with one
// line per dropped write.
droppedLogged bool
// reuseCount counts the extra logical owners that the span reuse rule in
// CreateEntrySpan/CreateExitSpan hands this span to. Every owner calls End
// exactly once and only the LAST End freezes and reports the span (see
// enterReuse and endSyncAndFreeze). Guarded by opLock.
reuseCount int
// reuses has one entry per extra owner that the span reuse rule in
// CreateEntrySpan/CreateExitSpan hands this span to, innermost last; only
// the LAST End freezes and reports the span. An entry is true when that
// owner is an interceptor that stored the span with SetContext. Guarded
// by opLock.
reuses []bool
}

func NewDefaultSpan(tracer *Tracer, parent TracingSpan) *DefaultSpan {
Expand Down Expand Up @@ -280,7 +284,7 @@ func (ds *DefaultSpan) enterReuse() {
ds.opLock.Lock()
defer ds.opLock.Unlock()
if !ds.ended {
ds.reuseCount++
ds.reuses = append(ds.reuses, false)
}
}

Expand All @@ -293,22 +297,30 @@ func (ds *DefaultSpan) enterReuse() {
// PrepareAsync/End pair free of data races).
func (ds *DefaultSpan) endSyncAndFreeze() bool {
ds.opLock.Lock()
if ds.reuseCount > 0 {
if n := len(ds.reuses); n > 0 {
// a nested owner of a reused span (see enterReuse) finished: this is
// not the last End, so keep the span open - the outer owner still
// writes to it (e.g. gorm tags db.statement after the sql driver's
// End) and will freeze it with its own End. Restoring the active span
// is kept here because the nested plugin expects its End to pop the
// span, exactly like before the freeze mechanism.
ds.reuseCount--
// End) and will freeze it with its own End.
stored := ds.reuses[n-1]
ds.reuses = ds.reuses[:n-1]
ds.opLock.Unlock()
if ctx := getTracingContext(); ctx != nil {
ctx.SaveActiveSpan(ds.Parent)
// An entry span that a nested interceptor stored, such as gin inside
// the net/http server, stays active for the outer owner's request.
// Otherwise it is popped, so the next call of the outer owner, such as
// a gorm hook query or a pull consumer's next message, gets its own span.
if ds.SpanType != SpanTypeEntry || !stored {
if ctx := getTracingContext(); ctx != nil {
ctx.SaveActiveSpan(ds.Parent)
}
}
return false
}
if !ds.EndTime.IsZero() { // already ended
ds.opLock.Unlock()
// A reused entry span stays active after its nested End, and
// AsyncFinish may have ended it since: the last End still pops it.
ds.popIfActive()
return false
}
ds.EndTime = time.Now()
Expand All @@ -328,6 +340,17 @@ func (ds *DefaultSpan) endSyncAndFreeze() bool {
return frozen
}

// popIfActive makes the parent active when this span is the active span.
func (ds *DefaultSpan) popIfActive() {
ctx := getTracingContext()
if ctx == nil {
return
}
if active, ok := ctx.ActiveSpan().(SegmentSpan); ok && active.GetDefaultSpan() == ds {
ctx.SaveActiveSpan(ds.Parent)
}
}

func (ds *DefaultSpan) IsEntry() bool {
return ds.SpanType == SpanTypeEntry
}
Expand Down
Loading
Loading