feat(boto3): Add response, retry, and error span attributes - #7499
pabloDeputter wants to merge 20 commits into
Conversation
Codecov Results 📊✅ 130355 passed | ⏭️ 7192 skipped | Total: 137547 | Pass Rate: 94.77% | Execution Time: 445m 57s 📊 Comparison with Base Branch
All tests are passing successfully. ✅ Patch coverage is 100.00%. Project has 2561 uncovered lines. Files with missing lines (1)
Coverage diff@@ Coverage Diff @@
## master #PR +/-##
==========================================
+ Coverage 90.28% 90.32% +0.04%
==========================================
Files 195 199 +4
Lines 26195 26460 +265
Branches 9758 9824 +66
==========================================
+ Hits 23649 23899 +250
- Misses 2546 2561 +15
- Partials 1483 1493 +10Generated by Codecov Action |
a15c641 to
56a7ab6
Compare
56a7ab6 to
176096b
Compare
af6b9a8 to
49474d0
Compare
2e63bac to
35f0dc3
Compare
76e7425 to
f922864
Compare
4fae1c9 to
cdd5c95
Compare
cdd5c95 to
d55060f
Compare
6c9e3a8 to
a72a791
Compare
8f65118 to
96a40ea
Compare
| _set_span_attributes(span, _get_error_attributes(error)) | ||
| raise | ||
| else: | ||
| with capture_internal_exceptions(): |
There was a problem hiding this comment.
Because this swallows exceptions, I don't think the else block is needed here, and this can instead be moved up to under line 115.
There was a problem hiding this comment.
I removed the else block, but kept it outside the inner try of orig_make_api_call() cause I think that represents the behavior better: 1. call original client 2. if it fails we add error-attributes and re-raise 3. otherwise, we add response-attributes
| def _set_span_attributes( | ||
| span: "Union[Span, StreamedSpan]", attributes: "Attributes" | ||
| ) -> None: | ||
| """Will be removed in the major.""" |
There was a problem hiding this comment.
This is a little vague 😅
Are you referring to the entire function? Or are you referring to a subset of the functionality that's here?
Generally speaking, for comments like these that are indicating the removal/modification of code at a particular point in time, it's worth being specific so that we know exactly which major the code is intended to be removed by.
There was a problem hiding this comment.
I left the comment to justify the set_span_attributes() helper, cause otherwise you would have multiple conditionals intertwined with the other logic. It also makes it easier to move for the next major (3.0).
The goal was to merge into main and then remove all the send_default_pii and legacy streaming stuff once again and merge into the major branch and the helper makes it a little bit easier to do that.
There was a problem hiding this comment.
Ah ok, sounds good. 👍🏻 Thanks!
| attributes: "Attributes" = {} | ||
|
|
||
| # botocore injects HTTP status into `ResponseMetadata` after parsing. | ||
| # https://github.com/boto/botocore/blob/develop/botocore/parsers.py#L273-L284 |
There was a problem hiding this comment.
Here and on line 111, we should use a permalink in case the upstream repository changes the file path
| attributes[SPANDATA.HTTP_STATUS_CODE] = status_code | ||
|
|
||
| retry_attempts = metadata.get("RetryAttempts") | ||
| # botocore represents retries as `attempts - 1`; OTel suggests "if and only if", so skip zero. |
There was a problem hiding this comment.
This comment isn't clear to me (specifically, the otel part). From my understanding of what this attribute (HTTP_REQUEST_RESEND_COUNT) is intended to represent, it's the number of retries, which does not include the original request. Is that correct?
If that's the case, I think the code that you've written below, from lines 113 to 119, is clear enough on its own that this extra comment is unnecessary.
There was a problem hiding this comment.
Yepp, correct: HTTP_REQUEST_RESEND_COUNT basically represents the number of attempts before success. I'll remove the extra comment
| # https://opentelemetry.io/docs/specs/semconv/http/http-spans/#http-client-span | ||
| if ( | ||
| isinstance(retry_attempts, int) | ||
| # avoid emitting `resend_count=True`. |
There was a problem hiding this comment.
Under what circumstances would reset_count be a boolean?
| headers = metadata.get("HTTPHeaders") | ||
| if not isinstance(headers, dict): | ||
| headers = {} |
There was a problem hiding this comment.
I don't think this check is necessary because, based on the type definition that I can see here, the HTTPHeaders will always be a dictionary.
| if isinstance(error, dict): | ||
| error_code = error.get("Code") | ||
| if isinstance(error_code, str) and error_code: |
There was a problem hiding this comment.
I'm surprised that you need to perform these isinstance checks because the types that I'm reading about these properties are indicating that the error will be a dictionary and the code will be a string.
Is the type checker telling you that these could be something different (like an Any)?
There was a problem hiding this comment.
I'm pretty sure that I gaslit myself by the clanker to believe that this was justified 🫣 I wanted to make the instrumentation-code as defensive as possible so it won't throw any exceptions or we won't lose any attributes, but this is already guaranteed by capture_internal_exceptions(); but you're right, these are pretty useless cause this will almost never happen (same for HTTPHeaders and RetryAttempts);
There are more of these useless conditionals, so I'll double check these and remove these as well in the next PR.
| exception_name = exception_type.__qualname__ | ||
| exception_module = exception_type.__module__ | ||
| if exception_module not in ("builtins", "__builtins__"): | ||
| return "%s.%s" % (exception_module, exception_name) |
There was a problem hiding this comment.
These can be updated to be f-strings
96a40ea to
fd30806
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit fd30806. Configure here.

Description
Add generic response, retry, and error attribute enrichment to boto3 spans. Previously, only recorded request and client metadata was instrumented; information after the call such as HTTP status, retry count, etc was omitted. The new attributes follow the OTel AWS SDK conventions.
Changes
SPANDATAconsts:AWS_REQUEST_ID,AWS_EXTENDED_REQUEST_ID,HTTP_REQUEST_RESEND_COUNT, andERROR_TYPE._get_response_attributes()to extracthttp.response_status_code,http.request_resend_count,aws.request_id, andaws.extended_request_id.erorr.typefromClientError.Eror.Codeor the exception class, without recording error messages.Issues
Resolves #7476