New serverless pattern - eventbridge-firehose-opensearch-cdk - #3284
Open
antoinebgit wants to merge 1 commit into
Open
New serverless pattern - eventbridge-firehose-opensearch-cdk#3284antoinebgit wants to merge 1 commit into
antoinebgit wants to merge 1 commit into
Conversation
A catch-all EventBridge rule streams every event on a custom bus to an Amazon OpenSearch Service domain through Amazon Data Firehose, giving full-text search over event payloads within about 60 seconds. - Firehose delivers to a daily-rotated index (events-YYYY-MM-DD) and backs up all documents to S3, with delivery errors logged to CloudWatch Logs. - An optional Lambda transform flattens the EventBridge envelope so detail-type becomes detail_type and detail fields are promoted to the top level, keeping envelope fields authoritative on key collisions. - The Firehose delivery role is authorized on both sides: an identity policy and the domain access policy, since a managed domain evaluates every request against its own policy. - Dashboards access is granted to one or more operator CIDRs, because browsers cannot sign requests with SigV4.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds the
eventbridge-firehose-opensearch-cdkpattern: a catch-all EventBridge rule streams every event on a custom bus to an Amazon OpenSearch Service domain through Amazon Data Firehose, giving full-text search over event payloads within about 60 seconds.How it works
{"source": [{"prefix": ""}]}) matches every event on the bus and targets a Firehose delivery stream.events-YYYY-MM-DD.detail-typebecomesdetail_typeanddetailfields are promoted to the top level. Envelope fields stay authoritative on key collisions, so a payload carrying its ownsourceis indexed asdetail_source.errors/prefix. Delivery errors are logged to CloudWatch Logs.Testing
Deployed and tested end-to-end in a live account: verified daily index rotation, envelope flattening, the collision guard, the S3 backup copy, and zero delivery errors.
Checklist
README.md,example-patternmetadata JSON, andarchitecture.pngcdk destroytested and removes all resources