feat: support log line content filtering via RELABEL_CONFIGS#98
Open
Omerlustig wants to merge 1 commit intografana:mainfrom
Open
feat: support log line content filtering via RELABEL_CONFIGS#98Omerlustig wants to merge 1 commit intografana:mainfrom
Omerlustig wants to merge 1 commit intografana:mainfrom
Conversation
Enable RELABEL_CONFIGS to filter log entries based on log line content using a synthetic __log_line__ label. This allows dropping noisy lines (e.g. "END RequestId:") before they are sent to Loki, reducing ingestion volume and cost. The __log_line__ label is injected temporarily before relabeling and stripped afterwards so it never reaches Loki. Also fixes a pre-existing bug where labels were not sorted before relabel.Process (which uses binary search), causing non-deterministic relabeling results. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
RELABEL_CONFIGSto filter log entries based on log line content using a synthetic__log_line__label, allowing users to drop noisy lines (e.g.END RequestId:) before they are sent to Loki__log_line__label is injected temporarily before relabeling and stripped afterwards so it never reaches LokiScratchBuilder.Sort()was missing beforerelabel.Process, which uses binary search internally and produced non-deterministic results on unsorted labelsHow it works
Users can now use the existing
RELABEL_CONFIGSenv var withsource_labels: ["__log_line__"]to match against log line content:[{"source_labels":["__log_line__"],"regex":"END RequestId:.*","action":"drop"}]All Prometheus relabel actions (
drop,keep,replace,labeldrop, etc.) are supported. Existing label-only relabel configs continue to work unchanged.Test plan
TestBatchAddDropsLogLinesViaRelabelConfig— drop action on log line contentTestBatchAddKeepsEntriesWithNoRelabelConfig— no-op when no configs setTestBatchAddKeepActionFiltersLogLines— keep action filteringTestBatchAddLogLineLabelNotInOutput—__log_line__label never in outputTestBatchAddLabelOnlyRelabelStillWorks— backward compat with label-only relabelTestBatchAddDropMultiplePatterns— multi-pattern drop🤖 Generated with Claude Code