Skip to content

Add Obs Pipeline Grok processor updates to OpenAPI Spec #3729

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
api-clients-generation-pipeline merged 1 commit into from
Jul 1, 2026
Merged

Add Obs Pipeline Grok processor updates to OpenAPI Spec #3729

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 47 additions & 2 deletions .generator/schemas/v2/openapi.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -67723,6 +67723,10 @@ components:
The `parse_grok` processor extracts structured fields from unstructured log messages using Grok patterns.

**Supported pipeline types:** logs
example:
id: "parse-grok-processor"
include: "service:my-service"
type: "parse_grok"
properties:
disable_library_rules:
default: false
Expand All @@ -67735,6 +67739,11 @@ components:
description: Indicates whether the processor is enabled.
example: true
type: boolean
field:
default: "message"
description: The log field to parse with the Grok rules.
example: "message"
type: string
id:
description: A unique identifier for this processor.
example: "parse-grok-processor"
Expand All @@ -67744,9 +67753,9 @@ components:
example: "service:my-service"
type: string
rules:
description: The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
description: The list of Grok parsing rules selected by either source field or include query.
items:
$ref: "#/components/schemas/ObservabilityPipelineParseGrokProcessorRule"
$ref: "#/components/schemas/ObservabilityPipelineParseGrokProcessorRuleItem"
type: array
type:
$ref: "#/components/schemas/ObservabilityPipelineParseGrokProcessorType"
Expand All @@ -67758,6 +67767,37 @@ components:
- enabled
type: object
x-pipeline-types: [logs]
ObservabilityPipelineParseGrokProcessorIncludeRule:
description: |-
A Grok parsing rule selected using the `include` query. Each rule defines how to extract structured fields
from logs matching a Datadog search query.
properties:
include:
description: A Datadog search query used to determine which logs this Grok rule targets.
example: "service:my-service"
type: string
match_rules:
description: |-
A list of Grok parsing rules that define how to extract fields from matching logs.
Each rule must contain a name and a valid Grok pattern.
example:
- name: "MyParsingRule"
rule: '%{word:user} connected on %{date("MM/dd/yyyy"):date}'
items:
$ref: "#/components/schemas/ObservabilityPipelineParseGrokProcessorRuleMatchRule"
type: array
support_rules:
description: A list of Grok helper rules that can be referenced by the parsing rules.
example:
- name: "user"
rule: "%{word:user.name}"
items:
$ref: "#/components/schemas/ObservabilityPipelineParseGrokProcessorRuleSupportRule"
type: array
required:
- include
- match_rules
type: object
ObservabilityPipelineParseGrokProcessorRule:
description: |-
A Grok parsing rule used in the `parse_grok` processor. Each rule defines how to extract structured fields
Expand Down Expand Up @@ -67790,6 +67830,11 @@ components:
- source
- match_rules
type: object
ObservabilityPipelineParseGrokProcessorRuleItem:
description: A single Grok parsing rule, selected by either source field or include query.
oneOf:
- $ref: "#/components/schemas/ObservabilityPipelineParseGrokProcessorRule"
- $ref: "#/components/schemas/ObservabilityPipelineParseGrokProcessorIncludeRule"
ObservabilityPipelineParseGrokProcessorRuleMatchRule:
description: |-
Defines a Grok parsing rule, which extracts structured fields from log content using named Grok patterns.
Expand Down
14 changes: 14 additions & 0 deletions docs/datadog_api_client.v2.model.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28004,13 +28004,27 @@ datadog\_api\_client.v2.model.observability\_pipeline\_parse\_grok\_processor mo
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_parse\_grok\_processor\_include\_rule module
---------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_parse_grok_processor_include_rule
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_parse\_grok\_processor\_rule module
------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_parse_grok_processor_rule
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_parse\_grok\_processor\_rule\_item module
------------------------------------------------------------------------------------------------

.. automodule:: datadog_api_client.v2.model.observability_pipeline_parse_grok_processor_rule_item
:members:
:show-inheritance:

datadog\_api\_client.v2.model.observability\_pipeline\_parse\_grok\_processor\_rule\_match\_rule module
-------------------------------------------------------------------------------------------------------

Expand Down
99 changes: 99 additions & 0 deletions examples/v2/observability-pipelines/ValidatePipeline_3345949653.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,99 @@
"""
Validate an observability pipeline with parse grok processor source rules returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi
from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig
from datadog_api_client.v2.model.observability_pipeline_config_processor_group import (
ObservabilityPipelineConfigProcessorGroup,
)
from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import (
ObservabilityPipelineDatadogAgentSource,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import (
ObservabilityPipelineDatadogAgentSourceType,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
ObservabilityPipelineDatadogLogsDestination,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import (
ObservabilityPipelineDatadogLogsDestinationType,
)
from datadog_api_client.v2.model.observability_pipeline_parse_grok_processor import (
ObservabilityPipelineParseGrokProcessor,
)
from datadog_api_client.v2.model.observability_pipeline_parse_grok_processor_rule import (
ObservabilityPipelineParseGrokProcessorRule,
)
from datadog_api_client.v2.model.observability_pipeline_parse_grok_processor_rule_match_rule import (
ObservabilityPipelineParseGrokProcessorRuleMatchRule,
)
from datadog_api_client.v2.model.observability_pipeline_parse_grok_processor_type import (
ObservabilityPipelineParseGrokProcessorType,
)
from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec
from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData

body = ObservabilityPipelineSpec(
data=ObservabilityPipelineSpecData(
attributes=ObservabilityPipelineDataAttributes(
config=ObservabilityPipelineConfig(
destinations=[
ObservabilityPipelineDatadogLogsDestination(
id="datadog-logs-destination",
inputs=[
"my-processor-group",
],
type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS,
),
],
processor_groups=[
ObservabilityPipelineConfigProcessorGroup(
enabled=True,
id="my-processor-group",
include="service:my-service",
inputs=[
"datadog-agent-source",
],
processors=[
ObservabilityPipelineParseGrokProcessor(
enabled=True,
id="parse-grok-processor",
include="*",
type=ObservabilityPipelineParseGrokProcessorType.PARSE_GROK,
rules=[
ObservabilityPipelineParseGrokProcessorRule(
source="message",
match_rules=[
ObservabilityPipelineParseGrokProcessorRuleMatchRule(
name="MyParsingRule",
rule="%{word:user}",
),
],
),
],
),
],
),
],
sources=[
ObservabilityPipelineDatadogAgentSource(
id="datadog-agent-source",
type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT,
),
],
),
name="Pipeline with Parse Grok Source Rules",
),
type="pipelines",
),
)

configuration = Configuration()
with ApiClient(configuration) as api_client:
api_instance = ObservabilityPipelinesApi(api_client)
response = api_instance.validate_pipeline(body=body)

print(response)
100 changes: 100 additions & 0 deletions examples/v2/observability-pipelines/ValidatePipeline_3928499240.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
"""
Validate an observability pipeline with parse grok processor include rules returns "OK" response
"""

from datadog_api_client import ApiClient, Configuration
from datadog_api_client.v2.api.observability_pipelines_api import ObservabilityPipelinesApi
from datadog_api_client.v2.model.observability_pipeline_config import ObservabilityPipelineConfig
from datadog_api_client.v2.model.observability_pipeline_config_processor_group import (
ObservabilityPipelineConfigProcessorGroup,
)
from datadog_api_client.v2.model.observability_pipeline_data_attributes import ObservabilityPipelineDataAttributes
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source import (
ObservabilityPipelineDatadogAgentSource,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_agent_source_type import (
ObservabilityPipelineDatadogAgentSourceType,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination import (
ObservabilityPipelineDatadogLogsDestination,
)
from datadog_api_client.v2.model.observability_pipeline_datadog_logs_destination_type import (
ObservabilityPipelineDatadogLogsDestinationType,
)
from datadog_api_client.v2.model.observability_pipeline_parse_grok_processor import (
ObservabilityPipelineParseGrokProcessor,
)
from datadog_api_client.v2.model.observability_pipeline_parse_grok_processor_include_rule import (
ObservabilityPipelineParseGrokProcessorIncludeRule,
)
from datadog_api_client.v2.model.observability_pipeline_parse_grok_processor_rule_match_rule import (
ObservabilityPipelineParseGrokProcessorRuleMatchRule,
)
from datadog_api_client.v2.model.observability_pipeline_parse_grok_processor_type import (
ObservabilityPipelineParseGrokProcessorType,
)
from datadog_api_client.v2.model.observability_pipeline_spec import ObservabilityPipelineSpec
from datadog_api_client.v2.model.observability_pipeline_spec_data import ObservabilityPipelineSpecData

body = ObservabilityPipelineSpec(
data=ObservabilityPipelineSpecData(
attributes=ObservabilityPipelineDataAttributes(
config=ObservabilityPipelineConfig(
destinations=[
ObservabilityPipelineDatadogLogsDestination(
id="datadog-logs-destination",
inputs=[
"my-processor-group",
],
type=ObservabilityPipelineDatadogLogsDestinationType.DATADOG_LOGS,
),
],
processor_groups=[
ObservabilityPipelineConfigProcessorGroup(
enabled=True,
id="my-processor-group",
include="service:my-service",
inputs=[
"datadog-agent-source",
],
processors=[
ObservabilityPipelineParseGrokProcessor(
enabled=True,
id="parse-grok-processor",
include="*",
type=ObservabilityPipelineParseGrokProcessorType.PARSE_GROK,
field="content",
rules=[
ObservabilityPipelineParseGrokProcessorIncludeRule(
include="service:foo",
match_rules=[
ObservabilityPipelineParseGrokProcessorRuleMatchRule(
name="MyParsingRule",
rule="%{word:user}",
),
],
),
],
),
],
),
],
sources=[
ObservabilityPipelineDatadogAgentSource(
id="datadog-agent-source",
type=ObservabilityPipelineDatadogAgentSourceType.DATADOG_AGENT,
),
],
),
name="Pipeline with Parse Grok Include Rules",
),
type="pipelines",
),
)

configuration = Configuration()
with ApiClient(configuration) as api_client:
api_instance = ObservabilityPipelinesApi(api_client)
response = api_instance.validate_pipeline(body=body)

print(response)
8 changes: 4 additions & 4 deletions src/datadog_api_client/v2/model/observability_pipeline_config_processor_item.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,11 @@ def __init__(self, **kwargs):
:param disable_library_rules: If set to `true`, disables the default Grok rules provided by Datadog.
:type disable_library_rules: bool, optional

:param rules: The list of Grok parsing rules. If multiple matching rules are provided, they are evaluated in order. The first successful match is applied.
:type rules: [ObservabilityPipelineParseGrokProcessorRule]
:param field: The log field to parse with the Grok rules.
:type field: str, optional

:param field: The name of the log field that contains a JSON string.
:type field: str
:param rules: The list of Grok parsing rules selected by either source field or include query.
:type rules: [ObservabilityPipelineParseGrokProcessorRuleItem]

:param always_use_text_key: Whether to always use a text key for element content.
:type always_use_text_key: bool, optional
Expand Down
Loading
Loading