Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
65 changes: 65 additions & 0 deletions documentation/ingestion/message-brokers/kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -524,6 +524,71 @@ entries.

:::

#### StructArrayExplode

The `StructArrayExplode` SMT converts arrays of structs into **separate 1D
`double[]` columns** — one per struct field. Unlike `OrderBookToArray` which
produces a single 2D array column, this transform "explodes" each struct field
into its own column.

**Input:**

```json
{
"symbol": "AAPL",
"vols": [
{ "strike": 150.0, "ivol": 0.25 },
{ "strike": 160.0, "ivol": 0.22 }
]
}
```

**Output:**

```json
{
"symbol": "AAPL",
"strikes": [150.0, 160.0],
"ivols": [0.25, 0.22]
}
```

**Configuration:**

```properties
transforms=explode
transforms.explode.type=io.questdb.kafka.StructArrayExplode$Value
transforms.explode.mappings=vols:strikes,ivols:strike,ivol
```

The `mappings` format is `sourceField:targetCol1,targetCol2:structField1,structField2;...`

Target columns and struct fields are paired positionally: `structField1` maps to
`targetCol1`, `structField2` maps to `targetCol2`, and so on. The number of
target columns must equal the number of struct fields.

Use semicolons to separate mappings from different source arrays:

```properties
transforms.explode.mappings=bids:bid_prices,bid_amounts:price,amount;asks:ask_prices,ask_amounts:price,amount
```

**Behavior:**
- All extracted values are converted to `double`
- Missing source fields are skipped (no error)
- Empty source arrays are skipped
- Null values inside structs cause an error
- If a target column name already exists in the input, it is replaced
- Works with both schema-based and schemaless messages

**Comparison with OrderBookToArray:**

| | OrderBookToArray | StructArrayExplode |
|---|---|---|
| Output | One 2D `double[][]` column | Separate 1D `double[]` columns |
| Mapping format | `source:target:field1,field2` | `source:target1,target2:field1,field2` |
| Use case | All fields in one array column | Each field as its own column |

### Sample projects

Additional examples are available on GitHub:
Expand Down