bloblang-authoring by redpanda-data

---
name: bloblang-authoring
description: This skill should be used when users need to create or debug Bloblang transformation scripts. Trigger when users ask about transforming data, mapping fields, parsing JSON/CSV/XML, converting timestamps, filtering arrays, or mention "bloblang", "blobl", "mapping processor", or describe any data transformation need like "convert this to that" or "transform my JSON".
---

# Redpanda Connect Bloblang Script Generator

Create working, tested Bloblang transformation scripts from natural language descriptions.

## Objective

Generate a Bloblang (blobl) script that correctly transforms the user's input data according to their requirements.
The script MUST be tested before presenting it.

## Setup

This skill requires `rpk` `rpk connect`, `python3`, and `jq`.
See the [SETUP](SETUP.md) for installation instructions.

## Tools

### Script format-bloblang.sh

Generates category-organized Bloblang reference files in XML format.
**Run once at the start of each session** before searching for functions/methods.

```bash
# Usage:
./resources/scripts/format-bloblang.sh
```
- No arguments
- Generates category files organized by type (e.g., `functions-General.xml`, `methods-String_Manipulation.xml`)
- Outputs generated files to a versioned directory
- Outputs the directory path to stdout (capture in `BLOBLREF_DIR` variable for later use)
- Each XML file contains structured function/method definitions with parameters, descriptions, and examples

#### Functions

Generated function files have `functions-<Category>.xml` names and contain functions relevant to that category.

- `functions-Encoding.xml` - Schema registry headers
- `functions-Environment.xml` - Environment vars, files, timestamps, hostname
- `functions-Fake_Data_Generation.xml` - Fake data generation
- `functions-General.xml` - Bytes, counter, deleted, ksuid, nanoid, uuid, random, range, snowflake
- `functions-Message_Info.xml` - Batch index, content, error, metadata, span links, tracing IDs
- etc.

**The `function` XML tag format:**
- `name` attribute - function name
- `params` attribute - comma-separated list of parameters with types, format `<name>:<type>` or empty string if no parameters
- body - description of function purpose and usage
- `example` XML subtag
  - `summary` attribute (optional) - brief description of the example
  - body - code block demonstrating usage

Example function definition:
```xml
<function name="random_int" params="seed:query expression, min:integer, max:integer">
Generates a pseudo-random non-negative 64-bit integer.
Use this for creating random IDs, sampling data, or generating test values.
Provide a seed for reproducible randomness, or use a dynamic seed like `timestamp_unix_nano()` for unique values per mapping instance.

Optional `min` and `max` parameters constrain the output range (both inclusive).
For dynamic ranges based on message data, use the modulo operator instead: `random_int() % dynamic_max + dynamic_min`.
<example>
root.first = random_int()
root.second = random_int(1)
root.third = random_int(max:20)
root.fourth = random_int(min:10, max:20)
root.fifth = random_int(timestamp_unix_nano(), 5, 20)
root.sixth = random_int(seed:timestamp_unix_nano(), max:20)
</example>
<example summary="Use a dynamic seed for unique random values per mapping instance.">
root.random_id = random_int(timestamp_unix_nano())
root.sample_percent = random_int(seed: timestamp_unix_nano(), min: 0, max: 100)
</example>
</function>
```

#### Methods

Generated method files have `methods-<Category>.xml` names and contain methods relevant to that category.

- `methods-Encoding_and_Encryption.xml` - Base64, compression, hashing, encryption
- `methods-General.xml` - Basic operations, type checking
- `methods-GeoIP.xml` - GeoIP lookups
- `methods-JSON_Web_Tokens.xml` - JWT operations
- `methods-Number_Manipulation.xml` - Arithmetic, rounding, formatting
- `methods-Object___Array_Manipulation.xml` - Filtering, mapping, sorting, merging
- `methods-Parsing.xml` - JSON, CSV, XML, protocol buffer parsing
- `methods-Regular_Expressions.xml` - Regex matching and replacement
- `methods-SQL.xml` - SQL operations
- `methods-String_Manipulation.xml` - Case, trimming, splitting, formatting
- `methods-Timestamp_Manipulation.xml` - Parsing, formatting, timezone conversion
- `methods-Type_Coercion.xml` - Type conversions
- etc.

**The `method` XML tag format:**
- `name` attribute - function name
- `params` attribute - comma-separated list of parameters with types, format `<name>:<type>` or empty string if no parameters
- body - description of function purpose and usage
- `example` XML subtag
  - `summary` attribute (optional) - brief description of the example
  - body - code block demonstrating usage

Example method definition:
```xml
<method name="ts_format" params="format:string, tz:string">
Formats a timestamp into a string using the specified format layout.
<example>
root.formatted = this.timestamp.ts_format("2006-01-02T15:04:05Z07:00")
</example>
</method>
```

### Grep Search

Lists Available functions and methods without loading full files.

```bash
# List all available functions and methods by name
grep -hE '<(function|method) name=' "$BLOBLREF_DIR"

# Search by keyword (searches names, descriptions, params, examples)
grep -i "timestamp" "$BLOBLREF_DIR"

# Search by parameter name (e.g., find all with "format" parameter)
grep 'params="[^"]*format' "$BLOBLREF_DIR"
```
- Requires `BLOBLREF_DIR` set to the directory output by `format-bloblang.sh`

### Script test-blobl.sh

Tests a Bloblang script against input data.
Executes the transformation and returns results or errors.
Can be run repeatedly during iteration.

```bash
# Usage:
./resources/scripts/test-blobl.sh <target-directory>
```
- Requires `data.json` (input) and `script.blobl` (transformation) in the target directory
- Returns transformed data or error messages

## Bloblang

**Bloblang** (blobl) is Redpanda Connect's native mapping language for transforming message data.
It's designed for readability and safely reshaping documents of any structure.

### Core Concepts

**Assignment**: Create new documents by assigning values to paths.
- `root` = the new document being created
- `this` = the input document being read

```bloblang
# Copy entire input
root = this

# Create specific fields
root.id = this.thing.id
root.type = "processed"

# In:  {"thing":{"id":"abc123"}}
# Out: {"id":"abc123","type":"processed"}
```

**Field Paths**: Use dot notation for nested fields. Use quotes for special characters:
```bloblang
root.user.name = this.customer.full_name
root."foo.bar".baz = this."field with spaces"
```

**Literals**: Numbers, booleans, strings, null, arrays, and objects:
```bloblang
root = {
  "count": 42,
  "active": true,
  "items": ["a", "b", "c"],
  "nested": {"key": "value"}
}
```

### Functions and Methods

**Functions** generate values (no target needed):
```bloblang
root.id = uuid_v4()
root.timestamp = now()
root.hostname = hostname()
```

**Methods** transform values (called on a target with `.`):
```bloblang
root.upper = this.name.uppercase()
root.formatted = this.date.ts_parse("2006-01-02").ts_format("Mon Jan 2")
root.sorted = this.items.sort()
```

Methods can be chained:
```bloblang
root.clean = this.text.trim().lowercase().replace_all("_", "-")
```

Methods require a target (called with `.`), while functions do not. 
Check the XML reference files to determine correct usage:

```bloblang
# Bad: floor() is a method, not a function
root.rounded = floor(this.value)  # Error: floor is not a function

# Good: Call floor() as a method on a value
root.rounded = this.value.floor()

# Bad: uuid_v4() is a function, not a method
root.id = this.uuid_v4()  # Error: uuid_v4 is not a method

# Good: Call uuid_v4() as a function
root.id = uuid_v4()
```

**Discovering Available Functions & Methods**

Bloblang provides hundreds of functions and methods organized into categories.
Start with these **foundational categories** that cover common use cases:
- `functions-General.xml` - Core utility functions (uuid_v4, timestamp, random, etc.)
- `functions-Message_Info.xml` - Message metadata access (hostname, env, content_type, etc.)
- `methods-General.xml` - Universal transformations (type conversions, existence checks, etc.)

For specialized needs, consult **domain-specific categories**: strings (uppercase, trim, regexp), timestamps (ts_parse, ts_format), arrays (map_each, filter), objects (keys, values), encoding (base64, json), and more.

**Discovery tools**:
- Run `format-bloblang.sh` to generate category-organized XML reference files in a versioned directory
- Use grep patterns to search function/method names, descriptions, parameters, and examples across categories
- Read specific category XML files for structured definitions with complete function signatures, parameter details, and usage examples

### Control Flow

**Conditionals** (if/else):
```bloblang
root.category = if this.score >= 80 {
  "high"
} else if this.score >= 50 {
  "medium"
} else {
  "low"
}
```

**Pattern Matching** (match):
```bloblang
root.sound = match this.animal {
  "cat" => "meow"
  "dog" => "woof"
  "cow" => "moo"
  _ => "unknown"  # Catch-all
}
```

**Coalescing** (try multiple paths with `|`):
```bloblang
# Use first non-null value from alternative fields
root.content = this.article.body | this.comment.text | "no content"

# Try different nested paths
root.id = this.data.(primary_id | secondary_id | backup_id)
```

Note: Use `|` for alternative field paths (missing fields), use `.catch()` for operation failures (parse errors, type mismatches).

### Common Operations

**Deletion**:
```bloblang
root = this
root.password = deleted()  # Remove field

# Or filter entire message
root = if this.spam { deleted() }
```

**Variables** (reuse values without adding to output):
```bloblang
let user_id = this.user.id
let enriched = this.user.name + " (" + $user_id + ")"

root.display_name = $enriched
root.user_id = $user_id
```

**IMPORTANT**: Variables must be declared at the top level, not inside `if`, `match`, or other blocks.

```bloblang
# Bad: Will cause "expected }" parse error
root.age = if this.birthdate != null {
  let parsed = this.birthdate.ts_parse("2006-01-02")  # let not allowed here!
  $parsed.ts_unix()
}

# Good: Declare variables at top level
let parsed = this.birthdate.ts_parse("2006-01-02").catch(null)
root.age = if $parsed != null {
  $parsed.ts_unix()
} else {
  null
}
```

**Named mappings**: (reusable scripts)
```bloblang
map extract_user {
  root.id = this.user_id
  root.name = this.full_name
  root.email = this.contact.email
}

root.customer = this.customer_data.apply("extract_user")
root.vendor = this.vendor_data.apply("extract_user")
```

**Error Handling** (provide fallback values):
```bloblang
# Catch errors from any point in the chain
root.count = this.items.length().catch(0)
root.parsed = this.data.parse_json().catch({})

# Catch missing/null values
root.name = this.user.name.or("anonymous")

# Multi-format parsing with catch chains
# Store value in variable for reliable access in catch fallbacks
let date_str = this.date
root.parsed = $date_str.ts_parse("2006-01-02").catch(
  $date_str.ts_parse("2006/01/02")
).catch(null)
```

**IMPORTANT**: When using `.catch()` with fallback expressions that reference `this.field`, store the field in a variable first.
Context references in catch chains can be unreliable:

```bloblang
# Risky: Context may not be preserved in catch
root.parsed = this.date.ts_parse("2006-01-02").catch(
  this.date.ts_parse("2006/01/02")  # this.date might not work here
)

# Safe: Store in variable first
let date_str = this.date
root.parsed = $date_str.ts_parse("2006-01-02").catch(
  $date_str.ts_parse("2006/01/02")  # variable reference is reliable
)
```

**Metadata**:
```bloblang
# Read metadata with @ or metadata()
root.topic = @kafka_topic
root.partition = @kafka_partition

# Set metadata
meta output_key = this.id
meta content_type = "application/json"
```

### Common Edge Case Patterns

**Safe field access with fallbacks**
```bloblang
# Bad: Will fail if user or name is missing
root.name = this.user.name

# Good: Provides fallback chain
root.name = this.user.name.or("anonymous")
root.name = this.(user.name | profile.display_name | "unknown")
```

**Safe collection operations**
```bloblang
# Bad: Will fail on empty array
root.first = this.items[0]

# Good: Handles empty arrays
root.first = if this.items.length() > 0 { this.items[0] } else { null }
root.first = this.items[0].catch(null)
```

**Safe parsing with error recovery**
```bloblang
# Bad: Will fail on invalid JSON
root.data = this.payload.parse_json()

# Good: Provides fallback on parse failure
root.data = this.payload.parse_json().catch({})
root.data = this.payload.parse_json().catch(this.payload)  # Keep original on failure
```

**Safe type coercion**
```bloblang
# Bad: Assumes field is already a string
root.id = this.user_id.uppercase()

# Good: Converts to string first
root.id = this.user_id.string().uppercase()
root.count = this.total.number().catch(0)
```

**IMPORTANT**: Arithmetic operations on null values fail silently.
Always check for null or use `.catch()` to provide fallbacks:

```bloblang
# Bad: Fails silently if price is null
root.total = this.price * this.quantity

# Good: Check for null before operations
root.total = if this.price != null && this.quantity != null {
  this.price * this.quantity
} else {
  null
}

# Also good: Use catch to handle null gracefully
root.total = (this.price * this.quantity).catch(null)
```

## Workflow

1. **Understand** - Analyze input structure, desired output, and required transformations
     - **Ambiguous requirements**: If transformation goal is unclear, ask clarifying questions before proceeding (e.g., "Should missing fields be omitted or set to null?", "How should arrays with mixed types be handled?")
     - **Missing sample data**: If user doesn't provide input example, request it explicitly - never proceed with assumptions
     - **Complex multistep transformations**: Break down into logical phases (parse → transform → filter → format) and confirm approach with user

2. **Discover** - Generate category files to versioned directory (capture `BLOBLREF_DIR` from script output), identify relevant categories, read specific category XML files to find actual Bloblang functions/methods (NEVER guess)

3. **Develop** - Write valid Bloblang syntax using discovered functions (root for output, this for input, chain methods, handle nulls)

4. **Validate** - Test script with sample input data, verify output matches expectations, iterate on errors until working
     - **Test edge cases**: Missing fields, null values, invalid formats, empty collections
     - **Iterate**: Fix syntax errors first (variable placement, method chains), then logic errors

5. **Deliver** - Write the working script and example input to files (`script.blobl`, `data.json`), present the tested output, document any assumptions

**Critical: Never present untested code. All scripts must be validated before showing to user.**