bloblang-authoring

Redpanda Connect Bloblang Script Generator

Create working, tested Bloblang transformation scripts from natural language descriptions.

Objective

Generate a Bloblang (blobl) script that correctly transforms the user's input data according to their requirements. The script MUST be tested before presenting it.

Setup

This skill requires rpk rpk connect, python3, and jq. See the SETUP for installation instructions.

Tools

Script format-bloblang.sh

Generates category-organized Bloblang reference files in XML format. Run once at the start of each session before searching for functions/methods.

# Usage:
./resources/scripts/format-bloblang.sh

• No arguments
• Generates category files organized by type (e.g., functions-General.xml, methods-String_Manipulation.xml)
• Outputs generated files to a versioned directory
• Outputs the directory path to stdout (capture in BLOBLREF_DIR variable for later use)
• Each XML file contains structured function/method definitions with parameters, descriptions, and examples

Functions

Generated function files have functions-<Category>.xml names and contain functions relevant to that category.

• functions-Encoding.xml - Schema registry headers
• functions-Environment.xml - Environment vars, files, timestamps, hostname
• functions-Fake_Data_Generation.xml - Fake data generation
• functions-General.xml - Bytes, counter, deleted, ksuid, nanoid, uuid, random, range, snowflake
• functions-Message_Info.xml - Batch index, content, error, metadata, span links, tracing IDs
• etc.

The function XML tag format:

• name attribute - function name
• params attribute - comma-separated list of parameters with types, format <name>:<type> or empty string if no parameters
• body - description of function purpose and usage
• example XML subtag
  • summary attribute (optional) - brief description of the example
  • body - code block demonstrating usage

Example function definition:

<function name="random_int" params="seed:query expression, min:integer, max:integer">
Generates a pseudo-random non-negative 64-bit integer.
Use this for creating random IDs, sampling data, or generating test values.
Provide a seed for reproducible randomness, or use a dynamic seed like `timestamp_unix_nano()` for unique values per mapping instance.

Optional `min` and `max` parameters constrain the output range (both inclusive).
For dynamic ranges based on message data, use the modulo operator instead: `random_int() % dynamic_max + dynamic_min`.
<example>
root.first = random_int()
root.second = random_int(1)
root.third = random_int(max:20)
root.fourth = random_int(min:10, max:20)
root.fifth = random_int(timestamp_unix_nano(), 5, 20)
root.sixth = random_int(seed:timestamp_unix_nano(), max:20)
</example>
<example summary="Use a dynamic seed for unique random values per mapping instance.">
root.random_id = random_int(timestamp_unix_nano())
root.sample_percent = random_int(seed: timestamp_unix_nano(), min: 0, max: 100)
</example>
</function>

Methods

Generated method files have methods-<Category>.xml names and contain methods relevant to that category.

• methods-Encoding_and_Encryption.xml - Base64, compression, hashing, encryption
• methods-General.xml - Basic operations, type checking
• methods-GeoIP.xml - GeoIP lookups
• methods-JSON_Web_Tokens.xml - JWT operations
• methods-Number_Manipulation.xml - Arithmetic, rounding, formatting
• methods-Object___Array_Manipulation.xml - Filtering, mapping, sorting, merging
• methods-Parsing.xml - JSON, CSV, XML, protocol buffer parsing
• methods-Regular_Expressions.xml - Regex matching and replacement
• methods-SQL.xml - SQL operations
• methods-String_Manipulation.xml - Case, trimming, splitting, formatting
• methods-Timestamp_Manipulation.xml - Parsing, formatting, timezone conversion
• methods-Type_Coercion.xml - Type conversions
• etc.

The method XML tag format:

• name attribute - function name
• params attribute - comma-separated list of parameters with types, format <name>:<type> or empty string if no parameters
• body - description of function purpose and usage
• example XML subtag
  • summary attribute (optional) - brief description of the example
  • body - code block demonstrating usage

Example method definition:

<method name="ts_format" params="format:string, tz:string">
Formats a timestamp into a string using the specified format layout.
<example>
root.formatted = this.timestamp.ts_format("2006-01-02T15:04:05Z07:00")
</example>
</method>

Grep Search

Lists Available functions and methods without loading full files.

# List all available functions and methods by name
grep -hE '<(function|method) name=' "$BLOBLREF_DIR"

# Search by keyword (searches names, descriptions, params, examples)
grep -i "timestamp" "$BLOBLREF_DIR"

# Search by parameter name (e.g., find all with "format" parameter)
grep 'params="[^"]*format' "$BLOBLREF_DIR"

• Requires BLOBLREF_DIR set to the directory output by format-bloblang.sh

Script test-blobl.sh

Tests a Bloblang script against input data. Executes the transformation and returns results or errors. Can be run repeatedly during iteration.

# Usage:
./resources/scripts/test-blobl.sh <target-directory>

• Requires data.json (input) and script.blobl (transformation) in the target directory
• Returns transformed data or error messages

Bloblang

Bloblang (blobl) is Redpanda Connect's native mapping language for transforming message data. It's designed for readability and safely reshaping documents of any structure.

Core Concepts

Assignment: Create new documents by assigning values to paths.

• root = the new document being created
• this = the input document being read

# Copy entire input
root = this

# Create specific fields
root.id = this.thing.id
root.type = "processed"

# In:  {"thing":{"id":"abc123"}}
# Out: {"id":"abc123","type":"processed"}

Field Paths: Use dot notation for nested fields. Use quotes for special characters:

root.user.name = this.customer.full_name
root."foo.bar".baz = this."field with spaces"

Literals: Numbers, booleans, strings, null, arrays, and objects:

root = {
  "count": 42,
  "active": true,
  "items": ["a", "b", "c"],
  "nested": {"key": "value"}
}

Functions and Methods

Functions generate values (no target needed):

root.id = uuid_v4()
root.timestamp = now()
root.hostname = hostname()

Methods transform values (called on a target with .):

root.upper = this.name.uppercase()
root.formatted = this.date.ts_parse("2006-01-02").ts_format("Mon Jan 2")
root.sorted = this.items.sort()

Methods can be chained:

root.clean = this.text.trim().lowercase().replace_all("_", "-")

Methods require a target (called with .), while functions do not. Check the XML reference files to determine correct usage:

# Bad: floor() is a method, not a function
root.rounded = floor(this.value)  # Error: floor is not a function

# Good: Call floor() as a method on a value
root.rounded = this.value.floor()

# Bad: uuid_v4() is a function, not a method
root.id = this.uuid_v4()  # Error: uuid_v4 is not a method

# Good: Call uuid_v4() as a function
root.id = uuid_v4()

Discovering Available Functions & Methods

Bloblang provides hundreds of functions and methods organized into categories. Start with these foundational categories that cover common use cases:

• functions-General.xml - Core utility functions (uuid_v4, timestamp, random, etc.)
• functions-Message_Info.xml - Message metadata access (hostname, env, content_type, etc.)
• methods-General.xml - Universal transformations (type conversions, existence checks, etc.)

For specialized needs, consult domain-specific categories: strings (uppercase, trim, regexp), timestamps (ts_parse, ts_format), arrays (map_each, filter), objects (keys, values), encoding (base64, json), and more.

Discovery tools:

• Run format-bloblang.sh to generate category-organized XML reference files in a versioned directory
• Use grep patterns to search function/method names, descriptions, parameters, and examples across categories
• Read specific category XML files for structured definitions with complete function signatures, parameter details, and usage examples

Control Flow

Conditionals (if/else):

root.category = if this.score >= 80 {
  "high"
} else if this.score >= 50 {
  "medium"
} else {
  "low"
}

Pattern Matching (match):

root.sound = match this.animal {
  "cat" => "meow"
  "dog" => "woof"
  "cow" => "moo"
  _ => "unknown"  # Catch-all
}

Coalescing (try multiple paths with |):

# Use first non-null value from alternative fields
root.content = this.article.body | this.comment.text | "no content"

# Try different nested paths
root.id = this.data.(primary_id | secondary_id | backup_id)

Note: Use | for alternative field paths (missing fields), use .catch() for operation failures (parse errors, type mismatches).

Common Operations

Deletion:

root = this
root.password = deleted()  # Remove field

# Or filter entire message
root = if this.spam { deleted() }

Variables (reuse values without adding to output):

let user_id = this.user.id
let enriched = this.user.name + " (" + $user_id + ")"

root.display_name = $enriched
root.user_id = $user_id

IMPORTANT: Variables must be declared at the top level, not inside if, match, or other blocks.

# Bad: Will cause "expected }" parse error
root.age = if this.birthdate != null {
  let parsed = this.birthdate.ts_parse("2006-01-02")  # let not allowed here!
  $parsed.ts_unix()
}

# Good: Declare variables at top level
let parsed = this.birthdate.ts_parse("2006-01-02").catch(null

---

*Content truncated.*