Building Messages

Table of Contents

• Introduction
• Example Code
• Explanation
  • Contexts
  • Building a message
  • Overriding data
  • Transmission
  • Inserting more data
  • Cleaning up
• Conclusion

Introduction

Welcome to the first guide in this series. This guide covers the basic ability of building arbitrary messages.

Lite³ is binary format without any schema definition or IDL. The structure of the message is built dynamically as you insert data into it.

In this example, we first build a message containing lap time of a race car driver:

{
    "lap": 55,
    "event": "lap_complete",
    "time_sec": 88.427
}

We then update the "lap" count field directly in-place to show serialized mutation capabilities:

"lap": 55, ===>  "lap": 56,

A 'network transmission' is simulated to race control, by copying it into another buffer.

Race control will verify the lap time and insert two extra fields to the message:

"fastest_lap": true,
"verified": "race_control"

When race control receives the transmitted data, the two extra field are inserted directly inside the serialized binary data. No parsing, transformation, or unmarshalling is ever required. This highlights the ability of Lite³ to treat serialized data as mutable rather than frozen.

Example Code

This guide is based on an example inside the Lite³ repository found in examples/context_api/01-building-messages.c:

/*
    Lite³: A JSON-Compatible Zero-Copy Serialization Format
 
    Copyright © 2025 Elias de Jong <elias@fastserial.com>
 
    Permission is hereby granted, free of charge, to any person obtaining a copy
    of this software and associated documentation files (the "Software"), to deal
    in the Software without restriction, including without limitation the rights
    to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
    copies of the Software, and to permit persons to whom the Software is
    furnished to do so, subject to the following conditions:
 
    The above copyright notice and this permission notice shall be included in all
    copies or substantial portions of the Software.
 
    THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
    IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
    FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
    AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
    LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
    OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
    SOFTWARE.
 
      __ __________________        ____
    _  ___ ___/ /___(_)_/ /_______|_  /
     _  _____/ / __/ /_  __/  _ \_/_ < 
      ___ __/ /___/ / / /_ /  __/____/ 
           /_____/_/  \__/ \___/       
*/
#include <stdio.h>
#include <string.h>
#include <stdbool.h>
 
#include "lite3_context_api.h"
 
 
int main() {
        lite3_ctx *ctx = lite3_ctx_create();
        if (!ctx) {
                perror("Failed to create lite3_ctx *ctx");
                return 1;
        }
 
        // Build message
        if (lite3_ctx_init_obj(ctx)                                     < 0
                || lite3_ctx_set_str(ctx, 0, "event", "lap_complete")   < 0
                || lite3_ctx_set_i64(ctx, 0, "lap", 55)                 < 0
                || lite3_ctx_set_f64(ctx, 0, "time_sec", 88.427)        < 0) {
                perror("Failed to build message");
                return 1;
        }
        printf("buflen: %zu\n", ctx->buflen);
        if (lite3_ctx_json_print(ctx, 0) < 0) { // Print Lite³ as JSON
                perror("Failed to print JSON");
                return 1;
        }
 
        printf("\nUpdating lap count\n");
        if (lite3_ctx_set_i64(ctx, 0, "lap", 56) < 0) {
                perror("Failed to update lap count");
                return 1;
        }
        printf("Data to send:\n");
        printf("buflen: %zu\n", ctx->buflen);
        if (lite3_ctx_json_print(ctx, 0) < 0) {
                perror("Failed to print JSON");
                return 1;
        }
 
        // Transmit data / copy to new context
        lite3_ctx *rx = lite3_ctx_create_from_buf(ctx->buf, ctx->buflen);
        if (!rx) {
                perror("Failed create lite3_ctx *rx");
                return 1;
        }
 
        // Mutate (zero-copy, no parsing)
        printf("\nVerifying fastest lap\n");
        if (lite3_ctx_set_str(rx, 0, "verified", "race_control")        < 0
                || lite3_ctx_set_bool(rx, 0, "fastest_lap", true)       < 0) {
                perror("Failed to verify lap");
                return 1;
        }
        printf("Modified data:\n");
        printf("rx_buflen: %zu\n", rx->buflen);
        if (lite3_ctx_json_print(rx, 0) < 0) {
                perror("Failed to print JSON");
                return 1;
        }
 
        // Ready to send:
        // send(sock, rx->buf, rx->buflen, 0);
 
        lite3_ctx_destroy(rx);
        lite3_ctx_destroy(ctx);
        return 0;
}

Output:

buflen: 154
{
    "lap": 55,
    "event": "lap_complete",
    "time_sec": 88.427
}
 
Updating lap count
Data to send:
buflen: 154
{
    "lap": 56,
    "event": "lap_complete",
    "time_sec": 88.427
}
 
Verifying fastest lap
Modified data:
rx_buflen: 197
{
    "lap": 56,
    "event": "lap_complete",
    "time_sec": 88.427,
    "fastest_lap": true,
    "verified": "race_control"
}

Explanation

We will walk through the example code step-by-step, explaining the use of Lite³ library functions.

Lite³ messages are just bytes, stored contiguously inside a buffer. If you want to allocate these messages inside your own custom allocators, you can using the Buffer API. However in this guide, we will be using the Context API so that memory is managed automatically.

Note that Lite³ is a binary format, but the examples print message data as JSON to stdout for better readability.

Contexts

We start by creating a context for our first message:

lite3_ctx *ctx = lite3_ctx_create();
if (!ctx) {
        perror("Failed to create lite3_ctx *ctx");
        return 1;
}

This is essentially just a memory allocation. Rare to fail, but we check anyways just in case.

Contexts are containers for Lite³ buffers, containing a single buffer for a single message. Instead of buffers being passed directly, functions take a *ctx variable as argument. The context will automatically resize when needed, similar to std::vector in C++.

If you want to access the buffer inside of a context, you can like so:

ctx->buf        // (uint8_t *) buffer pointer
ctx->buflen     // (size_t) message length in bytes

There is also a ctx->bufsz member that stores the total capacity inside the allocation. Contexts start out with a size of LITE3_CONTEXT_BUF_SIZE_MIN (default: 1024) and will reallocate 4X the current capacity if they run out of space.

Building a message

With a fresh context, we can insert data for the first time:

// Build message
if (lite3_ctx_init_obj(ctx)                                     < 0
        || lite3_ctx_set_str(ctx, 0, "event", "lap_complete")   < 0
        || lite3_ctx_set_i64(ctx, 0, "lap", 55)                 < 0
        || lite3_ctx_set_f64(ctx, 0, "time_sec", 88.427)        < 0) {
        perror("Failed to build message");
        return 1;
}

Notice how we group every function inside a single if-statement for error handling? Otherwise, each function would need a separate if-statement.

Before explaining this code, let's see what the result is. The next bit temporarily converts the Lite³ binary data to JSON and prints it to stdout:

printf("buflen: %zu\n", ctx->buflen);
if (lite3_ctx_json_print(ctx, 0) < 0) { // Print Lite³ as JSON
        perror("Failed to print JSON");
        return 1;
}

Resulting in this output:

buflen: 154
{
    "lap": 55,
    "event": "lap_complete",
    "time_sec": 88.427
}

Of course the binary data looks very different, but since raw bytes are not very friendly to the human eye, we much rather prefer the JSON representation. lite3_ctx_json_print() will be used a lot in code examples to visualize Lite³ messages.

Notice how many of these functions take ctx and 0 as their first 2 parameters? ctx is just the current context that we're dealing with (and that stores our message data), but what is this zero?

The zero is the ofs or 'offset' parameter and almost every function in Lite³ requires it. Basically, it tells the function at what 'offset' we start reading from. The root node is always stored at ofs == 0. Therefore, if we want to access the root-level object, we simply pass 0.
With nested structure, we will start passing non-zero values to say "don't target the root-level object, but a nested (internal) object located at this offset inside the buffer". This will be explained in the guide about Nesting.

But here we don't seem to pass any offset parameter:

lite3_ctx_init_obj(ctx) 

This function initializes an empty buffer as an object. This always targets the root node, therefore the ofs parameter is not required. If you are ever confused as to why inserting data doesn't work, it's probably because you forgot to initialize your empty buffer as an object or array. Recall from the JSON standard that the root-level must always be either one of those types. Since Lite³ is JSON compatible, this also applies to Lite³.

The 3rd parameter represents the key, and the 4th parameter the actual value to insert:

lite3_ctx_set_str(ctx, 0, "event", "lap_complete")
lite3_ctx_set_i64(ctx, 0, "lap", 55)
lite3_ctx_set_f64(ctx, 0, "time_sec", 88.427)

Integers in Lite³ are always stored as int64_t, and floating points are always stored as double (8 bytes). This also explains the function names.

Why these specific types? Because type ambiguity is undeseriable inside a schemaless format, and these types cover almost all use cases.

Overriding data

Now we move on to the next bit:

printf("\nUpdating lap count\n");
if (lite3_ctx_set_i64(ctx, 0, "lap", 56) < 0) {
        perror("Failed to update lap count");
        return 1;
}
 
printf("Data to send:\n");
printf("buflen: %zu\n", ctx->buflen);
if (lite3_ctx_json_print(ctx, 0) < 0) {
        perror("Failed to print JSON");
        return 1;
}

Which outputs the following:

Updating lap count
Data to send:
buflen: 154
{
    "lap": 56,
    "event": "lap_complete",
    "time_sec": 88.427
}

See how we updated the lap count? The value was updated in-place, all while the buffer length stayed unchanged at 154 bytes!

"lap": 55, ===>  "lap": 56,

This is one of the unique capabilities of Lite³. Despite being schemaless, we can directly modify serialized data. We don't need to actually parse or unmarshal first!

Transmission

Let us now transmit our message. Or more precisely, just copy the buffer data to a new context. Because if we start using actual sockets, the example becomes more complex than necessary.

// Transmit data / copy to new context
lite3_ctx *rx = lite3_ctx_create_from_buf(ctx->buf, ctx->buflen);
if (!rx) {
        perror("Failed create lite3_ctx *rx");
        return 1;
}

Here we declare a new context rx which is created by copying buffer data directly from ctx.

lite3_ctx_create_from_buf(): This function takes a pointer + length and allows you to create Lite³ contexts from any data source containing a Lite³ message. Note that this creates a copy of the data, leaving the original unaffected.

Inserting more data

Our transmission has successfully reached race control and been received inside the new context rx.

They will verify our lap time and insert extra fields into the message:

// Mutate in-place (zero-copy, no parsing)
printf("\nVerifying fastest lap\n");
if (lite3_ctx_set_str(rx, 0, "verified", "race_control")        < 0
        || lite3_ctx_set_bool(rx, 0, "fastest_lap", true)       < 0) {
        perror("Failed to verify lap");
        return 1;
}

Now let's see what this looks like in JSON:

printf("Modified data:\n");
printf("rx_buflen: %zu\n", rx->buflen);
if (lite3_ctx_json_print(rx, 0) < 0) {
        perror("Failed to print JSON");
        return 1;
}

Output:

Modified data:
rx_buflen: 197
{
    "lap": 56,
    "event": "lap_complete",
    "time_sec": 88.427,
    "fastest_lap": true,
    "verified": "race_control"
}

So race control was able to receive the message, then insert directly into the serialized binary data, even without parsing or demarshalling. Again, this illustraties the capabilities of Lite³.

The extra data increased buffer length from 154 to 197. Race control can now directly send the verified message elsewhere, perhaps to the scoreboard to show the fastest driver:

// Ready to send:
// send(sock, rx->buf, rx->buflen, 0);

Cleaning up

We are good citizens, so we clean up after ourselves:

lite3_ctx_destroy(rx);
lite3_ctx_destroy(ctx);

This destroys the context, freeing all the internal buffers so that the memory is released. When you create contexts, don't forget to destroy them, or else you will be leaking memory.

Conclusion

This was the first guide showing how to build messages using contexts.

The next guide will cover reading data from Lite³ messages:

Next Guide: Reading Messages