Introduction
The previous guide was about arrays. This guide will show iterators.
Iterators are an essential functionality for many kinds of applications. Examples include:
- range queries
- filtering
- validation
- aggregation
- discovering unknown data
Iterators can be created for both objects and arrays to traverse all entries or elements inside. The process consists of two steps:
- creating the iterator using
lite3_ctx_iter_create()
- looping over the items using
lite3_ctx_iter_next()
A lite3_iter struct (56 bytes) is used to store the state of an iterator. It can be stack- or heap-allocated.
Example Code
This guide is based on an example inside the Lite³ repository found in examples/context_api/06-iterators.c:
39const char names[NAME_COUNT][10] = {
51 perror(
"Failed to create lite3_ctx *ctx");
57 perror(
"Failed to initialize array");
60 for (
int i = 0; i < NAME_COUNT; i++) {
67 perror(
"Failed to build array");
72 perror(
"Failed to print JSON");
79 perror(
"Failed to create iterator");
91 perror(
"Failed to get object");
94 printf(
"id: %li\tname: %s\tvip_member: %s\tbenefits: %s\n",
97 vip_member ?
"true" :
"false",
98 benefits ?
"yes" :
"no"
105 perror(
"Failed to create iterator");
108 printf(
"\nObject keys:\n");
114 printf(
"key: %s\tvalue: ",
LITE3_STR(ctx->buf, key));
118 printf(
"%li\n", lite3_val_i64(val));
121 printf(
"%s\n", lite3_val_bool(val) ?
"true" :
"false");
127 printf(
"%s\n", lite3_val_str(val));
130 fprintf(stderr,
"Invalid object value type\n");
static int lite3_ctx_arr_append_obj(lite3_ctx *ctx, size_t ofs, size_t *__restrict out_ofs)
Append object to array.
#define lite3_ctx_get_str(ctx, ofs, key, out)
Get string value by key.
#define lite3_ctx_get_bool(ctx, ofs, key, out)
Get boolean value by key.
#define lite3_ctx_get_i64(ctx, ofs, key, out)
Get integer value by key.
static int lite3_ctx_init_arr(lite3_ctx *ctx)
Initialize a Lite³ context as an array.
static int lite3_ctx_iter_create(lite3_ctx *ctx, size_t ofs, lite3_iter *out)
Create a lite3 iterator for the given object or array.
static int lite3_ctx_iter_next(lite3_ctx *ctx, lite3_iter *iter, lite3_str *out_key, size_t *out_val_ofs)
Get the next item from a lite3 iterator.
static int lite3_ctx_json_print(lite3_ctx *ctx, size_t ofs)
Print Lite³ buffer as JSON to stdout
static lite3_ctx * lite3_ctx_create(void)
Create context with minimum size.
void lite3_ctx_destroy(lite3_ctx *ctx)
Destroy context.
#define lite3_ctx_set_i64(ctx, ofs, key, value)
Set integer in object.
#define lite3_ctx_set_null(ctx, ofs, key)
Set null in object.
#define lite3_ctx_set_bool(ctx, ofs, key, value)
Set boolean in object.
#define lite3_ctx_set_str(ctx, ofs, key, str)
Set string in object.
#define lite3_ctx_is_null(ctx, ofs, key)
Find value by key and test for null type.
#define LITE3_ITER_ITEM
Return value of lite3_iter_next(); iterator produced an item, continue;.
#define LITE3_STR(buf, val)
Generational pointer / safe access wrapper.
@ LITE3_TYPE_STRING
maps to 'string' type in JSON
@ LITE3_TYPE_BOOL
maps to 'boolean' type in JSON; underlying datatype: bool
@ LITE3_TYPE_I64
maps to 'number' type in JSON; underlying datatype: int64_t
@ LITE3_TYPE_NULL
maps to 'null' type in JSON
Lite³ Context API Header.
Struct containing iterator state.
Struct holding a reference to a string inside a Lite³ buffer.
Struct representing a value inside a Lite³ buffer.
Output:
[
{
"id": 0,
"benefits": null,
"name": "Boris",
"vip_member": false
},
{
"id": 1,
"benefits": null,
"name": "John",
"vip_member": false
},
{
"id": 2,
"benefits": null,
"name": "Olivia",
"vip_member": false
},
{
"id": 3,
"benefits": null,
"name": "Tanya",
"vip_member": false
},
{
"id": 4,
"benefits": null,
"name": "Paul",
"vip_member": false
},
{
"id": 5,
"benefits": null,
"name": "Sarah",
"vip_member": false
}
]
id: 0 name: Boris vip_member: false benefits: no
id: 1 name: John vip_member: false benefits: no
id: 2 name: Olivia vip_member: false benefits: no
id: 3 name: Tanya vip_member: false benefits: no
id: 4 name: Paul vip_member: false benefits: no
id: 5 name: Sarah vip_member: false benefits: no
Object keys:
key: id value: 5
key: benefits value: null
key: name value: Sarah
key: vip_member value: false
Explanation
We will walk through the example code step-by-step, explaining the use of Lite³ library functions.
Lite³ messages are just bytes, stored contiguously inside a buffer. If you want to allocate these messages inside your own custom allocators, you can using the Buffer API. However in this guide, we will be using the Context API so that memory is managed automatically.
Note that Lite³ is a binary format, but the examples print message data as JSON to stdout for better readability.
Building the message
As explained in the first guide, we use contexts to store Lite³ buffers (see: Context API):
if (!ctx) {
perror("Failed to create lite3_ctx *ctx");
return 1;
}
We will first construct an array of objects:
perror("Failed to initialize array");
return 1;
}
for (int i = 0; i < NAME_COUNT; i++) {
size_t obj_ofs;
perror("Failed to build array");
return 1;
}
}
We then convert Lite³ to JSON to see what this data looks like:
perror("Failed to print JSON");
return 1;
}
Output:
[
{
"id": 0,
"benefits": null,
"name": "Boris",
"vip_member": false
},
{
"id": 1,
"benefits": null,
"name": "John",
"vip_member": false
},
{
"id": 2,
"benefits": null,
"name": "Olivia",
"vip_member": false
},
{
"id": 3,
"benefits": null,
"name": "Tanya",
"vip_member": false
},
{
"id": 4,
"benefits": null,
"name": "Paul",
"vip_member": false
},
{
"id": 5,
"benefits": null,
"name": "Sarah",
"vip_member": false
}
]
Iterator creation
The first step is to create an iterator object like so:
perror("Failed to create iterator");
return 1;
}
Here we do a simple stack allocation for the lite3_iter struct, which has a default size of 56 bytes. The lite3_ctx_iter_create() will initialize initial parameters to make it ready for traversal.
- Note
- It is also possible to heap allocate the
lite3_iter struct like so: Perhaps the lifetime should outlive the current scope, or you are working with a small stack (sometimes you gotta work with what you have 😔).
Iterating over arrays
Now the actual traversal. We print out the values for each object:
size_t val_ofs;
int64_t id;
bool vip_member;
perror("Failed to get object");
return 1;
}
printf("id: %li\tname: %s\tvip_member: %s\tbenefits: %s\n",
id,
vip_member ? "true" : "false",
benefits ? "yes" : "no"
);
}
Output:
id: 0 name: Boris vip_member: false benefits: no
id: 1 name: John vip_member: false benefits: no
id: 2 name: Olivia vip_member: false benefits: no
id: 3 name: Tanya vip_member: false benefits: no
id: 4 name: Paul vip_member: false benefits: no
id: 5 name: Sarah vip_member: false benefits: no
There are three possible return values for the lite3_ctx_iter_next() function:
- LITE3_ITER_ITEM (== 1) on item produced
- LITE3_ITER_DONE (== 0) on success (no more items)
- < 0 (error)
In the loop we continuously check for LITE3_ITER_ITEM. The iterator will keep returning this everytime an item is produced successfully, or until there are no more items (LITE3_ITER_DONE). If an error occurs, the return value will be negative (< 0).
- Warning
- In this example we do not actually check for the error. However it is recommended to do so:
size_t val_ofs;
int ret;
}
if (ret < 0) {
perror("Failed to get iterator item");
return 1;
}
Remember that we are iterating over a series of objects inside an array. The 1st parameter is the current context storing our message, the 2nd is a pointer to the iterator object. The 3rd parameter is passed is NULL, as this is an out parameter for an object key.
Object key? But aren't we iterating over an array? Notice that iterator functions do not have 'arr' inside the name. This is because they work on objects as well as arrays. Since arrays do not have keys, this parameter is useless and we just pass NULL.
On each iteration, the offset of the current value (element) is written back to val_ofs (4th parameter). This value offset is very important. It is the exact offset inside the Lite³ buffer at which the current value is located. Since we are iterating over objects, it will be the offset of the current object. Remember from the Nesting guide that we can pass object offsets directly as the 2nd parameter to 'target' them for lookups, insertions etc.?
That is why all the following lookup functions use val_ofs as their 2nd parameter to lookup a field inside the current object:
Every iteration, this val_ofs will change to the next object, then we lookup the same fields on it.
Then each iteration we do a a printout of the values we just read:
printf("id: %li\tname: %s\tvip_member: %s\tbenefits: %s\n",
id,
vip_member ? "true" : "false",
benefits ? "yes" : "no"
);
Output:
id: 0 name: Boris vip_member: false benefits: no
Iterating over objects
With arrays, there are only elements. But objects are slightly different, as now there are keys as well. To show the difference, we will iterate over the key-value pairs inside the last object of the array.
It starts by creating another iterator object 'iter_2':
perror("Failed to create iterator");
return 1;
}
Then the traversal:
printf("\nObject keys:\n");
size_t val_ofs_2;
printf(
"key: %s\tvalue: ",
LITE3_STR(ctx->buf, key));
switch (val->type) {
printf("%li\n", lite3_val_i64(val));
break;
printf("%s\n", lite3_val_bool(val) ? "true" : "false");
break;
printf("null\n");
break;
printf("%s\n", lite3_val_str(val));
break;
default:
fprintf(stderr, "Invalid object value type\n");
return 1;
}
}
We take advantage of the situation by showcasing the use of lite3_val functions. Since Lite³ is a schemaless format where data can contain several different types (see enum lite3_type), we cannot know beforehand which type a value will be. A generic 'value reference' is required to point to the value, whatever type it may be.
lite3_val fulfils exactly this role, representing an opaque reference inside a Lite³ buffer that may be of any type. To find out, we can check val->type or lite3_val_type(val). This is explained more in detail in the guide about Reading Messages.
This line may at first seem puzzling:
Remember that lite3_ctx_iter_next() always writes back the offset of the current value. If the value turns out to be an object or array, it can be passed directly to other functions as 2nd parameter. But otherwise, it is still just an offset. To take advantage of lite3_val functions, we must obtain an actual pointer. To do this, we add the offset to the buffer pointer (start) of the message. The result is cast to (lite3_val *) and voilà we have ourselves a pointer.
Functions like this can now interpret the actual type:
lite3_val_i64(val)
lite3_val_bool(val)
lite3_val_str(val)
A switch/case statement illustrates differentiating between these types:
printf(
"key: %s\tvalue: ",
LITE3_STR(ctx->buf, key));
switch (val->type) {
printf("%li\n", lite3_val_i64(val));
break;
printf("%s\n", lite3_val_bool(val) ? "true" : "false");
break;
printf("null\n");
break;
printf("%s\n", lite3_val_str(val));
break;
default:
fprintf(stderr, "Invalid object value type\n");
return 1;
}
Output:
key: id value: 5
key: benefits value: null
key: name value: Sarah
key: vip_member value: false
The key string is printed, along with the value interpreted as the proper type.
- Warning
- lite3_val functions do not perform runtime type checking. It is up to the application to check for the type before interpreting it as such.
Cleaning up
We are good citizens, so we clean up after ourselves:
This destroys the context, freeing all the internal buffers so that the memory is released. When you create contexts, don't forget to destroy them, or else you will be leaking memory.
Conclusion
This was the sixth guide showing iterators.
The next and final guide will be about JSON conversion: Next Guide: JSON conversion