Stream parser

Streaming parser implementation is alternative option versus standard tokenized one, in the sense that:

  • There is no need to have full JSON available at one time to have successful parsing

  • It can be utilized to parse very large JSON strings on very small systems with limited memory

  • It allows users to take from the stream only necessary parts and store them to local more system-friendly variable

This type of parser does not utilize use of tokens, rather focuses on the callback function, where user is in charge to manually understand token structure and get useful data from it.

Stream parser introduces stack mechanism instead - to keep the track of depthness during parsing the process. 3 different element types are stored on local stack:

  • Start of object, with { character

  • Start of array, with [ character

  • Key from the object entry

Note

Stack is nested as long as JSON input stream is nested in the same way

Consider this input string: {"k1":"v1","k2":[true, false]}. During parsing procedure, at some point of time, these events will occur:

  1. Start of object detected - object pushed to stack

    1. key element with name k1 detected and pushed to stack

      1. string v1 parsed as string-value

    2. key element with name k1 popped from stack

    3. key element with name k2 detected and pushed to stack

      1. Start of array detected - array pushed to stack

        1. true primitive detected

        2. false primitive detected

      2. End of array detected - array popped from stack

    4. key element with name k2 popped from stack

  2. End of object detected - object popped from stack

Each of these events is reported to user in the callback function.

An example of the stream parsing:

Parse JSON data as a stream object
 1#include <stdio.h>
 2#include "lwjson/lwjson.h"
 3
 4/* Test string to parser */
 5static const char* json_str = "{\"k1\":\"v1\",\"k2\":[true, false]}";
 6
 7/* LwJSON stream parser */
 8static lwjson_stream_parser_t stream_parser;
 9
10/**
11 * \brief           Callback function for various events
12 * \param           jsp: JSON stream parser object
13 * \param           type: Event type
14 */
15void
16prv_example_callback_func(lwjson_stream_parser_t* jsp, lwjson_stream_type_t type) {
17    /* Get a value corresponsing to "k1" key */
18    if (jsp->stack_pos >= 2                                /* Number of stack entries must be high */
19        && jsp->stack[0].type == LWJSON_STREAM_TYPE_OBJECT /* First must be object */
20        && jsp->stack[1].type == LWJSON_STREAM_TYPE_KEY    /* We need key to be before */
21        && strcmp(jsp->stack[1].meta.name, "k1") == 0) {
22        printf("Got key '%s' with value '%s'\r\n", jsp->stack[1].meta.name, jsp->data.str.buff);
23    }
24    (void)type;
25}
26
27/* Parse JSON */
28void
29example_stream_run(void) {
30    lwjsonr_t res;
31    printf("\r\n\r\nParsing stream\r\n");
32    lwjson_stream_init(&stream_parser, prv_example_callback_func);
33
34    /* Demonstrate as stream inputs */
35    for (const char* c = json_str; *c != '\0'; ++c) {
36        res = lwjson_stream_parse(&stream_parser, *c);
37        if (res == lwjsonSTREAMINPROG) {
38        } else if (res == lwjsonSTREAMWAITFIRSTCHAR) {
39            printf("Waiting first character\r\n");
40        } else if (res == lwjsonSTREAMDONE) {
41            printf("Done\r\n");
42        } else {
43            printf("Error\r\n");
44            break;
45        }
46    }
47    printf("Parsing completed\r\n");
48}

Example

For the purpose of example, the following JSON input…

JSON input for streaming
 1{
 2  "test": "abc",
 3  "array": [
 4    "123",
 5    "def",
 6    "ghi"
 7  ],
 8  "array_in_array": [
 9    ["1", "2", "3"],
10    ["4", "5", "6"],
11    ["7", "8", "9"]
12  ]
13}

... will output the log as: