Packet buffer

Packet buffer (or pbuf) is buffer manager to handle received data from any connection. It is optimized to construct big buffer of smaller chunks of fragmented data as received bytes are not always coming as single packet.

Pbuf block diagram

Block diagram of pbuf chain

Image above shows structure of pbuf chain. Each pbuf consists of:

Pointer to next pbuf, or NULL when it is last in chain
Length of current packet length
Length of current packet and all next in chain
- If pbuf is last in chain, total length is the same as current packet length
Reference counter, indicating how many pointers point to current pbuf
Actual buffer data

Top image shows 3 pbufs connected to single chain. There are 2 custom pointer variables to point at different pbuf structures. Second pbuf has reference counter set to 2, as 2 variables point to it:

next of pbuf 1 is the first one
User variable 2 is the second one

Block structure
Block number	Next pbuf	Block size	Total size in chain	Reference counter
Block 1	Block 2	`150`	`550`	`1`
Block 2	Block 3	`130`	`400`	`2`
Block 3	`NULL`	`270`	`270`	`1`

Reference counter

Reference counter holds number of references (or variables) pointing to this block. It is used to properly handle memory free operation, especially when pbuf is used by lib core and application layer.

Note

If there would be no reference counter information and application would free memory while another part of library still uses its reference, application would invoke undefined behavior and system could crash instantly.

When application tries to free pbuf chain as on first image, it would normally call lwesp_pbuf_free() function. That would:

Decrease reference counter by 1
If reference counter == 0, it removes it from chain list and frees packet buffer memory
If reference counter != 0 after decrease, it stops free procedure
Go to next pbuf in chain and repeat steps

As per first example, result of freeing from user variable 1 would look similar to image and table below. First block (blue) had reference counter set to 1 prior freeing operation. It was successfully removed as user variable 1 was the only one pointing to it, while second (green) block had reference counter set to 2, preventing free operation.

Block diagram of pbuf chain after free from user variable 1

Block diagram of pbuf chain after free from *user variable 1*
Block number	Next pbuf	Block size	Total size in chain	Reference counter
Block 2	Block 3	`130`	`400`	`1`
Block 3	`NULL`	`270`	`270`	`1`

Note

Block 1 has been successfully freed, but since block 2 had reference counter set to 2 before, it was only decreased by 1 to a new value 1 and free operation stopped instead. User variable 2 is still using pbuf starting at block 2 and must manually call lwesp_pbuf_free() to free it.

Concatenating vs chaining

This section will explain difference between concat and chain operations. Both operations link 2 pbufs together in a chain of pbufs, difference is that chain operation increases reference counter to linked pbuf, while concat keeps reference counter at its current status.

Different pbufs, each pointed to by its own variable

Concat operation

Concat operation shall be used when 2 pbufs are linked together and reference to second is no longer used.

Structure after pbuf concat

After concating 2 pbufs together, reference counter of second is still set to 1, however we can see that 2 pointers point to second pbuf.

Note

After application calls lwesp_pbuf_cat(), it must not use pointer which points to second pbuf. This would invoke undefined behavior if one pointer tries to free memory while second still points to it.

An example code showing proper usage of concat operation:

Packet buffer concat example

lwesp_pbuf_p a, b;

/* Create 2 pbufs of different sizes */
a = lwesp_pbuf_new(10);
b = lwesp_pbuf_new(20);

/* Link them together with concat operation */
/* Reference on b will stay as is, won't be increased */
lwesp_pbuf_cat(a, b);

/*
 * Operating with b variable has from now on undefined behavior,
 * application shall stop using variable b to access pbuf.
 *
 * The best way would be to set b reference to NULL
 */
b = NULL;

/*
 * When application doesn't need pbufs anymore,
 * free a and it will also free b
 */
lwesp_pbuf_free(a);

Chain operation

Chain operation shall be used when 2 pbufs are linked together and reference to second is still required.

Structure after pbuf chain

After chainin 2 pbufs together, reference counter of second is increased by 1, which allows application to reference second pbuf separatelly.

Note

After application calls lwesp_pbuf_chain(), it also has to manually free its reference using lwesp_pbuf_free() function. Forgetting to free pbuf invokes memory leak

An example code showing proper usage of chain operation:

Packet buffer chain example

lwesp_pbuf_p a, b;

/* Create 2 pbufs of different sizes */
a = lwesp_pbuf_new(10);
b = lwesp_pbuf_new(20);

/* Chain both pbufs together */
/* This will increase reference on b as 2 variables now point to it */
lwesp_pbuf_chain(a, b);

/*
 * When application does not need a anymore, it may free it

 * This will free only pbuf a, as pbuf b has now 2 references:
 *  - one from pbuf a
 *  - one from variable b
 */

/* If application calls this, it will free only first pbuf */
/* As there is link to b pbuf somewhere */
lwesp_pbuf_free(a);

/* Reset a variable, not used anymore */
a = NULL;

/*
 * At this point, b is still valid memory block,
 * but when application doesn't need it anymore,
 * it should free it, otherwise memory leak appears
 */
lwesp_pbuf_free(b);

/* Reset b variable */
b = NULL;

Extract pbuf data

Each pbuf holds some amount of data bytes. When multiple pbufs are linked together (either chained or concated), blocks of raw data are not linked to contiguous memory block. It is necessary to process block by block manually.

An example code showing proper reading of any pbuf:

Packet buffer data extraction

const void* data;
size_t pos, len;
lwesp_pbuf_p a, b, c;

const char str_a[] = "This is one long";
const char str_a[] = "string. We want to save";
const char str_a[] = "chain of pbufs to file";

/* Create pbufs to hold these strings */
a = lwesp_pbuf_new(strlen(str_a));
b = lwesp_pbuf_new(strlen(str_b));
c = lwesp_pbuf_new(strlen(str_c));

/* Write data to pbufs */
lwesp_pbuf_take(a, str_a, strlen(str_a), 0);
lwesp_pbuf_take(b, str_b, strlen(str_b), 0);
lwesp_pbuf_take(c, str_c, strlen(str_c), 0);

/* Connect pbufs together */
lwesp_pbuf_chain(a, b);
lwesp_pbuf_chain(a, c);

/*
 * pbuf a now contains chain of b and c together
 * and at this point application wants to print (or save) data from chained pbuf
 *
 * Process pbuf by pbuf with code below
 */

/*
 * Get linear address of current pbuf at specific offset
 * Function will return pointer to memory address at specific position
 * and `len` will hold length of data block
 */
pos = 0;
while ((data = lwesp_pbuf_get_linear_addr(a, pos, &len)) != NULL) {
    /* Custom process function... */
    /* Process data with data pointer and block length */
    process_data(data, len);
    printf("Str: %.*s", len, data);

    /* Increase offset position for next block */
    pos += len;
}

/* Call free only on a pbuf. Since it is chained, b and c will be freed too */
lwesp_pbuf_free(a);

group LWESP_PBUF

Packet buffer manager.

Typedefs

typedef struct lwesp_pbuf *lwesp_pbuf_p: Pointer to lwesp_pbuf_t structure.

Functions

lwesp_pbuf_p lwesp_pbuf_new(size_t len)

Allocate packet buffer for network data of specific size.

Parameters: len – [in] Length of payload memory to allocate
Returns: Pointer to allocated memory, NULL otherwise

size_t lwesp_pbuf_free(lwesp_pbuf_p pbuf)

Free previously allocated packet buffer.

See also

lwesp_pbuf_free_s

Note

Application must not use reference to pbuf after the call to this function. It is advised to immediately set pointer to NULL or to call. Alternatively, call lwesp_pbuf_free_s, which will reset the pointer after free operation has been completed

Parameters: pbuf – [in] Packet buffer to free
Returns: Number of freed pbufs from head

size_t lwesp_pbuf_free_s(lwesp_pbuf_p *pbuf)

Free previously allocated packet buffer in safe way. Function accepts pointer to pointer and will set the pointer to NULL after the successful allocation.

Parameters: pbuf_ptr – [inout] Pointer to pointer to packet buffer
Returns: Number of packet buffers freed in the chain

void *lwesp_pbuf_data(const lwesp_pbuf_p pbuf)

Get data pointer from packet buffer.

Parameters: pbuf – [in] Packet buffer
Returns: Pointer to data buffer on success, NULL otherwise

size_t lwesp_pbuf_length(const lwesp_pbuf_p pbuf, uint8_t tot)

Get length of packet buffer.

Parameters

pbuf – [in] Packet buffer to get length for
tot – [in] Set to 1 to return total packet chain length or 0 to get only first packet length

Returns

Length of data in units of bytes

uint8_t lwesp_pbuf_set_length(lwesp_pbuf_p pbuf, size_t new_len)

Set new length of pbuf.

Note

New length can only be smaller than existing one. It has no effect when greater than existing one

Note

This function can be used on single-chain pbufs only, without next pbuf in chain

Parameters

pbuf – [in] Pbuf to make it smaller
new_len – [in] New length in units of bytes

Returns

1 on success, 0 otherwise

lwespr_t lwesp_pbuf_take(lwesp_pbuf_p pbuf, const void *data, size_t len, size_t offset)

Copy user data to chain of pbufs.

Parameters

pbuf – [in] First pbuf in chain to start copying to
data – [in] Input data to copy to pbuf memory
len – [in] Length of input data to copy
offset – [in] Start offset in pbuf where to start copying

Returns

lwespOK on success, member of lwespr_t enumeration otherwise

size_t lwesp_pbuf_copy(lwesp_pbuf_p pbuf, void *data, size_t len, size_t offset)

Copy memory from pbuf to user linear memory.

Parameters

pbuf – [in] Pbuf to copy from
data – [out] User linear memory to copy to
len – [in] Length of data in units of bytes
offset – [in] Possible start offset in pbuf

Returns

Number of bytes copied

lwespr_t lwesp_pbuf_cat(lwesp_pbuf_p head, const lwesp_pbuf_p tail)

Concatenate 2 packet buffers together to one big packet.

See also

lwesp_pbuf_cat_s

See also

lwesp_pbuf_chain

Note

After tail pbuf has been added to head pbuf chain, it must not be referenced by user anymore as it is now completely controlled by head pbuf. In simple words, when user calls this function, it should not call lwesp_pbuf_free function anymore, as it might make memory undefined for head pbuf.

Parameters

head – [in] Head packet buffer to append new pbuf to
tail – [in] Tail packet buffer to append to head pbuf

Returns

lwespOK on success, member of lwespr_t enumeration otherwise

lwespr_t lwesp_pbuf_cat_s(lwesp_pbuf_p head, lwesp_pbuf_p *tail)

Concatenate 2 packet buffers together to one big packet with safe pointer management.

See also

lwesp_pbuf_cat

See also

lwesp_pbuf_chain

Note

After tail pbuf has been added to head pbuf chain, tail pointer will be set to NULL

Parameters

head – [in] Head packet buffer to append new pbuf to
tail – [in] Pointer to pointer to tail packet buffer to append to head pbuf. Pointed memory will be set to NULL after successful concatenation

Returns

lwespOK on success, member of lwespr_t enumeration otherwise

lwespr_t lwesp_pbuf_chain(lwesp_pbuf_p head, lwesp_pbuf_p tail)

Chain 2 pbufs together. Similar to lwesp_pbuf_cat but now new reference is done from head pbuf to tail pbuf.

See also

lwesp_pbuf_cat

See also

lwesp_pbuf_cat_s

See also

lwesp_pbuf_chain_s

Note

After this function call, user must call lwesp_pbuf_free to remove its reference to tail pbuf and allow control to head pbuf: lwesp_pbuf_free(tail)

Parameters

head – [in] Head packet buffer to append new pbuf to
tail – [in] Tail packet buffer to append to head pbuf

Returns

lwespOK on success, member of lwespr_t enumeration otherwise

lwesp_pbuf_p lwesp_pbuf_unchain(lwesp_pbuf_p head)

Unchain first pbuf from list and return second one.

tot_len and len fields are adjusted to reflect new values and reference counter is as is

Note

After unchain, user must take care of both pbufs (head and new returned one)

Parameters: head – [in] First pbuf in chain to remove from chain
Returns: Next pbuf after head

lwespr_t lwesp_pbuf_ref(lwesp_pbuf_p pbuf)

Increment reference count on pbuf.

Parameters: pbuf – [in] pbuf to increase reference
Returns: lwespOK on success, member of lwespr_t enumeration otherwise

uint8_t lwesp_pbuf_get_at(const lwesp_pbuf_p pbuf, size_t pos, uint8_t *el)

Get value from pbuf at specific position.

Parameters

pbuf – [in] Pbuf used to get data from
pos – [in] Position at which to get element
el – [out] Output variable to save element value at desired position

Returns

1 on success, 0 otherwise

size_t lwesp_pbuf_memcmp(const lwesp_pbuf_p pbuf, const void *data, size_t len, size_t offset)

Compare pbuf memory with memory from data.

See also

lwesp_pbuf_strcmp

Note

Compare is done on entire pbuf chain

Parameters

pbuf – [in] Pbuf used to compare with data memory
data – [in] Actual data to compare with
len – [in] Length of input data in units of bytes
offset – [in] Start offset to use when comparing data

Returns

0 if equal, LWESP_SIZET_MAX if memory/offset too big or anything between if not equal

size_t lwesp_pbuf_strcmp(const lwesp_pbuf_p pbuf, const char *str, size_t offset)

Compare pbuf memory with input string.

See also

lwesp_pbuf_memcmp

Note

Compare is done on entire pbuf chain

Parameters

pbuf – [in] Pbuf used to compare with data memory
str – [in] String to be compared with pbuf
offset – [in] Start memory offset in pbuf

Returns

0 if equal, LWESP_SIZET_MAX if memory/offset too big or anything between if not equal

size_t lwesp_pbuf_memfind(const lwesp_pbuf_p pbuf, const void *data, size_t len, size_t off)

Find desired needle in a haystack.

See also

lwesp_pbuf_strfind

Parameters

pbuf – [in] Pbuf used as haystack
needle – [in] Data memory used as needle
len – [in] Length of needle memory
off – [in] Starting offset in pbuf memory

Returns

LWESP_SIZET_MAX if no match or position where in pbuf we have a match

size_t lwesp_pbuf_strfind(const lwesp_pbuf_p pbuf, const char *str, size_t off)

Find desired needle (str) in a haystack (pbuf)

See also

lwesp_pbuf_memfind

Parameters

pbuf – [in] Pbuf used as haystack
str – [in] String to search for in pbuf
off – [in] Starting offset in pbuf memory

Returns

LWESP_SIZET_MAX if no match or position where in pbuf we have a match

uint8_t lwesp_pbuf_advance(lwesp_pbuf_p pbuf, int len)

Advance pbuf payload pointer by number of len bytes. It can only advance single pbuf in a chain.

Note

When other pbufs are referencing current one, they are not adjusted in length and total length

Parameters

pbuf – [in] Pbuf to advance
len – [in] Number of bytes to advance. when negative is used, buffer size is increased only if it was decreased before

Returns

1 on success, 0 otherwise

lwesp_pbuf_p lwesp_pbuf_skip(lwesp_pbuf_p pbuf, size_t offset, size_t *new_offset)

Skip a list of pbufs for desired offset.

Note

Reference is not changed after return and user must not free the memory of new pbuf directly

Parameters

pbuf – [in] Start of pbuf chain
offset – [in] Offset in units of bytes to skip
new_offset – [out] Pointer to output variable to save new offset in returned pbuf

Returns

New pbuf on success, NULL otherwise

void *lwesp_pbuf_get_linear_addr(const lwesp_pbuf_p pbuf, size_t offset, size_t *new_len)

Get linear offset address for pbuf from specific offset.

Note

Since pbuf memory can be fragmented in chain, you may need to call function multiple times to get memory for entire pbuf chain

Parameters

pbuf – [in] Pbuf to get linear address
offset – [in] Start offset from where to start
new_len – [out] Length of memory returned by function

Returns

Pointer to memory on success, NULL otherwise

void lwesp_pbuf_set_ip(lwesp_pbuf_p pbuf, const lwesp_ip_t *ip, lwesp_port_t port)

Set IP address and port number for received data.

Parameters

pbuf – [in] Packet buffer
ip – [in] IP to assing to packet buffer
port – [in] Port number to assign to packet buffer

void lwesp_pbuf_dump(lwesp_pbuf_p p, uint8_t seq)

Dump and debug pbuf chain.

Parameters

p – [in] Head pbuf to dump
seq – [in] Set to 1 to dump all pbufs in linked list or 0 to dump first one only

struct lwesp_pbuf_t

#include <lwesp_private.h>

Packet buffer structure.

Public Members

struct lwesp_pbuf *next: Next pbuf in chain list

size_t tot_len: Total length of pbuf chain

size_t len: Length of payload

size_t ref: Number of references to this structure

uint8_t *payload: Pointer to payload memory

lwesp_ip_t ip: Remote address for received IPD data

lwesp_port_t port: Remote port for received IPD data