Packet buffer

Packet buffer (or pbuf) is buffer manager to handle received data from any connection. It is optimized to construct big buffer of smaller chunks of fragmented data as received bytes are not always coming as single packet.

Pbuf block diagram

Block diagram of pbuf chain

Block diagram of pbuf chain

Image above shows structure of pbuf chain. Each pbuf consists of:

  • Pointer to next pbuf, or NULL when it is last in chain

  • Length of current packet length

  • Length of current packet and all next in chain

    • If pbuf is last in chain, total length is the same as current packet length

  • Reference counter, indicating how many pointers point to current pbuf

  • Actual buffer data

Top image shows 3 pbufs connected to single chain. There are 2 custom pointer variables to point at different pbuf structures. Second pbuf has reference counter set to 2, as 2 variables point to it:

  • next of pbuf 1 is the first one

  • User variable 2 is the second one

Block structure

Block number

Next pbuf

Block size

Total size in chain

Reference counter

Block 1

Block 2

150

550

1

Block 2

Block 3

130

400

2

Block 3

NULL

270

270

1

Reference counter

Reference counter holds number of references (or variables) pointing to this block. It is used to properly handle memory free operation, especially when pbuf is used by lib core and application layer.

Note

If there would be no reference counter information and application would free memory while another part of library still uses its reference, application would invoke undefined behavior and system could crash instantly.

When application tries to free pbuf chain as on first image, it would normally call lwesp_pbuf_free() function. That would:

  • Decrease reference counter by 1

  • If reference counter == 0, it removes it from chain list and frees packet buffer memory

  • If reference counter != 0 after decrease, it stops free procedure

  • Go to next pbuf in chain and repeat steps

As per first example, result of freeing from user variable 1 would look similar to image and table below. First block (blue) had reference counter set to 1 prior freeing operation. It was successfully removed as user variable 1 was the only one pointing to it, while second (green) block had reference counter set to 2, preventing free operation.

Block diagram of pbuf chain after free from *user variable 1*

Block diagram of pbuf chain after free from user variable 1

Block diagram of pbuf chain after free from user variable 1

Block number

Next pbuf

Block size

Total size in chain

Reference counter

Block 2

Block 3

130

400

1

Block 3

NULL

270

270

1

Note

Block 1 has been successfully freed, but since block 2 had reference counter set to 2 before, it was only decreased by 1 to a new value 1 and free operation stopped instead. User variable 2 is still using pbuf starting at block 2 and must manually call lwesp_pbuf_free() to free it.

Concatenating vs chaining

This section will explain difference between concat and chain operations. Both operations link 2 pbufs together in a chain of pbufs, difference is that chain operation increases reference counter to linked pbuf, while concat keeps reference counter at its current status.

Different pbufs, each pointed to by its own variable

Different pbufs, each pointed to by its own variable

Concat operation

Concat operation shall be used when 2 pbufs are linked together and reference to second is no longer used.

Structure after pbuf concat

Structure after pbuf concat

After concating 2 pbufs together, reference counter of second is still set to 1, however we can see that 2 pointers point to second pbuf.

Note

After application calls lwesp_pbuf_cat(), it must not use pointer which points to second pbuf. This would invoke undefined behavior if one pointer tries to free memory while second still points to it.

An example code showing proper usage of concat operation:

Packet buffer concat example
 1lwesp_pbuf_p a, b;
 2
 3/* Create 2 pbufs of different sizes */
 4a = lwesp_pbuf_new(10);
 5b = lwesp_pbuf_new(20);
 6
 7/* Link them together with concat operation */
 8/* Reference on b will stay as is, won't be increased */
 9lwesp_pbuf_cat(a, b);
10
11/*
12 * Operating with b variable has from now on undefined behavior,
13 * application shall stop using variable b to access pbuf.
14 *
15 * The best way would be to set b reference to NULL
16 */
17b = NULL;
18
19/*
20 * When application doesn't need pbufs anymore,
21 * free a and it will also free b
22 */
23lwesp_pbuf_free(a);

Chain operation

Chain operation shall be used when 2 pbufs are linked together and reference to second is still required.

Structure after pbuf chain

Structure after pbuf chain

After chainin 2 pbufs together, reference counter of second is increased by 1, which allows application to reference second pbuf separatelly.

Note

After application calls lwesp_pbuf_chain(), it also has to manually free its reference using lwesp_pbuf_free() function. Forgetting to free pbuf invokes memory leak

An example code showing proper usage of chain operation:

Packet buffer chain example
 1lwesp_pbuf_p a, b;
 2
 3/* Create 2 pbufs of different sizes */
 4a = lwesp_pbuf_new(10);
 5b = lwesp_pbuf_new(20);
 6
 7/* Chain both pbufs together */
 8/* This will increase reference on b as 2 variables now point to it */
 9lwesp_pbuf_chain(a, b);
10
11/*
12 * When application does not need a anymore, it may free it
13
14 * This will free only pbuf a, as pbuf b has now 2 references:
15 *  - one from pbuf a
16 *  - one from variable b
17 */
18
19/* If application calls this, it will free only first pbuf */
20/* As there is link to b pbuf somewhere */
21lwesp_pbuf_free(a);
22
23/* Reset a variable, not used anymore */
24a = NULL;
25
26/*
27 * At this point, b is still valid memory block,
28 * but when application doesn't need it anymore,
29 * it should free it, otherwise memory leak appears
30 */
31lwesp_pbuf_free(b);
32
33/* Reset b variable */
34b = NULL;

Extract pbuf data

Each pbuf holds some amount of data bytes. When multiple pbufs are linked together (either chained or concated), blocks of raw data are not linked to contiguous memory block. It is necessary to process block by block manually.

An example code showing proper reading of any pbuf:

Packet buffer data extraction
 1const void* data;
 2size_t pos, len;
 3lwesp_pbuf_p a, b, c;
 4
 5const char str_a[] = "This is one long";
 6const char str_a[] = "string. We want to save";
 7const char str_a[] = "chain of pbufs to file";
 8
 9/* Create pbufs to hold these strings */
10a = lwesp_pbuf_new(strlen(str_a));
11b = lwesp_pbuf_new(strlen(str_b));
12c = lwesp_pbuf_new(strlen(str_c));
13
14/* Write data to pbufs */
15lwesp_pbuf_take(a, str_a, strlen(str_a), 0);
16lwesp_pbuf_take(b, str_b, strlen(str_b), 0);
17lwesp_pbuf_take(c, str_c, strlen(str_c), 0);
18
19/* Connect pbufs together */
20lwesp_pbuf_chain(a, b);
21lwesp_pbuf_chain(a, c);
22
23/*
24 * pbuf a now contains chain of b and c together
25 * and at this point application wants to print (or save) data from chained pbuf
26 *
27 * Process pbuf by pbuf with code below
28 */
29
30/*
31 * Get linear address of current pbuf at specific offset
32 * Function will return pointer to memory address at specific position
33 * and `len` will hold length of data block
34 */
35pos = 0;
36while ((data = lwesp_pbuf_get_linear_addr(a, pos, &len)) != NULL) {
37    /* Custom process function... */
38    /* Process data with data pointer and block length */
39    process_data(data, len);
40    printf("Str: %.*s", len, data);
41
42    /* Increase offset position for next block */
43    pos += len;
44}
45
46/* Call free only on a pbuf. Since it is chained, b and c will be freed too */
47lwesp_pbuf_free(a);
group LWESP_PBUF

Packet buffer manager.

Typedefs

typedef struct lwesp_pbuf *lwesp_pbuf_p

Pointer to lwesp_pbuf_t structure.

Functions

lwesp_pbuf_p lwesp_pbuf_new(size_t len)

Allocate packet buffer for network data of specific size.

Parameters

len[in] Length of payload memory to allocate

Returns

Pointer to allocated memory, NULL otherwise

size_t lwesp_pbuf_free(lwesp_pbuf_p pbuf)

Free previously allocated packet buffer.

Note

Application must not use reference to pbuf after the call to this function. It is advised to immediately set pointer to NULL or to call. Alternatively, call lwesp_pbuf_free_s, which will reset the pointer after free operation has been completed

Parameters

pbuf[in] Packet buffer to free

Returns

Number of freed pbufs from head

size_t lwesp_pbuf_free_s(lwesp_pbuf_p *pbuf)

Free previously allocated packet buffer in safe way. Function accepts pointer to pointer and will set the pointer to NULL after the successful allocation.

Parameters

pbuf_ptr[inout] Pointer to pointer to packet buffer

Returns

Number of packet buffers freed in the chain

void *lwesp_pbuf_data(const lwesp_pbuf_p pbuf)

Get data pointer from packet buffer.

Parameters

pbuf[in] Packet buffer

Returns

Pointer to data buffer on success, NULL otherwise

size_t lwesp_pbuf_length(const lwesp_pbuf_p pbuf, uint8_t tot)

Get length of packet buffer.

Parameters
  • pbuf[in] Packet buffer to get length for

  • tot[in] Set to 1 to return total packet chain length or 0 to get only first packet length

Returns

Length of data in units of bytes

uint8_t lwesp_pbuf_set_length(lwesp_pbuf_p pbuf, size_t new_len)

Set new length of pbuf.

Note

New length can only be smaller than existing one. It has no effect when greater than existing one

Note

This function can be used on single-chain pbufs only, without next pbuf in chain

Parameters
  • pbuf[in] Pbuf to make it smaller

  • new_len[in] New length in units of bytes

Returns

1 on success, 0 otherwise

lwespr_t lwesp_pbuf_take(lwesp_pbuf_p pbuf, const void *data, size_t len, size_t offset)

Copy user data to chain of pbufs.

Parameters
  • pbuf[in] First pbuf in chain to start copying to

  • data[in] Input data to copy to pbuf memory

  • len[in] Length of input data to copy

  • offset[in] Start offset in pbuf where to start copying

Returns

lwespOK on success, member of lwespr_t enumeration otherwise

size_t lwesp_pbuf_copy(lwesp_pbuf_p pbuf, void *data, size_t len, size_t offset)

Copy memory from pbuf to user linear memory.

Parameters
  • pbuf[in] Pbuf to copy from

  • data[out] User linear memory to copy to

  • len[in] Length of data in units of bytes

  • offset[in] Possible start offset in pbuf

Returns

Number of bytes copied

lwespr_t lwesp_pbuf_cat(lwesp_pbuf_p head, const lwesp_pbuf_p tail)

Concatenate 2 packet buffers together to one big packet.

See also

lwesp_pbuf_cat_s

See also

lwesp_pbuf_chain

Note

After tail pbuf has been added to head pbuf chain, it must not be referenced by user anymore as it is now completely controlled by head pbuf. In simple words, when user calls this function, it should not call lwesp_pbuf_free function anymore, as it might make memory undefined for head pbuf.

Parameters
  • head[in] Head packet buffer to append new pbuf to

  • tail[in] Tail packet buffer to append to head pbuf

Returns

lwespOK on success, member of lwespr_t enumeration otherwise

lwespr_t lwesp_pbuf_cat_s(lwesp_pbuf_p head, lwesp_pbuf_p *tail)

Concatenate 2 packet buffers together to one big packet with safe pointer management.

See also

lwesp_pbuf_cat

See also

lwesp_pbuf_chain

Note

After tail pbuf has been added to head pbuf chain, tail pointer will be set to NULL

Parameters
  • head[in] Head packet buffer to append new pbuf to

  • tail[in] Pointer to pointer to tail packet buffer to append to head pbuf. Pointed memory will be set to NULL after successful concatenation

Returns

lwespOK on success, member of lwespr_t enumeration otherwise

lwespr_t lwesp_pbuf_chain(lwesp_pbuf_p head, lwesp_pbuf_p tail)

Chain 2 pbufs together. Similar to lwesp_pbuf_cat but now new reference is done from head pbuf to tail pbuf.

See also

lwesp_pbuf_cat

See also

lwesp_pbuf_cat_s

See also

lwesp_pbuf_chain_s

Note

After this function call, user must call lwesp_pbuf_free to remove its reference to tail pbuf and allow control to head pbuf: lwesp_pbuf_free(tail)

Parameters
  • head[in] Head packet buffer to append new pbuf to

  • tail[in] Tail packet buffer to append to head pbuf

Returns

lwespOK on success, member of lwespr_t enumeration otherwise

lwesp_pbuf_p lwesp_pbuf_unchain(lwesp_pbuf_p head)

Unchain first pbuf from list and return second one.

tot_len and len fields are adjusted to reflect new values and reference counter is as is

Note

After unchain, user must take care of both pbufs (head and new returned one)

Parameters

head[in] First pbuf in chain to remove from chain

Returns

Next pbuf after head

lwespr_t lwesp_pbuf_ref(lwesp_pbuf_p pbuf)

Increment reference count on pbuf.

Parameters

pbuf[in] pbuf to increase reference

Returns

lwespOK on success, member of lwespr_t enumeration otherwise

uint8_t lwesp_pbuf_get_at(const lwesp_pbuf_p pbuf, size_t pos, uint8_t *el)

Get value from pbuf at specific position.

Parameters
  • pbuf[in] Pbuf used to get data from

  • pos[in] Position at which to get element

  • el[out] Output variable to save element value at desired position

Returns

1 on success, 0 otherwise

size_t lwesp_pbuf_memcmp(const lwesp_pbuf_p pbuf, const void *data, size_t len, size_t offset)

Compare pbuf memory with memory from data.

Note

Compare is done on entire pbuf chain

Parameters
  • pbuf[in] Pbuf used to compare with data memory

  • data[in] Actual data to compare with

  • len[in] Length of input data in units of bytes

  • offset[in] Start offset to use when comparing data

Returns

0 if equal, LWESP_SIZET_MAX if memory/offset too big or anything between if not equal

size_t lwesp_pbuf_strcmp(const lwesp_pbuf_p pbuf, const char *str, size_t offset)

Compare pbuf memory with input string.

Note

Compare is done on entire pbuf chain

Parameters
  • pbuf[in] Pbuf used to compare with data memory

  • str[in] String to be compared with pbuf

  • offset[in] Start memory offset in pbuf

Returns

0 if equal, LWESP_SIZET_MAX if memory/offset too big or anything between if not equal

size_t lwesp_pbuf_memfind(const lwesp_pbuf_p pbuf, const void *data, size_t len, size_t off)

Find desired needle in a haystack.

Parameters
  • pbuf[in] Pbuf used as haystack

  • needle[in] Data memory used as needle

  • len[in] Length of needle memory

  • off[in] Starting offset in pbuf memory

Returns

LWESP_SIZET_MAX if no match or position where in pbuf we have a match

size_t lwesp_pbuf_strfind(const lwesp_pbuf_p pbuf, const char *str, size_t off)

Find desired needle (str) in a haystack (pbuf)

Parameters
  • pbuf[in] Pbuf used as haystack

  • str[in] String to search for in pbuf

  • off[in] Starting offset in pbuf memory

Returns

LWESP_SIZET_MAX if no match or position where in pbuf we have a match

uint8_t lwesp_pbuf_advance(lwesp_pbuf_p pbuf, int len)

Advance pbuf payload pointer by number of len bytes. It can only advance single pbuf in a chain.

Note

When other pbufs are referencing current one, they are not adjusted in length and total length

Parameters
  • pbuf[in] Pbuf to advance

  • len[in] Number of bytes to advance. when negative is used, buffer size is increased only if it was decreased before

Returns

1 on success, 0 otherwise

lwesp_pbuf_p lwesp_pbuf_skip(lwesp_pbuf_p pbuf, size_t offset, size_t *new_offset)

Skip a list of pbufs for desired offset.

Note

Reference is not changed after return and user must not free the memory of new pbuf directly

Parameters
  • pbuf[in] Start of pbuf chain

  • offset[in] Offset in units of bytes to skip

  • new_offset[out] Pointer to output variable to save new offset in returned pbuf

Returns

New pbuf on success, NULL otherwise

void *lwesp_pbuf_get_linear_addr(const lwesp_pbuf_p pbuf, size_t offset, size_t *new_len)

Get linear offset address for pbuf from specific offset.

Note

Since pbuf memory can be fragmented in chain, you may need to call function multiple times to get memory for entire pbuf chain

Parameters
  • pbuf[in] Pbuf to get linear address

  • offset[in] Start offset from where to start

  • new_len[out] Length of memory returned by function

Returns

Pointer to memory on success, NULL otherwise

void lwesp_pbuf_set_ip(lwesp_pbuf_p pbuf, const lwesp_ip_t *ip, lwesp_port_t port)

Set IP address and port number for received data.

Parameters
  • pbuf[in] Packet buffer

  • ip[in] IP to assing to packet buffer

  • port[in] Port number to assign to packet buffer

void lwesp_pbuf_dump(lwesp_pbuf_p p, uint8_t seq)

Dump and debug pbuf chain.

Parameters
  • p[in] Head pbuf to dump

  • seq[in] Set to 1 to dump all pbufs in linked list or 0 to dump first one only

struct lwesp_pbuf_t
#include <lwesp_private.h>

Packet buffer structure.

Public Members

struct lwesp_pbuf *next

Next pbuf in chain list

size_t tot_len

Total length of pbuf chain

size_t len

Length of payload

size_t ref

Number of references to this structure

uint8_t *payload

Pointer to payload memory

lwesp_ip_t ip

Remote address for received IPD data

lwesp_port_t port

Remote port for received IPD data