Packet buffer

Packet buffer (or pbuf) is buffer manager to handle received data from any connection. It is optimized to construct big buffer of smaller chunks of fragmented data as received bytes are not always coming as single packet.

Pbuf block diagram

Block diagram of pbuf chain

Block diagram of pbuf chain

Image above shows structure of pbuf chain. Each pbuf consists of:

  • Pointer to next pbuf, or NULL when it is last in chain

  • Length of current packet length

  • Length of current packet and all next in chain

    • If pbuf is last in chain, total length is the same as current packet length

  • Reference counter, indicating how many pointers point to current pbuf

  • Actual buffer data

Top image shows 3 pbufs connected to single chain. There are 2 custom pointer variables to point at different pbuf structures. Second pbuf has reference counter set to 2, as 2 variables point to it:

  • next of pbuf 1 is the first one

  • User variable 2 is the second one

Block structure

Block number

Next pbuf

Block size

Total size in chain

Reference counter

Block 1

Block 2

150

550

1

Block 2

Block 3

130

400

2

Block 3

NULL

270

270

1

Reference counter

Reference counter holds number of references (or variables) pointing to this block. It is used to properly handle memory free operation, especially when pbuf is used by lib core and application layer.

Note

If there would be no reference counter information and application would free memory while another part of library still uses its reference, application would invoke undefined behavior and system could crash instantly.

When application tries to free pbuf chain as on first image, it would normally call esp_pbuf_free() function. That would:

  • Decrease reference counter by 1

  • If reference counter == 0, it removes it from chain list and frees packet buffer memory

  • If reference counter != 0 after decrease, it stops free procedure

  • Go to next pbuf in chain and repeat steps

As per first example, result of freeing from user variable 1 would look similar to image and table below. First block (blue) had reference counter set to 1 prior freeing operation. It was successfully removed as user variable 1 was the only one pointing to it, while second (green) block had reference counter set to 2, preventing free operation.

Block diagram of pbuf chain after free from *user variable 1*

Block diagram of pbuf chain after free from user variable 1

Block diagram of pbuf chain after free from user variable 1

Block number

Next pbuf

Block size

Total size in chain

Reference counter

Block 2

Block 3

130

400

1

Block 3

NULL

270

270

1

Note

Block 1 has been successfully freed, but since block 2 had reference counter set to 2 before, it was only decreased by 1 to a new value 1 and free operation stopped instead. User variable 2 is still using pbuf starting at block 2 and must manually call esp_pbuf_free() to free it.

Concatenating vs chaining

This section will explain difference between concat and chain operations. Both operations link 2 pbufs together in a chain of pbufs, difference is that chain operation increases reference counter to linked pbuf, while concat keeps reference counter at its current status.

Different pbufs, each pointed to by its own variable

Different pbufs, each pointed to by its own variable

Concat operation

Concat operation shall be used when 2 pbufs are linked together and reference to second is no longer used.

Structure after pbuf concat

Structure after pbuf concat

After concating 2 pbufs together, reference counter of second is still set to 1, however we can see that 2 pointers point to second pbuf.

Note

After application calls esp_pbuf_cat(), it must not use pointer which points to second pbuf. This would invoke undefined behavior if one pointer tries to free memory while second still points to it.

An example code showing proper usage of concat operation:

Packet buffer concat example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
esp_pbuf_p a, b;

/* Create 2 pbufs of different sizes */
a = esp_pbuf_new(10);
b = esp_pbuf_new(20);

/* Link them together with concat operation */
/* Reference on b will stay as is, won't be increased */
esp_pbuf_cat(a, b);

/*
 * Operating with b variable has from now on undefined behavior,
 * application shall stop using variable b to access pbuf.
 *
 * The best way would be to set b reference to NULL
 */
b = NULL;

/*
 * When application doesn't need pbufs anymore,
 * free a and it will also free b
 */
esp_pbuf_free(a);

Chain operation

Chain operation shall be used when 2 pbufs are linked together and reference to second is still required.

Structure after pbuf chain

Structure after pbuf chain

After chainin 2 pbufs together, reference counter of second is increased by 1, which allows application to reference second pbuf separatelly.

Note

After application calls esp_pbuf_chain(), it also has to manually free its reference using esp_pbuf_free() function. Forgetting to free pbuf invokes memory leak

An example code showing proper usage of chain operation:

Packet buffer chain example
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
esp_pbuf_p a, b;

/* Create 2 pbufs of different sizes */
a = esp_pbuf_new(10);
b = esp_pbuf_new(20);

/* Chain both pbufs together */
/* This will increase reference on b as 2 variables now point to it */
esp_pbuf_chain(a, b);

/*
 * When application does not need a anymore, it may free it

 * This will free only pbuf a, as pbuf b has now 2 references:
 *  - one from pbuf a
 *  - one from variable b
 */

/* If application calls this, it will free only first pbuf */
/* As there is link to b pbuf somewhere */
esp_pbuf_free(a);                               

/* Reset a variable, not used anymore */
a = NULL;

/*
 * At this point, b is still valid memory block,
 * but when application doesn't need it anymore,
 * it should free it, otherwise memory leak appears
 */
esp_pbuf_free(b);

/* Reset b variable */
b = NULL;

Extract pbuf data

Each pbuf holds some amount of data bytes. When multiple pbufs are linked together (either chained or concated), blocks of raw data are not linked to contiguous memory block. It is necessary to process block by block manually.

An example code showing proper reading of any pbuf:

Packet buffer data extraction
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
const void* data;
size_t pos, len;
esp_pbuf_p a, b, c;

const char str_a[] = "This is one long";
const char str_a[] = "string. We want to save";
const char str_a[] = "chain of pbufs to file";

/* Create pbufs to hold these strings */
a = esp_pbuf_new(strlen(str_a));
b = esp_pbuf_new(strlen(str_b));
c = esp_pbuf_new(strlen(str_c));

/* Write data to pbufs */
esp_pbuf_take(a, str_a, strlen(str_a), 0);
esp_pbuf_take(b, str_b, strlen(str_b), 0);
esp_pbuf_take(c, str_c, strlen(str_c), 0);

/* Connect pbufs together */
esp_pbuf_chain(a, b);
esp_pbuf_chain(a, c);

/*
 * pbuf a now contains chain of b and c together
 * and at this point application wants to print (or save) data from chained pbuf
 *
 * Process pbuf by pbuf with code below
 */

/*
 * Get linear address of current pbuf at specific offset
 * Function will return pointer to memory address at specific position
 * and `len` will hold length of data block
 */
pos = 0;
while ((data = esp_pbuf_get_linear_addr(a, pos, &len)) != NULL) {
    /* Custom process function... */
    /* Process data with data pointer and block length */
    process_data(data, len);
    printf("Str: %.*s", len, data);

    /* Increase offset position for next block */
    pos += len;
}

/* Call free only on a pbuf. Since it is chained, b and c will be freed too */
esp_pbuf_free(a);
group ESP_PBUF

Packet buffer manager.

Typedefs

typedef struct esp_pbuf *esp_pbuf_p

Pointer to esp_pbuf_t structure.

Functions

esp_pbuf_p esp_pbuf_new(size_t len)

Allocate packet buffer for network data of specific size.

Return

Pointer to allocated memory, NULL otherwise

Parameters
  • [in] len: Length of payload memory to allocate

size_t esp_pbuf_free(esp_pbuf_p pbuf)

Free previously allocated packet buffer.

Return

Number of freed pbufs from head

Parameters
  • [in] pbuf: Packet buffer to free

void *esp_pbuf_data(const esp_pbuf_p pbuf)

Get data pointer from packet buffer.

Return

Pointer to data buffer on success, NULL otherwise

Parameters
  • [in] pbuf: Packet buffer

size_t esp_pbuf_length(const esp_pbuf_p pbuf, uint8_t tot)

Get length of packet buffer.

Return

Length of data in units of bytes

Parameters
  • [in] pbuf: Packet buffer to get length for

  • [in] tot: Set to 1 to return total packet chain length or 0 to get only first packet length

uint8_t esp_pbuf_set_length(esp_pbuf_p pbuf, size_t new_len)

Set new length of pbuf.

Note

New length can only be smaller than existing one. It has no effect when greater than existing one

Note

This function can be used on single-chain pbufs only, without next pbuf in chain

Return

1 on success, 0 otherwise

Parameters
  • [in] pbuf: Pbuf to make it smaller

  • [in] new_len: New length in units of bytes

espr_t esp_pbuf_take(esp_pbuf_p pbuf, const void *data, size_t len, size_t offset)

Copy user data to chain of pbufs.

Return

espOK on success, member of espr_t enumeration otherwise

Parameters
  • [in] pbuf: First pbuf in chain to start copying to

  • [in] data: Input data to copy to pbuf memory

  • [in] len: Length of input data to copy

  • [in] offset: Start offset in pbuf where to start copying

size_t esp_pbuf_copy(esp_pbuf_p pbuf, void *data, size_t len, size_t offset)

Copy memory from pbuf to user linear memory.

Return

Number of bytes copied

Parameters
  • [in] pbuf: Pbuf to copy from

  • [out] data: User linear memory to copy to

  • [in] len: Length of data in units of bytes

  • [in] offset: Possible start offset in pbuf

espr_t esp_pbuf_cat(esp_pbuf_p head, const esp_pbuf_p tail)

Concatenate 2 packet buffers together to one big packet.

Note

After tail pbuf has been added to head pbuf chain, it must not be referenced by user anymore as it is now completelly controlled by head pbuf. In simple words, when user calls this function, it should not call esp_pbuf_free function anymore, as it might make memory undefined for head pbuf.

Return

espOK on success, member of espr_t enumeration otherwise

See

esp_pbuf_chain

Parameters
  • [in] head: Head packet buffer to append new pbuf to

  • [in] tail: Tail packet buffer to append to head pbuf

espr_t esp_pbuf_chain(esp_pbuf_p head, esp_pbuf_p tail)

Chain 2 pbufs together. Similar to esp_pbuf_cat but now new reference is done from head pbuf to tail pbuf.

Note

After this function call, user must call esp_pbuf_free to remove its reference to tail pbuf and allow control to head pbuf: esp_pbuf_free(tail)

Return

espOK on success, member of espr_t enumeration otherwise

See

esp_pbuf_cat

Parameters
  • [in] head: Head packet buffer to append new pbuf to

  • [in] tail: Tail packet buffer to append to head pbuf

esp_pbuf_p esp_pbuf_unchain(esp_pbuf_p head)

Unchain first pbuf from list and return second one.

tot_len and len fields are adjusted to reflect new values and reference counter is as is

Note

After unchain, user must take care of both pbufs (head and new returned one)

Return

Next pbuf after head

Parameters
  • [in] head: First pbuf in chain to remove from chain

espr_t esp_pbuf_ref(esp_pbuf_p pbuf)

Increment reference count on pbuf.

Return

espOK on success, member of espr_t enumeration otherwise

Parameters
  • [in] pbuf: pbuf to increase reference

uint8_t esp_pbuf_get_at(const esp_pbuf_p pbuf, size_t pos, uint8_t *el)

Get value from pbuf at specific position.

Return

1 on success, 0 otherwise

Parameters
  • [in] pbuf: Pbuf used to get data from

  • [in] pos: Position at which to get element

  • [out] el: Output variable to save element value at desired position

size_t esp_pbuf_memcmp(const esp_pbuf_p pbuf, const void *data, size_t len, size_t offset)

Compare pbuf memory with memory from data.

Note

Compare is done on entire pbuf chain

Return

0 if equal, ESP_SIZET_MAX if memory/offset too big or anything between if not equal

See

esp_pbuf_strcmp

Parameters
  • [in] pbuf: Pbuf used to compare with data memory

  • [in] data: Actual data to compare with

  • [in] len: Length of input data in units of bytes

  • [in] offset: Start offset to use when comparing data

size_t esp_pbuf_strcmp(const esp_pbuf_p pbuf, const char *str, size_t offset)

Compare pbuf memory with input string.

Note

Compare is done on entire pbuf chain

Return

0 if equal, ESP_SIZET_MAX if memory/offset too big or anything between if not equal

See

esp_pbuf_memcmp

Parameters
  • [in] pbuf: Pbuf used to compare with data memory

  • [in] str: String to be compared with pbuf

  • [in] offset: Start memory offset in pbuf

size_t esp_pbuf_memfind(const esp_pbuf_p pbuf, const void *data, size_t len, size_t off)

Find desired needle in a haystack.

Return

ESP_SIZET_MAX if no match or position where in pbuf we have a match

See

esp_pbuf_strfind

Parameters
  • [in] pbuf: Pbuf used as haystack

  • [in] needle: Data memory used as needle

  • [in] len: Length of needle memory

  • [in] off: Starting offset in pbuf memory

size_t esp_pbuf_strfind(const esp_pbuf_p pbuf, const char *str, size_t off)

Find desired needle (str) in a haystack (pbuf)

Return

ESP_SIZET_MAX if no match or position where in pbuf we have a match

See

esp_pbuf_memfind

Parameters
  • [in] pbuf: Pbuf used as haystack

  • [in] str: String to search for in pbuf

  • [in] off: Starting offset in pbuf memory

uint8_t esp_pbuf_advance(esp_pbuf_p pbuf, int len)

Advance pbuf payload pointer by number of len bytes. It can only advance single pbuf in a chain.

Note

When other pbufs are referencing current one, they are not adjusted in length and total length

Return

1 on success, 0 otherwise

Parameters
  • [in] pbuf: Pbuf to advance

  • [in] len: Number of bytes to advance. when negative is used, buffer size is increased only if it was decreased before

esp_pbuf_p esp_pbuf_skip(esp_pbuf_p pbuf, size_t offset, size_t *new_offset)

Skip a list of pbufs for desired offset.

Note

Reference is not changed after return and user must not free the memory of new pbuf directly

Return

New pbuf on success, NULL otherwise

Parameters
  • [in] pbuf: Start of pbuf chain

  • [in] offset: Offset in units of bytes to skip

  • [out] new_offset: Pointer to output variable to save new offset in returned pbuf

void *esp_pbuf_get_linear_addr(const esp_pbuf_p pbuf, size_t offset, size_t *new_len)

Get linear offset address for pbuf from specific offset.

Note

Since pbuf memory can be fragmented in chain, you may need to call function multiple times to get memory for entire pbuf chain

Return

Pointer to memory on success, NULL otherwise

Parameters
  • [in] pbuf: Pbuf to get linear address

  • [in] offset: Start offset from where to start

  • [out] new_len: Length of memory returned by function

void esp_pbuf_set_ip(esp_pbuf_p pbuf, const esp_ip_t *ip, esp_port_t port)

Set IP address and port number for received data.

Parameters
  • [in] pbuf: Packet buffer

  • [in] ip: IP to assing to packet buffer

  • [in] port: Port number to assign to packet buffer

void esp_pbuf_dump(esp_pbuf_p p, uint8_t seq)

Dump and debug pbuf chain.

Parameters
  • [in] p: Head pbuf to dump

  • [in] seq: Set to 1 to dump all pbufs in linked list or 0 to dump first one only

struct esp_pbuf_t
#include <esp_private.h>

Packet buffer structure.

Public Members

struct esp_pbuf *next

Next pbuf in chain list

size_t tot_len

Total length of pbuf chain

size_t len

Length of payload

size_t ref

Number of references to this structure

uint8_t *payload

Pointer to payload memory

esp_ip_t ip

Remote address for received IPD data

esp_port_t port

Remote port for received IPD data