Packet buffer
Packet buffer (or pbuf) is buffer manager to handle received data from any connection. It is optimized to construct big buffer of smaller chunks of fragmented data as received bytes are not always coming as single packet.
Pbuf block diagram
Image above shows structure of pbuf chain. Each pbuf consists of:
Pointer to next pbuf, or
NULL
when it is last in chainLength of current packet length
Length of current packet and all next in chain
If pbuf is last in chain, total length is the same as current packet length
Reference counter, indicating how many pointers point to current pbuf
Actual buffer data
Top image shows 3
pbufs connected to single chain.
There are 2
custom pointer variables to point at different pbuf structures. Second pbuf has reference counter set to 2
, as 2
variables point to it:
next of pbuf 1 is the first one
User variable 2 is the second one
Block number |
Next pbuf |
Block size |
Total size in chain |
Reference counter |
---|---|---|---|---|
Block 1 |
Block 2 |
|
|
|
Block 2 |
Block 3 |
|
|
|
Block 3 |
|
|
|
|
Reference counter
Reference counter holds number of references (or variables) pointing to this block. It is used to properly handle memory free operation, especially when pbuf is used by lib core and application layer.
Note
If there would be no reference counter information and application would free memory while another part of library still uses its reference, application would invoke undefined behavior and system could crash instantly.
When application tries to free pbuf chain as on first image, it would normally call lwesp_pbuf_free()
function. That would:
Decrease reference counter by
1
If reference counter
== 0
, it removes it from chain list and frees packet buffer memoryIf reference counter
!= 0
after decrease, it stops free procedureGo to next pbuf in chain and repeat steps
As per first example, result of freeing from user variable 1 would look similar to image and table below.
First block (blue) had reference counter set to 1
prior freeing operation.
It was successfully removed as user variable 1 was the only one pointing to it,
while second (green) block had reference counter set to 2
, preventing free operation.
Block number |
Next pbuf |
Block size |
Total size in chain |
Reference counter |
---|---|---|---|---|
Block 2 |
Block 3 |
|
|
|
Block 3 |
|
|
|
|
Note
Block 1 has been successfully freed, but since block 2 had reference counter set to 2
before, it was only decreased by 1
to a new value 1
and free operation stopped instead.
User variable 2 is still using pbuf starting at block 2 and must manually call lwesp_pbuf_free()
to free it.
Concatenating vs chaining
This section will explain difference between concat and chain operations.
Both operations link 2
pbufs together in a chain of pbufs,
difference is that chain operation increases reference counter to linked pbuf,
while concat keeps reference counter at its current status.
Concat operation
Concat operation shall be used when 2
pbufs are linked together and reference to second is no longer used.
After concating 2 pbufs together, reference counter of second is still set to 1
, however we can see that 2
pointers point to second pbuf.
Note
After application calls lwesp_pbuf_cat()
, it must not use pointer which points to second pbuf.
This would invoke undefined behavior if one pointer tries to free memory while second still points to it.
An example code showing proper usage of concat operation:
1lwesp_pbuf_p a, b;
2
3/* Create 2 pbufs of different sizes */
4a = lwesp_pbuf_new(10);
5b = lwesp_pbuf_new(20);
6
7/* Link them together with concat operation */
8/* Reference on b will stay as is, won't be increased */
9lwesp_pbuf_cat(a, b);
10
11/*
12 * Operating with b variable has from now on undefined behavior,
13 * application shall stop using variable b to access pbuf.
14 *
15 * The best way would be to set b reference to NULL
16 */
17b = NULL;
18
19/*
20 * When application doesn't need pbufs anymore,
21 * free a and it will also free b
22 */
23lwesp_pbuf_free(a);
Chain operation
Chain operation shall be used when 2
pbufs are linked together and reference to second is still required.
After chainin 2 pbufs together, reference counter of second is increased by 1
, which allows application to reference second pbuf separatelly.
Note
After application calls lwesp_pbuf_chain()
,
it also has to manually free its reference using lwesp_pbuf_free()
function.
Forgetting to free pbuf invokes memory leak
An example code showing proper usage of chain operation:
1lwesp_pbuf_p a, b;
2
3/* Create 2 pbufs of different sizes */
4a = lwesp_pbuf_new(10);
5b = lwesp_pbuf_new(20);
6
7/* Chain both pbufs together */
8/* This will increase reference on b as 2 variables now point to it */
9lwesp_pbuf_chain(a, b);
10
11/*
12 * When application does not need a anymore, it may free it
13
14 * This will free only pbuf a, as pbuf b has now 2 references:
15 * - one from pbuf a
16 * - one from variable b
17 */
18
19/* If application calls this, it will free only first pbuf */
20/* As there is link to b pbuf somewhere */
21lwesp_pbuf_free(a);
22
23/* Reset a variable, not used anymore */
24a = NULL;
25
26/*
27 * At this point, b is still valid memory block,
28 * but when application doesn't need it anymore,
29 * it should free it, otherwise memory leak appears
30 */
31lwesp_pbuf_free(b);
32
33/* Reset b variable */
34b = NULL;
Extract pbuf data
Each pbuf holds some amount of data bytes. When multiple pbufs are linked together (either chained or concated), blocks of raw data are not linked to contiguous memory block. It is necessary to process block by block manually.
An example code showing proper reading of any pbuf:
1const void* data;
2size_t pos, len;
3lwesp_pbuf_p a, b, c;
4
5const char str_a[] = "This is one long";
6const char str_a[] = "string. We want to save";
7const char str_a[] = "chain of pbufs to file";
8
9/* Create pbufs to hold these strings */
10a = lwesp_pbuf_new(strlen(str_a));
11b = lwesp_pbuf_new(strlen(str_b));
12c = lwesp_pbuf_new(strlen(str_c));
13
14/* Write data to pbufs */
15lwesp_pbuf_take(a, str_a, strlen(str_a), 0);
16lwesp_pbuf_take(b, str_b, strlen(str_b), 0);
17lwesp_pbuf_take(c, str_c, strlen(str_c), 0);
18
19/* Connect pbufs together */
20lwesp_pbuf_chain(a, b);
21lwesp_pbuf_chain(a, c);
22
23/*
24 * pbuf a now contains chain of b and c together
25 * and at this point application wants to print (or save) data from chained pbuf
26 *
27 * Process pbuf by pbuf with code below
28 */
29
30/*
31 * Get linear address of current pbuf at specific offset
32 * Function will return pointer to memory address at specific position
33 * and `len` will hold length of data block
34 */
35pos = 0;
36while ((data = lwesp_pbuf_get_linear_addr(a, pos, &len)) != NULL) {
37 /* Custom process function... */
38 /* Process data with data pointer and block length */
39 process_data(data, len);
40 printf("Str: %.*s", len, data);
41
42 /* Increase offset position for next block */
43 pos += len;
44}
45
46/* Call free only on a pbuf. Since it is chained, b and c will be freed too */
47lwesp_pbuf_free(a);
- group LWESP_PBUF
Packet buffer manager.
Typedefs
-
typedef struct lwesp_pbuf *lwesp_pbuf_p
Pointer to lwesp_pbuf_t structure.
Functions
-
lwesp_pbuf_p lwesp_pbuf_new(size_t len)
Allocate packet buffer for network data of specific size.
- Parameters
len – [in] Length of payload memory to allocate
- Returns
Pointer to allocated memory,
NULL
otherwise
-
size_t lwesp_pbuf_free(lwesp_pbuf_p pbuf)
Free previously allocated packet buffer.
See also
Note
Application must not use reference to pbuf after the call to this function. It is advised to immediately set pointer to
NULL
or to call. Alternatively, call lwesp_pbuf_free_s, which will reset the pointer after free operation has been completed- Parameters
pbuf – [in] Packet buffer to free
- Returns
Number of freed pbufs from head
-
size_t lwesp_pbuf_free_s(lwesp_pbuf_p *pbuf)
Free previously allocated packet buffer in safe way. Function accepts pointer to pointer and will set the pointer to
NULL
after the successful allocation.- Parameters
pbuf_ptr – [inout] Pointer to pointer to packet buffer
- Returns
Number of packet buffers freed in the chain
-
void *lwesp_pbuf_data(const lwesp_pbuf_p pbuf)
Get data pointer from packet buffer.
- Parameters
pbuf – [in] Packet buffer
- Returns
Pointer to data buffer on success,
NULL
otherwise
-
size_t lwesp_pbuf_length(const lwesp_pbuf_p pbuf, uint8_t tot)
Get length of packet buffer.
- Parameters
pbuf – [in] Packet buffer to get length for
tot – [in] Set to
1
to return total packet chain length or0
to get only first packet length
- Returns
Length of data in units of bytes
-
uint8_t lwesp_pbuf_set_length(lwesp_pbuf_p pbuf, size_t new_len)
Set new length of pbuf.
Note
New length can only be smaller than existing one. It has no effect when greater than existing one
Note
This function can be used on single-chain pbufs only, without
next
pbuf in chain- Parameters
pbuf – [in] Pbuf to make it smaller
new_len – [in] New length in units of bytes
- Returns
1
on success,0
otherwise
-
lwespr_t lwesp_pbuf_take(lwesp_pbuf_p pbuf, const void *data, size_t len, size_t offset)
Copy user data to chain of pbufs.
-
size_t lwesp_pbuf_copy(lwesp_pbuf_p pbuf, void *data, size_t len, size_t offset)
Copy memory from pbuf to user linear memory.
- Parameters
pbuf – [in] Pbuf to copy from
data – [out] User linear memory to copy to
len – [in] Length of data in units of bytes
offset – [in] Possible start offset in pbuf
- Returns
Number of bytes copied
-
lwespr_t lwesp_pbuf_cat(lwesp_pbuf_p head, const lwesp_pbuf_p tail)
Concatenate
2
packet buffers together to one big packet.See also
See also
Note
After
tail
pbuf has been added tohead
pbuf chain, it must not be referenced by user anymore as it is now completely controlled byhead
pbuf. In simple words, when user calls this function, it should not call lwesp_pbuf_free function anymore, as it might make memory undefined forhead
pbuf.
-
lwespr_t lwesp_pbuf_cat_s(lwesp_pbuf_p head, lwesp_pbuf_p *tail)
Concatenate
2
packet buffers together to one big packet with safe pointer management.See also
See also
Note
After
tail
pbuf has been added tohead
pbuf chain,tail
pointer will be set toNULL
-
lwespr_t lwesp_pbuf_chain(lwesp_pbuf_p head, lwesp_pbuf_p tail)
Chain 2 pbufs together. Similar to lwesp_pbuf_cat but now new reference is done from head pbuf to tail pbuf.
See also
See also
See also
lwesp_pbuf_chain_s
Note
After this function call, user must call lwesp_pbuf_free to remove its reference to tail pbuf and allow control to head pbuf:
lwesp_pbuf_free(tail)
-
lwesp_pbuf_p lwesp_pbuf_unchain(lwesp_pbuf_p head)
Unchain first pbuf from list and return second one.
tot_len
andlen
fields are adjusted to reflect new values and reference counter isas is
Note
After unchain, user must take care of both pbufs (
head
andnew returned one
)- Parameters
head – [in] First pbuf in chain to remove from chain
- Returns
Next pbuf after
head
-
lwespr_t lwesp_pbuf_ref(lwesp_pbuf_p pbuf)
Increment reference count on pbuf.
-
uint8_t lwesp_pbuf_get_at(const lwesp_pbuf_p pbuf, size_t pos, uint8_t *el)
Get value from pbuf at specific position.
- Parameters
pbuf – [in] Pbuf used to get data from
pos – [in] Position at which to get element
el – [out] Output variable to save element value at desired position
- Returns
1
on success,0
otherwise
-
size_t lwesp_pbuf_memcmp(const lwesp_pbuf_p pbuf, const void *data, size_t len, size_t offset)
Compare pbuf memory with memory from data.
See also
Note
Compare is done on entire pbuf chain
- Parameters
pbuf – [in] Pbuf used to compare with data memory
data – [in] Actual data to compare with
len – [in] Length of input data in units of bytes
offset – [in] Start offset to use when comparing data
- Returns
0
if equal,LWESP_SIZET_MAX
if memory/offset too big or anything between if not equal
-
size_t lwesp_pbuf_strcmp(const lwesp_pbuf_p pbuf, const char *str, size_t offset)
Compare pbuf memory with input string.
See also
Note
Compare is done on entire pbuf chain
- Parameters
pbuf – [in] Pbuf used to compare with data memory
str – [in] String to be compared with pbuf
offset – [in] Start memory offset in pbuf
- Returns
0
if equal,LWESP_SIZET_MAX
if memory/offset too big or anything between if not equal
-
size_t lwesp_pbuf_memfind(const lwesp_pbuf_p pbuf, const void *data, size_t len, size_t off)
Find desired needle in a haystack.
See also
- Parameters
pbuf – [in] Pbuf used as haystack
needle – [in] Data memory used as needle
len – [in] Length of needle memory
off – [in] Starting offset in pbuf memory
- Returns
LWESP_SIZET_MAX
if no match or position where in pbuf we have a match
-
size_t lwesp_pbuf_strfind(const lwesp_pbuf_p pbuf, const char *str, size_t off)
Find desired needle (str) in a haystack (pbuf)
See also
- Parameters
pbuf – [in] Pbuf used as haystack
str – [in] String to search for in pbuf
off – [in] Starting offset in pbuf memory
- Returns
LWESP_SIZET_MAX
if no match or position where in pbuf we have a match
-
uint8_t lwesp_pbuf_advance(lwesp_pbuf_p pbuf, int len)
Advance pbuf payload pointer by number of len bytes. It can only advance single pbuf in a chain.
Note
When other pbufs are referencing current one, they are not adjusted in length and total length
- Parameters
pbuf – [in] Pbuf to advance
len – [in] Number of bytes to advance. when negative is used, buffer size is increased only if it was decreased before
- Returns
1
on success,0
otherwise
-
lwesp_pbuf_p lwesp_pbuf_skip(lwesp_pbuf_p pbuf, size_t offset, size_t *new_offset)
Skip a list of pbufs for desired offset.
Note
Reference is not changed after return and user must not free the memory of new pbuf directly
- Parameters
pbuf – [in] Start of pbuf chain
offset – [in] Offset in units of bytes to skip
new_offset – [out] Pointer to output variable to save new offset in returned pbuf
- Returns
New pbuf on success,
NULL
otherwise
-
void *lwesp_pbuf_get_linear_addr(const lwesp_pbuf_p pbuf, size_t offset, size_t *new_len)
Get linear offset address for pbuf from specific offset.
Note
Since pbuf memory can be fragmented in chain, you may need to call function multiple times to get memory for entire pbuf chain
- Parameters
pbuf – [in] Pbuf to get linear address
offset – [in] Start offset from where to start
new_len – [out] Length of memory returned by function
- Returns
Pointer to memory on success,
NULL
otherwise
-
void lwesp_pbuf_set_ip(lwesp_pbuf_p pbuf, const lwesp_ip_t *ip, lwesp_port_t port)
Set IP address and port number for received data.
- Parameters
pbuf – [in] Packet buffer
ip – [in] IP to assing to packet buffer
port – [in] Port number to assign to packet buffer
-
void lwesp_pbuf_dump(lwesp_pbuf_p p, uint8_t seq)
Dump and debug pbuf chain.
- Parameters
p – [in] Head pbuf to dump
seq – [in] Set to
1
to dump allpbufs
in linked list or0
to dump first one only
-
struct lwesp_pbuf_t
- #include <lwesp_private.h>
Packet buffer structure.
Public Members
-
struct lwesp_pbuf *next
Next pbuf in chain list
-
size_t tot_len
Total length of pbuf chain
-
size_t len
Length of payload
-
size_t ref
Number of references to this structure
-
uint8_t *payload
Pointer to payload memory
-
lwesp_ip_t ip
Remote address for received IPD data
-
lwesp_port_t port
Remote port for received IPD data
-
struct lwesp_pbuf *next
-
typedef struct lwesp_pbuf *lwesp_pbuf_p