XZ Utils 5.3.4alpha
Data Structures | Typedefs | Functions
outqueue.h File Reference

Output queue handling in multithreaded coding. More...

#include "common.h"

Data Structures

struct  lzma_outbuf_s
 
struct  lzma_outq
 

Typedefs

typedef struct lzma_outbuf_s lzma_outbuf
 Output buffer for a single thread. More...
 

Functions

uint64_t lzma_outq_memusage (uint64_t buf_size_max, uint32_t threads)
 Calculate the memory usage of an output queue. More...
 
lzma_ret lzma_outq_init (lzma_outq *outq, const lzma_allocator *allocator, uint32_t threads)
 Initialize an output queue. More...
 
void lzma_outq_end (lzma_outq *outq, const lzma_allocator *allocator)
 Free the memory associated with the output queue. More...
 
void lzma_outq_clear_cache (lzma_outq *outq, const lzma_allocator *allocator)
 Free all cached buffers that consume memory but aren't in use. More...
 
void lzma_outq_clear_cache2 (lzma_outq *outq, const lzma_allocator *allocator, size_t keep_size)
 Like lzma_outq_clear_cache() but might keep one buffer. More...
 
lzma_ret lzma_outq_prealloc_buf (lzma_outq *outq, const lzma_allocator *allocator, size_t size)
 Preallocate a new buffer into cache. More...
 
lzma_outbuflzma_outq_get_buf (lzma_outq *outq, void *worker)
 Get a new buffer. More...
 
bool lzma_outq_is_readable (const lzma_outq *outq)
 Test if there is data ready to be read. More...
 
lzma_ret lzma_outq_read (lzma_outq *restrict outq, const lzma_allocator *restrict allocator, uint8_t *restrict out, size_t *restrict out_pos, size_t out_size, lzma_vli *restrict unpadded_size, lzma_vli *restrict uncompressed_size)
 Read finished data. More...
 
void lzma_outq_enable_partial_output (lzma_outq *outq, void(*enable_partial_output)(void *worker))
 Enable partial output from a worker thread. More...
 
static bool lzma_outq_has_buf (const lzma_outq *outq)
 Test if there is at least one buffer free. More...
 
static bool lzma_outq_is_empty (const lzma_outq *outq)
 Test if the queue is completely empty. More...
 
static uint64_t lzma_outq_outbuf_memusage (size_t buf_size)
 Get the amount of memory needed for a single lzma_outbuf. More...
 

Detailed Description

Output queue handling in multithreaded coding.

Typedef Documentation

◆ lzma_outbuf

typedef struct lzma_outbuf_s lzma_outbuf

Output buffer for a single thread.

Function Documentation

◆ lzma_outq_memusage()

uint64_t lzma_outq_memusage ( uint64_t  buf_size_max,
uint32_t  threads 
)

Calculate the memory usage of an output queue.

Returns
Approximate memory usage in bytes or UINT64_MAX on error.

◆ lzma_outq_init()

lzma_ret lzma_outq_init ( lzma_outq outq,
const lzma_allocator allocator,
uint32_t  threads 
)

Initialize an output queue.

Parameters
outqPointer to an output queue. Before calling this function the first time, *outq should have been zeroed with memzero() so that this function knows that there are no previous allocations to free.
allocatorPointer to allocator or NULL
threadsNumber of buffers that may be in use concurrently. Note that more than this number of buffers may actually get allocated to improve performance when buffers finish out of order. The actual maximum number of allocated buffers is derived from the number of threads.
Returns
- LZMA_OK
  • LZMA_MEM_ERROR

◆ lzma_outq_end()

void lzma_outq_end ( lzma_outq outq,
const lzma_allocator allocator 
)

Free the memory associated with the output queue.

◆ lzma_outq_clear_cache()

void lzma_outq_clear_cache ( lzma_outq outq,
const lzma_allocator allocator 
)

Free all cached buffers that consume memory but aren't in use.

◆ lzma_outq_clear_cache2()

void lzma_outq_clear_cache2 ( lzma_outq outq,
const lzma_allocator allocator,
size_t  keep_size 
)

Like lzma_outq_clear_cache() but might keep one buffer.

One buffer is not freed if its size is equal to keep_size. This is useful if the caller knows that it will soon need a buffer of keep_size bytes. This way it won't be freed and immediately reallocated.

◆ lzma_outq_prealloc_buf()

lzma_ret lzma_outq_prealloc_buf ( lzma_outq outq,
const lzma_allocator allocator,
size_t  size 
)

Preallocate a new buffer into cache.

Splitting the buffer allocation into a separate function makes it possible to ensure that way lzma_outq_get_buf() cannot fail. If the preallocated buffer isn't actually used (for example, some other error occurs), the caller has to do nothing as the buffer will be used later or cleared from the cache when not needed.

Returns
LZMA_OK on success, LZMA_MEM_ERROR if allocation fails

◆ lzma_outq_get_buf()

lzma_outbuf * lzma_outq_get_buf ( lzma_outq outq,
void *  worker 
)

Get a new buffer.

lzma_outq_prealloc_buf() must be used to ensure that there is a buffer available before calling lzma_outq_get_buf().

◆ lzma_outq_is_readable()

bool lzma_outq_is_readable ( const lzma_outq outq)

Test if there is data ready to be read.

Call to this function must be protected with the same mutex that is used to protect lzma_outbuf.finished.

References lzma_outbuf_s::finished, lzma_outq::head, lzma_outbuf_s::pos, and lzma_outq::read_pos.

Referenced by wait_for_work().

◆ lzma_outq_read()

lzma_ret lzma_outq_read ( lzma_outq *restrict  outq,
const lzma_allocator *restrict  allocator,
uint8_t *restrict  out,
size_t *restrict  out_pos,
size_t  out_size,
lzma_vli *restrict  unpadded_size,
lzma_vli *restrict  uncompressed_size 
)

Read finished data.

Parameters
outqPointer to an output queue
outBeginning of the output buffer
out_posThe next byte will be written to out[*out_pos].
out_sizeSize of the out buffer; the first byte into which no data is written to is out[out_size].
unpadded_sizeUnpadded Size from the Block encoder
uncompressed_sizeUncompressed Size from the Block encoder
Returns
- LZMA: All OK. Either no data was available or the buffer being read didn't become empty yet.
  • LZMA_STREAM_END: The buffer being read was finished. *unpadded_size and *uncompressed_size were set if they were not NULL.
Note
This reads lzma_outbuf.finished and .pos variables and thus calls to this function need to be protected with a mutex.

◆ lzma_outq_enable_partial_output()

void lzma_outq_enable_partial_output ( lzma_outq outq,
void(*)(void *worker)  enable_partial_output 
)

Enable partial output from a worker thread.

If the buffer at the head of the output queue isn't finished, this will call enable_partial_output on the worker associated with that output buffer.

Note
This reads a lzma_outbuf.finished variable and thus calls to this function need to be protected with a mutex.

References lzma_outbuf_s::finished, lzma_outq::head, and lzma_outbuf_s::worker.

◆ lzma_outq_has_buf()

static bool lzma_outq_has_buf ( const lzma_outq outq)
inlinestatic

Test if there is at least one buffer free.

This must be used before getting a new buffer with lzma_outq_get_buf().

References lzma_outq::bufs_in_use, and lzma_outq::bufs_limit.

Referenced by wait_for_work().

◆ lzma_outq_is_empty()

static bool lzma_outq_is_empty ( const lzma_outq outq)
inlinestatic

Test if the queue is completely empty.

References lzma_outq::bufs_in_use.

◆ lzma_outq_outbuf_memusage()

static uint64_t lzma_outq_outbuf_memusage ( size_t  buf_size)
inlinestatic

Get the amount of memory needed for a single lzma_outbuf.

Note
Caller must check that the argument is significantly less than SIZE_MAX to avoid an integer overflow!