In practice we often have to encode/decode data which does not fit
into a single encoder or decoder. To support this use-case Kodo
provides the kodo::object::storage_encoder and
kodo::object::storage_decoder classes.

It is recommended that you first familiarize yourself with using a
single encoder/decoder pair. You will notice that extending
to several encoders and decoders requires only a few changes to the
code. We will not explain all parameters in detail in this example
only those relevant to using the kodo::object::storage_encoder and
kodo::object::storage_decoder classes. If you find some
information missing, please check the The Basics example as it is likely
you find it there.

// Copyright Steinwurf ApS 2011.// Distributed under the "STEINWURF EVALUATION LICENSE 1.0".// See accompanying file LICENSE.rst or// http://www.steinwurf.com/licensing#include<cstring>#include<utility>#include<iostream>#include<vector>//! [0]#include<kodo_core/object/storage_decoder.hpp>#include<kodo_core/object/storage_encoder.hpp>#include<kodo_rlnc/full_vector_codes.hpp>//! [1]/// @example encode_decode_storage.cpp////// Often we want to encode / decode data that exceed a single/// encoding/decoding block. In this case we need to divide the data/// into manageable blocks and then encode and decode each block/// separately. This examples shows how to use the storage encoder and/// decoder in Kodo.////// A single block consists of symbol and each symbol has a size in bytes./// So the total size of a block is number of symbols multiplied by/// the symbol size. We use the word "block" here, in litterature several/// other words are used to refer to the same thing, other popular words/// are generation or chunk. They are all different words for the same/// thing.////// One important thing to note here is that data encoded from one block/// cannot be mixed with data encoded from a different block. So if we/// have two encoders we need also two decoders and we must pass data/// between the corresponding encoders and decoders.////// +---------+ +---------+ +---------+ +---------+/// | | | | | | | |/// | | | encoder | | decoder | | |/// | | +--> | one | +--> | one | +--> | |/// | input | | | | | | output |/// | data | +---------+ +---------+ | data |/// | | | | | | | |/// | | +--> | encoder | +--> | decoder | +--> | |/// | | | two | | two | | |/// | | | | | | | |/// +---------+ +---------+ +---------+ +---------+////// If the encoded data is to be trasmitted over a network or stored/// we do have to make sure we can identify which block the data comes/// from. This can be done by adding a "block id" to each encoded/// packet e.g. an integer value which is written on the encoder side/// and which can later be read by the receving process to understand/// which decoder it needs to pass the data to.////// As an example a header could look like:////// +--------+----------------------------+/// | block | payload |/// | id | data |/// +--------+----------------------------+////// Most likely other application specific headers would need to be/// added, this is just to illustrate in it's simplets form how it/// could look like. The example below does not write a "block id" to/// each encoded payload, but one could be added with minimal effort.////// Both the encoder and decoder uses a shallow storage which means/// that they operate directly on the memory provided.intmain(){//! [2]// Set the number of symbols (i.e. the generation size in RLNC// terminology) and the size of a symbol in bytesuint32_tsymbols=42;uint32_tsymbol_size=64;fifi::api::fieldfield=fifi::api::field::binary;uint32_tobject_size=23456;//! [3]//! [4]usingstorage_encoder=kodo_core::object::storage_encoder<kodo_rlnc::full_vector_encoder>;usingstorage_decoder=kodo_core::object::storage_decoder<kodo_rlnc::full_vector_decoder>;//! [5]//! [6]storage_encoder::factoryencoder_factory(field,symbols,symbol_size);storage_decoder::factorydecoder_factory(field,symbols,symbol_size);autoencoder=encoder_factory.build();autodecoder=decoder_factory.build();std::vector<uint8_t>data_in(object_size,'x');std::vector<uint8_t>data_out(object_size,'\0');encoder->set_const_storage(storage::storage(data_in));decoder->set_mutable_storage(storage::storage(data_out));assert(encoder->object_size()==object_size);assert(decoder->object_size()==object_size);std::cout<<"object_size = "<<object_size<<std::endl;std::cout<<"encoder blocks = "<<encoder->blocks()<<std::endl;std::cout<<"decoder blocks = "<<decoder->blocks()<<std::endl;//! [7]//! [8]for(uint32_ti=0;i<encoder->blocks();++i){storage_encoder::stack_pointere=encoder->build(i);storage_decoder::stack_pointerd=decoder->build(i);std::vector<uint8_t>payload(e->payload_size());while(!d->is_complete()){e->write_payload(payload.data());// Here we would send and receive the payload over a// network. Lets throw away some packet to simulate.if(rand()%2){continue;}d->read_payload(payload.data());}}// Check we properly decoded the dataif(data_in==data_out){std::cout<<"Data decoded correctly"<<std::endl;}else{std::cout<<"Unexpected failure to decode "<<"please file a bug report :)"<<std::endl;}//! [9]}

As in most other examples we have to specify the number of symbols and
the size of each symbol which we would like to use for the individual
encoders and decoders. One thing to notice here is that these values
are maximum values (i.e. we will never exceed these). However,
depending on the block partitioning scheme used we might not use
exactly those values.

Note

When encoding/decoding large objects we have to assign
different parts of the data to different encoders/decoders,
the strategy for how this is done is called the block
partitioning scheme.

In addition we will also specify the size of the object we want to code.

1
2
3
4
5
6
7

// Set the number of symbols (i.e. the generation size in RLNC// terminology) and the size of a symbol in bytesuint32_tsymbols=42;uint32_tsymbol_size=64;fifi::api::fieldfield=fifi::api::field::binary;uint32_tobject_size=23456;

The kodo::object::storage_encoder and
kodo::object::storage_decoder classes take one template argument
which is the actual type of the erasure correcting code to use. In
this case we are using the kodo::shallow_full_rlnc_encoder for
encoding and kodo::shallow_full_rlnc_decoder for decoding. These
are standard RLNC (Random Linear Network Coding) codes.

Note

We use the shallow variant of the RLNC codes. This simply
means that Kodo will not copy the data into the
encoder/decoder, but operate directly on the user provided
buffer (this is currently the only supported mode).

The encoding/decoding loop has changed a bit since we now have several
encoders and decoders that need to finish before the entire object has
been encoded and decoded. However, the general structure is very
similar to using just a single encoder and decoder.