LeoStorage's features depend on its configuration. If once a LeoFS system is launched, you cannot modify the following LeoStorage's configurations because the algorithm of the data operation strictly adheres to the settings.

Able to change the directory of the container(s) but not able to add or remove the directory(s). You need to move the data files which are <obj_containers.path>/avs/object and <obj_containers.path>/avs/metadata, which adhere to this configuration.

obj_containers.num_of_containers

Yes

Not able to change the configuration because LeoStorage cannot retrieve objects or metadatas. If you want to modify this setting in order to add additional disk volumes for LeoFS then follow the instruction here3.

obj_containers.metadata_storage

Yes

As above

num_of_vnodes

Yes

As above

MQ

mq.backend_db

Modifiable with condition

Lose all the MQ's data after changing

mq.num_of_mq_procs

Modifiable with condition

As above

Replication and Recovery object(s)

replication.rack_awareness.rack_id

Yes

Not able to change the configuration because LeoFS cannot retrieve objects or metadatas.

Other Directories Settings

queue_dir

Modifiable with condition

Able to change the MQ's directory but you need to move the MQ's data, which adhere to this configuration.

A number of object-containers of each directory. As backend_db.eleveldb.write_buf_size * obj_containers.num_of_containers memory can be consumed in total, take both into account to meet with your memory footprint requirements on LeoStorage.

The MQ storage feature is pluggable which depends on bitcask and leveldb.

( Default: leveldb )

mq.num_of_mq_procs

A number of mq-server's processes

( Default: 8 )

mq.num_of_batch_process_max

Maximum number of bach processes of message

( Default: 3000 )

mq.num_of_batch_process_regular

Regular value of bach processes of message

( Default: 1600 )

mq.interval_between_batch_procs_max

Maximum value of interval between batch-procs

( Default: 3000, Unit: msec )

mq.interval_between_batch_procs_regular

Regular value of interval between batch-procs

( Default: 500, Unit: msec )

Backend DB / eleveldb

backend_db.eleveldb.write_buf_size

Write Buffer Size. Larger values increase performance, especially during bulk loads.Up to two write buffers may be held in memory at the same time, so you may wish to adjust this parameter to control memory usage.Also, a larger write buffer will result in a longer recovery time the next time the database is opened. As backend_db.eleveldb.write_buf_size * obj_containers.num_of_containers memory can be consumed in total, take both into account to meet with your memory footprint requirements on LeoStorage.

( Default: 62914560 )

backend_db.eleveldb.max_open_files

Max Open Files. Number of open files that can be used by the DB. You may need to increase this if your database has a large working set (budget one open file per 2MB of working set).

( Default: 1000 )

backend_db.eleveldb.sst_block_size

The size of a data block is controlled by the SST block size. The size represents a threshold, not a fixed count. Whenever a newly created block reaches this uncompressed size, leveldb considers it full and writes the block with its metadata to disk. The number of keys contained in the block depends upon the size of the values and keys.

( Default: 4096 )

Replication and Recovery object(s)

replication.rack_awareness.rack_id

Rack-Id for the rack-awareness replica placement feature

replication.recovery.size_of_stacked_objs

Size of stacked objects. Objects are stacked to send as a bulked object to remote nodes.

( Default: 5242880, Unit: byte )

replication.recovery.stacking_timeout

Stacking timeout. A bulked object are sent to a remote node after reaching the timeout.

( Default: 1, Unit: sec )

Multi Data Center Replication / Basic

mdc_replication.size_of_stacked_objs

Size of stacked objects. Objects are stacked to send as a bulked object to a remote cluster.

( Default: 33554432, Unit: byte )

mdc_replication.stacking_timeout

Stacking timeout. A bulked object are sent to a remote cluster after reaching the timeout.

-smp enable and -smp start the Erlang runtime system with SMP support enabled.

( Default: enable )

erlang.schedulers.compaction_of_load

Enables or disables scheduler compaction of load. If it's enabled, the Erlang VM will attempt to fully load as many scheduler threads as mush as possible.

( Default: true )

erlang.schedulers.utilization_balancing

Enables or disables scheduler utilization balancing of load. By default scheduler utilization balancing is disabled and instead scheduler compaction of load is enabled, which strives for a load distribution that causes as many scheduler threads as possible to be fully loaded (that is, not run out of work).

( Default: false )

erlang.distribution_buffer_size

Sender-side network distribution buffer size (unit: KB)

( Default: 32768 )

erlang.fullsweep_after

Option fullsweep_after makes it possible to specify the maximum number of generational collections before forcing a fullsweep, even if there is room on the old heap. Setting the number to zero disables the general collection algorithm, that is, all live data is copied at every garbage collection.

( Default: 0 )

erlang.secio

Enables or disables eager check I/O scheduling. The flag effects when schedulers will check for I/O operations possible to execute, and when such I/O operations will execute.

( Default: true )

process_limit

The maxinum number of Erlang processes. Sets the maximum number of simultaneously existing processes for this system if a Number is passed as value. Valid range for Number is [1024-134217727]

Without setting object_storage.is_strict_check to true, there is a little possibility your data could be broken without any caution even if a LeoFS system is running on a filesystem like ZFS1 that protect both the metadata and the data blocks through the checksum when bugs of any unexpected or unknown software got AVS files broken.

mq.num_of_mq_procs can affect not only the performance/load during recover/rebalance operations but the load while there is at least one node suspended/downed in the cluster. so that setting mq.num_of_mq_procs to an appropriate value based on the amount of expected traffic and hardware specs is really important. This section would give you the brief understanding on mq.num_of_mq_procs and how to choose the optimal value for your requirements.

How the mq.num_of_mq_procs setting affect the system operations

High

Fast recover/rebalance time

High CPU/Load on storage during recover/rebalance and also existing suspended/stopped nodes in the cluster

Low

Slow recover/rebalance time

Low CPU/Load on storage during recover/rebalance and also existing suspended/stopped nodes in the cluster

Recommend settings

If you have enough CPU resources on storage nodes then set it to a higher one as long as it doesn't affect the operations coming from LeoGateway

If you don't then set it to somewhat a lower one unless the recover take too much time

LeoStorage's MQ mechanism depends on the watchdog mechanism to reduce costs of a message consumption. The MQ dynamically updates a number of batch processes and an interval of a message consumption.

Figure: Number-of-batch-processes and interval:

As of Figure: Relationship of Watchdog and MQ, the watchdog can automatically adjust a value of a number of batch processes between mq.num_of_batch_process_min and mq.num_of_batch_process_max, which is increased or decreased with mq.num_of_batch_process_step.

On the other hands, a value of an interval is adjusted between mq.interval_between_batch_procs_min and mq.interval_between_batch_procs_max, which is increased or decreased with mq.interval_between_batch_procs_step.

When the each value reached the min value, the MQ changes the status to suspending, after that the node’s processing costs is changed to low, the MQ updates the status to running, again.

LeoStorage's auto-compaction mechanism also depends on the watchdog mechanism to reduce costs of processing. The Auto-compaction can dynamically update a number of batch processes and an interval of a processing of seeking an object. The basic design of the relationship with the watchdog is similar to the MQ.

Figure: Number-of-batch-processes and interval

As of Figure: Relationship of the watchdog and the auto-compaction, the watchdog automatically adjusts the value of a number of batch processes between compaction.batch_procs_min and compaction.batch_procs_max, which is increased or decreased with compaction.batch_procs_step.

On the other hand, the value of an interval is adjusted between compaction.waiting_time_min and compaction.waiting_time_max, which is increased or decreased with compaction.waiting_time_step.

When the each value reached the min value, the auto-compaction changes the status to suspending, after that the node’s processing costs is changed to low, the auto-compaction updates the status to running, again.