Internal call graph ▹

Internal call graph ▾

In the call graph viewer below, each node
is a function belonging to this package
and its children are the functions it
calls—perhaps dynamically.

The root nodes are the entry points of the
package: functions that may be called from
outside the package.
There may be non-exported or anonymous
functions among them if they are called
dynamically from another package.

Click a node to visit that function's source code.
From there you can visit its callers by
clicking its declaring func
token.

Functions may be omitted if they were
determined to be unreachable in the
particular programs or tests that were
analyzed.

Constants

const (
// CompactionTempExtension is the extension used for temporary files created during compaction.
CompactionTempExtension = "tmp"
// TSMFileExtension is the extension used for TSM files.
TSMFileExtension = "tsm"
)

const (
// MagicNumber is written as the first 4 bytes of a data file to// identify the file as a tsm1 formatted file
MagicNumber uint32 = 0x16D116D1
// Version indicates the version of the TSM file format.
Version byte = 1
)

var (
//ErrNoValues is returned when TSMWriter.WriteIndex is called and there are no values to write.
ErrNoValues = fmt.Errorf("no values written")
// ErrTSMClosed is returned when performing an operation against a closed TSM file.
ErrTSMClosed = fmt.Errorf("tsm file closed")
// ErrMaxKeyLengthExceeded is returned when attempting to write a key that is too long.
ErrMaxKeyLengthExceeded = fmt.Errorf("max key length exceeded")
// ErrMaxBlocksExceeded is returned when attempting to write a block past the allowed number.
ErrMaxBlocksExceeded = fmt.Errorf("max blocks exceeded")
)

ErrFileInUse is returned when attempting to remove or close a TSM file that is still being used.

var ErrFileInUse = fmt.Errorf("file still in use")

var (
// ErrSnapshotInProgress is returned if a snapshot is attempted while one is already running.
ErrSnapshotInProgress = fmt.Errorf("snapshot in progress")
)

CanReadBitFast returns true if calling ReadBitFast() is allowed.
Fast bit reads are allowed when at least 2 values are in the buffer.
This is because it is not required to refilled the buffer and the caller
can inline the calls.

WriteMulti writes the map of keys and associated values to the cache. This
function is goroutine-safe. It returns an error if the cache will exceeded
its max size by adding the new values. The write attempts to write as many
values as possible. If one key fails, the others can still succeed and an
error will be returned.

Load returns a cache loaded with the data contained within the segment files.
If, during reading of a segment file, corruption is encountered, that segment
file is truncated up to and including the last valid byte, and processing
continues with the next segment file.

DefaultPlanner implements CompactionPlanner using a strategy to roll up
multiple generations of TSM files into larger files in stages. It attempts
to minimize the number of TSM files on disk while rolling up a bounder number
of files.

type DefaultPlanner struct {
FileStore interface {
Stats() []FileStat
LastModified() time.Time
BlockCount(path string, idx int) int
}
// CompactFullWriteColdDuration specifies the length of time after// which if no writes have been committed to the WAL, the engine will// do a full compaction of the TSM files in this shard. This duration// should always be greater than the CacheFlushWriteColdDuraion
CompactFullWriteColdDuration time.Duration
// contains filtered or unexported fields
}

type Engine struct {
WAL *WAL
Cache *Cache
Compactor *Compactor
CompactionPlan CompactionPlanner
FileStore *FileStore
MaxPointsPerBlock int
// CacheFlushMemorySizeThreshold specifies the minimum size threshodl for// the cache when the engine should write a snapshot to a TSM file
CacheFlushMemorySizeThreshold uint64
// CacheFlushWriteColdDuration specifies the length of time after which if// no writes have been committed to the WAL, the engine will write// a snapshot of the cache to a TSM file
CacheFlushWriteColdDuration time.Duration
// contains filtered or unexported fields
}

Backup writes a tar archive of any TSM files modified since the passed
in time to the passed in writer. The basePath will be prepended to the names
of the files in the archive. It will force a snapshot of the WAL first
then perform the backup with a read lock against the file store. This means
that new TSM files will not be able to be created in this shard while the
backup is running. For shards that are still acively getting writes, this
could cause the WAL to backup, increasing memory usage and evenutally rejecting writes.

BlockCount returns number of values stored in the block at location idx
in the file at path. If path does not match any file in the store, 0 is
returned. If idx is out of range for the number of blocks in the file,
0 is returned.

type IndexEntry struct {
// The min and max time of all points stored in the block.
MinTime, MaxTime int64
// The absolute position in the file where this block is located.
Offset int64
// The size in bytes of the block in the file.
Size uint32
}

type TSMFile interface {
// Path returns the underlying file path for the TSMFile. If the file// has not be written or loaded from disk, the zero value is returned.
Path() string
// Read returns all the values in the block where time t resides.
Read(key string, t int64) ([]Value, error)
// ReadAt returns all the values in the block identified by entry.
ReadAt(entry *IndexEntry, values []Value) ([]Value, error)
ReadFloatBlockAt(entry *IndexEntry, values *[]FloatValue) ([]FloatValue, error)
ReadIntegerBlockAt(entry *IndexEntry, values *[]IntegerValue) ([]IntegerValue, error)
ReadStringBlockAt(entry *IndexEntry, values *[]StringValue) ([]StringValue, error)
ReadBooleanBlockAt(entry *IndexEntry, values *[]BooleanValue) ([]BooleanValue, error)
// Entries returns the index entries for all blocks for the given key.
Entries(key string) []IndexEntry
ReadEntries(key string, entries *[]IndexEntry)
// Returns true if the TSMFile may contain a value with the specified// key and time.
ContainsValue(key string, t int64) bool
// Contains returns true if the file contains any values for the given// key.
Contains(key string) bool
// TimeRange returns the min and max time across all keys in the file.
TimeRange() (int64, int64)
// TombstoneRange returns ranges of time that are deleted for the given key.
TombstoneRange(key string) []TimeRange
// KeyRange returns the min and max keys in the file.
KeyRange() (string, string)
// KeyCount returns the number of distinct keys in the file.
KeyCount() int
// KeyAt returns the key located at index position idx.
KeyAt(idx int) ([]byte, byte)
// Type returns the block type of the values stored for the key. Returns one of// BlockFloat64, BlockInt64, BlockBoolean, BlockString. If key does not exist,// an error is returned.
Type(key string) (byte, error)
// Delete removes the keys from the set of keys available in this file.
Delete(keys []string) error
// DeleteRange removes the values for keys between timestamps min and max.
DeleteRange(keys []string, min, max int64) error
// HasTombstones returns true if file contains values that have been deleted.
HasTombstones() bool
// TombstoneFiles returns the tombstone filestats if there are any tombstones// written for this file.
TombstoneFiles() []FileStat
// Close closes the underlying file resources.
Close() error
// Size returns the size of the file on disk in bytes.
Size() uint32
// Rename renames the existing TSM file to a new name and replaces the mmap backing slice using the new// file name. Index and Reader state are not re-initialized.
Rename(path string) error
// Remove deletes the file from the filesystem.
Remove() error
// InUse returns true if the file is currently in use by queries.
InUse() bool
// Ref records that this file is actively in use.
Ref()
// Unref records that this file is no longer in use.
Unref()
// Stats returns summary information about the TSM file.
Stats() FileStat
// BlockIterator returns an iterator pointing to the first block in the file and// allows sequential iteration to each and every block.
BlockIterator() *BlockIterator
}

TSMIndex represent the index section of a TSM file. The index records all
blocks, their locations, sizes, min and max times.

type TSMIndex interface {
// Delete removes the given keys from the index.
Delete(keys []string)
// DeleteRange removes the given keys with data between minTime and maxTime from the index.
DeleteRange(keys []string, minTime, maxTime int64)
// Contains return true if the given key exists in the index.
Contains(key string) bool
// ContainsValue returns true if key and time might exist in this file. This function could// return true even though the actual point does not exists. For example, the key may// exist in this file, but not have a point exactly at time t.
ContainsValue(key string, timestamp int64) bool
// Entries returns all index entries for a key.
Entries(key string) []IndexEntry
// ReadEntries reads the index entries for key into entries.
ReadEntries(key string, entries *[]IndexEntry)
// Entry returns the index entry for the specified key and timestamp. If no entry// matches the key and timestamp, nil is returned.
Entry(key string, timestamp int64) *IndexEntry
// Key returns the key in the index at the given position.
Key(index int) (string, byte, []IndexEntry)
// KeyAt returns the key in the index at the given position.
KeyAt(index int) ([]byte, byte)
// KeyCount returns the count of unique keys in the index.
KeyCount() int
// Size returns the size of the current index in bytes.
Size() uint32
// TimeRange returns the min and max time across all keys in the file.
TimeRange() (int64, int64)
// TombstoneRange returns ranges of time that are deleted for the given key.
TombstoneRange(key string) []TimeRange
// KeyRange returns the min and max keys in the file.
KeyRange() (string, string)
// Type returns the block type of the values stored for the key. Returns one of// BlockFloat64, BlockInt64, BlockBool, BlockString. If key does not exist,// an error is returned.
Type(key string) (byte, error)
// UnmarshalBinary populates an index from an encoded byte slice// representation of an index.
UnmarshalBinary(b []byte) error
}

ContainsValue returns true if key and time might exists in this file. This function could
return true even though the actual point does not exist. For example, the key may
exist in this file, but not have a point exactly at time t.

type TSMWriter interface {
// Write writes a new block for key containing and values. Writes append// blocks in the order that the Write function is called. The caller is// responsible for ensuring keys and blocks are sorted appropriately.// Values are encoded as a full block. The caller is responsible for// ensuring a fixed number of values are encoded in each block as well as// ensuring the Values are sorted. The first and last timestamp values are// used as the minimum and maximum values for the index entry.
Write(key string, values Values) error
// WriteBlock writes a new block for key containing the bytes in block. WriteBlock appends// blocks in the order that the WriteBlock function is called. The caller is// responsible for ensuring keys and blocks are sorted appropriately, and that the// block and index information is correct for the block. The minTime and maxTime// timestamp values are used as the minimum and maximum values for the index entry.
WriteBlock(key string, minTime, maxTime int64, block []byte) error
// WriteIndex finishes the TSM write streams and writes the index.
WriteIndex() error
// Close closes any underlying file resources.
Close() error
// Size returns the current size in bytes of the file.
Size() uint32
}

type Tombstone struct {
// Key is the tombstoned series key.
Key string
// Min and Max are the min and max unix nanosecond time ranges of Key that are deleted. If// the full range is deleted, both values are -1.
Min, Max int64
}