shaman

README.md

Shaman

Shaman is a file storage server. It accepts uploaded files via HTTP, and stores them based on their
SHA256-sum and their file length. It can recreate directory structures by symlinking those files.
Shaman is intended to complement Blender Asset
Tracer (BAT) and
Flamenco, but can be used as a standalone component.

Uploading: the file is being streamed over HTTP and in the process of being stored to disk. The {checksum} and {filesize} fields are as given by the user. While the file is being streamed to disk the SHA256 hash is calculated. After upload is complete the user-provided checksum and file size are compared to the SHA256 hash and actual size. If these differ, the file is rejected.

Stored: after uploading is complete, the file is stored in the stored directory. Here the {checksum} and {filesize} fields can be assumed to be correct.

Garbage Collection

To prevent infinite growth of the File Store, the Shaman will periodically
perform a garbage collection sweep. Garbage Collection can be configured by
setting the following settings in shaman.yaml:

garbageCollect.period: this is the sleep time between garbage collector sweeps. Default is 8h. Set to 0 to disable garbage collection.

garbageCollect.maxAge: files that are newer than this age are not considered for garbage collection. Default is 744h or 31 days.

garbageCollect.extraCheckoutPaths: list of directories to include when searching for symlinks. Shaman will never create a checkout here. Default is empty.

Every time a file is symlinked into a checkout directory, it is 'touched'
(that is, its modification time is set to 'now').

Files that are not referenced in any checkout, and that have a modification
time that is older than garbageCollectMaxAge will be deleted.

To perform a dry run of the garbage collector, use shaman -gc.

Key file generation

SHAman uses JWT with ES256 signatures. The public keys of the JWT-signing
authority need to be known, and stored in jwtkeys/*-public*.pem.
For more info, see jwtkeys/README.md

Source code structure

Makefile: Used for building Shaman, testing, etc.

main.go: The main entry point of the Shaman server. Handles CLI arguments, setting up logging, starting & stopping the server.

auth: JWT token handling, authentication wrappers for HTTP handlers.

checkout: Creates (and deletes) checkouts of files by creating directories and symlinking to the file storage.

config: Configuration file handling.

fileserver: Stores uploaded files in the file store, and serves files from it.

httpserver: The HTTP server itself (other packages just contain request handlers, and not the actual server).

libshaman: Combines the other modules into one Shaman server struct. This allows main.go to start the Shaman server, and makes it possible in the future to embed a Shaman server into another Go project.

_py_client: An example client in Python. Just hacked together as a proof of

concept and by no means of any official status.

Non-source directories

jwtkeys: Public keys + a private key for JWT sigining. For now Shaman can create its own dummy JWT keys, but in the future this will become optional or be removed altogether.

static: For serving static files for the web interface.

views: Contains HTML files for the web interface. This probably will be merged with static at some point.