Yet another dev blog

Menu

MegaFS, a FUSE filesystem wrapper for Mega. Part 2: Reading and writing files.

In the previous article, we implemented the beginning of a filesystem where we can list our Mega files and navigate through them. It is now time to read/download them, and create/upload new ones! In order to to that, we need to implement 5 FUSE callbacks:

mknod(path, mode, dev): called to create a new, empty file ;

open(path, flags): called to open a file ;

read(path, size, offset, fh): called to read size bytes from a file at a given offset ;

write(path, buf, offset, fh): called to write the contents of buf to a file at a given offset ;

release(path, flags, fh): called to release a file (the opposite of open, see the FUSE FAQ to know the relation between release and the close system call).

Reading files

The read implementation can be tricky because of the size and offset parameters. They can take any value, but since the contents of the file contents are encrypted, we can only deal with data starting and ending on an AES block boundary (16 bytes). We could handle that by downloading the blocks containing the requested range (the Mega API supports partial downloads via the HTTP Range header and/or a suffix appended to the download URL), decrypting them and returning only the part really requested.

But for now, we will follow a simpler approach: just download the entire file and store it in a temporary file when open is called. Then, we can just seek/read this temporary file whenever read is called.

We currently only support files opened in read-only mode (flag O_RDONLY), so we start by checking that (the access mode is stored in the two least significant bits of flags, which explains the “& 3” mask). We then create the temporary file, download the entire Mega file to it (the downloadfile method is pretty much the same as in my first article, with an additional parameter to specify the target path), and check that the download succeeded by checking the meta-MAC (a thing we wouldn’t be able to do with partial downloads because we don’t know the MACs of each chunk). We finally return a handle to this temporary file, which will be used by read and release:

Our approach makes the read implementation very simple: just seek to the desired offset and read the temporary file. We are now almost done, and only need to delete the temporary file when the file is released:

def release(self, path, flags, fh):
fh.close()os.unlink(fh.name)

That’s all! We can now move to the write implementation.

Writing files

The writing part is even more tricky than the reading part: we have not only the encryption problem (we can only handle data aligned on AES block boundaries to be able to encrypt it, so we would need to buffer the data), but we also have to know the final size of the file before writing it! It is indeed required by the API ‘u‘ method, used to request an upload URL. Moreover, partial uploads must start and end on a chunk boundary (see the Mega developer’s guide, section 5 – “File encryption and integrity checking”, but this is in fact pretty much the same problem as the encryption one, with largest boundaries).

We will address these problems while keeping our code as simple as possible by using quite the same technique as earlier: we will first write all the data to a temporary file, and then upload it when the file is released. That way, we will know its final size when requesting an upload URL from the API.

Let’s start with the changes to the open method. The only writing mode allowed is O_WRONLY | O_CREAT | O_EXCL (see the Mega developer’s guide, section 1.3 – “Storage model”), that is, write-only with creation of the file: we can’t write to an existing file, nor can we open a file for both reading and writing. But in fact, open is not in charge of creating the file: a call to open(path, flags | O_CREAT) will be turned into a call to mknod (to create the file) followed by a call to open(path, flags). Thus, we have to implement mknod:

When mknod(path) is called, we add a new, empty file to our filesystem at path, so that it can be found by the getattr and open methods (they are called just after mknod and need to see that the file has been created). It obviously has no Mega ID (‘h‘ field), and this will enable us to distinguish these “special” files from the others. We can now make changes to open to handle openings in write-only mode:

For the O_RDONLY case, we just add a condition to check that the file is a “real” file (with a Mega ID, and not an empty file created by mknod) before downloading it.

For the O_WRONLY case, we first check that the file is an empty file created by mknod: if it’s not the case, that means that we are trying to write to an existing file, and that is not allowed by the API. Then we create the temporary file we are going to write all the data into, and we return a handle to this file that will be used by write and release:

If the temporary file was opened for writing, we upload it and create the new node on Mega. The uploadfile method is the same as in my first article, with an additional parameter to specify the source path. It returns the information on the new node, but with its key and attributes still encrypted. Thus, we give it to a processfile method that is just the main loop of the getfiles method (from the first article) extracted to a method (you can find the complete code on GitHub):

In the next article, we will see how to rename/move files, change their attributes, and delete them. Meanwhile, you can find the complete source code on the MegaFS project page on GitHub: https://github.com/CyberjujuM/MegaFS!

While i tried to make a folder i created a non-existing folder, pretty weird… to get MegaFs working again with my account i had to modify megafs.py:
line:48
path = self.getpath(files, files[hash]['p']) + “/” + files[hash]['a']['n']