Add S3 capabilities to Azure Blob Storage using Minio

First of all, why can't we use Azure API?

If you already use AWS S3 as an object storage and want to migrate your applications on Azure, you would like to reduce the risk of it. For that, you could move your object storage without changing a line of code in your application. This is the solution we were checking and use now for some of our services.

If you have problems uploading large files to Azure Blob Storage, like a MySQL Dump, the Azure API can limit you. For example, the PUT API support blob size lesser to 256 MB, you can upload multiple blobs and assemble them into a single blob, but it's more complicated than a simple multi-part upload like with S3.

What is an Object Storage

Object storage is a computer data storage architecture that manages data as objects, as opposed to other storage architectures like file systems which manage data as a file hierarchy and block storage which manages data as blocks within sectors and tracks.

Each object typically includes the data itself, a variable amount of metadata, and a globally unique identifier. Object storage can be implemented at multiple levels, including the device level (object storage device), the system level, and the interface level.

In each case, object storage seeks to enable capabilities not addressed by other storage architectures, like interfaces that can be directly programmable by the application, a namespace that can span multiple instances of physical hardware, and data management functions like data replication and data distribution at object-level granularity.

In my case, I mostly use object storage for storing attachments, backups and other unstructured data.

What is Minio

Minio is an open source object storage server with Amazon S3 compatible API.

You can use Minio for building your own distributed object storage, or you can use it for adding an S3 compatible API to a storage backend like Azure Blob Storage, Google Cloud Storage or a NAS.In this blog post, we will use an Azure Blob storage with Minio.

Azure limitation

Can't have ACL on an object, only on the container (bucket) or on the account

Can't have keys with many account storage access, nor different keys with different access for one account storage

Do not have plenty of tools as AWS S3 have

Configuring an Azure blob storage

We will add a new storage account storage. The name will be the access key.

Once the Account storage is created, we will need one of the Access keys:

I give 2 different versions, minio_with_traefik use some custom labels for traefik. With it, we can have some https easily and listen on a standard web port. In this case, Minio can be used for serving assets or other document types to your applications.

s3cmd tools with Minio

You can access your new S3 gateway with the AWS CLI or by using other compatible tools like s3cmd, a popular CLI tools.

If you expose Minio on port http/80 or https/443, you can skip the port.

Conclusion

We use one Minio gateway in production. It's an internal gateway for our backups, and we use s3cmd for uploading them without issues.

We try to have a second gateway for exposing an object storage using the S3 protocol for our rails application. We added Traefik in-front-of it for easily adding an address on standard HTTPs port and an SSL certificate without effort. But due to the lack of ACL by objects or by patterns, these forces us to rewrite some part of our application for splitting the document into two different storage accounts. During the rewrite, we will check if we can't use the Storage Account directly without using Minio.

If you find a typo, have a problem when trying what you find on this article, please contact me!