News and Development Blog

It’s been awhile since we communicated, sorry about that! In between the last information post and now, we had a position seeking post that had some insights. It’s December 1st, and I want to update you all on what has happened, what is happening, and some stuff on what will happen.

New Infrastructure

We got a new server!

We waited until black friday rolled around to try to get a great deal to fix a few major expansion issues:

We were running out of outgoing bandwidth on the 2 servers we already had, despite Cloudflare handling the majority of the bandwidth! At the time of writing this post, Cloudflare records us transmitting out 400TB of data in the last 30 days. Cloudflare handles 95.5% of the outgoing bandwidth. The rest is taken from our servers. Our servers also transfer files between each other for storage needs, and this is at a rate of about 1-2 terabytes per month of inter-server transfers, minimum.

We were running out of reliable storage. Our secondary storage server, before this new purchase, was not ideal in any form for storing files in. The storage drive configuration was not resilient to losing a drive and we had no real remote backup server. With the new server, we have a ton of space and can use the old storage server as a live backup for the foreseeable future.

We were at risk of losing everything if Cloudflare decided to kick us off their systems. Handling 400TB a month is no joke. The new server can handle 75% of that by itself. The rest of that can be made up buy our primary and backup server if in the case that we can no longer use Cloudflare’s excellent caching system.

The last point is of considerable concern. Last month, an image hosting site called Postimage.org went into a panic mode. [You can read about it here]. Postimage was using Cloudflare’s Pro plan for their CDN and to help alleviate bandwidth issues. Well it got to a point where Postimage.org was using way way more than Cloudflare liked.

Postimage.org posted this image of using 1.8 petabytes in the last 30 days on Cloudflare before Cloudflare shutdown their account.

Cloudflare doesn’t have an official limit of how much bandwidth an account or domain can use for any of their plans. But Cloudflare was extremely upset at Postimage.org for using 1.8 Petabytes of data. They demanded Postimage.org start paying $12,000 per month for using Cloudflare in this fashion, up from $20 per month. Obviously Postimage did not have this kind of money nor did they have the server infrastructure to handle this kind of issue.

So what does this have to do with us? Compared to Postimage.org, we use just 25% of what they use in bandwidth per month, which is still a crap ton! Like I said 400TB is not a joke, and that number is steadily climbing every week. We also only use a free Cloudflare plan because we can’t even afford to use an upgraded plan if we wanted to. So we were in a bad spot where at any time, Cloudflare could have audited our account and kicked us off their service and Mixtape.moe would have to shut down or move to a very limited server host that could support an unmetered but limited connection. This is mainly why we got the new server. We are prepared in the emergency case scenario that Cloudflare decides to send Mixtape’s account to the gulags.

Specifications of server

Intel Xeon E3-1230 V2 3.30 GHz (4 cores, 8 threads)

32GB RAM

4 x 8TB Hard Drives

300TB of outgoing Bandwidth

1 Gbps Uplink Port

The new server costs us $99 per month.

To explain a bit on how our setup will look like with this new server, I made a quick and pretty crappy diagram. There’s more happening than what is pictured, but this is the general idea.

New Plans and Directions

We’ve been planning and slowly doing work to progress Mixtape to have not only accounts, but premium paid accounts that offer additional capabilities for the user (and helps keeps up running and expanding!). We had a July 2016 user survey where people gave us valuable feedback on what they wanted to see happen and what they want to continue having on Mixtape.moe. We have to make decisions on how we want Mixtape to be in the future.

For the most part, I don’t anticipate the usage of Mixtape as it stands currently to change as we make these changes. My ultimate goal is to provide and improve the current offering and features of Mixtape while expanding beyond it. People are using Mixtape for a reason at this stage, and I don’t intend to abandon the users and use cases that jumpstarted Mixtape’s growth.

How we want to proceed on our plans for expanding Mixtape is slowly changing from our conception. Because these are not finalized plans and changes, I don’t want to make misconstrued statements, unfulfilled hopes, or give ideas that may cause concern. I would rather wait until the New Year or a bit afterwards to talk about these things. Below however I will talk about some stuff that is more immediate and decided.

New People

We reached out looking for a development partner to join. We’ve brought onboard a couple of people to help us develop Mixtape (and Sapphire as well) out in multiple areas. When things are finalized with their onboarding, we’ll make another blog post introducing them, their position, and specializations.

Mixtape

Mixtape.moe, as part of the first step of creating a new system, will be rewritten and based on Go (currently PHP and Pomf code base). The goal is to relaunch Mixtape.moe with at least the same functionality as it does now, fixed and improved API, and an account and login system that stores upload urls and the ability to delete your own uploads from within the account.

Mixtape is currently suffering some issues with the inhouse storage management system. Files on one of the storage containers show up as 404-Not Found. The files exist but are not being served. We’re trying to fix this as soon as we can.

Stats displayed are less than they actually are. We use a simple deduplication system and the stats use a simple file count, so the amount of files being uploaded is less than the reality.

GitGud

GitGud.io is just cruising along with no new issues or changes planned. GitGud is always updated to the latest GitLab version within 24 hours of a release.

GitGud still suffers from email confirmations going into Gmail spam and not being delivered at all on Mail.com addresses.

TweetSave

TweetSave.com is siting steady. The site will receive a frontend update in the future for visual improvements as well as giving it responsive layouts for mobile, tablet, and smaller width use cases.

TweetSave.com is gearing towards expanding to include extra functionality with premium paid accounts.

I’d like to give a big thanks to the people who donated to us, especially the people who help us out every month! We would be completely gone and shutdown if not for your support.

When it comes to finances, we are almost always just scratching by. Paid accounts will be critical for surviving 2017.

We have big plans, but we need to walk before we can run. I don’t wish to get ahead of myself on these plans and ideas before finalization and or the means to execute them. I hope to announce some of them early next year! 🙂

The file isn’t gone. We just haven’t had enough free time to work out the issue with how the system is mapping where files are located. We do this as part time and we’ve been slammed with personal and work responsibilities the past month.

Kiwimen

Ok, I understand. Is there a chance that if I give you the URL of the clip you can track the clip? If is too much work or is not possible it’s okay, is not that important, just wondering.