Today we proudly announce the official release of TKLDev, the new mothership of all TurnKey Linux appliance development. With TKLDev, building TurnKey Core from scratch is as easy as running make. If you can handle that you're ready to roll:

As you might expect, we're pretty stoked to finally (finally!) be able to make this announcement. It's the culmination of a couple of years worth of unglorious work behind the scenes upgrading the development plumbing that makes TurnKey possible.

I personally believe it's also a critically important milestone for TurnKey, a project I've poured my heart and soul into and would like to see grow up to embrace it's full potential.

So what's the big deal with this new build system?

Here's the problem: as much as we'd like to be the omnipresent, omnipotent, and omniscient developers that can do everything ourselves there is way more awesome open source software out there than we (or any other closed group of developers!) could ever hope to bottle up into a working library of ready-to-use solutions. Not without a great deal of help anyhow!

We started this project about five years ago. It took us a couple of years to start realizing we had a problem and another year or so to realize we had a really big problem. TurnKey was all about open source but it wasn't truly open source itself because the relationship of the core development team with the community was pretty much one sided. We could look out, but nobody could look in.

We tried plugging the hole with TKLPatch, at least as a temporary stopgap, but that didn't work and we think part of the reason was that we hadn't gone nearly far enough. It wasn't enough to make it possible for the community to customize appliances. That just made the community into limited second class citizens and while it may have been better than nothing nobody really wants to be a second class citizen in a purported open source project. Give me full access to the source code and build chain or give me death! (or... a casual hrmpf of disinterest).

We realized that meant TurnKey was either doomed to stagnate or we figure out a way to re-engineer the development process in a way that truly democratized it so that everyone could join in on the same level.

So as many of you know (we've been waving our hands around about this for a while) a couple of years ago we started working on this in earnest and we've been gradually re-engineering all our legacy build infrastructure with the ultimate goal of creating the mother of all appliances - a self-contained appliance that can be used to build any appliance (including itself) from source code.

We've actually had this in our sights for quite a while now. We made some bad technical decisions early on with regards to how we setup our legacy build infrastructure and realized a bit too late that we would have to redo everything if we wanted to get the open source community truly on board with TurnKey's development.

With everything on our plate (e.g., developing over a hundred appliances and the TurnKey Hub) it sometimes feels like we're running in place. So it's taken longer than we would have liked to make this happen.

TKLDev and appliance sources released

Long story short we've re-engineered our entire build infrastructure, bottled it up into a self-contained appliance and battle tested it by building the entire TurnKey 12.1 maintenance release. We also moved to GitHub to streamline collaboration with the community.

Although TKLDev is relatively easy to use, it'll probably save you some time if you read through the documentation first to get an overview of how things works, development best practices, etc.

TKLDev lets you build any TurnKey appliance from scratch, directly from source. You can find the source code for all appliances in the TurnKey library at github.com/turnkeylinux-apps. All our other repositories live at github.com/turnkeylinux. We put appliances in a separate GitHub account because with so many of them they need their own namespace.

The way forward - TKL 13.0 Wheezy, a community effort?

A while back we announced release candidates for TurnKey Core 13.0, but then back tracked a little with the 12.1 maintenance release, and then side tracked again with TKLDev and the move to GitHub.

Well, it was all for a very good cause but I think we're done playing hop-scotch. With all the pieces now in place, we're moving full force on the TKL 13.0 release. The first things we'll be doing is updating all the TurnKey specific components, Core and TKLDev for 13.0. With that, the infrastructure is set to start upgrading the appliances.

We're hoping with the release of TKLDev and full appliance source code that the upcoming release will be more of a community effort. Thanks to the new infrastructure there is no special sauce, no wizard behind the curtain driving TurnKey development. We're giving Linux geeks everywhere the (super?) power to build any appliance, including Core, from scratch. What you do with it is limited only by your imagination but we're hoping at least a small part of that imagination will go towards helping us improving TurnKey for everyone's benefit.

Hopefully, we've made this stuff easy enough to use so nearly anyone with the inclination can jump in and make cool stuff happen. All the hard, really technical bits are automated so you don't need a lot of Linux experience to work with TKLDev. The same amount of experience you'd need to install and configure a regular Debian installation will take you a long way. The difference being that you can mass reproduce the results and collaborate with others on the development in ways you couldn't do otherwise.

If we've done our job well, we won't be able to anticipate all the different ways the community is going to put TKLDev to use. Some of the basic uses should be pretty obvious though. Let's say you want to improve an existing appliance. Maybe upgrade an appliance to use the latest upstream versions of your favorite open source application. Or maybe you've come up with a cool new feature that will make TurnKey Core (and by extension all other appliances) more useful? Fork, update, test, send a pull request, and your commit becomes an official part of TurnKey. It's that easy.

Deployed and setup easy enough; took me about 30 minutes to convert a tklpatch to the new fab system. I took my three most popular applainces (subsonic, nzbapp and newznabapp) and migrated them without difficulty. My github repos are updated so others can try them out if they'd like:

Adrian, I like your vision. There are a few features missing for TKLDev to become a fully fledged autonomous, planet colonizing von Neumann probe, but eventually, why not! I promise to try and integrate it with first viable open source AI, then maybe it will improve itself recursively. That would be a big win.

No 32-bit DEV version? That means I have to go buy a 64-bit machine to build a 32-bit appliance, or is that even an option anymore? Remember, there are a lot of 32-bit machines sitting out there running TKL...

The 64 bit version is the default (from the links on the site) but 32 bit version is available from the TKL SourceForge repo. If you have a look in the files section under your preferred medium (e.g. ISO) you'll find it (e.g. turnkey-tkldev-12.1-squeeze-i386.iso)

Thanks for pointing out that it's not so obvious where the 32-bit image is. Other people might be wondering about that. I've added a couple of links to the blog post and tkldev app page so that they you don't have to read through to JedMeister's comment (helpful as always) to find it.

Thanks Tim, I'm looking forward to seeing what you come up with so keep us in the loop.

BTW, we've been thinking about multi-app appliances for about as long as we've been thinking about TKLDev and friends. Unfortunately we haven't had the time to tackle this one ourselves yet but we do have a few ideas. To sketch the outline, I envision a meta TurnKey appliance that supports running multiple TurnLey inside thin containers that share the same kernel. LXC would be a good candidate component for this. I gather it's improved a lot in recent Linux versions. Last I checked there were issues with the quality of isolation between containers but you don't necessarily need good isolation to get this to work, that would just give you a big security bonus.

Thanks Liraz. Had a quick look at LXC and it looks pretty interesting.

FYI - my motivation for multi-app appliances is to minimise the cost for my SMB clients. There is a minimum cost for a VM since any VM is always doing something even if doing nothing (obviously). Add on a network monitor and an external heartbeat poll and the cost of a VM doing "nothing" goes up. Where an SMB I.T. infrastructure includes multiple apps that play well together (repository, network monitor hub, document management system), the overall service cost is reduced considerably by grouping apps in fewer VMs. LXC seems like it would help to simplify the lifecycle management of a multi-app VM (assuming that LXC can be run within a VMWare, AWS, etc VM of course).

Unlike other virtualization techniques, LXC is implemented inside the Linux kernel as an operating system level feature. Like chroot, but with more features. All containers share the same kernel so for security's sake you have to limit certain permissions inside the container even from root (e.g., the ability to insert/remove kernel modules, see processes outside the container, etc.) but you get much higher efficiency in return.

Thanks Jeremy! Alon and I have tried pushing the envelope in terms of how far we could stretch out our resources, but we also ran many times head first into our limitations and I'm hoping with TKLDev in the hands of the community our limitations will no longer be TurnKey's limitations.

Fingers crossed, we'll see the open source development model finally unleashed in this neck of the woods. That would be exciting!

This is a tremendous milestone that so many of us must be excited to start working with. can't wait to let students loose on this. hopefully I can get moving on the queue of appliances I've been intending to develop and really get to take advantage of what tkldev has to offer.

TKLdev is an awesome initiative. Been aware of turnkey linux for years, and now with tkldev I am finally sold ..

Multi-app appliances is an important development progression from single-app appliances .. So, I am keen to play with LXC in tkldev to see what can be achieved, but ultimately it would be best to have LXC integrated natively within tkldev.

Don't know if you guys (TKL dev team) are aware of Docker (see http://www.slideshare.net/dotCloud/intro-docker-october-2013) .. I have been quite excited by that platform and all the new possibilities it presents. Perhaps, this can be integrated into TKL in future?

I made some experiments in the past with LXC and TKL, but never get to something useful. Now the idea of Docker for TKL sounds very good, considering that the open PAAS environment may be moving towards Docker also as a way to enable portability between platforms...

I have been reading through the documentation and It took about 11 pages to get to the "answer" of my first question. How do I run TKLDEV on a New Appliance Build?

I had been thinking that, like TKLPatch I could install TKLDEV into the system and do a build. That appears not to be the case.

As I understand it the new Appliance needs to be built, then Forked back up to Git, then pulled from Git into TKLDEV for the ISO to be built.

Question though, if part of the appliance I am building is hosted on LaunchPad am I "out of luck" or can I somehow specify the bzr sources in the new System Build?

I am wanting to do an OpenERP appliance on top of the PostgreSQL appliance. Perhaps there are some Git Sources for OpenERP (a confusing number but none that I can find is CLEARLY the source).

All guidance is appreciated.

(small feedback, docs should perhaps have an outline structure... and as an aside not sure why I would want to make an ISO of Core, other than going through the steps.. Perhaps illustrating early on How to convert Core into Lamp, and make an ISO of that would be more telling of the process involved) My thoughts anyway. Thanks for all the hard work here!

Thanks for weighing in. There's definitely room for improvement in the docs. We should look into that. The process for using TKLDev is as follows:

Deploy thel TKLDev app and follow the setup instructions. I propose building TurnKey core as a test case. If that works you know you're ready to start building your own apps.

Fork the TurnKey app sources for the app that's closest to your use case. If you want to develop OpenERP you should probably look into the LAPP or PostgreSQL apps.

I propose taking a look at the sources for a few apps to understand what is going on. You can install software from any source, including Launchpad. You just have to provide the installation instructions (e.g., like you would have to do in a TKLPatch) and test that they work.

It would be great if I could look at a source which is calling Git, Launchpad and Python sources (I expect Python is in them all).

If you knew one of the Git/Source/Appliances I could look at that would be great.

OpenERP, btw, should be forked from PostgreSQL Appliance as it has its own Web Server, so Apache is not needed (and is somewhat in the way). PostGresSQL uses the "lightspeed" (or similar named) server for the Database interface.

Ideally Ports should be contollable in Webmin so that perhaps OpenERP took 80/443 and PostGre took a off highway port.

The source code for all the TKLDev builds are on GitHub so there are plenty of examples (the whole TKL library of them!). Note that it is a separate repo to the TKL one (IIRC it is called TKL-apps or something similar).

There is no issue basing your appliance on the PostgreSQL appliance (rather than the LAPP appliance). Just use the PostgreSQL appliance as your base (similar to what you may have done with TKLPatch).

As for your source code on LaunchPad, is that code you have created yourself or the code for the OpenERP software? If it's code you created then you may be able to import that into GitHub. If it's the source for OpenERP itself then as Liraz said you should be able to install it, same as you would have done with TKLPatch.

I just downloaded the TKLDev appliance (VMDK), followed the instructions in the Setup guide at GitHub and I believe my first wordpress-build is building. The one on this website is really behind, so I'm curious to see if making a new ISO is really this simple :D