Every technology market has a set of questions that is good to have on hand when you look at the products that serve the market. This is true of every space I’ve worked in, and true with every vendor I’ve worked for. The key is that there are not golden answers to most of these questions. The importance of a given question deeply depends upon your environment. With that said, I’ll throw a bunch of starting questions out here, with the goal of helping some projects get moving along a little faster. Remember to weight these based on importance in your organization, and be aware that these are in no particular order – because ordering by importance should be based on your priorities.

What type of installations does the tool support? Most server provisioning tools fall into one of two categories – golden image, and package based.

Golden Image: An ISO taken from a working machine is known to be good for a given Hardware/OS/Patch Level/Apps combination, and is used for all machines that meet those specs. Other golden images exist for differing hardware, app distributions, and OS versions. After install, things like IP are updated to reflect that this is not the machine the image was originally taken from.

Package Based: The collection of install files needed to install an OS version (Ubuntu 14.04 on X86, 64 bit, for example) are kept in a repository on the provisioning server, and all servers that use that OS version are installed from it. The next step depends upon tools chosen – either apps are immediately installed and configured, or an application provisioning tool is called to install apps.

What install communications mechanisms are supported? While the vast majority of these tools use local subnet PXE to perform installations, knowing which ones allow alternative methods for remote or cross subnet installation is a necessary step for organizations with distributed managed servers and centralized operations. Also, if PXE is restricted in your organization for security reasons, knowing what your other options are is important.

What are the installation steps? Believe it or not, most of these tools designed to make installation easier are… Difficult to install and configure. While long-term this is not an issue, knowing before hand the level of complexity you are embarking upon is deeply relevant to an evaluation project that may have limited man-hours.

What level of virtualization/cloud/container support does the application have for spinning up servers? Most of these tools will support virtualization as long as it presents on the network like a physical server (eg: Standard network interfaces), but cloud and container support gets more varied from product to product. Some are deeply tied into cloud and container interfaces and will spin up a machine in those locations the same as spinning up a physical or virtual machine. Others are really focused on traditional datacenter servers – physical and virtual – and provide limited or no support for cloud or container instances. Along these same lines, because “server provisioning” isn’t really a thing with cloud – the cloud image does most of that work – many application provisioning tools do cloud and container spin-up. If yours does, this question is not as important, but read on.

What level of support does the tool have for virtualization/cloud/container infrastructure. Think of OpenStack on OpenStack (OoO) – a provisioning tool whose sole purpose is to spin up OpenStack instances. Most provisioning tools are not as single-purpose as OoO, but do provide a level of support for this type of advanced configuration of complex systems. The amount of support varies wildly, and it is best to check with the vendors in question to understand this support if you need it.

Server intake. Another area where usability is impacted is getting servers into the system. What options are available, and what are the requirements for including/excluding specific servers? Because PXE is physical subnet based, the answers to these questions might surprise you. Relevant questions are:

Import: What options are there to get a server into the system?

Exclusion: How can I specifically exclude a server from the system?

Movement: How can I move servers between management systems (From OpenStack into Hadoop for example)

What operating systems does it support? These tools range from “one or two variants of Linux” to “Linux, BSD, AIX, and Windows”. Of course, the more operating systems are added, the more likely that the quality/level of support is not equal amongst them (a vendor that develops and primarily tests on BSD, for example, might not have AIX support that is as thorough). Don’t hesitate to ask what the primary development platform is, just like you would do with a closed-source product.

All of the items in the “What to look for in an open source project” section of my “Free as in Beer” blog. As I point out in that article, it may not be as bad moving from failed open source project to thriving open source project as it is to move from failing commercial product to thriving commercial product, but in the end, there will still be a ton of man-hours invested converting data, policies, etc. and learning the new system. Make sure you’re adopting a project with a future.

Security, security, security. Review the product with the security team. These products are going to install your servers. Obviously this is not “some app used only by one team in Marketing”, it has a huge impact on your security infrastructure. Make sure it fits. Areas where some organizations have had issues with this particular market segment include:

Passwords: Most of these products default to a single root password for all installed servers. If that’s not in line with security policy, check to see if this default can easily be changed.

Install Manifest: These products install a ton of software all at once. This can make security teams (rightfully) nervous. Ask to see if a list of what is in a “standard” install can be generated for security to review.

PXE: As mentioned above, some organizations don’t want it on their network. If you work at one of them, ask what alternative options there are.

RBAC: Does the system support RBAC? At what level? If it has an API, does the API support RBAC? It is true that in some orgs the desire to let “Group X” only install/reinstall servers in “Server Rack A” or to only reinstall a server during certain hours is an actual need. If you’re one of those organizations, find out what is available to you. The market isn’t very mature in this space.

Policies: Security policy support. Can it roll out your security policies to servers? If you use IPTables, for example, can the tool auto-configure it during install and first boot? If you disallow IPTables, can the tool disable the service during install? Know what support you have.

Application Provisioning Support. My series on DevOps.com covers this topic at the 20,000 foot level, but really dig in if you are trying to create seamless automation across the datacenter. At a minimum, ask if the tool can install agents for your flavor of application provisioning. After that, the application provisioning tool can take over and tweak the installation as part of spinning up the apps.

What’s “in” and what’s “out”? All of these tools are open source with a support option offered by the company sponsoring the open source project. Is there a difference between free and supported versions? If so, what are the details?

Availability of user-created add-ons. Are users actively creating add-ons for the system that help make your job easier? Does the project team or sponsoring org showcase these add-ons? More development, and development outside the chosen direction of the sponsor, is generally a good thing, just make certain you evaluate these add-ons carefully, as they don’t necessarily go through rigorous testing.

Specialized support. If you use some variant hardware or OS, make certain the tools you choose to evaluate give you some path to implementation. A specialized NIC can wreak havoc with these tools if it is the primary connectivity of the box, for example. RAID cards are another area where support varies. Make sure you know if you can provide support for any unique bits in your environment – and how hard providing that support may be.

There will no doubt be other questions you want to ask, but this list should get you started when you’re gearing up evaluations. As always, your environment will dictate what questions I didn’t even think of, but you need to ask.

Overall, this market is coming to maturity, and the products are improving at a rapid pace. It’s a good time to be evaluating them, just make certain you’re asking the right questions.

Hopefully, by the end of your project, you’ll be on the beach sipping some cool beverage, and simply text “Reinstall server #42” when errors occur. But even if we’re not there yet, hopefully your life is made easier.

Don MacVittie is founder of Ingrained Technology, A technical advocacy and software development consultancy. He has experience in application development, architecture, infrastructure, technical writing,DevOps, and IT management. MacVittie holds a B.S. in Computer Science from Northern Michigan University, and an M.S. in Computer Science from Nova Southeastern University.