Learn with our tutorials and training

developerWorks provides tutorials, articles and other
technical resources to help you grow your development skills
on a wide variety of topics and products. Learn about a specific
product or take a course and get certified. So, what do you want to learn
about?

Featured products

Featured destinations

Find a community and connect

Learn from the experts and share with other developers in one of our
dev centers. Ask questions and get answers with dW answers. Search for local events
in your area. All in developerWorks communities.

This content is part # of 8 in the series: DevOps best practices, Part
2

This content is part of the series:DevOps best practices, Part
2

Stay tuned for additional content in this series.

DevOps is a set of principles and practices that focus on improving
communication and collaboration between development and operations teams.
These practices have become popular with technology experts who wish to
develop robust technology solutions for which reliability can be ensured
and that can be easily upgraded. The initial focus of DevOps is to help
developers and operations professionals to communicate and work more
effectively together, but other stakeholders such as the Quality
Assurance, testing, and information security (InfoSec) teams are equally
important. This article explains how to use DevOps best practices to make
your information security robust and effective.

Why information security is a key part of DevOps

DevOps has been extremely successful at enabling developers and operations
to work together more effectively. With DevOps, the operations team gets
the information that they need to understand how to establish effective
and reliable application build, package, and deployment procedures. The
information security group has many of the same requirements as the
operations group. In addition, InfoSec needs to get the information that
they need to ensure that the entire system is secure and reliable. Just as
DevOps helps development and operations to work effectively together,
DevOps helps development and information security to be more effective as
well. Continuous deployment has been a key practice within DevOps and
focuses on automating the deployment pipeline through automated build,
package, and deployment. Information security can benefit significantly
from the deployment pipeline by providing a platform to assess and address
security concerns early in the development lifecycle. Effective security
should always start with an assessment of the risks involved.

Security risks that DevOps can help address

Risks need to be understood and addressed as part of a software or systems
development effort. Security cannot be just added on to a system at the
end of a development process. (See Related links at the bottom of this
article for
more on building secure software.) Systems need to be designed and
developed with security in mind from the very beginning of the development
lifecycle. DevOps provides the necessary structure to help address many of
the security risks that are inherent in creating any complex technology
system.

Security vulnerabilities can be the consequence of a variety of
circumstances. For example, inadequate coding practices in C and C++
systems can result in buffer overflow conditions that give a malicious
attacker a command prompt with the privileges of the program. Buffer
overflow attacks have been used by attackers to gain control of a system
and even effectively to gain root privileges.

Runtime issues can occur due to inadequate security controls such as
authentication and authorization in the interfaces between components. One
common source of issues is incorrect permissions applied during a
deployment.

Another area of concern is ensuring that the correct code is deployed.
Mistakes during a deployment can be exploited by a malicious attacker.

Configuration issues between interfaces can be exploited by a hacker trying
to penetrate a system. Once a system is compromised, inadequate security
controls can make it harder to identify unauthorized changes and perform
forensics to ascertain what has been changed through error or malicious
intent.

InfoSec rarely has the technical expertise and enough adequate test
environments early in the development process to conduct effective
security- related testing. Many organizations also cannot provision their
servers from scratch and deploy the applications as needed, a limitation
that means that those responding to a security breach often have a tough
job trying to understand exactly what code has been compromised and how
the threat can be eliminated.

These risks are all too common. They cause many system outages that have
impacted major banking and trading systems and even the trading exchanges
themselves. If we cannot simply add in security at the end of a project
then how exactly do we build secure systems from the beginning?

Effective source code management to build secure
systems

Quality management guru, W. Edwards Deming, wisely noted that quality has
to be built in from the beginning. DevOps depends upon effective source code management
to create the automated build, package, and deployment scripts that
comprise the deployment pipeline.

Effective source code management includes:

Version control system

The ability to reliably baseline the code

Techniques to model the software architecture using streams and
components

Support for multiple variants in the code.

Having all of the source code in a version control system enables InfoSec
to scan the code for vulnerabilities. Robust version control systems also
enable the security analysts to understand how the system is constructed,
by modeling the component structure as well as the interfaces between
components, which are often the target of a malicious attack.

DevOps and the deployment pipeline help to efficiently create and provision
test environments that can be used to evaluate and test for
vulnerabilities in the interfaces between components. Providing robust
test environments in a timely manner enhances security by providing a
platform for automated testing that can identify security issues that need
to be addressed. Once issues are found in the code, they can be addressed
as defects or requests for change and tracked throughout their lifecycle
to ensure that identified risks have been addressed.

Robust source code management solutions can help manage the application
lifecycle by providing a set of tools and processes that help drive the
entire development process from choosing the right requirements to
managing defects found while the system is in production. Tracking
requirements and defects are an essential aspect of task-based
development.

DevOps best practices for robust and effective information
security

The following sections describe recommendations from DevOps principles to
help build in information security:

The information security team rarely has enough technical expertise
required to understand common security vulnerabilities that are inherent
in complex systems. Just as DevOps improves communication between
development and operations, DevOps also empowers the information security
team by allowing them to fully understand the entire application and how
it is built, packaged and deployed. This knowledge can help InfoSec
maintain a focus that is relevant and effective. By understanding the
systems infrastructure, InfoSec can also understand when security has been
breached and precisely what needs to be done to address the security
weaknesses in the event that the infrastructure itself has been
compromised. In many cases, it is essential to rebuild the servers using
automated procedures.

Develop software using a task-based approach

Task-based development involves creating work items such as tasks, defects,
and requirements and then tracking these work items to the change sets
created to develop the code. By tracking requirements to tasks and
defects, security issues that could have been introduced as a result of
failing to implement an essential requirement can be avoided.

Tracking the requirements to test cases is also an important aspect of
task-based development, because it ensures that each requirement is
verified and validated. This traceability is essential in preventing
defects that could result in security vulnerabilities. Defining and
tracking work items helps to manage the complexity of the development
effort. Code can also be instrumented to help facilitate security
testing.

Automate the application build, package, and
deployment

Creating automated procedures to build, package, and deploy the code
supports agile iterative development in many ways. Automated build
procedures are a prerequisite for continuous integration and continuous
delivery. The build procedures should automatically embed immutable
version ID into each configuration item (CI) that is used or created by
the build procedure. This is also where cryptographic hashes should be
created for use in verification during the application deployment
described below. Sprint milestone releases can be used for testing and
validating requirements that may not be completely understood at project
inception. Most importantly, the technical details of how to build each
component are implicitly documented and can be reviewed by interested
stakeholders including information security. Creating variants in the code
facilitates application testing including instrumenting the code to
facilitate security related testing.

Instrument the code with libraries

Instrumenting the code involves running the code with libraries that make
it easier to scan the runtime code for vulnerabilities. DevOps plays a key
role by enabling the code to be built, packaged, and deployed to a test
machine for the purpose of identifying security vulnerabilities. In
keeping with the DevOps focus to address issues as early as possible,
analysis of the security of an application needs to be conducted
throughout the lifecycle of the application. Trying to fix a security
issue late in the lifecycle is much more difficult than addressing it up
front and early in the development effort. By being involved in the full
application lifecycle, the security analyst can more effectively identify
any issues that exist in the system.

Identify the weakest link

As the old adage goes, "a chain is only as strong as its weakest link". By
taking a comprehensive view of the system, the security analyst can
identify issues that are specific to a particular component or interface
between components. Interfaces to legacy systems are often necessary. They
are also the weakest link in the chain. After vulnerability has been
identified as a risk, the application can be tested to assess the
potential security issue. Steps can then be taken to mitigate the risk and
address the security vulnerability. These techniques are particularly
useful in facilitating early penetration testing.

Conduct penetration testing

The deployment pipeline provides an effective framework for creating the
required test environment to conduct penetration testing. With DevOps,
penetration testing can occur throughout the entire software and systems
lifecycle. By improving the communication between development and
information security, the DevOps InfoSec team can design and execute more
effective penetration testing within the short time frames that are
typically allocated for such purposes. The deployment pipeline also
enables information security to gain the technical information that is
essential for effectively assessing security vulnerabilities.

Programmatically build the infrastructure

Security experts recognize that almost any system can be compromised and
once the environment has been penetrated, malicious code can creep in. It
is common for hackers to compromise inadequately secured systems and then
use them to attack other machines that are more secure. These types of
attacks may simply involve a denial of service attack, which can then be
traced back to the machines that were used in the attack. In denial of
service attacks, unsuspecting victims may find themselves accused of
mounting an attack on another system and required to explain why their
machines were used for malicious intent.

When a system is compromised, the best course of action is to completely
clean and re-provision the servers. DevOps provides some excellent
procedures to build the servers from the operating system to the
application base through a fully automated procedure. The ability to
programmatically build the infrastructure is known as infrastructure
as code. This key practice within DevOps is commonly used to
support cloud-based infrastructures. These techniques have also been
called agile systems administration, a term that refers to the value that
DevOps provides in an agile development environment.

Use automated systems to configure and verify the operating
system

Infrastructure as code can be used to implement secure systems by providing
automated procedures to configure and verify that the operating system
conforms to established industry security standards such as the
well-respected Center for Internet Security (CIS) Security benchmark,
which defines precisely how the operating system should be configured.
(See See Related links at the bottom of this
article for a link to the Center for
Internet Security.)

Both the provisioning of the operating system and the verification of the
security configuration can be fully scripted using the guidance provided
in the CIS specification. Scripts enable servers to be fully provisioned
and supported through an automated procedure. To ensure a reliable and
secure platform, start with servers that are completely secure and that
are configured correctly. After your server infrastructure has been
provisioned and is verifiable, it is time to use the same principles to
deploy the application base in a way that is secure and verifiable. The
automated build procedures should ensure that the code can be verified
once it is deployed by embedding version IDs into each configuration item
including code components, configuration, and properties files.

Cryptographic hashes should be created for use in verifying the application
deployment.

Deploying the application itself is just as important to provide a
completely secure trusted application base that is not only verifiable,
but also enables the detection of security vulnerabilities, including
unauthorized changes. Building code that contains embedded version IDs for
deployment verification and using cryptography to identify unauthorized
changes ensures that you can create a secure trusted application base.

Provide a secure, trusted application base

To secure the trusted application base, build the application code using
procedures that securely identify the exact binary code and all other
configuration items (including XML and properties configuration files) so
that you can definitely prove that the correct code has been deployed and
confirm that no unauthorized changes have been made.

These procedures usually make use of cryptographic hashes that are first
created during the automated build and packaging of the application. Once
the code has been deployed, the hashes can be recalculated to verify that
all of the code has been correctly deployed.

These same procedures can be used to establish reliable baselines and to
identify any unauthorized changes that could potentially occur as a result
of human error or even malicious intent.

Working from a secure, trusted, application base ensures that the systems
have been reliably and accurately deployed and makes it easier to identify
and address any possible security breaches. When authorized changes are
implemented, the baselines must be updated to record and track the
changes. These techniques are described in industry standards and
frameworks, including the well-respected ITIL v3 framework, which provides
guidance on how to build and maintain reliable services.

Rely on standards and frameworks

Standards and frameworks provide expert guidance on how to implement secure
and reliable systems. There are many functions and processes described in
these guidelines including the ability to monitor and track code
baselines. The ITIL v3 framework describes Service Asset &
Configuration Management (SACM), which captures baselines that can be
stored in the configuration management system (CMS). These baselines can
be monitored by the configuration management database (CMDB) with
discrepancies reported to the CMS. The definitive media library (DML)
contains the baseline, official releases that can be deployed as part of
provisioning a server.

These techniques, including the practices of monitoring baselines using
intrusion detection systems (IDS), are well-regarded as industry best
practices. The DevOps approach provides the automation and framework for
implementing these procedures. Even with the best preventive practices,
intrusions can happen. Being able to detect and respond to incidents is
essential.

Enable early detection of unauthorized changes

When unauthorized changes occur either because of human error or due to a
malicious attack, they should be immediately detected and they should
trigger an incident response. The DevOps approach provides the procedures
so that unauthorized changes can be easily detected and proactively
addressed with minimal impact to the users. They also provide the basis
for provisioning servers and deploying systems immediately to restore the
system to a reliable baseline. These techniques require collaboration and
excellent communication among the key stakeholders including development,
operations, Quality Assurance, testing and information security teams.

Conclusion

DevOps is well known for its focus on helping developers and operations to
work together more effectively together. Information security, along with
Quality Assurance and testing teams, are also key stakeholders and need to
be represented within the DevOps framework. DevOps helps to improve
communication and helps to transfer essential technical knowledge from the
developers who are writing the code, to the operations and security
professionals who are responsible for ensuring that there is uninterrupted
service. With the DevOps structure in place, InfoSec can make informed
decisions that will help protect the system infrastructure and
applications from any unexpected occurrences that might compromise the
system, whether due to unintentional error or malicious attack. Keeping
systems secure helps to prevent system glitches and also ensures
uninterrupted service. The DevOps approach and the deployment pipeline
enable your team to roll out features that both meet your business needs
and provide the secure trusted application base.