Clemens Reijnen - DevOpsDevOps, ALM and Software Developmenthttp://clemensreijnen.nl/
http://www.rssboard.org/rss-specificationBlogEngine.NET 3.1.1.0en-UShttp://clemensreijnen.nl/opml.axdhttp://www.dotnetblogengine.net/syndication.axdClemens ReijnenClemens Reijnen0.0000000.000000The 4 focus areas of DevSecOps.<p>DevOps will make systems more secure. In opposite to what many think, adopting DevOps, with its fast release cadence, will result in hardened systems which are fully compliant with security guidelines and which can stand the modern hackers. <h1>Summary.</h1><p>Teams must follow, must inject, secure guidelines and practices in their way of working. This way of working needs to be highly automated, supported by machine learning and role playing. <p>Fast, flexible, innovative, cheap, compliant and secure are the common requirements the business has on systems. In the past these requirements where a tradeoff from each other. Fast, flexible and innovative never went hand in hand with compliant and secure. With the current set of development practices, tool and platform capabilities these tradeoffs are gone, teams which follow a Secure DevOps way of working (DevSecOps) will deliver and run secure systems. <p>For a secure system DevOps teams and the connected business need to focus on the areas of: <p><strong>Automation, Machine learning, Platform and Culture.</strong><strong></strong><p>Automation will help on fast reliable and repeatable provisioning and validation of systems. Machine learning is the new kid on the block and will help analyze and understand the behavior of the system. Platforms are appearing in many flavors with many capabilities, many already with security baked in. Culture eats automation, platform capabilities and machine learning for breakfast, without the proper mindset of the whole company all the countermeasures won’t help a thing.<p>As the DevOps Handbook mentions in chapter 22:<b> </b><p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image002_4.jpg"><img width="314" height="480" title="clip_image002" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image002" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image002_thumb_4.jpg" border="0" hspace="12"></a></p><blockquote><p>The bad guys are already delivering malicious code continuously. Security can respond faster by working within the DevOps patterns.</p><p>The DevOps paradigm shift may give security pros the opportunity to finally bake security into IT processes rather than add it on as an afterthought<h3></h3><h3></h3></blockquote><h1>Automation</h1><p>When the whole team and business is comfortable to release as often as they want, multiple times a day whenever a feature or patch is ready, will make a system more secure. <p>Automation is key to make a team capable of releasing as often as they want. Automate the provisioning of the system from start to finish with validation between every step. Automation will give the team the ability to quickly push a security patch when a breach is detected. <p>Too often companies are in the news about system hacks via unpatched breaches. For example the Equifax Breach where the hackers used a vulnerability, <a href="https://www.symantec.com/security_response/attacksignatures/detail.jsp?asid=29972">Apache Struts CVE-2017-5638</a>, that was found months before.<blockquote><p>Equifax said that it was aware of the vulnerability two months earlier and worked to patch the bug then. <a href="https://www.nytimes.com/2017/09/14/business/equifax-hack-what-we-know.html">https://www.nytimes.com/2017/09/14/business/equifax-hack-what-we-know.html</a></p></blockquote><p>Automated provisioning is one part of the game, validating the work is the other part. Every activity executed during the creation and provision process of a system requires validation. <p>Coding activities requires unit testing with a good coverage, executed every time during the build. Not only the handmade code needs to be validated also the used packages. Packages and framework components are almost all opensource. Knowing if these opensource components have vulnerabilities requires specific tools which scan the used package every time the automated build runs. Sonatype, <a href="https://www.whitesourcesoftware.com/">Whitesource</a>, Flexera and other tool vendors offers these capabilities.<p>The golden rule of automation that every used artifact should be under version control doesn’t count for secrets. Secrets never ever should be stored in a Git repository, even if they are development and test secrets, still they easily can be used to track the path to the production environment. An example is explained in this blogpost.<blockquote><p>The GitHub extension that ships with Visual Studio 2015 exposed my source code to a public repository which allowed Bitcoin miners to spend $6,500 on my AWS account. <a href="https://www.humankode.com/security/how-a-bug-in-visual-studio-2015-exposed-my-source-code-on-github-and-cost-me-6500-in-a-few-hours">https://www.humankode.com/security/how-a-bug-in-visual-studio-2015-exposed-my-source-code-on-github-and-cost-me-6500-in-a-few-hours</a></p></blockquote><p>Nowadays there are tools and build steps that can be executed every build run which validates if the code has secrets of any kind. An example is the free <a href="https://blogs.msdn.microsoft.com/visualstudio/2017/11/17/managing-secrets-securely-in-the-cloud/">CredScan</a> from Microsoft<p>Automated vulnerability scanning is an important part and a benefit DevOps brings to system development next to automated provisioning. <a href="https://www.owasp.org/index.php/OWASP_Zed_Attack_Proxy_Project">OWASP Zed Attack Proxy Project</a> and <a href="https://software.microfocus.com/en-us/products/application-security-testing/overview">Micro focus Fortify</a> are vulnerability scanners which must be integrated in the automated release pipeline of systems. <p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image004_8.png"><img width="640" height="332" title="clip_image004" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image004" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image004_thumb_8.png" border="0"></a><p>Automation is often the main DevSecOps focus area. The fast delivery next to the automated validation helps secure systems. Automation is also the area where many tool vendors are entering the DevSecOps market, helping mature this area for teams. <h1>Machine learning</h1><p>Knowing the default behavior of the system is a must to know when it is hacked and behaves different. Too often companies don’t know they are hacked and only face it when their data is set public. Even hackers are missing they are hacked.<blockquote><p>It's the summer of 2014. A hacker from the Dutch intelligence agency AIVD has penetrated the computer network of a university building next to the Red Square in Moscow, oblivious to the implications. One year later, from the AIVD headquarters in Zoetermeer, he and his colleagues witness Russian hackers launching an attack on the Democratic Party in the United States. <a href="https://www.volkskrant.nl/tech/dutch-agencies-provide-crucial-intel-about-russia-s-interference-in-us-elections~a4561913/">https://www.volkskrant.nl/tech/dutch-agencies-provide-crucial-intel-about-russia-s-interference-in-us-elections~a4561913/</a></p></blockquote><p>Capturing log data, monitoring the whole system, its components and business functionality is a starting point. There are great log tools available which can monitor from traditional systems till large services-based systems and everything in between. <p>The log capturing, and visualization tools are expanding their capabilities with Machine Learning, with a reason. The huge number of small chunks of log data are a rich set of information. With Machine learning applied on this set of data systems can get more secure.<p>For cloud providers it is already a common capability in their products. Microsoft which positions his Azure Cloud offering as the Intelligent Cloud, has machine learning in many offerings. For example, Azure Active Directory Identity protection uses it to identify compromised identities:<blockquote><p>Discovering compromised identities is no easy task. Identity Protection uses adaptive machine learning algorithms and heuristics to detect anomalies and risk events that may indicate that an identity has been compromised.<a href="https://blog.route443.eu/2016/03/10/azure-active-directory-identity-protection-2/">https://blog.route443.eu/2016/03/10/azure-active-directory-identity-protection-2/</a></p></blockquote><p>Azure Security Center uses machine learning to understand the behavior of the applications in your subscription to understand if they are candidates for hacks.<p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image006_6.jpg"><img width="610" height="480" title="clip_image006" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image006" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image006_thumb_6.jpg" border="0"></a><p><a href="https://azure.microsoft.com/en-us/blog/how-azure-security-center-uses-machine-learning-to-enable-adaptive-application-control/">https://azure.microsoft.com/en-us/blog/how-azure-security-center-uses-machine-learning-to-enable-adaptive-application-control/</a><p>Making systems for monitoring, configuring the environment for monitoring and creating the machine learning algorithms to secure the system are a main task of DevOps teams and shouldn’t be an afterthought. <p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image008_6.jpg"><img width="537" height="331" title="clip_image008" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image008" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image008_thumb_5.jpg" border="0" hspace="12"></a>Connecting machine learning techniques to automation for self-healing (self-defending) systems when an attack takes place is a near future scenario, we’re not there yet. It won’t be good when an algorithm mistake results in system failures, like with automatic stock trading systems where an algorithm mistake results in spontaneous price drop. <h1>Platform capabilities</h1><p>A platform comes in many forms. When talking Cloud, the platform exists of IaaS, containers, PaaS, Serverless and SaaS. Each platform has different features and capabilities which makes the platform valuable for business systems. Also, every platform flavor has its specific security characteristics and needs for protection.<p>From a security perspective IaaS surrounded with traditional networks for security can be of a good security level. There is a lot of experiences how to configure this kind of environments. Compared with cloud native systems like PaaS and Serverless where ‘no network’ is the default. <p>Cloud native systems require a different kind of security via authentication, encryption and other technologies. For example, when making the connection between an Azure WebApp and a SQL PaaS database a connecting string with username and password over https isn’t enough. This communication can better be secured via Azure Active Directory authentication or with certificates and an Azure KeyVault. It is more secure but also require more effort to accomplish, the platform requires it. <p>Platforms also offers capabilities easy to configure and which makes use of the experience of the platform provider to protect the platform. <p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image010_1.png"><img width="640" height="429" title="clip_image010" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image010" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image010_thumb_1.png" border="0"></a><p>AWS WAF rules configuration is a set of default capabilities to protected web applications against attacks to exploit a vulnerability, take control of a server or DDOS attacks.<p>Other platform types like container technology also have their own tradeoff with security. Containers are easy shareable over different stages, by sharing the hypervisor of the container host. The smaller the container size the easier to distribute, while memory overloads can cause data to be leaked to other instances. Windows Hyper-V container have their kernel level isolation which solves this security problem, the tradeoff is that the containers are significantly bigger.<p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image012_3.jpg"><img width="640" height="318" title="clip_image012" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image012" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image012_thumb_3.jpg" border="0"></a><p>Companies adopting platforms need to think about these tradeoffs when selecting the platform for their infrastructure, on-premise or in the Cloud. <h2>The compliant Platform.</h2><p>To make the selected Cloud platform compliant with regulations, this platform needs to be configured, and consume resource from the platform in a specific way. For example, Microsoft has built several reference architectures on Azure for specific compliances.<p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image014_5.jpg"><img width="640" height="460" title="clip_image014" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image014" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image014_thumb_5.jpg" border="0"></a><p><a href="https://docs.microsoft.com/en-us/azure/security/blueprints/financial-services-regulated-workloads">Azure Blueprint Automation: Financial Services Blueprint for Regulated Workloads</a>.<h2>Cloud CoE and the Service Catalog.</h2><p>Companies will need such a compliant platform on top of the cloud platform with additional capabilities and configuration to be compliant to the company security rules. The Cloud Center of Excellence (or any other name) will be set in place for business units to maximize the usages and speed on delivering business value by using the cloud while staying compliant.<blockquote><p>“A Cloud Center of Excellence (CCoE) is a cross-functional team of people responsible for developing and managing the cloud strategy, governance, and best practices that the rest of the organization can leverage to transform the business using the cloud. The CCoE leads the organization as a whole in cloud adoption, migration, and operations. It may also be called a Cloud Competency Center, Cloud Capability Center, or Cloud Knowledge Center. “ — <a href="https://cloudcheckr.com/document/cloud-management-report/">https://cloudcheckr.com/document/cloud-management-report/</a></p></blockquote><p>Cloud technology adoption combined with a focus on value delivery and feedback will bring the goal of a flexible, cheap, innovative and secure business closer. The combination of these two are a catalysator for change.<p>The Cloud CoE is a service and practice center with deep and width knowledge on cloud platforms, design, delivery and run of cloud systems. The breadth and combination of development and operational knowledge makes the Cloud CoE an accelerator for businesses innovation.<p>Automation, templates, practices, solutions and services supports organizations in the adoption of cloud. The Cloud CoE must make an Service Catalog available for business units with these artifacts. <p>An implementation of a Service Catalog, Platform and Business projects are in the image below. In yellow the Service Catalog team with artifacts ready for use by the business teams (purple) and the platform team (blue) which uses the same service catalog artifacts to build and release the compliant platform.<p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image016_3.png"><img width="640" height="380" title="clip_image016" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image016" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image016_thumb_3.png" border="0"></a><p>With the out of the box ready solutions and automation scripts in the Service Catalog, teams can start to focus on business functionality immediately. Setup and organizational practices will make sure systems are ready for operations with the feedback loop back to the business ready to use.<p>AWS has a Service Catalog service which tracks the lifecycle of the products in the service catalog and provides access to the catalog.<h2>Continuous Platform Compliancy.</h2><p>Monitoring the platform, the service catalog and the business projects during their lifecycle on compliancy on security frameworks such as NIST, CIS/SANS 20 or ISO 27001 is a daunting task. <p>Cloud platforms are taking care that they are compliant on many frameworks, making it possible to inherit some of the compliance to the business projects, service catalog and company platform. Still continuous monitoring is required.<p>Azure Security Center monitors the platform continuous on threats and vulnerabilities. Policies can be configured to ensure compliancy with security frameworks. <p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image018.png"><img width="640" height="388" title="clip_image018" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image018" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image018_thumb.png" border="0"></a><p>Multiple vendors are offering additional capabilities on top of these security policies with auditing and compliancy management. In the platform marketplaces these offerings can be found.<p>Azure Marketplace on Security and Compliance:<p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image020_3.jpg"><img width="640" height="390" title="clip_image020" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image020" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image020_thumb_3.jpg" border="0"></a><p>AWS partnering with Allgress <a href="https://aws.amazon.com/config/partners/allgress/">https://aws.amazon.com/config/partners/allgress/</a><p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image022.jpg"><img width="640" height="478" title="clip_image022" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image022" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image022_thumb.jpg" border="0"></a><p>Platforms and tools help to stay compliant with security frameworks, they are supportive and reduce the complexity to be compliant. <h1>Culture</h1><p>Just opt-in for a cloud subscription, automated the delivery, setup the compliant platform configuration and learn from the system behavior isn’t enough for a secure platform. Getting a real secure environment requires a combined mindset from all the organization, its people, its processes, the whole cultural mindset of the organization should be secure. Cloud technologies with a DevOps way of working offer powerful capabilities to make business systems secure. But, to get the most out of these efforts a clear and structured secure way of working is required.<b></b><p>The question “<i>What is the weakest point in my organization?</i>” results in interesting answers. From admin passwords on shares and production data for test runs on less secure environments till old habits. <p>Having the whole organization focusing on security is challenging, there are many roadblocks. Secure DevOps Practices like automation, feedback loops from operations will help teams to clear many of these barriers, but there are more practices to adopt.<blockquote><p>We believe that boards and senior business leaders should be asking the technology team a different question—namely, “Are we ready to respond to a cyberattack?” <a href="https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/playing-war-games-to-prepare-for-a-cyberattack">https://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/playing-war-games-to-prepare-for-a-cyberattack</a></p></blockquote><p>Cyber Security War Games is a practice companies are starting to adopt. One team tries to hack the system and the other team reacts to it.<p>These games have a twofold benefit, they train teams in fast responding to intrusions. The alerts, log information will be added, and Machine Learning algorithms will be tuned to make the team more responsive to hacks. The other benefit is obvious, finding holes internally before the bad guys do.<p>An exercise which makes the whole company secure aware, is to do an internal phishing exercise and publish the shocking results. <h1>Closing</h1><p>Security should be baked in the whole company. DevSecOps with practices like automation, machine learning, platform and culture should be adopted by the whole organization. Still be aware hackers move fast, they think different. Be as fast as the hackers, know<a name="_GoBack"></a> when you are hacked and responsive to these hacks will make you win the security war. Because:<blockquote><p>You just have to accept it. The hackers are going to get in. The question is, what are you going to do once they are in? <a href="http://www.trustedsoftwarealliance.com/2016/03/10/security-war-games-with-sam-guckenheimer-at-rugged-devops-rsac-2016/">http://www.trustedsoftwarealliance.com/2016/03/10/security-war-games-with-sam-guckenheimer-at-rugged-devops-rsac-2016/</a></p></blockquote>http://clemensreijnen.nl/post/2018/04/02/The-4-focus-areas-of-DevSecOps
info@clemensreijnen.nlhttp://clemensreijnen.nl/post/2018/04/02/The-4-focus-areas-of-DevSecOps#commenthttp://clemensreijnen.nl/post.aspx?id=5d862065-a8a2-4bc2-becc-f2f63a1a610fMon, 02 Apr 2018 11:07:28 +0100DevOpsClemensReijnenhttp://clemensreijnen.nl/pingback.axdhttp://clemensreijnen.nl/post.aspx?id=5d862065-a8a2-4bc2-becc-f2f63a1a610f50http://clemensreijnen.nl/trackback.axd?id=5d862065-a8a2-4bc2-becc-f2f63a1a610fhttp://clemensreijnen.nl/post/2018/04/02/The-4-focus-areas-of-DevSecOps#commenthttp://clemensreijnen.nl/syndication.axd?post=5d862065-a8a2-4bc2-becc-f2f63a1a610fUse Azure availability tests and load tests to analyze the decoupling needs of your system (handson).<p>Azure load testing can set load on your system from any location around the world. <br><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image002_9.png"><img width="640" height="148" title="clip_image002" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image002" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image002_thumb_9.png" border="0"></a><p>Azure availability tests can monitor and analyze the responsiveness of your system anywhere in the world.<br><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image004_6.jpg"><img width="640" height="163" title="clip_image004" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image004" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image004_thumb_6.jpg" border="0"></a><p>Interesting capabilities even more interesting when you combine them in an investigation of your system. When you put some load on one part of your system and monitor the availability of another part you can analyze if that load impacts the availability of the other part. <p>When the analytics are positive, there is an impact, it is a candidate for decoupling so the parts can scale independent. PaaS services, Containers or a CDN to the rescue.<p><br><h2>The scenario</h2><p>The sales of PartsUnlimited is expanded to Asia. Due to interest in Jumper Leads from this new location a lot of traffic is expected on the search part of the website. <p>The product owner from PartsUnlimited wants to know if the high usages of the search feature impacts the responsiveness in Europe.<p>The DevOps team wants to monitor the availability of the website from both regions and they want to investigate if the website stays responsive in Europe during the enormous load from Asia on the search screen. <p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image002[6].png"><img width="644" height="339" title="clip_image002[6]" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image002[6]" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image002[6]_thumb.png" border="0"></a></p><p>For the proof of this take the PartsUnlimited website.</p><ul><li>Go to <a href="https://github.com/Microsoft/PartsUnlimited">https://github.com/Microsoft/PartsUnlimited</a>, select clone or download option.<u></u></li><li>Open the solution in Visual Studio and Publish the PartsUnlimited website tot Azure Web Apps.<u></u></li></ul><p>You are now ready for the power of Cloud testing.<br></p><u></u><h2>Setup Azure Availability Tests</h2><p>Go to your PartsUnlimited Azure Web App and configure the availability tests to monitor availability from different regions.</p><p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image006_5.jpg"><img width="640" height="256" title="clip_image006" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image006" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image006_thumb_5.jpg" border="0"></a><p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image004[6].jpg"><img width="328" height="338" title="clip_image004[6]" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image004[6]" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image004[6]_thumb.jpg" border="0"></a><p>After several minutes, you will see the results in the availability graph.<p>Next to the simple availability test, it is also interesting to know the status of a specific scenario. For example, the login process or the checkout process. We want these scenarios always to be fast, high available so our returning and paying customers have a smooth experience. <p>Azure availability tests can validate these scenarios via a Visual Studio performance test a Multi-step web tests.<p><a href="https://docs.microsoft.com/en-us/azure/application-insights/app-insights-monitor-web-app-availability">https://docs.microsoft.com/en-us/azure/application-insights/app-insights-monitor-web-app-availability</a><ul><li>Record the scenario </li><li>set a validation rule </li><li>upload the VSTest file to Azure. </li></ul><p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image008_5.jpg"><img width="267" height="480" title="clip_image008" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image008" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image008_thumb_4.jpg" border="0"></a></p><h2>VSTS Cloud Load test.</h2><p>With VSTS Load tests you can create load from any region in the world, for our scenario we put load from an Asia region.<p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image010_7.jpg"><img width="640" height="401" title="clip_image010" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image010" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image010_thumb_7.jpg" border="0"></a><ul><li>Create performance tests with Visual Studio.</li><li>Add Load test to you Visual Studio solution.</li><li>Run the load test to play the Asia search load scenario...</li></ul><p>During the execution of the load test you can open the streaming analytics in the Azure portal, to follow the load and possible errors.<p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image012_2.jpg"><img width="640" height="382" title="clip_image012" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image012" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image012_thumb_2.jpg" border="0"></a><h2>Impact.</h2><p>When you look at the previous create multi step availability test reports you will see that errors popup and maybe you also got some emails when notifications are configured.<p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image014_4.jpg"><img width="640" height="339" title="clip_image014" style="margin: 0px; border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image014" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image014_thumb_4.jpg" border="0"></a><p>The load on our search page impacted the responsiveness of our login and checkout scenario. Something you can solve by separating these scenario’s in separate services, de-couple them, split the data sources, introduce a CDN, a separate search database or know Asia is in a different time zone . Everything to make the components independent and can scale independently.<h2>App Monitoring and Feedback Loops Microsoft DevOps Course.</h2><p>This scenario is in detail explained in the <a href="https://openedx.microsoft.com/courses/course-v1:Microsoft+DEVOPS200.7+2017_T3/about">Microsoft App Monitoring and Feedback Loops DevOps course</a> I contributed to. It is free and you can download the script to replay this scenario (with more detailed steps and explanation) from the course page. <p><a href="http://www.clemensreijnen.nl/image.axd?picture=clip_image016_1.jpg"><img width="640" height="383" title="clip_image016" style="border: 0px currentcolor; border-image: none; display: inline; background-image: none;" alt="clip_image016" src="http://www.clemensreijnen.nl/image.axd?picture=clip_image016_thumb_1.jpg" border="0"></a><p>Keep on learning, feedback and monitoring is the DevOps trend of 2017.http://clemensreijnen.nl/post/2017/09/04/Use-Azure-availability-tests-and-load-tests-to-analyze-the-decoupling-needs-of-your-system-(handson)
info@clemensreijnen.nlhttp://clemensreijnen.nl/post/2017/09/04/Use-Azure-availability-tests-and-load-tests-to-analyze-the-decoupling-needs-of-your-system-(handson)#commenthttp://clemensreijnen.nl/post.aspx?id=c45cf834-fa00-424c-a96e-c0e78a7ecb86Mon, 04 Sep 2017 21:19:32 +0100AzureDevOpsClemensReijnenhttp://clemensreijnen.nl/pingback.axdhttp://clemensreijnen.nl/post.aspx?id=c45cf834-fa00-424c-a96e-c0e78a7ecb869http://clemensreijnen.nl/trackback.axd?id=c45cf834-fa00-424c-a96e-c0e78a7ecb86http://clemensreijnen.nl/post/2017/09/04/Use-Azure-availability-tests-and-load-tests-to-analyze-the-decoupling-needs-of-your-system-(handson)#commenthttp://clemensreijnen.nl/syndication.axd?post=c45cf834-fa00-424c-a96e-c0e78a7ecb86Provision AWS Resources with a VSTS release pipeline.<p>Infrastructure as Code is one of the many practices teams have to fulfill the needs of modern systems. The provisioning of resources for systems in an automated versioned way supports also the need of consistent environments across different stages of system development, making it much more comfortable to develop, validate and test systems. <blockquote> <p><b>Keep development, staging, and production as similar as possible.<br></b>See also the <a href="https://12factor.net/dev-prod-parity">Dev/prod parity</a> practice of the 12 factor methodology.</p></blockquote> <p>The principles of a pipeline also fit on pipelines which provision infrastructures. The artifacts should be versioned, validated, automated and more, see <a href="http://clemensreijnen.nl/post/2017/01/16/CICD-for-a-cloud-native-services-based-system-on-Azure">pipelines principles</a>. <h3>Visual Studio Team Services and Amazon AWS</h3> <p>VSTS covers many needs for teams and pipelines out of the box. VSTS is optimized for releasing systems and provisioning environments on Azure, many out of the box capabilities which speed up teams. <p>Great are also the integration capabilities of VSTS, it supports many platforms and many languages, also Amazon AWS. AWS offers similar template and cloud platform provisioning capabilities to what we have on Azure using ARM. This is named CloudFormation which can be used to create 95% of the AWS cloud resources. For authoring such an CloudFormation template you can download the <a href="https://aws.amazon.com/visualstudio/">AWS toolkit for Visual Studio</a>. In this way we can create and update our CloudFormation templates using our familiar Microsoft tooling you might be already using for ARM. <p><a href="http://clemensreijnen.nl/image.axd?picture=1_2.png"><img title="1" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="1" src="http://clemensreijnen.nl/image.axd?picture=1_thumb_2.png" width="624" height="480"></a> <p>You can simply open up an example AWS Cloud Formation template and author to make it fit for your use case. <p><a href="http://clemensreijnen.nl/image.axd?picture=2_2.png"><img title="2" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="2" src="http://clemensreijnen.nl/image.axd?picture=2_thumb_1.png" width="640" height="445"></a> <p>To test your AWS CloudFormation templates you can also deploy to AWS platform from the Visual Studio client directly or by copy and pasting it in the AWS portal: <p><a href="http://clemensreijnen.nl/image.axd?picture=3_2.png"><img title="3" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="3" src="http://clemensreijnen.nl/image.axd?picture=3_thumb_2.png" width="640" height="376"></a> <p><a href="http://clemensreijnen.nl/image.axd?picture=4_1.png"><img title="4" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="4" src="http://clemensreijnen.nl/image.axd?picture=4_thumb_1.png" width="640" height="378"></a> <p><font color="#666666">We will cover the details on how to author a CloudFormation template in different post.</font> <p>So now it comes to the Build and Release pipeline for AWS. To get build and release working for Amazon hosted systems it requires a dedicated build agent. This custom build agent needs to contain the AWS CLI. <p><a href="http://clemensreijnen.nl/image.axd?picture=5_2.png"><img title="5" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; float: none; padding-top: 0px; padding-left: 0px; margin-left: auto; border-left: 0px; display: block; padding-right: 0px; margin-right: auto" border="0" alt="5" src="http://clemensreijnen.nl/image.axd?picture=5_thumb_2.png" width="377" height="299"></a> <p>The AWS VSTS Build Agent can be any VM (a Linux VM on AWS or Azure, a Windows VM on Azure or AWS it doesn’t matter). It needs to contain the AWS CLI and the VSTS Linux (or Windows) Build Agent software installed and configured. <p><a href="http://clemensreijnen.nl/image.axd?picture=6_2.png"><img title="6" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="6" src="http://clemensreijnen.nl/image.axd?picture=6_thumb_2.png" width="640" height="195"></a> <p>Together with the AWS CLI also the AWS Account profiles are stored. This is one of the main differences with releasing to Azure. When you want to release to azure via VSTS the connection can be made via the Services configuration screen. <p><a href="http://clemensreijnen.nl/image.axd?picture=7_2.png"><img title="7" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="7" src="http://clemensreijnen.nl/image.axd?picture=7_thumb_2.png" width="640" height="442"></a> <p>The AWS Account profile is needed to configure the connection to the corresponding AWS accounts. While we are deploying to multiple accounts we have multiple profiles. <p><a href="http://clemensreijnen.nl/image.axd?picture=8_2.png"><img title="8" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="8" src="http://clemensreijnen.nl/image.axd?picture=8_thumb_2.png" width="640" height="84"></a> <p>The release uses the Shell Script release step utility, this one runs a Bash command on the system. Our system is the build agent, with the connection profiles. <p>The Bash command is coming from the Git repository where also the AWS Cloud Formation template are stored. <p><a href="http://clemensreijnen.nl/image.axd?picture=9_2.png"><img title="9" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="9" src="http://clemensreijnen.nl/image.axd?picture=9_thumb_2.png" width="640" height="319"></a> <p>To make the release flexible enough to release to multiple AWS accounts release variables are used in which are used as argument during release in the Bash command. <p><a href="http://clemensreijnen.nl/image.axd?picture=10_1.png"><img title="10" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="10" src="http://clemensreijnen.nl/image.axd?picture=10_thumb_1.png" width="640" height="307"></a> <p>Finally, this release takes care of provisioning the infrastructure and network for the system. It provides feedback when provisioning fails. <p><a href="http://clemensreijnen.nl/image.axd?picture=11_2.png"><img title="11" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="11" src="http://clemensreijnen.nl/image.axd?picture=11_thumb_2.png" width="640" height="374"></a> <p>And succeeds <img class="wlEmoticon wlEmoticon-smile" style="border-top-style: none; border-left-style: none; border-bottom-style: none; border-right-style: none" alt="Smile" src="http://clemensreijnen.nl/image.axd?picture=wlEmoticon-smile_1.png">. <p><a href="http://clemensreijnen.nl/image.axd?picture=12_2.png"><img title="12" style="border-top: 0px; border-right: 0px; background-image: none; border-bottom: 0px; padding-top: 0px; padding-left: 0px; border-left: 0px; display: inline; padding-right: 0px" border="0" alt="12" src="http://clemensreijnen.nl/image.axd?picture=12_thumb_2.png" width="330" height="480"></a> <p>The network which is setup is our default network configuration with VPC and Zones. <p>The default network configuration contains trusts levels for applications to land in: <ul> <li>Internal: all trusted networks accessed from inside the local network.</li> <li>External: all networks hosting applications facing untrusted zones (e.g. Internet).</li> <li>Public: all networks which are not under control.</li></ul> <p>(see also the release steps) <p>Next step the release of applications in either VM’s, Containers (first the host) or PaaS (<a href="http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/vpc-bastion-host.html">Beanstalk</a>). The releases of these ‘application’ components are in separate release because you want to evolve these in depend from the network (the core resources). See also <a href="http://clemensreijnen.nl/post/2017/01/16/CICD-for-a-cloud-native-services-based-system-on-Azure">this post</a>, paragraph resource groups. <p>As long as ‘application’ AWS resources can be provisioned via a <a href="https://aws.amazon.com/cloudformation/aws-cloudformation-templates/">AWS CloudFormation Templates</a> it can be done with <a href="https://www.visualstudio.com/team-services/">Visual Studio Team Services</a>. <p>thanks to: <a title="http://www.identityandcloud.com/" href="http://www.identityandcloud.com/">http://www.identityandcloud.com/</a></p>http://clemensreijnen.nl/post/2017/02/08/Provision-AWS-Resources-with-a-VSTS-release-pipeline
info@clemensreijnen.nlhttp://clemensreijnen.nl/post/2017/02/08/Provision-AWS-Resources-with-a-VSTS-release-pipeline#commenthttp://clemensreijnen.nl/post.aspx?id=f9c15f5d-7c4f-4fd8-b917-6651f85e3c8fWed, 08 Feb 2017 21:43:17 +0100ALMCloudDevOpsAWSClemensReijnenhttp://clemensreijnen.nl/pingback.axdhttp://clemensreijnen.nl/post.aspx?id=f9c15f5d-7c4f-4fd8-b917-6651f85e3c8f92http://clemensreijnen.nl/trackback.axd?id=f9c15f5d-7c4f-4fd8-b917-6651f85e3c8fhttp://clemensreijnen.nl/post/2017/02/08/Provision-AWS-Resources-with-a-VSTS-release-pipeline#commenthttp://clemensreijnen.nl/syndication.axd?post=f9c15f5d-7c4f-4fd8-b917-6651f85e3c8f