Topics

Featured in Development

Understandability is the concept that a system should be presented so that an engineer can easily comprehend it. The more understandable a system is, the easier it will be for engineers to change it in a predictable and safe manner. A system is understandable if it meets the following criteria: complete, concise, clear, and organized.

Featured in Architecture & Design

Sonali Sharma and Shriya Arora describe how Netflix solved a complex join of two high-volume event streams using Flink. They also talk about managing out of order events and processing late arriving data, exploring keyed state for maintaining large state, fault tolerance of a stateful application, strategies for failure recovery, data validation batch vs streaming, and more.

Featured in Culture & Methods

Tim Cochran presents research gathered from ThoughtWorks' varied clients and projects, and shows some of the metrics their teams have identified as guides to creating the platform and the culture for high performing teams.

AWS re:Invent Recap

At their annual re:Invent conference in Las Vegas, AWS unleashed a flurry of announcements about upcoming cloud services. Amazon outlined over two dozen new capabilities coming to the public cloud, including directly querying data in S3 object storage, building code as part of deployment pipelines, provisioning cheap virtual private servers, and moving data in bulk, ETL-style.

In a subset of product announcements, AWS addressed the lifecycle of data: migration, storage, integration, query, and analysis. AWS Snowball Edge is an appliance that builds on the previous Snowball that focused on physically migrating bulk data to cloud data centers. Each Snowball Edge appliance has 100TB of storage and offers more connectivity protocols than the previous version. It also offers integration with AWS Lambda for data processing. Also part of the winter-themed lineup is AWS Snowmobile, a secure truck that hauls 100PB of data to AWS data centers over the span of weeks.

Amazon Aurora originally launched as a specialized version of MySQL focused on performance and availability. Based on intense customer demand, AWS addedAurora compatibility for PostgreSQL. That service is now in preview. Developers are familiar with using Amazon S3 object storage to stage data as it moves between processing steps. With Amazon Athena, developers use standard SQL syntax to query data at rest in S3. This service works with CSV, JSON, log files, delimited files, and more, and is available today in the United States. Amazon also gave a sneak peek at AWS Glue, an ETL service that connects to Amazon data stores and JDBC-compliant databases. This service is not yet available, but promises to include data transformation, job scheduling, execution history, and more. The final data-related service announced by AWS was Amazon Pinpoint, a tool for mobile developers that want to run targeted campaigns based on user engagement.

Amazon also announced a handful of managed services targeted at application developers. Three of them were Artificial Intelligence (AI) focused, and specifically highlighted in a blog post by Amazon CTO, Werner Vogels. Amazon Polly is a cloud service that does text-to-speech conversion in 24 languages. Developers can use either the AWS Console or API to “synthesize speech” into an audio format. Trustworthy image detection requires deep learning against a comprehensive model, and Amazon says that the new Amazon Rekognition managed service “already analyzes billions of images daily.” This service is available in the United States and Europe, and returns image labels as a result of analyzing the scene, objects, and faces. The third AI-related service announced at re:Invent was Amazon Lex. Amazon Lex takes the speech recognition engine Alexa from the Amazon Echo consumer device and makes it available to developers. This chat bot-friendly service is in preview in the United States.

Cloud operations was front and center in a handful of AWS announcements. In partnership with configuration management vendor Chef, AWS released AWS OpsWorks for Chef Automate. This managed Chef environment is available today and differs from the previously available Chef-based OpsWorks service. The previous product, now branded AWS OpsWorks Stacks, was “built on a forked version of open-source Chef” and didn’t work with the full Chef ecosystem. Debugging distributed systems is not easy, and AWS is positioning AWS X-Ray as one solution. This tracing service integrates with the Amazon API Gateway as well as EC2 instances, containers, and Elastic Beanstalk. Since 2008, AWS customers used the Service Health Dashboard to see the current health of each AWS service around the world. Amazon is now offering a Personal Health Dashboardtailored to the services used by each customer. It also has built-in notifications and visibility into infrastructure changes that may cause customer impact. The final operations-oriented service announced by AWS is called AWS Shield. It provides DDoS protection at the DNS, CDN, and load balancer tiers and is offered in free and premium flavors.

As is tradition at re:Invent, Amazon rolled out a new set of EC2 instances. There are now F1 instances that support programmable hardware, memory-focused R4 instances, more simple T2 instances, high performing C5 instances, and I/O intensive I3 instances. Also, Amazon discussed an upcoming service called Elastic GPUs that adds graphics acceleration to any existing EC2 instance type. Additionally, EC2 instances in a Virtual Private cloud now supportIPv6 network addresses.

For years, developers who wanted virtual servers quickly flocked to providers like Digital Ocean and Linode. At re:Invent, Amazon jumped into this Virtual Private Server (VPS) space by launching Amazon Lightsail. This service is designed to simplify provisioning for those who want a server with basic features and predictable pricing. Batch processing has been a staple of computing for decades, and Amazon thinks the cloud can simplify the model. They introduced AWS Batch,a service that manages all the provisioning, management, monitoring and maintenance of batch clusters.

Amazon made a pair of announcements that defy simple categorization in this summary. AWS Greengrass aims to bring the “AWS programming model to small, simple, field-based devices.” This service is built for offline operations and is in a private preview. Blox is a new open source project from Amazon for developers who want to build their own containers schedulers. The project’s home is on GitHub.