Topics

Featured in Development

Alex Bradbury gives an overview of the status and development of RISC-V as it relates to modern operating systems, highlighting major research strands, controversies, and opportunities to get involved.

Featured in Architecture & Design

Will Jones talks about how Habito, the leading digital mortgage broker, benefited from using Haskell, some of the wins and trade-offs that have brought it to where it is today and where it's going next. He also talks about why functional programming is beneficial for large projects, and how it helps especially with migrating the data store.

Featured in AI, ML & Data Engineering

Katharine Jarmul discusses research related to fair-and-private ML algorithms and privacy-preserving models, showing that caring about privacy can help ensure a better model overall and support ethics.

Featured in Culture & Methods

This personal experience report shows that political in-house games and bad corporate culture are not only annoying and a waste of time, but also harm a lot of initiatives for improvement. Whenever we become aware of the blame game, we should address it! DevOps wants to deliver high quality. The willingness to make things better - products, processes, collaboration, and more - is vital.

Featured in DevOps

Service mesh architectures enable a control and observability loop. At the moment, service mesh implementations vary in regard to API and technology, and this shows no signs of slowing down. Building on top of volatile APIs can be hazardous. Here we suggest to use a simplified, workflow-friendly API to shield organization platform code from specific service-mesh implementation details.

The new ADLS Gen2 service combines scalability, cost-effectiveness, and a security model with rich analytics capabilities using the Hadoop Distributed File System (HDFS). Moreover, with the HDFS customers can store both structured and unstructured data, along with an Azure Blob File System driver (ABFS) that allows files and folders to be distinctly addressed on the server side – eliminating the need for a complex client-side driver, and ensuring high fidelity file system transactions.

We implemented a hierarchical namespace (HNS) which supports atomic file and folder operations. This is important because it reduces the overhead associated with processing big data on blob storage. This speeds up job execution and lowers cost because fewer compute operations are required. The ABFS driver and HNS significantly improve ADLS’ performance, removing scale and performance bottlenecks.

Next, in regard to the performance boost, Microsoft also offers the same robust data security capabilities built into Azure Blob Storage, such as:

Currently, ADLS is available in almost all Azure regions except for US DOD Central and US DOD East. Furthermore, the pricing details for ADLS are available on the pricing page.

With the new ADX, customers can leverage a fully managed data analytics service for real-time analysis on large volumes of streaming data. This service is, according to the blog post by Willis, capable of querying 1 billion records in under a second with no modification of the data or metadata required. Furthermore, ADX includes native connectors to Azure Data Lake Storage, Azure SQL Data Warehouse, and Power BI and comes with an intuitive query language allowing customers to obtain insights in minutes.

Microsoft made the design for ADX with speed and simplicity in mind – it combines two distinct services that work in tandem:

The Engine, a service responsible for processing the incoming raw data and serving user queries, and

A Data Management (DM) service, which allows the ingestion of various types of raw data. Furthermore, the DM is also responsible for managing failures, backpressure, and data grooming tasks when necessary.

Note that both services are deployed as clusters of compute nodes (virtual machines) in Azure.

ADX is currently available in 41 Azure regions, and pricing details are available on the pricing page.

With the two new services, customers can have greater flexibility in managing unstructured data or data generated from interactions on the web, software-as-a-service apps, social media, mobile apps, and internet of things devices. According to John Chirapurath, general manager of Azure data, blockchain, and AI at Microsoft in a VentureBeat article:

We always strive to make it very easy for IT staff to adopt analytics and for line-of-business people to utilize and deliver powerful insights using beautiful products.

Lastly, Microsoft also released a preview of a new Mapping Data Flow capability in Azure Data Factory (ADF) - a hybrid cloud-based data integration service for orchestrating and automating data movement and transformation. With the new capability, customers can visually design, build, and manage data transformation processes without learning Spark or having a deep understanding of their distributed infrastructure. Currently, ADF is available in 21 regions and pricing details are available on the pricing page.