AWS Private/Air-gapped Regions Overview

AWS air-gapped regions have increased in popularity over the past several years, and we expect that trend to continue for quite some time. At DAE, we’ve seen firsthand the profound impact air-gapped/private regions have had on customers’ missions and have seen our ISV and SaaS partners thrive in these environments. We get a lot of questions from our partners regarding what it takes to succeed in AWS air-gapped regions, so we are launching a blog series to help Systems Integrators, SaaS providers, and ISVs thrive in these regions.

When companies are looking (or are asked) to launch their product or code in an AWS air-gapped/private region, they may not know where to start. They may be surprised to learn that their commercial products or resources built in commercial AWS environments may not work as designed out-of-the-box in these special regions. Or, companies may have a high-level idea of how to operate in these regions and may understand some of the constraints, but they may not know every nuance, any one of which could mean the difference between success and failure. Proper planning and testing will ensure success when launching into these regions.

Because of this, ISVs (independent software vendor), SaaS providers, and systems integrators often find themselves in a situation where:

They don’t know how much configuration may be needed for their product to function in these air-gapped/private regions.

They don’t understand the AWS service availability of these regions and what impact that may have on their product

They think they’ve tested their code/product, but when they go to deploy in these regions, their product fails to launch or perform as designed.

They don’t have the resources (i.e., people with accesses) to triage bugs within the air-gapped/private region.

They do not have a dev/test environment in commercial AWS that mimics the nuances of these air-gapped regions.

This often results in many months of back-and-forth, rework, and wasted effort, drastically increasing the cost to operating in these air-gapped regions, the length of time between deployments, and the risk of products not working as intended once in these air-gapped regions and damaging their products’ reputation. While this sounds dire, with the right education, tools, and expert support, preparing a product or system for an air-gapped region is achievable!

Below are the top differences between air-gapped and commercial AWS regions to understand:

Internet restriction – your system will have no access to the internet

The term “air-gapped” typically stems from no network connectivity to the internet. This means your app/service cannot call out to the internet. For example, your web assets, such as font files or JS libraries, may be pulling down from external distributions on the internet. These would not work in air-gapped regions as those external distributions are not accessible. The same goes for configuration files from external sources or even things like licensing calls back to a central server. If your system is deployed to these air-gapped regions while trying to pull down web assets from the internet, it is almost guaranteed the frontend of your system will not look as designed, be riddled with JavaScript errors, or may not even load at all.

Some AWS services are not available in these regions

If a system is reliant on an AWS service that is not available in these regions and that service doesn’t exist, the system will not work. We often find that systems are making AWS calls behind the scenes and may not realize that deep in their code there is a call to an AWS service that is fundamental to the system. A good example is AWS’s System Services Manager (ssm) service and the Parameter Store. Many systems today use ssm with the parameter store to hold system configuration items that are used during system start-up. The ssm service isn’t necessarily available in these air-gapped regions and if deployed, will fail on start-up.

Some AWS services are limited/deprecated

While an AWS service itself may be available, some actions and features within that service may not be. (Fun Fact: Did you know that EC2 has around 277 different actions you can call through the CLI?). Similar to if the service itself wasn’t available, if the specific action/feature your system is reliant on isn’t available in these air-gapped regions, your system will not work as intended. For example, it is common practice to use NAT Gateways commercially to protect resources in private subnets but allow outbound access to the internet. If your system tried to create NAT Gateways as part of its infrastructure, it would likely fail in these air-gapped regions as even though the VPC service is available, the NAT Gateway feature may not be.

Air-gapped regions have different endpoints that are not accessible commercially

Service endpoints are the primary way your system talks to AWS services (i.e., S3, EC2, etc.). Pointing to the correct service endpoints is mandatory for your system to function correctly. If your system cannot connect to the proper endpoint, it won’t be able to use AWS services in air-gapped regions. Furthermore, since these endpoints do not exist commercially, the endpoint mappings also do not exist. This means out-of-the-box AWS SDKs, JDKs, and CLI are unaware of these endpoints and will fail to connect to these endpoints without configuration.

Air-gapped regions have different region names

Many systems are developed assuming regions will never change or that new regions will be automatically updated in all the AWS SDKs and CLI. These assumptions are incorrect and can lead to severe errors within your system. For example, many products allow the user to select the region from the drop-down list. The problem is, this list doesn’t include these air-gapped regions as they’re based on commercial regions and thus will fail. Getting the region(s) right within your system is essential. Additionally, many systems use the default region (us-east-1) which does not exist in these air-gapped environments.

Air-gapped regions have endpoints that signed by a custom certificate authority

It is important to consider the possibility that air-gapped regions could use custom certificate authorities (CA) to sign their endpoint communications. If endpoints do use a custom CA, your system will need to be configurable to work with custom certificates signed by that CA. If not, even if your system is pointing to the correct endpoint and region, it will still fail as it is not able to validate the endpoint certificate and be unable to create a secure TLS/SSL connection.

Air-gapped regions often have unique custom attributes (i.e., ARNs)

An example of custom attributes are changes made to Amazon Resources Names (ARN) which could occur in air-gapped regions. ARNs follow a very established format that developers can use to reference AWS resources. In an air-gapped region, these ARN formats can be different. These custom changes could cause your system to fail in odd and unexpected ways that make it extremely hard to debug.

If any nuanced difference referenced above is not completely remediated within your system, it will not perform as intended in air-gapped regions. That’s why we’ve designed SHIFT to monitor, analyze, and suggest remedial action for ALL the functions listed above. In our experience, with the proper education and training early in a development cycle, we’ve seen vast increases in success and time to market. In some cases, configuration can occur within a matter of days vs. months of back and forth.

Our recommendation: Utilize SHIFT as your full-scale emulation and testing platform. Failure cannot be an option in these private regions, where customers are relying on your product to work as designed on the first deployment. If you are new to tailoring your product to thrive in air-gapped regions, we recommend that you find a trusted partner who can provide services and guidance throughout this journey. Digital Age Experts (DAE) entire mission is to support customers thrive in air-gapped regions and we are always happy to help.