The second issue is see a lot of people having is how do you reference a resource already created in ARM and get some of that objects properties e.g. FQDN on a public IP already created etc.

The clue to solve this issue, so that ARM Template B can reference a resource created in ARM Template A can be found here:

By using the reference function, you implicitly declare that one resource depends on another resource if the referenced resource is provisioned within same template and you refer to the resource by its name (not resource ID). You don’t need to also use the dependsOn property. The function isn’t evaluated until the referenced resource has completed deployment.

Or use linked templates (Linked templates is a huge rework and you need to host the files on the net). Lets see if we can do it via resourceId.

Therefore if we do reference a resource by resourceId, we will remove the implicit “depends on”, allowing ARM Template B to use a resource created in a totally different ARM template.

The important thing here is to ensure you always include the API Version. This pattern is a very powerful way to create smaller and more modular ARM templates.

Note: In the above pattern, you do not need to define DependsOn in ARM Template B, as we are explicitly defining a reference to an existing resource. ARM Template B is not responsible for creating a public IP. If you need it, you run ARM Template A.

So if you need a reference to existing resources use the above. If you need a reference to resources created in the SAME ARM template use:

How do you upgrade the existing Data Disk on a primary Node Type Virtual Machine ScaleSet in Service Fabric?

How do you upgrade the existing Operating System on a primary Node Type VMSS in Service Fabric?

How do you move the Data Disk on a primary Node Type VMSS in Service Fabric?

How do you monitor the status during the upgrade, so you know exactly how many seed nodes have migrated over to the new scale set?

note – We successfully increased the SKU size as well, however this is not supported by Microsoft. However just increase your SKU in ARm and later, after the successful transfer to the new VMSS, run Update-AzureRmServiceFabricDurability.

Considerations

You have knowledge to use ARM to deploy an Azure Load Balancer

You have knowledge to use ARM to deploy a VMSS Scale Set

Service Fabric Durability Tier/Reliability Tier must be at least Silver

Keep the original Azure DNS name on the Load Balancer that is used to connect to the Service Fabric Endpoint. Very Important to write it down as a backup

You will need to reduce the TTL of all your DNS settings to reduce downtime during the upgrade which will just be the TTL value e.g. 10 minutes. (Ensure you have access to your primary DNS provider to do this)

Prepare an ARM template to add the new Azure Load Balancer that the new VMSS scaleset will attach to (Backend Pool)

Prepare an ARM template to add the new VMSS to an existing Service Fabric primary Node Type

Deploy the new Azure Load Balancer + Virtual Machine Scale Set to the Service Fabric Primary node

Run the RemoveScaleSetFromClusterController.ps1 – Run this script on the NEW node in the NEW VMSS. This script will monitor and facilitate moving the Primary Node Type to the new VMSS for you. It will show you the status of the Seed nodes moving from the original Primary Node Type to the new VMSS.

When it completed, the last part will be to update DNS.

Run MoveDNSToNewPublicIPController.ps1

ARM Templates

You will need only 2 templates. One to Deploy a new Azure Load Balancer and one to Deploy the new VMSS Scale Set to the existing Service Fabric Cluster.

You will also need a powershell script that will run a custom script extension.

Once you have a new VMSS scale set attached to the existing NodeType, you should see in Service Fabric the extra nodes. the next step is to disable and remove the existing VMSS scaleset. This is an online operation, so you should be fine. However later we will need to update DNS for the Cluster Endpoint. This is important for Powershell Admin tools to still connect to the Service Fabric cluster.

RemoveScaleSetFromClusterController.ps1

Remote into one of the NEW VMSS virtual machines and run the following command. It will make dead sure that your seed nodes migrate over. it can take a long time (Microsoft docs say it takes a long time, how long?). it depends, for a cluster with 5 seed nodes, it took nearly 4 hours! So be patient and update the loop timeout to match your environment, increase the timeout if you have more than 5 seed nodes. My general rule is allow 45 minutes per seed node transfer.

This script is extremely useful, you can see the progress of the transfer of seed nodes and disabling of existing primary node types.

You know it is successful, when the old nodes have ZERO seed nodes. All SEED nodes must transfer over to the new nodes, and all nodes in the old scale set shoul dbe set to false by the end of the script execution.

MoveDNSToNewPublicIPController.ps1

Lastly you MUST update DNS to use the original CNAME . This script can help with this, what it does is actually detach the original internal Azure CNAME from the old public IP and move it to your new public IP attached to the new load balancer.

What you are doing is creating a variable that can be used by the hiera.yaml file to dynamically load data for the correct environment when the agent runs.

On the Puppet Master we need to setup our environments to match the Control Repository and add the magic variable. Any Node that runs the puppet agent will then have this variable set. This can then be used to load the corresponding Hiera config file.

Here we can see Environment1 has a variable defined called geography that matches the Environment name. We can then leverage this convention:

As you can see above. As long as your profiles and parameters match, Hiera will automatically inject the correct parameter for each environment.

Hiera will resolve – %{geography}/jumpbox/conf.yaml

In the Puppet master you had setup your classifications, so when the puppet agent runs on Environment1 nodes, it will get the jumpbox/conf.yaml that matches the variable name geography=”Environment1″

Encrypted Data – eyaml

Encrypted data is just as easy to store.
* Generate the encrypted data.
* Store the data in an eyaml file in the same folder as the yaml data.
* Add a path to the data in the hiera.yaml file.

We have encrypted data e.g. the default local admin account setup via the profile – include profile::base
We use the Puppet Master private key to generate the encrypted data, see the end of this blog on how to create encrypted data.

We use Azure Active Directory Domain Services and wanted a single sign on solution for Windows and Linux. The decision was made to join all servers to the Windows Domain in addition to having SSH Key auth.

I assume you know how to use Puppet Code Manager and have a Puppet Control Repository to manage Roles and Profiles.

Ensure you have a Active Directory Service Account that has permissions to join a computer to a domain. You can store this user in Hiera.

The module we need is called Realmd, however the current version (Version 2.3.0 released Sep 3rd 2018) does not support Ubuntu 16.04. So I have a forked repro here that you can use.

PSQL has some awesome tools pg_dump and pg_restore that can assist and cut down restore times in the event of a recovery event.

Scenario

User error – If a table was accidentally emptied at noon today:

• Restore point in time just before noon and retrieve the missing table and data from that new copy of the server (Azure Portal or CLI) , Powershell not supported yet.
• Update firewall rules to allow traffic internally on the new SQL Instance

Then log onto a JumpBox in the Azure cloud.
• Dump the restored database to file (pg_dump)

Problem

Update Management in Azure does not support system workspace. If you are using Security Center in Azure, the chances are; all your VM’s are allocated to the system workspace that Security Center created.

The selected workspace is a system workspace and cannot be linked to this account

The other error you can get is when you try enable Update Management on VM, and that VM is in another Workspace.

The selected Automation account is already linked to a Long Analytics workspace that is not the selected workspace

Solution

Deallocate All VM’s that you want to use Update Management from the System Workspace to a New Workspace.

The first step is for Development and Staging, which do not use an ELB

The next 4 steps is for Web Servers in Group 1

The last 4 steps is for Web Servers in Group 2

A Powershell script is used to poll the web server, after the NUGET package is used to deploy the website. It will poll the Web Servers in the group until the homepage returns an HTTP 200 code for all of them. The homepage does various calls to the database, so this page is great to use. You could also create a custom up-time endpoint to query. A really cool way, is to have an API endpoint that returns which servers are up and use this to validate the step for polling.

I ran into a few issues when I wanted to install Puppet Enterprise 2-17 in AWS as an EC2 instance. The main issues were

Summary

Need to use hostnamectl and cloud.cfg to change my hostname, as I wanted puppet on a public address, not private address, just for a POC

I was using a t2.nano and t2.micro, which will not work with Puppet Enterprise 2017 (puppet-enterprise-2017.2.2-el-7-x86_64). The error you get is just Failed to run PE Installer…… So I used a t2.medium to get around the issue.

The usual /etc/hosts file needs some settings and DNS registration (Route53 for me)

Disabled SELinux (We usually use a VPN)

Configure security groups and have 4433 as backup port (Probably not needed)

Preliminary Install Tasks

Spin up an instance with at least 4GB memory, I had a lot of installation issues with applying the catalog with low memory. T2.Medium should work. Bigger is better![puppet.rangerrom.com] Failed to run PE installer on puppet.rangerrom.com.

If you not using a VPN then ensure you setup an elastic IP mapped to the instance for the public DNS name

Register the hostname and elastic IP in DNS

Add you hostnames to /etc/hosts (Important!), note I also added puppet as this is the default for installs. This is a crucial step, so make sure you add your hostnames that you want to use. Put the public hostname first. As this is our primary hostname127.0.0.1 puppet.rangerrom.com puppet localhost

This is the error I got when I first installed puppet (Due to low memory), therefore we will add port 4433 as well to the AWS security in the next step. I think this was due to insufficient memory, so use a T2.Medium instance size, so you have a minimum of 4GB of memory, else java kills itself. However I add it as a backup here in case you run some other service on 443.

Install NC and use it to test if your ports are accessible.
sudo yum install ncnc -nlvp 3000 (Run in one terminal)

nc puppet 3000 ( Run from another terminal)
This is a great way to ensure firewall rules are not restricting your installation. Secondly we testing that the local server can resolve itself, as it is important that you can resolve puppet and also your custom FQDN before running PE install.

Ensure in DNS Alias, you add puppet and all other DNS names you want to use. Otherwise the installation will fail.

You should see the correct default hostname, if not, you got issues…. I added some alias names such as puppet and my internal and external ec2 addresses.

Set an Admin password and click next

Check and double check the settings to confirm.

Check the validation rules, since this is for testing, I am happy with the warnings. It would be awesome if puppetlabs did DNS name resolution validation checks on the HostName. Anyways, here we get a warning about memory, 4GB is what is needed, so if you have install failures it may be due to memory!