Remote management of VMs hosted in colocation

I'm taking a look at how I'm currently accessing and managing VMs on my colo lab server. What I currently have works, but I see some flaws I need to address. I figure this could turn into a discussion of methods of remote management for colo infrastructure. For me, my goal is to treat my lab as I would any business asset in colo as far as the practices I follow when doing $tasks.

This is currently how I access my lab -- using ZeroTier. I have ZeroTier installed on my laptop at home and my host. I run Virt Manager on my laptop at home to get console access to the host and the subsequent VMs.

The major problem I see with this:
If my laptop becomes compromised (think ransomware), then my host is pwn'd.

Some options I've considered for handling this better

Create a VM for management and connect to it with a tool like ScreenConnect

I've experimented with this with some success. I've tried a Fedora VM with the ScreenConnect agent installed. Opening a terminal session and SSHing into the host was fine. I found that connecting to the host with VirtManager to be a pain (probably my GUI environment is to blame -- LXQT), and using a web browser (Falkon) was a slow, painful experience.

What I like about this approach is that it's using a jump box (the management VM). It's also using an on-demand connection to it, which I can access from anywhere with an Internet connection. It also reduces the risk of my pwn'd laptop pwning my server.

Allowing an SSH connection to the managementVM from the Internet

I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.

I thought virt-manager only used ssh to connect to the host. What about using key-based auth to the host and disable password auth? Seems it would be faster than going through ZT. I guess the laptop compromise still poses an issue in that scenario.

I thought virt-manager only used ssh to connect to the host. What about using key-based auth to the host and disable password auth? Seems it would be faster than going through ZT. I guess the laptop compromise still poses an issue in that scenario.

Virt-manager does use SSH. I connect virt-manager to the host via SSH to its ZeroTier IP address.

I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.

SSH is an always encrypted connection. It doesn't even pass keys until an encrypted tunnel is setup, so using over VPN is.... 2 levels of encrypted communication.

I thought virt-manager only used ssh to connect to the host. What about using key-based auth to the host and disable password auth? Seems it would be faster than going through ZT. I guess the laptop compromise still poses an issue in that scenario.

If you realize your laptop has been Pwned or stolen then kick it out of the ZT network.

I thought virt-manager only used ssh to connect to the host. What about using key-based auth to the host and disable password auth? Seems it would be faster than going through ZT. I guess the laptop compromise still poses an issue in that scenario.

Keys or not, I will not open SSH to the public internet if I have a simple way around it.

That is what I like about using something like ZeroTier for. His KVM host has zero presence on the Public internet. It only uses an outbound SSL session to connect to the ZT network.

Perhaps if I wanted an extra thick tinfoil hat, I could have my managementVM connected to ZeroTier, setup $remoteDesktopMechanism on the VM, and connect to the managementVM from my laptop via ZeroTier to manage the kvm host and all of the other VMs.

I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.

Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.

I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.

Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.

I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.

Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.

Or add Salt and open/close based on need so it doesn't stay open.

I could see that in a situation where your office has a static IP. For me, I wouldn't be able to lock down the allowed IP, since there's a chance it'll change.

I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.

Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.

Or add Salt and open/close based on need so it doesn't stay open.

I could see that in a situation where your office has a static IP. For me, I wouldn't be able to lock down the allowed IP, since there's a chance it'll change.

Not really an issue, use Salt to maintain it so not a problem even for dynamic.

ZeroTier makes it super easy and safe. If you want, you could make a ZeroTier Bridge VM. I would still setup Fail2Ban and use key-based authentication.

I'd argue that it doesn't really make it safer. ZT is just key based access, like SSH is. If you are concerned that SSH with keys isn't enough, ZT isn't either. ZT exposes even more should it be compromised, as well. In both cases, being compromised is pretty bad, but one is a lot more limited and able to be limited. If you feel someone can breach your keys with SSH, why not ZT just the same?

I have not tried this approach yet, and it appears more risky than the Screen Connect approach, since SSH to that VM would be open to the Internet. Unless I'm missing some benefit to this approach, I'll not be using it.

Use a strong key, lock to your IP. Very safe. Add Fail2Ban, of course.

Or add Salt and open/close based on need so it doesn't stay open.

I could see that in a situation where your office has a static IP. For me, I wouldn't be able to lock down the allowed IP, since there's a chance it'll change.

Not really an issue, use Salt to maintain it so not a problem even for dynamic.

That would be another thread. Spin up a Salt VM for maintaining stuff on my host and accessing it from afar.

ZeroTier makes it super easy and safe. If you want, you could make a ZeroTier Bridge VM. I would still setup Fail2Ban and use key-based authentication.

I'd argue that it doesn't really make it safer. ZT is just key based access, like SSH is. If you are concerned that SSH with keys isn't enough, ZT isn't either. ZT exposes even more should it be compromised, as well. In both cases, being compromised is pretty bad, but one is a lot more limited and able to be limited. If you feel someone can breach your keys with SSH, why not ZT just the same?

ZT is access without a public inbound allowance. Someone would have to gain access to your ZT network before being able to then try and get your keys for the SSH session.

that's my point. How is the access to your ZT network more secure than the keyed access to an SSH port? Both have the risk of vulnerabilities (but nothing has the eyes on it and reviews of OpenSSH, so if that's a concern that makes SSH the clean place to start), and both have access from the outside. Both rely on keys and both have known open ports.

ZT is access without a public inbound allowance. Someone would have to gain access to your ZT network before being able to then try and get your keys for the SSH session.

that's my point. How is the access to your ZT network more secure than the keyed access to an SSH port? Both have the risk of vulnerabilities (but nothing has the eyes on it and reviews of OpenSSH, so if that's a concern that makes SSH the clean place to start), and both have access from the outside. Both rely on keys and both have known open ports.

I was wondering this exact same thing - why is ZT more trusted in this case than SSH?

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

If you're using an infested machine - when you use the apps to make that remote session to the colo, be it SSH or SC, the malware can likely grab the needed information and provide it to the hackers.

I'm guessing that the Salt idea might give the greatest protection here when combined with the IP lock option that Scott mentioned - but then I ask - how do you securely manage the Saltmaster, where ever it lives?

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.

but then I ask - how do you securely manage the Saltmaster, where ever it lives?

Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).

Why create all this overhead for no reason, when ZT already handles it?

Answer? Because Salt is SAM's shiny toy.

Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.

Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.

but then I ask - how do you securely manage the Saltmaster, where ever it lives?

Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).

Why create all this overhead for no reason, when ZT already handles it?

Answer? Because Salt is SAM's shiny toy.

Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.

Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."

I'm not sure the always connected bit matters - does it? If you have an SSH client with the key stored in it - and why wouldn't you - this is likely just as bad as ZT. The attacker can just launch the SSH client on your machine and tada - he's got a connection to your host.

The same might be say able about SC - your laptop is compromised - they still your SC creds - hell, they now use your creds from their own machine to connect to SC.

I'm not seeing anything that offers protection from you using a compromised machine.

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.

but then I ask - how do you securely manage the Saltmaster, where ever it lives?

Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).

Why create all this overhead for no reason, when ZT already handles it?

Answer? Because Salt is SAM's shiny toy.

Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.

Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.

More work than your gimmicky Salt setup? As if.

Well - I think Scott was dumbing it down to simply using SSH, and not using salt to open IPs to allow connections, etc. That would be simpler than using ZT.

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."

I'm not sure the always connected bit matters - does it? If you have an SSH client with the key stored in it - and why wouldn't you - this is likely just as bad as ZT. The attacker can just launch the SSH client on your machine and tada - he's got a connection to your host.

The same might be say able about SC - your laptop is compromised - they still your SC creds - hell, they now use your creds from their own machine to connect to SC.

I'm not seeing anything that offers protection from you using a compromised machine.

You can password protect the key. Or if you use Apps like MobaXTerm and such, they can be set to require a password before the app is usable.

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

Umm, no. Exactly the opposite. The point is the reduced attack surface to the host.

but then I ask - how do you securely manage the Saltmaster, where ever it lives?

Because ZT has account level auth (your password to sign in to your account) and then device auth (approving the device that wants to join the network).

Why create all this overhead for no reason, when ZT already handles it?

Answer? Because Salt is SAM's shiny toy.

Pot calling the kettle black there. SSH is the simple, safe, battle tested option. ZT is great and I really like it, but if someone has a toy here, that's it. Yes, it has multiple log in spots, but you just up the key strength to match that and you are at the same security level.

Yes, ZT already handles some of this, but SSH is simple and easy and built in. ZT takes quite a bit more work and is much more complex to do something really simple, it's just not necessary or beneficial when the simple solution handles it.

More work than your gimmicky Salt setup? As if.

Well - I think Scott was dumbing it down to simply using SSH, and not using salt to open IPs to allow connections, etc. That would be simpler than using ZT.

But that is not what I was arguing. Thus, as I mentioned before, strawman.

The main thing I see the OP trying to save himself from is his control workstation being pwned. But in reality, I'm not sure there is anything one can do if they are using a pwned machine connect to their colo.

The OP's goal is discovering good practices for managing remotely hosted VMs. I identified a potential risk with how I'm doing stuff now as "if my laptop gets compromised, then it's game over for my host, since there is an always-on connection to it via ZT."

I'm not sure the always connected bit matters - does it? If you have an SSH client with the key stored in it - and why wouldn't you - this is likely just as bad as ZT. The attacker can just launch the SSH client on your machine and tada - he's got a connection to your host.

The same might be say able about SC - your laptop is compromised - they still your SC creds - hell, they now use your creds from their own machine to connect to SC.

I'm not seeing anything that offers protection from you using a compromised machine.

You can password protect the key. Or if you use Apps like MobaXTerm and such, they can be set to require a password before the app is usable.

sure, but if the machine is comp'ed, then there is likely a keylogger there capturing that password too.