Backup Critical wiki pages

There is a script in sysadmin@home.cs.earlham.edu:~/wiki_critical/ called send_wiki.sh that specifies which pages to pull down and send out via email. This is important to do before anyone starts shutting down the machines because the wiki will go offline.

General Info

babbage should be the very last machine brought down

to get to sysadmin@control first ssh into home or hopper.

Make sure all virtual machines are shut down before restarting the bare metal hardware

CS virtual machies

The different VMs mount from eachother, so just be patient and hopefully everything will work out.

Tools

We may have to restart nginx, jupyter, and sage by hand. Using history | grep <command> is helpful here. (make sure to grab the entire command including ampersand)

order to bring up

The reverse of shutdown, again make sure to wait before proceeding at the appropriate steps.

CS

order of shutdowns

If hopper is back online, ssh sysadmin@cluster.cs.earlham.edu and then ssh sysadmin@control.cs.earlham.edu. This way we can shutdown all the VMs directly without being knocked off line or being in the machine room.

If it's an issue, you'll need to enable the vmstore-shared volume group as well as any logical volumes in there.

Starting VMs

The VMs on control should be brought up in the reverse order they were shutdown. It is very important to bring up net first.

control-# xm create -c /home/sysadmin/eccs-<hostname>.cfg
# To exit to the hypervisor shell you can press Ctrl + ]
# To start VM without going into console for boot messages, forego the -c

The configuration files used for starting the virtual machines are in /mnt/vmdata-shared/config/. There are 4: eccs-home.cfg, eccs-web.cfg, eccs-net.cfg, and eccs-tools.cfg. There are backup copies in ~sysadmin/xen-configs.

Connect to VM console

console-# xm console <hostname>.cs.earlham.edu

You'll need to make sure that their DNS resolver settings are correctly set. I've had trouble with them using the incorrect DNS server settings sometimes and I'm not sure if the issues are resolved or not.

-Eamon

Update 24/6/2017: The new procedure for starting and shutting down net and web are as follow.

To shut down:

# xl destroy net.cs.earlham.edu
# xl destroy web.cs.earlham.edu

Idealy though, the VM's should be shutdown from inside (by ssh'ing into them). After that, run "xl list" to see if they're still listed as domains, then run the "xl destroy" commands as above if needed.