Production debugging

Debugging failures in MAAS is hard, we should aim so that everything we ship with MAAS - i.e. outside of its core - should work through the API, even debugging tools. Customers - or PES - should not have to perform brain surgery on MAAS to diagnose a tagging issue, for example. Any solution we come up with has to work on site, in production.

Examples of failures:
* node fails to boot
* node fails to commission
* power scripts not working
* DNS not working

Sprints

Whiteboard

= Debugging MAAS =

== Ideas ==

- IPMI console (conserver), view it in the UI when requested
- Kernel param net console
- Better error messages
- Keep track of the major events in the lifecycle of a node
- Add a command line option ("debug mode") to "block" during enlistment/commissioning to let the user log in and debug (i.e. similar to the backdoor feature that "exists" today).
- MaaSTest permanently in debug mode so you can always get a back door into the system.
- cloud-init logs on region/UI
- Consolidate logs into one file (→ (r)syslog)
- Log level changes on the fly (this needs a bug)
- Set software clock on enlistment
- Customize the DHCP template from the UI.
- Improved notifications UI.
- SOS report (in saucy main / https://github.com/sosreport/sosreport): gathers logs/db dump and creates a tarball with all that to help remote debugging.