* Install machine with 14.04.5 and fully upgrade it.
* Execute 'do-release-upgrade' to upgrade to 16.04.
* When the upgrade is finished, exit the release upgrade tool (do not allow the tool to reset the system).
* Manually attempt to reboot the system with "sudo /sbin/shutdown -r"

The machine will not reboot and the shutdown command will exit with:
Failed to set wall message, ignoring: Method "SetWallMessage" with signature "sb" on interface "org.freedesktop.login1.Manager" doesn't exist
Failed to call ScheduleShutdown in logind, proceeding with immediate shutdown: Method "ScheduleShutdown" with signature "st" on interface "org.freedesktop.login1.Manager" doesn't exist

Seems like there's two issues here which I'll address in separate PRs to the landscape-client:

First is the issue that the landscape-client fails to report the completion (or failure) of the release upgrade process. This turns out to be due to a delayed import in the version of twisted distributed with trusty. Because the import (of twisted.internet.unix) is delayed, by the time we *do* import it (implicitly by calling another twisted method), the twisted version on disk has been replaced with the xenial version and things (unsurprisingly) break. The trivial fix is to pre-emptively import the module prior to starting the release upgrade; I'll push a PR for this in a moment.

The second issue is that landscape-client fails to reboot (or shutdown) the machine immediately following the release upgrade procedure. This is due to the fact landscape-client currently relies upon /sbin/shutdown to handle reboot or shutdown requests. This utility operates happily in its upstart variant (in trusty), and systemd variant (in xenial and beyond), but fails in the post-trusty-to-xenial upgrade environment when the system on disk is effectively xenial (so /sbin/shutdown is a link to systemctl), but the running environment is still partially trusty. It transpires that systemctl (without the double --force --force option) requires both systemd and dbus to be operational in order to successfully (and cleanly) shutdown or restart.

The fix for this is a little more complex as the landscape-client relies upon the scheduling function of /sbin/shutdown to ensure it can request a shutdown and still report back to the server before the machine actually goes down. I'm still investigating the best method which will work reliably in all potential environments.

I've now pushed a PR which should alleviate the reboot issue *partially*. The complexity in fixing this arises from the differing behaviours of implementations of the shutdown, poweroff, and reboot commands; upstart's poweroff & reboot commands for example, immediately return (briefly) permitting the caller to query their exit codes. In contrast, systemd's don't return at all unless the poweroff or reboot fails. None of the reboot or poweroff implementations provide a scheduling facility like the /sbin/shutdown implementation (unsurprisingly), but systemd's poweroff and reboot commands *do* work even in the post-trusty upgrade environment (even though systemd's shutdown command doesn't).

The quandry is as follows: we can reliably detect when the shutdown command fails (it exits with an exit code and some error messages), and when it succeeds ... sort of. We currently schedule the shutdown for 4 minutes time and if it hasn't died within 10 seconds we assume it's going to work; this assumes that the implementation won't fail at the end of the schedule but so far that seems a reliable assumption under both upstart and systemd. We cannot *reliably* detect when the poweroff or reboot commands either succeed or fail, but they do seem reliable in practice even in aberrant environments like the post-trusty upgrade.

So, the method I've chosen is as follows: treat shutdown and restart requests as we presently do (schedule /bin/shutdown for 4 minutes time, monitor for 10 seconds). If it succeeds, nothing changes. If it fails, report the failure with an extra message indicating we are going to attempt to force the procedure, then run /sbin/poweroff or /sbin/reboot as appropriate. The result is that in the post-trusty environment (testing on containers and VMs) a reboot request does succeed, but is still reported as failing (albeit with an extended message explaining we're going to try and force it).

We *could* report it as succeeding, but frankly there's no guarantees so it's a choice between giving the user bad news ("sorry your reboot request failed ...") with a possible silver lining ("... but we're going to try another way which *might* work, however we won't be able to tell"), or giving the user good news ("your reboot request worked after we forced it") and hoping we're not lying! The former sounds like the more sensible approach to me, and hence is what is in the PR (https://github.com/CanonicalLtd/landscape-client/pull/55).

@xnox Scheduling with /sbin/shutdown normally does work ... however, what we're dealing with here is the anomalous situation of an instance which starts off as trusty and is upgraded to xenial. During the upgrade process, the /sbin/shutdown implementation is changed from the upstart one to the systemd one. At the end of the process, /sbin/shutdown is (unsurprisingly) a link to systemctl ... but it doesn't work because it's expecting to be able to talk to systemd which isn't fully operational until a reboot. As Landscape is expecting to be able to reboot via /sbin/shutdown the user relying on Landscape in such a scenario finds themselves in a bit of a bind (having to have some other means of dealing with the server, like SSH).

However, /sbin/reboot and /sbin/poweroff *do* work in this (admittedly rare) scenario, hence the patches committed for this issue leave the system using /sbin/shutdown by default but if/when it fails, they fall back to trying /sbin/reboot or /sbin/poweroff (as appropriate) instead.

As to why -h is used instead of -P, in systemd's implementation of /sbin/shutdown they're exactly the same thing. In the upstart implementation it's more vague; quoting from the man-page: "Requests that the system be either halted or powered off after it has been brought down, with the choice as to which left up to the system". I'm not sure what that means in practice though!

/sbin/shutdown, post upgrade not working, imho is a severe bug. I'll investigate that. I suspect that because we cannot restart dbus things don't show up there, but /sbin/shutdown should be smart enough to do something. I guess the scheduling of timeout is failing, but /sbin/shutdown now works?

Re differences between halt & poweroff, in systemd, eventually it results in:

I suspect no-one's bothered to report it because as soon as the machine is (somehow) rebooted, the issue goes away (and reproduction then involves the pain of going through a trusty install + xenial upgrade cycle). Still, I'm not entirely convinced this *is* a bug; I'll come back to this in a bit below...

> I guess the scheduling of timeout is failing, but /sbin/shutdown now works?

I thought I'd tested this and concluded it didn't, but it turns out I just didn't wait long enough. When trying "shutdown -r now" there's a fair delay, then it complains about a connection time out, exits with an error, and the system reboots anyway!

So, it is mostly likely the scheduling portion that's failing, and if given an immediate request the operation does still work. I haven't got time to test the -h switch as well, but I'd assume it's probably a similar story.

Now consider those two cases from the perspective of a human administrator:

If you've typed "shutdown -r now" you're expecting an immediate reboot. Even if some bits of the shutdown command fail, you still expect it to do what you asked: reboot immediately. However, if you've just typed "shutdown -r +5" you're expecting a 5 minute delay before reboot. If it can't schedule that delay would you rather it exit with an error and tell you, or just immediately reboot the system? Probably the former given you're not expecting an immediate reboot.

The issue in this case was that it isn't a human scheduling the reboot (directly) and the only reason landscape uses the schedule at all is to determine whether the shutdown command has (likely) worked. Which is a bit of an abuse of the shutdown command's scheduling facility, but I can't think of a better way off the top of my head (that'll reliably work in all implementations).

Anyway, in landscape's case we *would* rather the system immediately reboots when shutdown fails because the delay's only there for the Landscape client's benefit, and we can't guarantee the administrator has any other means of accessing the system for the purpose of rebooting it. So in a sense we're dealing with an interface mismatch: shutdown is (understandably) written for the (direct) use of a human administrator, and landscape is (ab)using it to provide reboot/poweroff facilities on (indirect) behalf of a human administrator.

So we come back to "I can't think of a better way of doing it" (that'll reliably work in all our aforementioned scenarios of upstart, systemd, and post-upgrade-almost-sort-of-systemd).

Heh, I believe there are historical options re -H -h. As a wild guess, i looked at the sysvinit implementation in Debian (we have removed the package in Ubuntu) it has this:

-h Halt or power off after shutdown.
-H Modifier to the -h flag. Halt action is to halt or drop into boot monitor on sys‐
tems that support it. Must be used with the -h flag.

So clear as mud, and i guess the "portable" way is to call "shutdown -h -H" or something.

RE: scheduling

I guess you could inspect the system from landscape as follows:
- check system has re-execed into systemd init ( [ -d /run/systemd/system ] )
- check that upstart is gone ( UPSTART_SESSION= initctl version )
- poke the dbus interface to see if logind is there and has support for timing things (probably using busctl http://manpages.ubuntu.com/manpages/xenial/en/man1/busctl.1.html)
- realise the scheduling will not work
- request shutdown now

Imho, failing to schedule thing to shutdown, should shut the machine down instantly. Or i should upgrade machine from trusty to xenial; and like snapshot it; and figure out why scheduling does not work / logind is not there.

I have still have two machines that need upgrading from 14.04 -> 16.04 -> 18.04. I want to replace them and test the upgrade in the lab.
The upgrade from 16.04->18.04 also fails. I'll report it in a new ticket.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, details of your testing will help us make a better decision.

The verification of the Stable Release Update for systemd has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Landscape is supposed to be Enterprise Grade software. I should be able to upgrade the releases of an entire farm of machines. This has been broken since the first time I tried this feature with Landscape.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-cosmic to verification-done-cosmic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-cosmic. In either case, without details of your testing we will not be able to proceed.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-trusty to verification-done-trusty. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-trusty. In either case, without details of your testing we will not be able to proceed.