controlling systemd shutdown order

Please save my sanity; I've been reading manpages and articles forhours without finding a successful solution. This may be becauseI'm trying to do something which sort of violates the spirit ofsystemd - but it should still be possible!!!

When the system starts to shut down (or reboot), I'd like it torun a script, and I'd like systemd to wait for my script to finish,*before* systemd starts to kill user processes.

I'm able to define a service (Type=oneshot, RemainAfterExit=yes)and enable it, and my ExecStop is duly called during shutdown, buttoo late. Since the shutdown order is supposed to be the reverseof the start-up order, I would have thought that I could come upwith some appropriate "After=" specifications that would cause myservice to be stopped before the "Stopping Session" tasks startedto kick in, but so far, no go. I've even tried:

Re: controlling systemd shutdown order

Thank you very much, Docbroke and Brebs, for taking the time to reply.

Yes, of course I could replace the shutdown and reboot scripts; thatwould be trivial. However, it would be a hack, and I'm trying tolearn how to write systemd units correctly.

Docbroke wrote:

Once the shutdown process starts you cannot run your script. Onlyway is to run your script before calling the process of shutdown.

That's not the case at all - in fact, my script does run.It just runs too late. According to the manpage for "systemd.unit":

when two units with an ordering dependency between them are shut
down, the inverse of the start-up order is applied. i.e. if a
unit is configured with After= on another unit, the former is
stopped before the latter if both are shut down.

So it should be possible to ensure that my service is shut down(and therefore, my script as specified via ExecStop runs) beforeanother service gets shut down, by making my script "want" or"require" that other service, and start "after" it.

What I need to figure out (I think!) is: what is the service thatis responsible for user processes, and therefore that kills userprocesses on shutdown? Then I could make my service start afterthat, and thus be shut down before it.

Alternatively, if user processes are killed not by shutting down aparent service of theirs, but by starting some other service thatitself is required/wanted by the shutdown.target, then what is thisservice that kills the user processes?

If there's documentation out there that describes the systemd shutdownprocess in more detail than the manpages, I haven't managed to findit yet. I'm happy to read the docs and the various unit definitionsto figure this out, but I'm stuck as to where to look.

Re: controlling systemd shutdown order

Trilby wrote:

At least let us know how long this script is likely needed to complete. Might it be more than 90 seconds? If so, have you set TimeoutStopSec? or DefaultTimeoutStopSec?

Not likely more than 5 seconds, but you're barking up the wrong tree.It starts and it completes, but by the time it starts, the system hasalready started killing user processes, and I want this script to runbefore any end-user processes are killed.

What I can't seem to figure out is how to set up a dependency toaccomplish this.

Re: controlling systemd shutdown order

You could add an 'ExecStop=' line for the service for which you want the script to run before it stops, using a drop-in conf file for this service.

I'd be happy to do this, if I could figure out which service iskilling the end-user processes during shutdown. ... but if Icould figure that out, I could probably write the dependency!

(That is, if "Type=oneshot" successfully makes follow-up serviceswait for its completion before starting their own shutdown - themanpage explains that "the process has to exit before systemd startsfollow-up units" during start, but it's not clear that thisrestriction is also enforced during shutdown.)

At the moment, though, I can't even make my "service" start shuttingdown first, so making sure that any other service waits for it tofinish shutting down is rather moot.

Re: controlling systemd shutdown order

anne wrote:

... but you're barking up the wrong tree.

Yeah, I guess you fooled me. Not providing relevant information, and not sharing the purpose of the script after two requests will have that effect. Good luck - I'm not sure anyone will care to try to help you if you persist like this.

Re: controlling systemd shutdown order

Trilby wrote:

anne wrote:

... but you're barking up the wrong tree.

Yeah, I guess you fooled me. Not providing relevant information, and not sharing the purpose of the script after two requests will have that effect. Good luck - I'm not sure anyone will care to try to help you if you persist like this.

I'm sorry, I'm not trying to be mysterious or difficult, but Iam trying to understand systemd; to some extent this is anexercise for me for that purpose. I'm eventually going to needto make sure that production systems get their services shut downcleanly: apps before databases, etc. This seemed as though itwould be straightforward, based on the manpages, and quite possiblyit will be, and perhaps I'm stuck now only because the "service"I'm trying to depend on isn't really a service - though I thoughtthat most things were services.

The actual problem I'm trying to solve with this is not thatimportant in itself, but, since you folks really want to know:

I start my X session manually with "startx", and I'd like it toclose down gracefully, with certain applications (xterms) beingkilled first, with a short sleep between each kill. The originalreason for this was that I would get hangs during system shutdownand even during first login after a reboot, which I eventually tracedto contention by tcsh over the .history.lock file. That particularproblem was eliminated for logging in by adding code to .cshrc toblow that file away if it persists for more than one second, toavoid .login blocking indefinitely trying to obtain this lockfile.I also reduced the DefaultTimeoutStopSec to reduce the pain when thishappens on shutdown. Finally, if I exit X gracefully, all's well: myxinitrc now has a script that runs right after the preferred clientexits, which kills the xterms one by one to reduce the chances oftcsh tripping over itself with the lock files. (It also looks cool tohave the windows go away one by one, but that's neither here nor there!)

Of course, all of this suggests a bug in tcsh whereby file locking isnot done correctly, but it's beyond my ability in the short term totrack that down, and my workarounds work well enough. If I manageddesktops in production, I'd have to dig into why tcsh doesn't cleanup its .history.lock file properly, but fortunately, in productionI manage only servers, where I'm unlikely to encounter this problem. ;-)

In any case, my original problem of tcsh hangs on my home desktop hasbeen worked around as long as I exit X before shutting down the system.

However, while working on that problem, I became determined to learnhow to make systemd run a script before killing user processes,with the intention of having a script kill my preferred clientand wait for the X server to disappear. I have a script thatdoes exactly that, but no matter what I do, I can't seem to makesystemd's killing of the user processes wait for my script to run.

I defined a service and tried various dependency and orderingstatements to try to convince systemd to stop my service beforekilling the user processes. "systemctl status" shows my Xorg andxterms and other X apps, as well as my shells, as children of"session-c2.scope", which is in turn a child of "user-1043.slice",which is a child of "user.slice". So in desperation I threweverything at it and tried:

... hoping that my service would be stopped before the user slice wasstopped. No go.

I also tried alternative approaches where I made a service that waswanted by and to be started before the shutdown.target, where thestart of the service would have been intended to run my script.(By this point, I was simply creating a bunch of bogus units thatcalled "/usr/bin/logger" for ExecStart, ExecStop, ExecStop, andExecStopPost, just to see what was happening.) In pretty muchall cases, the starts happened at the right moment, and the stopshappened later than I wanted.

Finally, I really do appreciate efforts to help; people's time isvaluable and I don't want to waste it. But with all due respect,I did provide relevant information in my original posting.I explained that I was able to define a service and my ExecStopwas duly called during shutdown, but too late. I asked for help indetermining a dependency that would cause my service to be stoppedbefore the "Stopping Session" tasks started to kick in.

I assumed that I was missing some simple piece of knowledge withrespect to what triggers the user process kills during shutdown, andthat if I had this piece of knowledge, I'd be able to write a unitfile to call my script at the right moment.

The reason why I want to call a script before user processes arekilled isn't all that important: I might be taking system diagnosticsnapshots, for example, or automatically sending e-mail to everyonelogged in, to apologize for an unannounced reboot. No matter whatarbitrary reason I have, it should still be possible to do this, andI'm quite astonished that it has turned out to be so difficult.

If someone knows what triggers the user session stops, and how todepend on it, I hope that they will share that knowledge.

Thanks to all who've piped up, and I apologize for annoying some of you.

Re: controlling systemd shutdown order

ExecStop= doesn't seem to wait for the process to finish (even on type=oneshot) so other units might kick in earlier. Even though 'After=multi-user.target' starts the unit very early in the shutdown process already the script invocation itself (*.sh) might need too much time so other (parallel started) units might come in too fast. Thus the 'sleep 1' that keeps them away (oddly, this is not parallelized. The 'sleep' properly blocks the whole systemd). Just a dirty hack but i haven't found anything proper either.

Edit: Correction. Its not reliable. 'After=multi-user.target' still starts it too late sometimes.

Re: controlling systemd shutdown order

Brebs, I had already tried using shutdown.target to start the script, but it still starts too late.

Damjan, I was able to start the service very late in the boot sequence, but still couldn't get it to start shutting down early enough.

Maniaxx, my experience mirrors yours, that even "After=multi-user.target" isn't sufficient to make the shutown start first. I'm glad that you're also finding that "ExecStop= doesn't seem to wait for the process to finish (even on type=oneshot)", and I really wonder whether or not "oneshot" should wait on shutdown; as I recall from studying the manpage, no explicit statement is made one way or the other. I might try to find a spot to report that omission; perhaps the systemd people can at least clarify what's supposed to happen there. I'm intrigued by the idea that the "sleep" blocks everything, though - I'll experiment with that in case it gives me useful ideas.

I probably won't work on this again for the next few weeks, but if I get anything to work, I'll report back.

Re: controlling systemd shutdown order

Maniaxx wrote:

ExecStop= doesn't seem to wait for the process to finish (even on type=oneshot) so other units might kick in earlier.

anne wrote:

Maniaxx, my experience mirrors yours, that even "After=multi-user.target" isn't sufficient to make the shutown start first. I'm glad that you're also finding that "ExecStop= doesn't seem to wait for the process to finish (even on type=oneshot)", and I really wonder whether or not "oneshot" should wait on shutdown; as I recall from studying the manpage, no explicit statement is made one way or the other. I might try to find a spot to report that omission; perhaps the systemd people can at least clarify what's supposed to happen there.

Uhhuh. Surely it does (per the ordering dependencies). It's just specifying After=multi-user.target in a unit WantedBy=multi-user.target is creating an ordering dependency loop (because if a target Wants= a unit, then that unit is implicitly Before= that target — unless you say DefaultDependencies=no), and systemd has to throw out one of the conflicting dependencies. Unfortunately, it does not have any concept of "priority" to prefer throwing out an implicit dependency — so it may very well prefer the implicit one and discard what you wrote explicitly.

Now replying to the original thread. Anne, you seem to want to run your unit before the session scopes are killed, correct? This, I'm afraid, is impossible with current systemd: session scope units are autogenerated, and there is no way to specify an After= dependency on all of them.

However, the dependency mechanism per se is not flawed in any way, it works, and if you magically knew the name of the needed session scope, then you could simply write After= to that scope (and After=user@<uid>.service to work around an unrelated bug which is going to be fixed before next release when LP reviews my pull requests), and your oneshot unit will be stopped before systemd even attempts to kill that session.

You can try to propose a RFE to make systemd add an "include" statement to the autogenerated session scope units, so that you can put your own snippets with ordering dependencies into that file and have them included in all autogenerated session scope units.

Re: controlling systemd shutdown order

intelfx wrote:

Uhhuh. Surely it does (per the ordering dependencies). It's just specifying After=multi-user.target in a unit WantedBy=multi-user.target is creating an ordering dependency loop (because if a target Wants= a unit, then that unit is implicitly Before= that target — unless you say DefaultDependencies=no), and systemd has to throw out one of the conflicting dependencies.

are you sure about this? the systemd.target man page says:

AUTOMATIC DEPENDENCIES
Unless DefaultDependencies= is set to no in either of related units or an explicit ordering dependency is already defined,
target units will implicitly complement all configured dependencies of type Wants= or Requires= with dependencies of type
After=. Note that Wants= or Requires= must be defined in the target unit itself — if you for example define Wants=some.target
in some.service, the implicit ordering will not be added.

Re: controlling systemd shutdown order

i think anne's request is very reasonable, which is to say i had the same desire and i am still dismayed that systemd has the attitude it has towards processes. in essence, it demands that every single process be part of its grand plan for the universe or it kills it off immediately. Yuck.

anyways, one thing I found that at least shines a little light on that is "systemctl status <pid>", which shows you which unit it thinks the process belongs to. so if you want to make sure your unit's ExecStop runs before a specific process is killed, you can at least find out which unit you have to depend on.

but i think madness is at the end of that road - systemd really doesn't want to admit there's a value to processes it doesn't explicitly know. so i counsel letting systemd know the exact dependency by putting the process that you care about inside of a unit. i don't like it, but it is the systemd way, and for as stubborn as systemd is, it is not hard to make a new unit. so instead of starting your process directly, you would wrap it in foo.service and do "systemctl start foo.service" to start it, or something like that.

assuming it can run without the terminal, right?

that's easy to do if you're running everything as root. if you need to depend on a non-root process, i'm not entirely clear on that, but systemd.exec man page's User= looks promising? supposedly polkit can let you use systemctl as a user..

Re: controlling systemd shutdown order

oh and as an aside, once i finally got the Wants=/Requires= and After= lines down how i needed them, i found systemd totally reliable about waiting until the Type=oneshot ExecStop= command has finished before stopping the next unit. so i think once you find the right thing to depend upon, it really will wait until your script is finished.