apt-get upgrade, systemd refuses to reload daemons

My dockstar server running debian stretch complains when doing updates. I got this a couple months ago, didn't get any useful hits on the problem, and avoided it since the server appeared to still do everything fine.

I updated just now and got several errors like this:

Warning: smbd.service changed on disk. Run 'systemctl daemon-reload' to reload units.
Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd. Currently, 10.0M are free, but a safety buffer of 16.0M is enforced.

Still, I get relatively few hits on the topic. I don't know what to do about it... it seems I need to have a bigger /run mount, but I'm not sure. Maybe these errors don't matter? They look horrible, and I even get them multiple times for the same daemons as it continues to try to restart them during apt-get upgrade.

Do you have everything in a single file system? If so you are dangerously short of available disk space. You could try running:

apt-get autoremove
apt-get clean

The first command will auto remove packages that were installed to satisfy dependencies for other packages but are now no longer required. The second command deletes all .deb files from /var/cache/apt/archives and can free-up significant volume of disk space. You should also look into what is consuming the available disk space. The 'du' command can be helpful in determining what is using your disk space.

Yes, it is the /run mount. The reason rebooting works fine is that systemd isn't trying to reload anything, simply start the services. I can work around the problem by stopping each service and then starting it as such: [edit] 'restart' works too. the issue is 'systemctl daemon-reload' which isn't called during a reboot

One of the responders ("Michael Biebl") indicates that minimum memory for Debian now is 256MB, which I think is incorrect, the link he points to says actually a minimum of 128MB (and I checked for ARM, as this was a AMD64 link) ... Anyhow ...

A semi-permanent fix, as suggested later on in the same thread above, is to add this to /etc/fstab :
... bla bla bla ...
tmpfs /run tmpfs nosuid,noexec,size=18M,nr_inodes=4096 0 0
... bla bla bla ...

I tested below 18M and it didn't fix the problem, 18M seems to be the lowest for "systemctl daemon-reload" command to still work without complaint (and therefore presumably apt-get install, update, upgrade, remove etc)

Can anyone confirm I have all the correct options - "nosuid,noexec,size=18M,nr_inodes=4096" as that was just a copy and paste, and differs from what @Nematocyst suggests in this thread above :) ... also happy to know if there's "a better" solution ...

Hey bodhi ... yes, I understand, sorry I didn't mean to mislead anyone ! ... the link cited in the debian bugs site ( https://www.debian.org/releases/wheezy/amd64/ch03s04.html.en ) suggests that Debian may run in as little as 20MB in some configs ... the same page for Stretch indicates 60MB ... 128MB is their "recommended bare minimum" for both wheezy and stretch, although they'd generally prefer that we had more RAM.

I haven't tried it. My reported solution didn't involve fstab, b/c I just tried to match what 'mount' reported for /run, only with more memory. The problem with fstab is that it reduces overall resources elsewhere permanently, instead of during upgrades only.

> so happy to know if there's "a better" solution ..

the nr_inodes discussed in that thread might be a good idea instead of the default. I don't know.

However, I have since removed systemd from my server, and I no longer need any workarounds.
edit: this may not be advisable for everyone. after removing systemd, I had a power outage, and the dockstar failed to boot. I have modded my dockstar so that it has an easily accessible serial console, and I was able to figure out the problem-- one of the mounts was FAT32, and the mandatory fsck ran out of memory! anyway, be careful with this solution unless you have a way to access the console, netconsole should be ok

rayknight Wrote:
-------------------------------------------------------
> I ran into this issue again after upgrading a
> Dockstar from jessie to stretch and decided to try
> to find a more permanent solution. I resolved it
> as follows:
>

Revisiting this issue yet again as it keeps popping up on my systems with only 128MB RAM. It appears that an update to initramfs-tools has a new configurable option in the /etc/initramfs-tools/initramfs.conf file. This option is RUNSIZE and is set to the default value of 10% which means less than 12MB will be available in /run on a 128MB system causing issues when reloading services. So to permanently resolve this problem take the following steps:

Edit /etc/initramfs-tools/initramfs.conf file and change the RUNSIZE=10% line to RUNSIZE=20M

This will set the size of the /run mount point to 20M which should be more than adequate to ensure the 16M needed for restarting services is available. The next two steps will ensure the updated size of the mountpoint is in your uInitrd image. After the reboot your /run mount point should show a size of 20M.

NOTE: This should only be necessary on systems with less than 256M of RAM as the 10% default would be adequate for systems with 256M or more.

rayknight Wrote:
-------------------------------------------------------
> Revisiting this issue yet again as it keeps
> popping up on my systems with only 128MB RAM. It
> appears that an update to initramfs-tools has a
> new configurable option in the
> /etc/initramfs-tools/initramfs.conf file. This
> option is RUNSIZE and is set to the default value
> of 10% which means less than 12MB will be
> available in /run on a 128MB system causing issues
> when reloading services. So to permanently
> resolve this problem take the following steps:
>
Apparently the updated initramfs-tools package is not available in stretch its only available in buster and sid!

bodhi Wrote:
-------------------------------------------------------
> > Apparently the updated initramfs-tools package
> is
> > not available in stretch its only available in
> > buster and sid!
>
> That does not sound right! It is available on
> Debian stretch.

bodhi the updated package version 0.132 is only available on buster and sid. Version 0.130 is the latest available for stretch and doesn't have the new RUNSIZE parameter which can be modified to increase the size of the /run mount point to prevent the "Failed to reload daemon: Refusing to reload, not enough space available on /run/systemd" messages.

There are ways to install buster packages on stretch, but it would be nice if they would include this package in stretch updates.

rayknight Wrote:
-------------------------------------------------------
> bodhi Wrote:
> -------------------------------------------------------
> > > Apparently the updated initramfs-tools
> package
> > is
> > > not available in stretch its only available
> in
> > > buster and sid!
> >
> > That does not sound right! It is available on
> > Debian stretch.
>
> bodhi the updated package version 0.132 is only
> available on buster and sid. Version 0.130 is the
> latest available for stretch and doesn't have the
> new RUNSIZE parameter which can be modified to
> increase the size of the /run mount point to
> prevent the "Failed to reload daemon: Refusing to
> reload, not enough space available on
> /run/systemd" messages.
>
> There are ways to install buster packages on
> stretch, but it would be nice if they would
> include this package in stretch updates.
>
> Ray

Ah. In that case, instead of using the initramfs-tool, couldn't you just use fstab tmpfs to deal with this?

DonCharisma Wrote:
-------------------------------------------------------
> Can anyone confirm I have all the correct options - "nosuid,noexec,size=18M,nr_inodes=4096" as that was just a copy and paste, and differs from what @Nematocyst suggests in this thread above :) ... also happy to know if there's "a better" solution ...
>
I added the above with size=32M to my Seagate Dockstar running debian stretch (see below) and so far it works. I will update my Seagate GoFLEX Home a few days later on if this addition doesn't give any problems.

Please, enter the code that you see below in the input field.
This is for blocking bots that try to post this form automatically. If the code is hard to read, then just try to guess it right.
If you enter the wrong code, a new image is created and you get
another chance to enter it right.