Bug Description

When using default settings for unattended-upgrade i.e.
Unattended-Upgrade::Remove-Unused-Dependencies "false";
# default "false"
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
# default "true"
in configuration file /etc/apt/apt.conf.d/50unattended-upgrades,
unattended-upgrade is unable to remove packages that become unused in conjunction with updating by other software such as update-manager or apt full-upgrade. This is because unattended-upgrade compares the list of unneeded packages before and after it upgrades packages to detect which packages are new unused ones.

Consequently, if user installs new kernels using e.g. update-manager, the excessive kernels will not be removed by unattended-upgrade, and eventually (small) /boot will become full.

Expected behavior: handle removing of unused packages differently at least until other package management software installed by default can handle removing of new unused packages.

- Unable to remove packages that become unused in conjunction with- updating by other software+ Unable to automatically remove packages that become unused in+ conjunction with updating by other software

A fix could be adding line
Unattended-Upgrade::Remove-Unused-Dependencies "true";
in /etc/apt/apt.conf.d/50unattended-upgrades
But does it remove too many packages? Why was
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
setting ever added?

Besides:
If you do not want automatic updates, but just automatic removal of unnecessary software including kernels, you could have 'Download and install updates automatically' set, but edit /etc/apt/apt.conf.d/50unattended-upgrades to have only comments within Unattended-Upgrade::Allowed-Origins section.

@Jarno: IMO Unattended-Upgrade::Remove-Unused-Dependencies is already a risky option and I don't recommend enabling it because it may remove packages which are not used according the to package-dependency chain but which users rely on using software that is not packaged.
The only place I would use this option are dedicated server systems where everything is packaged and which run services only without allowing user shell/GUI logins.

This bug is about removing packages which became unused as a result of running other package management tools, not u-u.

I consider this out of scope for u-u because it can be solved easily by the used other package management tool and this feature will most likely unpleasantly surprise users by removing unexpected packages.

I think less advanced users do not install software that is not packaged. Advanced users can configure system to work as they wish. IMO currently Unattended-Upgrade::Remove-New-Unused-Dependencies is more risky as default.

@Jarno I added update-manager based on my experience with (less experienced) users, who kept their system up-to-date by saying yes to everything regarding updates which popped up on their system.

They never touched apt or synaptic nor installed packages by themselves by other means.

I think removing newly unused packages would be a reasonable default for u-u and update-manager.

I agree that Remove-New-Unused-Dependencies=true is riskier than having it disabled but it still removes a subset of what Remove-Unused-Dependencies=true would thus Remove-Unused-Dependencies=true would be more risky.

Remove-New-Unused-Dependencies=true is needed to clean up old kernel packages filling up /boot (and the whole disk).
If update-manager defaults to that by default, too, the system won't break itself.

I think removing "newly unused" packages is an ugly hack. Deciding which packages are excessive or unneeded should not depend on how some packages were removed previously. If the reason for this automatic remove thing is to get rid of excessive kernels, a specific tool for that could be used. I have written such a tool. It is called linux-purge (https://launchpad.net/linux-purge)

Anyway, it would be good, if other software could optionally remove new unused packages.
I do not consider Unattended-Upgrade::Remove-New-Unused-Dependencies as an reasonable default option for unattended-upgrades before the feature has been released for at least apt, update-manager, gnome-software and possible other related software installed by default.

- Unable to automatically remove packages that become unused in- conjunction with updating by other software+ By default settings unattended-upgrade is unable to automatically remove+ packages that become unused in conjunction with updating by other+ software.

I found the description somewhat confusing because 'Unattended-Upgrade::Remove-New-Unused-Dependencies "true"; doesn't actually exist in /etc/apt/apt.conf.d/50unattended-upgrades, rather that's the default setting in unattended-upgrades since 0.90. Regardless, I was able to recreate the issue on an Ubuntu 16.04 system via the following procedure.

Brian, however, as you installed the kernel by apt-get install, the kernel becomes manually installed, and will not later be removed by 'sudo apt autoremove', unless you change it to be marked as automatically installed.

I question the use of 'sudo apt autoremove' in this case. If its action was the desired one, setting
Unattended-Upgrade::Remove-Unused-Dependencies "false";
would be the setting to use, instead of the current default
Unattended-Upgrade::Remove-New-Unused-Dependencies "true";
Balint Reczey tries to explain in #11 why that setting is not desired.

On Thu, Sep 21, 2017 at 02:03:41PM -0000, Jarno Suni wrote:
> Brian, however, as you installed the kernel by apt-get install, the
> kernel becomes manually installed, and will not later be removed by
> 'sudo apt autoremove', unless you change it to be marked as
> automatically installed.

No, this is not the behavior I observed. After installing a new kernel
version via update-manager or 'sudo apt-get install' on Ubuntu 16.04, 'sudo
apt autoremove' does the right thing and wants to remove my third newest
kernel.

Anyway, in my view
Unattended-Upgrade::Remove-Unused-Dependencies "true";
would be the default setting to have now, and advanced users should mark the dependencies of their unpackaged software as "manual".

The get_auto_removable function in unattended-ugprades just checks to see if the package has the "is_auto_removable" flag set. So setting "Unattended-Upgrade::Remove-Unused-Dependencies" to "true" will produce the same outcome as using "sudo apt autoremove" e.g.:

@Balint re "@Jarno: IMO Unattended-Upgrade::Remove-Unused-Dependencies is already a risky option and I don't recommend enabling it because it may remove packages which are not used according the to package-dependency chain but which users rely on using software that is not packaged."

Could you give me an example of how people would install dependencies for software that is not package that would also set the "is_auto_removable" flag in the apt cache to True? I can't think of a situation like this.

On Wed, Sep 27, 2017 at 1:49 PM, Brian Murray <email address hidden> wrote:
> @Balint re "@Jarno: IMO Unattended-Upgrade::Remove-Unused-Dependencies
> is already a risky option and I don't recommend enabling it because it
> may remove packages which are not used according the to package-
> dependency chain but which users rely on using software that is not
> packaged."
>
> Could you give me an example of how people would install dependencies
> for software that is not package that would also set the
> "is_auto_removable" flag in the apt cache to True? I can't think of a
> situation like this.

I had the opposite situation in my mind. People would remove packages
which are not auto removable, like ubuntu-desktop (for example because
something ubuntu-desktop depends on is acting badly) and then many
package in ubuntu-desktop's dependency change become auto removable:

$ sudo apt-get remove ubuntu-desktop
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages were automatically installed and are no longer required:
a11y-profile-manager-indicator activity-log-manager adium-theme-ubuntu
aisleriot app-install-data-partner apturl apturl-common baobab bluez-cups
...
xorg
Use 'sudo apt autoremove' to remove them.
The following packages will be REMOVED:
ubuntu-desktop
0 upgraded, 0 newly installed, 1 to remove and 7 not upgraded.
After this operation, 46,1 kB disk space will be freed.
Do you want to continue? [Y/n]

It is a simpler problematic situation than the one I originally mentioned, u-u
would remove software which is probably in use, no additional software is
broken.

On Wed, Sep 27, 2017 at 2:42 PM, Julian Andres Klode
<email address hidden> wrote:
> Hmm, I thought we would not do that anymore, and packages get marked as
> manual when removing a meta package, but I might be missing something.

I ran this on zesty, but if meta packages are handled differently then
I could imagine a similar
case when meta packages are not involved.

On Wed, Sep 27, 2017 at 08:36:13PM -0000, Balint Reczey wrote:
> On Wed, Sep 27, 2017 at 2:42 PM, Julian Andres Klode
> <email address hidden> wrote:
> > Hmm, I thought we would not do that anymore, and packages get marked as
> > manual when removing a meta package, but I might be missing something.
>
> I ran this on zesty, but if meta packages are handled differently then
> I could imagine a similar
> case when meta packages are not involved.

I ran the same metapackage removal test on Xenial, so if Julian is right
and this is a regression its a rather old one. Do you have any more
details about your thoughts Julian?

- By default settings unattended-upgrade is unable to automatically remove+ By default settings unattended-upgrade does not automatically remove packages that become unused in conjunction with updating by other- software.+ software

Observe #1267059 about 'Unattended-Upgrade::Remove-Unused-Dependencies' not working as expected for old versions of unattended-upgrades, also resulting e.g. in obsolete kernel packages not getting removed.

* Merge from Debian unstable
- Remaining changes:
- unattended-upgrades: Do not automatically upgrade the development
release of Ubuntu unless Unattended-Upgrade::DevRelease is true.
- Dropped changes, included in Debian:
- Run upgrade-between-snapshots only on amd64.
The test exercises only unattented-upgrade's Python code and uses dependencies from the frozen Debian snapshot archive thus running
it on all architectures would provide little benefit.

unattended-upgrades (1.0) unstable; urgency=medium

[ Simon Arlott ]
* Revert sending mails on WARNINGS when in MailOnlyOnError mode"
* Consider conffile prompts to be errors (Closes: #852465)
Flag packages that have to be upgraded manually because of a conffile
prompt and consider this to be an error when sending email or exiting.

[ Simon McVittie ]
* Add python, python3, setuptools, DistutilsExtra to Build-Depends.
They are needed for `clean`, so Build-Depends-Indep is not enough.
* Add .gitignore and debian/.gitignore
* Remove bzr configuration.
This is unnecessary now that u-u is in git.

[ Balint Reczey ]
* Run upgrade-between-snapshots only on amd64.
The test exercises only unattented-upgrade's Python code and uses
dependencies from the frozen Debian snapshot archive thus running
it on all architectures would provide little benefit.
* Clean up processes started for getting md5 sums
* Don't keep /var/lib/dpkg/status open multiple times
* Adjust candidates in UnattendedUpgradesCache.open()
* Perform autoremovals in minimal steps, too.
Also add check to remove only the set of packages selected for autoremoval.
Without that check unattended-upgrades when (by default) configured to
remove newly unused packages could also remove auto removable packages
which were unused before starting starting the upgrade step.
* Remove unused automatically installed kernel packages
(LP: #1357093, #1624644, #1675079, #1698159)
* Stop including Python syntax in the report (Closes: #876796)
* Do not auto remove packages related to the running kernel (LP: #1615381)
* Check packages to be autoremoved against blacklists, whitelists.
Also check if the packages are held.
* Report package removals in the summary email (Closes: #876797)
* Run upgrade-between-snapshots test with debugging enabled
* Don't create new UnattendedUpgradesCache for checking for autoremovals
.open() refreshes the state in each cache_commit(), this is enough
* Update .pot and .po files
* Update .travis.yml to actually build and test u-u from the repo
* Run only a simple installation test on Travis, the system upgrade
test was always failing