How To Restore Broken Arch Linux To Previous Working State

If you’re a Arch Linux user, you probably know that the recent Linux Kernel 4.13.x branch is causing so many issues. Yesterday, I did a full system upgrade using command “sudo pacman -Syu”, and ended up with a broken system. My Arch Linux won’t boot into graphical mode, and kept displaying the error messages “dependency failed multi-user system” and “dependency failed for graphical interface”. The only option I had is to login to single user mode and try to rollback the updated packages to their previous versions.

From the single user mode, I tried the following things:

I can’t install any packages from official repositories. My Network card is not recognized in single user mode, so Internet didn’t work.

The “downgrade” command doesn’t work. I have no idea why. So, I couldn’t downgrade any packages to their previous versions. It simply displayed an error message “sudo pacman command is not found”.

I was helpless. All I have is a Tablet PC with an Internet connection. After extensive search on Arch Linux wiki and forums, I thought to give a one-last try. I decided to downgrade all the packages which are previously updated one by one. Because, the “pacman -Syu” command updated many packages, so I couldn’t confirm exactly which package is causing the problem.

If you’re in this similar situation, this guide might be help you to get a working Arch Linux system back.

Restore Broken Arch Linux To Previous Working State

First, login in single user mode. To do so, enter “e” when you see the Grub menu.

Then, press F10 or CTRL+X to continue. After couple seconds, you will be landed into single user mode.

Type the following command to mount your root (/) file system in read/write mode.

mount -n -o remount,rw /

Now, you need to find when you did the full system upgrade. You can easily find this by looking at your pacman.log file. This file saves all pacman entries you did in the past.

tail -n 200 /var/log/pacman.log | less

Here, -n 200 will display the last 200 lines in your pacman.log file. Because, my pacman.log file so many entries. As you may know, tail command will display the last 10 entries only. So, replace 200 with your own number to go through the pacman.log file. And “less” option displays the results page by page.

After going through the pacman.log file, I found the exact time of the full system upgrade. Check the timestamp in the below screenshot (Sorry for the low quality image). I have run “sudo pacman -Syu” command at 12:06 pm.

As you can see, “pacman -Syu” command has updated 40+ packages. Since, I can’t use “downgrade” command and downgrade all packages to any specific date, I had no choice, but to downgrade all updated packages one by one using “pacman -U” command from the cache. As you already know, pacman saves all downloaded packages in /var/cache/pacman/pkgs/ folder.

I noted down all updated packages from the screenshot and downgraded them one by one to earlier version from the cache like below.

pacman -U /var/cache/pacman/pkg/qt5-base-5.9.1-3-x86_64.pkg.tar.xz

The above command will install qt5-base package from the cache folder. It took me nearly half an hour to downgrade all packages. If you cleared the cache folder already, you have no choice. Reinstallation might be the only option. So, I recommend you to keep at least two old versions in the cache, so you can restore them if there are any problems in the new packages.

After downgrading all packages, type the following command to apply the changes and start your Arch Linux in to normal mode.

exec /sbin/init

Now, I can be able to login to my Arch Linux desktop without any problems. I think I should hold a couple of days and then try to update again when this is fixed.

If you have time and patience, downgrade one package and run “exec /sbin/init” to boot your system into normal mode and check if it solves the problem. If not, try another package. Do this until you find the problematic package. Then simply leave the problematic package and update all other packages.

Some of you may find this method is not so good and/or complicated. You might know some other easy ways than this. If there are any such methods, please let me know in the comment section below. I am open to learn new methods.

And, that’s all. I hope this helps. More good stuffs to come. Stay tuned!

One of my personal work habits is to always do a backup of the root filesystem before doing kernel or graphics device driver updates. It takes me 10 minutes with Clonezilla (backup and verify), and when the kernel horks I do a 5-minute restore and put an ignore on the update and wait for the next release.
This has saved me much time and much grief over the years.