The backup is not working anymore

Hi. I write this to you cPanel staff because I would like to avoid a law suit for losing data for hundreds of websites.

After the recent backups to WHM 11.48.0 (build 11 and earlier), something went wrong with the backup system, and with other things too.

But I am interested to fix the backup system.

So I had this server installed since over two years ago with no problems. Everything was working great until Tuesday 10th of February 2015.

On Tuesday, I accessed whm and I received an update inform about mod_security and I went through.
Once I went through this setup, my server has started to malfunction. Every request took ages, but there was no high load on the server.
I tried to fix it by disabling firewall, I disabled mod_security, I reinstalled apache, I changed settings in apache, I have done several tweaks for about 15 hours and the results were the same.
Whenever I restarted cPanel service, everything were working great for the first 5 minutes and then everything started to respond harder and harder, up to non-responding.

I mention that there was no flood or something because I also use an external firewall and I was looking at the requests on tcp and udp.
So no abnormal requests on network, MySQL, ftp or apache.
No higher than 2 load on server, but still everything went wrong.

So I decided that is time to reinstall the server to have a fresh one.
In order to avoid complications and because I didn't had the time to wait for a forced backup to complete, I used some older backups to restore my accounts.
By older I mean that I used backups generated on Sunday February 8th 2015.
Logic, it supposed to be safer because problems appeared on 10, it should been better anyway to use those old backups.
I also had backups from 4th of February, but I checked now and they still have the same errors.

So what is the problem here?

After I restored the accounts, I noticed that websites has started to act different than they suppose.
For example, cPanel has decided not to back-up some of the files for some websites.

Some websites are missing the databases completely
Some websites came with corrupted tables of databases
Some websites came with missing email addresses
Some websites came without subdomains
Some websites came without the public_html files.

One website was missing most of the files inside the public_html folder and some databases since August 2014.
I only could found a backup in august 2014 to have all the files and the databases. That was a luck because this customer didn't updated his website.

Some other websites, had replaced files with older (I don't know where it took it from).
So inside the backup of 7th of February, it was a contact.php file that was removed from server on January 5th. .... But the new contact.php page was missing. How can that be possible?

Anyway... everything was messed up, but somehow I managed to fix some websites.

But the real problems are coming now.

1: I decided to install a second server and to move one of the websites to this new server for doesn't matter reasons.
So I installed the 2nd new server and used transfer told to transfer that website from the new server #1 to the new server #2.
Both servers with fresh installation since two days ago.

After 5 hours of restoration, this website came on the server 2 with some stuff missing like:
1: over 800 subdomains were missing
2: the php files of the website were missing (but the pdf, jpg, zip... were there)
3: some of the databases including the database of the main website.

But everything was in place on the 1st server.
So after the restoration was complete, I trusted cPanel and I removed the website from the server 1.
I could not access the website on server #2 because I had to move the IPs of the nameservers from one server to another, and cPanel is not smart enough to understand that an ip is free to be used once is not mounted on another server.
Which this is another issue.
So I had two IP on server #1, I removed them from there and then I added them to server #2. But they were not working.
I tried to use these IPs on a windows server and everything were great.
I used the IPs on another centos machine and everything was great.
But with cPanel, I had to wait 4 hours for something to refresh there or I don't know what you do and why you do that.
When I add an IP in cPanel, if it has been used by another cPanel for the last 4 hours, it will not work. ... That’s another issue that you must fix because I met this since 5 years ago.

Now, regarding the website I moved, many things are missing and I checked with the transferred archived and everything is there.
I am lucky that I kept the cpmove backup from the transfer, otherwise I would been losing all the files.
So why in the hell are you not restoring all the files if they are to be found inside the backup? a backup made by cPanel 2 hours earlier on another fresh installed machine.

Now let’s go to the second problem.
I have about 235 more websites that was backed up without databases.
I checked with the backups from 7th of February and there are no databases
I checked with the backups from 4th and there are no databases.
Backups before 4th are irrelevant.

So where are the databases? Why were they not been included in the backups?

So what I need is someone from cPanel to contact me and fix all these things.
My customers are patient until Monday 8am GMT time. If their websites are not working proper, they will all sue cPanel because you messed up with the backup system. And they are a few hundred sues.

Staff Member

On Tuesday, I accessed whm and I received an update inform about mod_security and I went through.
Once I went through this setup, my server has started to malfunction.

Click to expand...

Did you enable the Vendor Rules, is that what you went though?

I'm not sure I'd have gone to the extreme of deciding to reload the entire server from backups after an update that didn't go as planned. Yikes. Stop what you're doing on the existing server with the updates issue, and open a ticket.

So what I need is someone from cPanel to contact me and fix all these things.

Click to expand...

You should open a ticket to cPanel Technical Support. From within your WHM, in the top right corner is an life preserver icon, click that. On it, click Contact cPanel.

It's Saturday, Technical Support via the ticket system is suggested over waiting on a forum reply from someone.

hello , i have same issue with my cpanel , so how was your problem right now ? is that any update from your trouble ticket ?

Click to expand...

Hi. they didn't solved anything.
I gave them access in my server, i explained to them where are the old files, where are the new files. What to do, what not to do.

I explained to them each problem again and again and i only received emails stating things that i already knew or general things.

Like for example i told them "the backups were corupted, please tell me why and how to fix this.
And they replied to me like " the backups were corupted"... I guess they have seen a bug that may cause many law suits and they act dumb. they act like they don't understand what i repeat several times.

I have no ideea of why if i already knew that the databases are not working and i was the one to tell them that and i was the one that came to them for answers. instead of giving me answers they were giving me the same result "databases are corupted". No kidding !!. thanks for letting me know that

I will reply again to this thread explaining on how i did solved the problem and what i have found to may be the cause.

As cPanel had no idea of why this happen and how to fix it, i made my own researches:

1: Why this happen?
To be honest, i don't know exactly.
But i did noticed some things as:

- CPanel make the backup as a user. If your files were uploaded by root, it will not be included in backups because it has no rights. So this may be a reason.
CPanel has introduced this security back in September and since then many backups has suffered.

- This security is however not well optimized and there are some bugs in cPanel.
For example if you create databases with a script or as root, or from a reseller with root access, or sometimes if your user has the same password with root (even if it’s a user), that database might not be included in backup.
I don't know exactly why, because databases must be owned by MySQL to work properly. Otherwise you cannot do anything with that database because it cannot be executed by MySQL.
So, somehow cPanel create databases for owner MySQL, but it keeps some information that it might be owned by root.
It’s a bug which i believe they had no idea about it. Otherwise it would been fixed.

- Because of the same security, uploaded files are also affected, but there is another bug too.
Sometimes, cPanel forget to include in the pack files in folders that contains files owned by nobody.
So if your folder /public_html/ contains another folder like /public_html/cache/ and inside /cache there are html files owned by nobody, your entire /public_html/ folder will be omitted.
Not always, but sometimes. I tested this several times.

- Another reason for not having all the files inside your backup, it is another security from cPanel "Restricted Restore", which i don't know if its active when is making backups, but i noticed that sometimes some files are omitted from backup in purpose. The exactly same files that are omitted on restoration.

- Not let’s pretend your backup contains all the files, but when you restore there are missing files.
This is because of "Restricted Restore" which sometimes is active by default on "backup restoration" and sometimes is active on "restore a full backup cpmove" even if you uncheck it.
I had a 48GB backup and cPanel has restored everything except .php and .js files. Somehow cPanel considered that these files should be restricted.
So another bug to be fixed.

- And the last reason for not including some files in your backup, is when you have WordPress files. There might be two reasons for doing this:
1) It happens the same thing with nobody or root files
2) They are paid but competitors of WordPress to make things hard for webmasters so another platform to overcome. It is possible i have seen it happen with oscommerce a few years ago.

- Another problem you may have is with the corrupted tables.
CPanel told me that the databases were corrupted because of hardware failure, which is untrue because i use the exactly same hardware for this new server and everything works like a charm.
My guess is that this problem with corrupted databases came from CSF which acted strange since the update from 6th of February or so.
There might also be because of WHMextra and some other plugins installed from there.
I was able to replicate the same problem on a test server by installing CSF and WHMextra with all plugins... etc and in two days i got my databases corrupted.
So i guess there might be some conflicts with something. But i don't know exactly for sure what.

1: If you have a corrupted backup and it cannot be restored because is corrupted, download it into your computer, extract the files with WinRAR and then repackage the extracted files with 7zip in .tar.gz format.
Then use "restore a full backup cpmove" to restore the files and don't forget to uncheck the "restricted restore" security when you do the restoration.

HINT: After you extract the files from backup in your computer, make sure you don't get doubled folder.
If your backup name is username.tar.gz you should have a folder called "username". When you go inside "username" folder, you must see many other folders. But in case there is another folder called "username" and no other folder, it means that you have a doubled folder and the restoration will not work. So you must get rid of the first folder by moving "username" folder out of the first "username" folder.

So you should not have /username/username/allotherfilesandfolders but /username/allotherfilesandfolders

2: If your backup is restoring and inside the backup you have all the files and folders, but when you restore them, files are missing, you must use "restore a full backup cpmove" to restore it and to uncheck the "restricted restore".
If is not working from the first time, terminate that account and try again. It should work 2nd or 3rd time for sure.

3: If your backup is incomplete, you may try these two things:

a) If you still have access to your server, try to back up again using this command on putty window:
/usr/local/cPanel/bin/backup --force

b) If you don't have access anymore to the server or if your backups are still missing files, all you can do is to merge several backups.

So let’s say the username you want to restore is called "john". So your backups should be named john.tar.gz (with or without backup date).

- First you have to find an older backup that has all the files up to that date. (I found backups from August that included all files till August).
- restore to that backup using restore a full backup cpmove and uncheck that protected restore.
- Now you must update your user with new files from more recent backups, following the order of oldest to newest. So the most recent backup should be the last one you process and the oldest should be the oldest one.

How to update files?
1: create a folder called /updaterestore/
You can create it where you want except the username's folder. So i presume you create it in /backup/
So you should now have a new folder that you can call via SSH as /backup/updaterestore/

2: Your website on server should be on /home/john/public_html/
That’s because your username is john. Right?
So you know that all your files should be inside /home/john/

3: After you are sure you identified your user, you must create several folders inside /backup/updaterestore/
If you want to update from 3 backups, you must create 3 folders like:
/backup/updaterestore/1/
/backup/updaterestore/2/
/backup/updaterestore/3/

If you want more, you can do more.

4: Now you must start to extract your backups inside those folders.
/backup/updaterestore/1/ should be the oldest backup and will be processed first
And /backup/updaterestore/3/ should be the newest backup and should be processed last.

5: you must now have these folders:
/backup/updaterestore/1/john/
/backup/updaterestore/2/john/
/backup/updaterestore/3/john/

And inside each one of them you should have other folders like mail, public_html, public_ftp... etc.
Once you have these folders you must start to update your user with newer and newer files.

Please notice that you should not replace other files than /public_html/* and /mail/ because they are not all owned by your user, some of them are owned by cPanel or by MySQL or by mailman and more.
I would recommend that if you want to update other folders that those to, check first the owner and owner group so you will know latter to grant access to the same users.

6: let’s start the update process.
You have two options
- To update files, but to keep the already existing files, or to update files and to replace the existing files (for example, for e-mails is no reason to replace existing files because the e-mails are the same.

These commands will update your files and will only replace existing files from the last backup. (if you wish to. otherwise use no instead of yes too ). And it will not ask you to confirm overwrite or not. It will do it automatically.
The 4th command will grant access to user john to see those new files and write inside folders.

As you can see, for public_html i did it different because if you use /public_html/* it will not copy the .htaccess files or any other files that starts with "."
And i chowned the entire public_html folder to affect .htaccess files too.
And then changed chown of public_html folder ONLY (without using -R ) to nobody.john ( group should be nobody and owner john )
And then changed the mod to 0750 for folder public_html (without using -R ) because this folder should have 0750 rights. Otherwise won't work.
But folders inside public_html/ should be 0755 which will be like that by default.

I don't know if you needed all this, but i took the time to write in case someone else will need to fix this in the feature.