I'm going to stop the access to all the servers in a couple of hours, and this is probably going to be for a good part of the weekend because I'm planning to do a complete evaluation of how things are, including operating system upgrade, drive changes, backups verification, etc...

In the meantime there will be a secondary server serving a status page to show what is happening.

I know it's annoying, but it's probably less annoying than these freezes that happen now and then.

15:00 - Server offline and unplugged
15:30 - Disk #2 content have been copied onto Disk #1
16:00 - Disk #2 is now replaced by a new and much larger SSHD drive
16:30 - Formating the SSHD to Ext3 format
16:40 - Copying the content of Disk #1 to the newly formated SSHD
18:50 - ...still copying...
19:10 - Done copying
19:15 - Did some fstab magic to point the old drive to the new drive and rebooted the system
19:39 - Switched back the port forwarding to Miniserve

I'm done for today, but I'm going to continue tomorrow, this time this will be the operating system upgrade/fix (including php and phpbb to the latest version), and then reworking all the backup processes.

Normally we should not have lost anything, and all sub-domains should be up and running, and SVN(*) should be ok as well.

You should be able to use everything now until sometimes tomorrow morning when I will start messing with stuff again

Not sure how many of you actually saw the "work in progress" log, I tried to maintain it while stuff was happening.

Would be interesting to see if the freezes are gone, if that's the case then it was probably one of the harddrives having issues.

(*) anyone trying to put one of the SHA1 pdf on the SVN depot will be dealt with extreme prejudice

Regarding the performance, I'm not sure why it had got better.
Technically compared to before the maintenance what has changed is:
- The server is not in the server room anymore (it's on my desk), so different power, ethernet cable, no UPS, etc...
- The files have been moved from the old 750 GB drive to a new 2TB SSHD

I guess that could have been some hardware issue, maybe the HDD on the brink of dying?
I was running the performance tool to see if there was any spikes in CPU when the server was not answering, but from a UI point of view everything seemed fine.

So, the second stage is done, which means we now have the following improvements:
- The system disk was upgraded from a (getting old) Intel X25 to a Samsung 540 SSD
- The storage drives have been upgraded from a standard (and getting old as well) 7200 RPM 750GO HDD to a 2TB SSHD
- Operating system upgraded from a broken Ubuntu 12/14 to a fully up-to-date 16.04 LTS
- PHP upgraded from version 5 to 7
- PhpBB upgraded from version 3.0 to 3.2 (we have unfortunately lost the skins, will have to be redone)
- DokuWiki had not been upgraded since 2006, it's now running the very last version

For the timeline of the events:

11:00 - Server offline and unplugged
11:10 - Installed the new SSD
11:15 - Full update of the OS (Ubuntu 16.04 LTS)
11:50 - Did a new backup of the phpbb database
13:20 - Fighting with php, samba and mysql
13:44 - Trying to get webdav/svn up and running
14:25 - mmuman confirmed that the SVN access was working
14:55 - Still fighting with phpbb which is not happy with my version of php and mysql
16:10 - Only took 3 hours to figure out that I should have replaced $dbms = 'mysql'; by $dbms = 'mysqli'; in the config.php file...
16:20 - Updated from phpbb 3.0.12 to 3.1.10 (lost all my visual configuration... snifff... :'(')
16:40 - Updated from phpbb 3.1.10 to 3.2.0
17:00 - Tried to get the styling sorted, and failed, so it's going to stay default blue for a while
17:20 - Working on adding the crontab scripts for the dynamic dns
17:45 - Updating Doku Wiki to the latest stable release. It's not like if it was 10 years old...
19:00 - Correctly setup the .htaccess apache2 configuration
19:30 - Time to rescrew everything

We are running without any backup script, I did not have time to do that, but hopefully everything should be running.

So your mission is now to go around and poke at everything to try to find what I forgot, what is broken or missing, being pictures, messages, pages, redirects, subdomains, etc... the faster we find stuff the better it will be.

Thanks for the huge effort to keep us all updated and the systems all on course .. looking pretty good so far. The only thing I miss is the "New Posts.." link that used to be available all over the forums, but I guess I can set up smart filters with the new phpBB .. anyway, thanks again for the work! defence-force.org is a great destination every week ..