I just wanted to give you all a quick and more technical update on what we have been dealing with over the last week.

Our project has been running for upwards of 10 years now and we have been crunching literally billions of workunits over those years. As a result of all of your hard work and dedication, we have actually calculated enough results that we have run out of room to store the IDs of all of these results in a normal unsigned integer value (the default data type used for storing IDs in BOINC databases). As a result, on Tuesday night, I updated our database to be able to store IDs in a much larger data type to prevent this issue from happening again during the remaining life of the project. As a result, I also had to quickly patch the BOINC code we run on the server to allow it to use this newly available data type in the database.

During this process, I missed one of the foreign keys that refers to the results, specifically in the validation process. This led to an issue in validation of work units over the last couple days. I had trouble diagnosing this issue because before the workunit queue clear, everything was running fine however, after the workunit queue clear, when new work with large IDs was being returned, it silently failed to validate workunits.

I have implemented a fix for this last issue and my hope is that things will be smooth sailing from here on with regards to this issue. Of course, there is always the possibility I missed another bug in this pipeline, or I might realize I need to force validation on workunits received over the last couple days. In any case, I will be watching the server pretty closely over the next couple days to make sure its running correctly.

Sorry for all of the trouble over the last couple days and I hope you all continue to support the important research we are doing into the future.

Thanks for the update Jake. Ask me sometime about when I ruined a university license server with a bit of misjudged scripting.

Just to be clear: how can we help you get things up and running smooth again--Should we clear out all currently-downloaded tasks and start fresh, keep crunching what we've got at the moment even if they're not validating, or some other option I've not considered yet?

I just got the validator fixed. I've tested it on a few workunits and I watched it validate them. I'm going to go ahead and try to come up with a query to tell the validator to recheck all validation inconclusive workunits as well as workunits that were errored for running too many runs. You should see a massive validation run here in a few minutes and you should see retroactive credits even on dead workunits.

As a note some units which we cancelled over the last couple days still will not validate as they were cancelled before a consensus could have been reached on the correct answer.

I just got the validator fixed. I've tested it on a few workunits and I watched it validate them. I'm going to go ahead and try to come up with a query to tell the validator to recheck all validation inconclusive workunits as well as workunits that were errored for running too many runs. You should see a massive validation run here in a few minutes and you should see retroactive credits even on dead workunits.

As a note some units which we cancelled over the last couple days still will not validate as they were cancelled before a consensus could have been reached on the correct answer.

Jake

Good luck and I hope it's gonna work, fingers crossed for you guys and I hope you have a good sunday regardless!

I just got the validator fixed. I've tested it on a few workunits and I watched it validate them. I'm going to go ahead and try to come up with a query to tell the validator to recheck all validation inconclusive workunits as well as workunits that were errored for running too many runs. You should see a massive validation run here in a few minutes and you should see retroactive credits even on dead workunits.

As a note some units which we cancelled over the last couple days still will not validate as they were cancelled before a consensus could have been reached on the correct answer.

Jake

Good luck and I hope it's gonna work, fingers crossed for you guys and I hope you have a good sunday regardless!

UPDATE - I now have VALID results and a number of units PENDING. My number of Inconclusive hasn't changed ... most of them were completed within the last 24 hours. THANKS!

I've had a couple of systems running MW on autopilot for the last few days while I address other issues. I just got back to all the commotion. WOW!! I don't know about the others, but I crunch for the science, not for the worthless credits. So, I just brought 4 more machines back onto MW because so many were whinning they were dropping out.

MW and Einstein were my first BOINC projects and they are still my favorite GPU projects.

Thanks for the hard work Jake and keep you head up, but duck when you have to!!!

I just checked all of those listed workunits. It seems the workunits that is still waiting to validate is de_modfit_fast_19_3s_146_bundle5_ModfitConstraintsWithDiskandUpdateStreams_1_Bouncy_3_1508646134_281802_0.

I can confirm this one is just waiting for its sister result to return to check your answer against.