Since last I wrote a lot has happened. Looking at the traffic graphs it's like feast or famine - either we are unable to create/send out workunits, or we're sending out as many as we can fit through the pipe. Mostly it's been the usual gremlins.

However regarding the past 24 hours it was a new problem: the result space on the upload server filled up unexpectedly, which would have been fine except this (perhaps) inspired some RAID freakout on the system. We couldn't really sort it out until this morning. From the looks of it we had something like a six drive simultaneous failure. Jeff and I beat on it for a while - we eventually assumed this was just a hardware blip, and the data was more or less intact on the drives, but the RAID metadata got a little screwed up. Long story short we were able to carefully bring down the RAID and recreate the meta devices from scratch with the data intact, and all was well. Phew. For the record we do have a virtually-up-to-date result storage backup at all times in case of catastrophic failure on this system.

In any case, the main culprit was our disks filling up, so as I write this we're keeping the project down until major queues drain and the constituent workunit/result files can be deleted.

On a more happy (perhaps) note, yesterday the core group of us were in the same place at the same time (which is rare) and we had an ad hoc meeting about our current project status/plans, especially in light of many recent server problems, increasingly random schedules, and embarrassingly low funding. We're all kind of tired and beaten up and wanting some results already - so I like to think this paved the way for several large and ultimately positive changes in the future.

Also Jeff has been working on this nagging mysterious problem where some of our raw data files are only getting partially processed (which vastly increases our "burn rate" and leads to unexpected workunit shortages). He found some major clues today, and we brainstormed why this is happening and what the exact effect is. At least there's a smoking gun on that front.

- Matt

____________
-- BOINC/SETI@home network/web/science/development person
-- "Any idiot can have a good idea. What is hard is to do it." - Jeanne-Claude

Thanks for the update. You guys do a "Yeoman" job. I am curious, what would it cost for a new server? It seems that if the people who seem to be venting the most are those who haven't given cash. That is not to say they haven't donated time and electrical costs, but the project could use more of the cash. Maybe a "fundraiser" for a new server or two would shake loose a few dollars that may make your lives a bit easier.

Again thanks for the update and the day to day work you do to keep the project going.

and we had an ad hoc meeting about our current project status/plans, especially in light of many recent server problems, increasingly random schedules, and embarrassingly low funding. We're all kind of tired and beaten up and wanting some results already - so I like to think this paved the way for several large and ultimately positive changes in the future.

We all await to hear further, it sounds positive.

He found some major clues today, and we brainstormed why this is happening and what the exact effect is. At least there's a smoking gun on that front.

Matt, thank you very much for the update, it really shed's a lot of light on the recent problems. Here's hoping you guys get it running smooth again, or as smooth as you can for a bit. LOL Have a great rest of the week!

Thanks for the update and the great job you guys do to keep things running.

Since Blurf decided to reduce his involvement we don't seem to have anyone to coordinate hardware (or other) funding drives. Is is possible to get the list of required equipment updated so we have an idea of what is currently needed please.

On behalf of the SETI@Netherlands team I would like to thank you for all your hard work you are putting in.
No budget, no manpower and no luck. It's a bitch.
But please, don't get discouraged and keep up the good work.

When you've got the flow running again, we will again do our part in the search :)
____________
Treasurer of team SETI@Netherlands

Hey folks. We need to do something about this funding issue. SETI has more crunchers than just about any other project. This is a huge resource, and it feels like the operation at Berkeley is slowly deteriorating. The guys there can only do so much to keep things running and eventually, critical hardware will fail that they cannot afford to replace.

Given the broad spectrum of people who contribute to this project, there have to be some profs who can help pursue grants to reinvigorate the project. Any ideas?

____________I'm only running one computer. Using 2 cores of an old Q8200 CPU for CPU tasks, and 2 cores feeding a single Mid-range GPU, ATI HD7870.
Look at the RAC folks, and ask yourselves why it beats so many multi GPU monster computers :-)