SETI@homehttp://setiathome.berkeley.edu/
BOINC project SETI@home: Technical NewsUniversity of CaliforniaThu, 24 May 2018 19:00:08 GMTen-ushttp://setiathome.berkeley.edu/rss_image.gifSETI@homehttp://setiathome.berkeley.edu/
Technical News 3 May 2018, 0:15:54 UTChttp://setiathome.berkeley.edu/tech_news.php#326
http://setiathome.berkeley.edu/tech_news.php#326Of course it's been a busy time as always, but I'll quickly say I posted something on the general SETI blog:
http://seti.berkeley.edu/blog/trip-to-parkes-april-2018/
The short story: Last week I and others on the Breakthrough Listen team took a trip to Parkes Observatory in NSW, Australia. A short but productive visit, getting much needed maintenance done. However, we're still not exactly quite there for generating/distributing raw data from Parkes to be used for SETI@home. This will happen, though!
- Mattsee commentsThu, 03 May 2018 00:15:54 GMTTechnical News 4 Mar 2018, 17:15:26 UTChttp://setiathome.berkeley.edu/tech_news.php#325
http://setiathome.berkeley.edu/tech_news.php#325I'm still here! Well, still working 200% time for Breakthrough Listen. Yes I know long time no see. I've been quite busy, and less involved with the day-to-day operations of SETI@home... but I miss writing technical news updates. And clearly we (as in all the projects of the Berkeley SETI Research Center) could always use more engagement. So I want to get back into that habit.
However, I would like a different forum than these message boards. I'll be writing mostly of Breakthrough Listen stuff, so this may not be the most appropriate place. Also non-SETI@home participants would be unable to post responses.
Any thoughts on preferred communication methods in this day and age? It's hard because people have their favorites (blogs, facebook, twitter, reddit, instagram, tumblr, ello, mastodon, vero, etc.).
Basically I'm looking for places I can post stuff where anybody can view, anybody can respond in kind and ask questions (not that I'll have much time to answer but I'll try since I'll be likely asking questions too), but I won't have to moderate or spam filter or what-have-you. I know there's a lot of hostility towards facebook and the like, but it *works* (at least for now). There are general Berkeley SETI Research Channels already, but I don't want to overwhelm them with my frequent informal nerdish ramblings.
Now that I'm writing all this maybe I should just start a Breakthrough Listen forum here on the SETI@home page, echoing whatever I write elsewhere. Still, I'm curious about what people think about this sort of thing these days.
In the meantime, if you want - I encourage you to sign up for the general Berkeley SETI Research Center social media channels:
Facebook
Twitter
Instagram
YouTube
Thank you all -
- Mattsee commentsSun, 04 Mar 2018 17:15:26 GMTTechnical News 17 May 2016, 23:00:11 UTChttp://setiathome.berkeley.edu/tech_news.php#324
http://setiathome.berkeley.edu/tech_news.php#324I haven't written in a long long while, with good reason: As of December 2015 I moved entirely to working on Breakthrough Listen, and Jeff and Eric heroically picked up all the slack. Of course we are all one big SETI family here at Berkeley and the many projects overlap, so I'm still helping out on various SETI@home fronts. But keeping Breakthrough moving forward has been occupying most of my time, and thus I'm not doing any of the day-to-day stuff that was fodder for many past tech news items.
That said, I thought it would be fun to chime in again on some random subset of things. I guess I could look and see what Eric already reported on over the past few months, but I won't so I apologize if there's any redundancy.
First, thanks to gaining access on a free computing cluster (off campus) and a simultaneous influx of free time from Dave and Eric we are making some huge advances in reducing the science database. All the database performance problems I've been moaning about for years are finally getting solved, or worked around, basically. This is really good news.
Second, obviously we are also finally splitting Green Bank data for SETI@home. While Jeff and Eric are managing all that, it's up to me to pass data from the Breakthrough Listen pipeline at Green Bank to our servers here at Berkeley. This is no small feat. We're collected 250TB of data a day with Breakthrough Listen - and maybe eventually recording as much as 1000TB (or 1PB a day). When we aren't collecting data we need every cycle we have to reduce the data to some manageable size. It's still in my court to figure out how to get some of this unreduced data to Berkeley. Shipping disks is not possible, or at least as easy as it is at Arecibo (because we aren't recording to single, shippable disks, but to giant arrays that aren't going anywhere. We may be able to do the data transfers over the net, and in theory have 10GB links between Berkeley and Green Bank, but in practice there'sa 1GB chokepoint somewhere. We're still figuring that out, but we have lots of data queued up so no crisis... yet.
Third, our new (not so new anymore) server centurion has been a life-saver, taking over as both the splitter for Green Bank data (turns out we needed a hefty server like this - the older servers would have fallen behind demand quickly) as well as our web server muarae1 when that system went bonkers around the start of the year. Well, we finally got a new muarae1 so centurion is back to being centurion - a dedicated splitter, a storage server, and potentially a science database clone and analysis machine. We also got a muarae2 server which is a back up (and eventual replacement) for the seti.berkeley.edu web server.
Fourth, our storage server bucket is having fits. All was well for a while but this is an old clunker of a machine so it's no surprise its internal disk controllers are misbehaving (we've seen similar behavior on similar oldSun servers). No real news here, as it doesn't have any obvious effect on public data services, but it means that Jeff and I have to wake up early and meet at the data center to deal with it tomorrow.
And on that note.. there's lots more of course but I should get back to it...see commentsTue, 17 May 2016 23:00:11 GMTTechnical News 10 Nov 2015, 23:15:41 UTChttp://setiathome.berkeley.edu/tech_news.php#323
http://setiathome.berkeley.edu/tech_news.php#323So one thing I left off yesterday's catchup technical news item was the splitter snafu from last week which caused a bunch of bogus broken workunits to be generated (and will continue to gum up the system until they pass through).
Basically that was due to us having the splitter code cracked open to eventually work with Green Bank data. Progress is being made on this front. However some code changes for Green Bank affected our current splitter for Arecibo, as we needed to change some things to make the splitter telescope agnostic (i.e. generalized to work with any data from any telescope). These changes were tested in beta, or at least we thought were thoroughly tested, but things definitely broke in the public project. We fixed that, but not after a ton of bad workunits made its way into the world. We still have some clean up to do on that front.
BUT ALSO we needed to update some fields in the current science database schema to also make the database itself telescope agnostic. Just a few "alter table" commands to lengthen the tape name fields beyond 20 characters. We thought these alters would take a few hours (and completed before the end of today's Tuesday outage). Now it looks like it might take a day. We can't split/assimilate any new work until the alters are finished. Oh well. We're going to run out of work tonight, but should have fresh work sometime tomorrow morning. It is a holiday tomorrow, so cut us some slack, if it's later than tomorrow morning :).
- Mattsee commentsTue, 10 Nov 2015 23:15:41 GMTTechnical News 10 Nov 2015, 0:29:29 UTChttp://setiathome.berkeley.edu/tech_news.php#322
http://setiathome.berkeley.edu/tech_news.php#322Okay. Every time I put off writing a tech news item a bunch more stuff happens that causes me to continue putting off even further! So here's a quick stream-of-consciousness update, though I'm sure I'm missing some key bits.
First off, the AP migration is officially finished! As a recap, things got corrupted in the database late last year - and to uncorrupt it required a long and slow migration of data among various servers and temporary tables. A lot of the slowness was just us being super careful, and we were largely able to continue normal operations during most of this time. Anyway I literally just dropped the last temporary table and its database spaces a few hours ago. Check that off!
One of the temporary servers used for the above is now being repurposed as a desperately needed file server just for internal/sysadmin use (temporary storage for database backups, scratch space, etc.). For this I just spent a couple hours last week unloading and reloading slightly bigger hard drives in 48 drive trays.
A couple months ago we also checked another big thing off our list: getting off of Hurricane Electric and going back to using the campus network for our whole operation. The last time campus supported all our bandwidth needs was around 2002 (when the whole campus had a 100Mbit limit, and paid for bandwidth by the bit). The upshot of this is that we no longer have to pay for our own bandwidth ($1000/month) and we can also manage our own address space instead of relying on campus. Basically it's all much cheaper and easier to maintain now. Plus we're also no longer relying on various routers out of our control including one at the PAIX that we haven't been able to log into for years.
But! Of course there were a couple unexpected snags with this network change. First, our lustre file server had a little fit when we changed its IP addresses to this new address space. So we changed it back, but it still wouldn't work! Long story short, we learned a lot about lustre and the voodoo necessary to keep these things behaving. Making matters more confusing was a switch that was part of this lustre complex having its own little fit.
The other snag was moving some campus management addresses into our address space, which also should have been trivial, but unearthed this maddening, and still not completely understood, problem where one of the two routers directing all the traffic in and out of the campus data center seemed unhappy with a small random subset of our addresses, and people all over the planet were intermittently unable to reach our servers. I think the eventual solution was campus rebooting the problem router.
Those starving for new Astropulse work - I swear new data from Arecibo will be coming. Just waiting for enough disks to make a complete shipment. Meanwhile Jeff is hard at work making a Green Bank splitter. Lots of fresh data from fresh sources coming around the bend... Part of the reason I bumped up the ceiling for results-ready-to-send was to do a little advance stress testing on this front.
Oh yeah there was that boinc bug (in the web code) that caused the mysql replica to break every so often. Looks like that's fixed.
Over the weekend lots of random servers had headaches due to one of the GALFA machines going down. It's on the list to separate various dependencies such that this sort of thing doesn't keep happening. Didn't help that me and Eric were both on vacation when this went down.
Meanwhile my daily routine includes a large helping of SERENDIP 6 development, a lesser helping of messing around with VMs (as we start entering the modern age), and taking care of various bits of the Breakthrough Listen project that have fallen on my plate.
- Mattsee commentsTue, 10 Nov 2015 00:29:29 GMTTechnical News 31 Aug 2015, 21:01:09 UTChttp://setiathome.berkeley.edu/tech_news.php#321
http://setiathome.berkeley.edu/tech_news.php#321Right now there's a whole bunch of activity taking place regarding the Astropulse database cleanup project. Basically this week all AP activity will be off line (possibly longer than a week) as I'm rebuilding the server/OS from scratch as we're upgrading to larger disks, then merging everything together onto this new system. So all the assimilators and splitters will be offline until this is finished.
The silver lining is we're currently mostly splitting data from 2011 which has already been processed by AP, so it wouldn't be doing much anyway. Good timing.
There will be new data from Arecibo eventually, and progress continues on a splitter for data collected at Green Bank.
Uh oh, looks like the master science database server crashed. Garden variety crash at first glance (i.e. requiring a simple reboot). I guess I better go deal with that...
- Mattsee commentsMon, 31 Aug 2015 21:01:09 GMTTechnical News 21 Aug 2015, 22:47:12 UTChttp://setiathome.berkeley.edu/tech_news.php#320
http://setiathome.berkeley.edu/tech_news.php#320Those panicking about a coming storm due to lack of data... The well is pretty dry but Jeff and I just uncovered a stash of tapes from 2011 that require some re-analysis, so that's why you'll see a bunch showing up in splitter queue over the weekend (hopefully before the the results-to-send queue drops to zero).
In the meantime, we are still recording data at AO (not fast enough to keep our crunchers supplied), but.... this situation has really pushed us to devote more resources to finally finishing the GBT splitter, which will avail to us another reserve supply of data in case we hit another dry spell.
The network switch on Tuesday seems to have gone fairly well. We are now sending all our bits over the campus net just like the very old days <waxes nostalgic>.
- Mattsee commentsFri, 21 Aug 2015 22:47:12 GMTTechnical News 17 Aug 2015, 17:56:43 UTChttp://setiathome.berkeley.edu/tech_news.php#319
http://setiathome.berkeley.edu/tech_news.php#319I've been meaning to do a tech news item for a while. Let's just say things have been chaotic.
Some big news is that campus is, for the first time in more than a decade, allowing SETI@home traffic back on the campus network infrastructure, thus obviating our need to pay for our own ISP. We are attempting this switchover tomorrow morning. Thus there will be more chaos and outages and DNS cache issues but this has been in the works for literal years so we're not stopping now. I apologize if this seems sudden but we weren't sure if this was actually going to happen until this past weekend.
We are finally seeming to get beyond the 2^32 result id problem and its various aftershocks. Due to various residual bugs after deploying the first wave of back-end process upgrades we have a ton of orphan results in the database (hence the large purge queue) which I'll clean up as I can.
Re: BOINC server issues galore, all I gotta say is: Ugh. Lots of bad timing and cursed hardware.
The Astropulse database cleanup continues, though progress has stalled for several months due to one Informix hurdle requiring us to employ a different solution, then simply just failing to coordinate the schedules between me, Jeff, Eric, and our various other projects. But we will soon upgrade the server and start merging all the databases back into one. This hasn't slowed the public facing part of the project, or reduced science, but it will be wonderful to get this behind us someday.
So much more to write about, but as I wait for dust to settle ten more dust clouds are being kicked up...
- Mattsee commentsMon, 17 Aug 2015 17:56:43 GMTTechnical News 23 Jun 2015, 22:16:16 UTChttp://setiathome.berkeley.edu/tech_news.php#318
http://setiathome.berkeley.edu/tech_news.php#318Catching up on some news...
We suddenly had a window of opportunity to get another SERENDIP VI instrument built and deployed at Green Bank Telescope in West Virginia. So we were all preoccupied with that project for the past month or so, culminating in most of the team (myself included) going to the site to hook everything up.
So what does this mean? Currently we have three instruments actively collecting data. One SETI@home data recorder at Arecibo - where all our workunits come from, and two SERENDIP VI instruments - one at Arecibo and one at Green Bank - collecting data in a different format. Once the dust settles on the recent install and we get our bearings on the SERENDIP VI data and bandwidth capabilities we will sort out how to get the computing power of SETI@home involved. Lots of work ahead of us and a very positive period of scientific growth and potential.
We also are in a very positive period of general team growth as the previously disparate hardware/software groups have slowly been merging together the past couple of years, and now that we all have a place to work here at Campbell Hall on campus - proximity changes everything. Plus we have the bandwidth to pick up some students for the summer. Basically the new building and the Green Bank project rekindled all kinds of activity. I hope this yields scientific and public-outreach improvements we've been sorely lacking for way too long (getting Steve Croft on board has already helped on these and many other fronts). We still need some new hardware, though. More about all this at some point soon...
Meanwhile, some notes about current day-to-day operations. Same old, same old. We got some new data from Arecibo which Jeff just threw into the hopper. I just had to go down to the colo and adjust some loose power cables (?!?!) that caused our web server to be down for about 12 hours this morning. Some failed drives were replaced, some more failed drives need to be replaced. Now that Jeff and I are back to focusing on SETI@home the various database-improvement projects are back on our plates...
Speaking of databases, the Astropulse rebuild project continues! As predicted the big rebuild project on the temporary database completed in early June. To speed this up (from a year to a mere 3 months) I did all the rebuilding in 8 table fragments and ran these all in parallel. I thought the merging of the fragments into one whole table would take about an hour. In practice it took 8 days. Fine. That finished this past weekend, and I started an index build that is still running. When that completes we then have to merge the current active database with this one. So there are many more steps, but the big one is behind us. I think. It needs to be restated that we are able to acheive normal public-facing operations on Astropulse during all this, outside of some brief (i.e. less than 24 hour) outages in the future.
Speaking of outages, this Saturday (the 27th) we will be bringing the project down for the day as the colo is messing with power lines and while they are confident we shouldn't lose power during their upgrades we're going to play it safe and make sure our databases are quiescent. I'll post something on the front page about this.
Still no word about new cricket traffic graphs, but that's rolled up with various campus-level network projects so there's not much we can do about that.
- Mattsee commentsTue, 23 Jun 2015 22:16:16 GMT