This week, I've been busting my brain about some interference we are seeing in the multibeam data recorder. Anyone who runs the beta may have noticed that both Astropulse and SETI@home multibeam work units have been hitting overflow conditions fairly regularly. Regularly enough that a lot of machines were hitting their 100 workunit per day limit. So I
set about the task of figuring out what was going on. I'll ask you to forgive me in advance because this might get a bit technical...

I grabbed one of the recorder data files that showed this effect and did a little spectral analysis to see if I could figure something out. This is what I saw.

This is what we call a waterfall plot of about 0.42 seconds of data from one of the data recorder channels. The vertical axis is frequency difference in MHz from 1.420 GHz. The horizontal axis is time. Generally it looks OK. The darkish stripe at zero frequency is due to a high-pass filter that cuts out any constant voltage offset in the system. The gradual darkening at the top and bottom are the high and low frequency edges of our bandpass filter. You might notice a slightly brighter stripe at about 0.5 MHz that is due to the Hydrogen line that Kevin spends his time making maps of.

What I didn't expect to see is the 7 or so vertical lines that cross zero frequency. That's our problem. Pulsed interference of some sort. From the width I can see that the pulses have time scales of 20 to 50 microseconds or so. They aren't entirely regularly spaced, but they are close enough and strong enough to register as pulses in astropulse and as triplets in SETI@home. I looked at all of the receivers and I saw something similar in each.

The next thing to check was whether the interference appeared at the same time in all the bands. So I did a complex sum of all of the channels to see if the signals were coherent, meaning they have the same phase and timing in all the channels.

Spectral analysis of the result showed this:

That convinced me that there is some source of interference that affects all of the channels in nearly the same way. It's also more common than you would think if you just looked at one channel. So I did the obvious thing and subtracted the summed waveform from the channel to see if that would remove the pulses. It did. Perfectly.

That is, until I tried to turn the subtracted channel into a work unit. In order to do that, I needed to turn my floating point data back into the 1 bit samples. That restored the pulses. It took a little work to figure out why. I was subtracting a waveform with values between -1 and 1 from values that were either -1 or 1. Then I would take the result and convert it to -1 or 1 depending upon whether it was less than zero. So if I subtracted 1 from 1, I would get zero, which is not less than zero so the result would be 1. 1-1=1. Josh is exploring ways to fix this. The one I'm currently working with is: if the result of the subtraction is between -0.5 and 0.5, then assign a random value. So far it seems to work pretty well, I'll be testing it out in beta next week.
Another option would be to use 2 bit samples in the work units. That also appears to work well, but it doubles the size of the workunit files. So we'll try the randomizer first.

Next we'll try to figure out where these pulses are coming from. Since they are coherent across channels and polarizations I suspect that that are somewhere in the electronics chain after the mixer. Maybe even in the data recorder itself. We'll be talking with engineers at Arecibo in order to find out if we can get rid of them at the source.

Dan suggested this may be the first time that an RFI waveform has been removed from a channel by using multibeam data. I doubt it, but it's certainly the first time we've done it, and that's good enough for me.

Those FFTs look like the pulses are rounded, and they are symmetrically centred mid-band, so I'd suspect first signal wiring pickup or voltage rail disturbance near/on the ADC(s) in the data recorder, or pickup on the way to the datarecorder.

Do those pulses match HDD activity?!

A second wild shot is if the datarecorder CPU is overloaded and is missing picking up or responding to a proportion of its interrupts.

Dan suggested this may be the first time that an RFI waveform has been removed from a channel by using multibeam data. I doubt it, but it's certainly the first time we've done it, and that's good enough for me.

-- Eric

Eric ,

the "normal" Multibeam WU in beta testing usually on average has an angle range of somewhere between 3 and 0.03

Dan suggested this may be the first time that an RFI waveform has been removed from a channel by using multibeam data. I doubt it, but it's certainly the first time we've done it, and that's good enough for me.

-- Eric

Eric ,

the "normal" Multibeam WU in beta testing usually on average has an angle range of somewhere between 3 and 0.03

Byron, I saw someone mentioning one of these extremely long angle range units a year or so ago. On that one it appeared that the recorder was turned off for several hours, the scope then moved most of that angle and then the recorder turned back on. Probably that was the way the cookie crumbled when somebody divided the tape into workunits. There are probably 255 other units with similar ID numbers that have that long angle.

Byron, I saw someone mentioning one of these extremely long angle range units a year or so ago. On that one it appeared that the recorder was turned off for several hours, the scope then moved most of that angle and then the recorder turned back on. Probably that was the way the cookie crumbled when somebody divided the tape into workunits. There are probably 255 other units with similar ID numbers that have that long angle.

We have been seeing batches of Multibeam WUs with extremely high ARs in Beta, often returning â€œâ€“9 overflowsâ€; the highest I noticed was aroud 75Â°, with start and end RAs nearly 6 h apart.
____________

Somewhere, records of telescope (receiver) position at time of day were available for SetiClassic. The same might be true for Alfa. Every few seconds the position in declination and right ascension were given. But with the units with long angle there was a big discontinuity of several hours and many tens of degrees, as I remember. The receiver was probably turned off several hours during which the receiver moved that big angle. I'd have my doubts that any of those 256 discontinuous units would be any good. But to search for those long interruptions might be a bigger waste of time than to crunch 256 worthless units.
____________

The high angle range workunits are the result of a different problem. Apparently the telescope sends a bad coordinate packet on rare occasion. I'm going to fix that one by checking whether the distance between any two coordinates exceeds the maximum rate the telescope can move in that amount of time.

Those FFTs look like the pulses are rounded, and they are symmetrically centred mid-band, so I'd suspect first signal wiring pickup or voltage rail disturbance near/on the ADC(s) in the data recorder, or pickup on the way to the datarecorder.

My bet is fluctuations in the ground plane or a loose connection somewhere.

Do those pulses match HDD activity?!

That's another one we can check on. The files are temporarily stored on a RAM disk before being transferred to the HDD. We should be able to run for a period with no HDD activity.

A second wild shot is if the datarecorder CPU is overloaded and is missing picking up or responding to a proportion of its interrupts.

Have you figured out what the pulse repitition interval and the pulse duration is?

Personally, my opinion is any pulse repitition less than 100usecs would or should indicate a pulse doppler radar? Or something with a fairly predictable duty cycle...it doesn't even have to be pulse constant...

Is there a weather radar near by? Perhaps your picking up sidelobes or back lobes getting into the sidelobes of the receiving antenna? Sidelobe poke through or sibelobe of a separate transmitter getting into the sidelobe of the receiver? I am not actually suggesting radar, but even perhaps unshielded equipment nearby?

Have you figured out what the pulse repitition interval and the pulse duration is?

Personally, my opinion is any pulse repitition less than 100usecs would or should indicate a pulse doppler radar? Or something with a fairly predictable duty cycle...it doesn't even have to be pulse constant...

I half thought that but the spacing is more of the order of 10ms, and variable (100 pulses per second) which is more like the clicks from a mobile phone or HDD activity...

There are FFT waterfall plots posted earlier in the thread.

Capacitive pickup across cables or ground bounce can be... "difficult"...

Happy tracing!

Regards,
Martin
____________
See new freedom: Mageia4Linux Voice See & try out your OS Freedom!
The Future is what We make IT (GPLv3)

Is there a weather radar near by? Perhaps your picking up sidelobes or back lobes getting into the sidelobes of the receiving antenna? Sidelobe poke through or sibelobe of a separate transmitter getting into the sidelobe of the receiver? I am not actually suggesting radar, but even perhaps unshielded equipment nearby?

Many years ago very quiet night I could pick up the airport radar at Lindberg Field (or possibly the North Island Naval Station) on my CB radio. Gotta watch out for those harmonics.
____________Join BOINC Synergy!

There seems to be a "ghost" of the central group of vertical lines at about the 0.12 to 0.17 Mhz range of your graph. To me, that looks possibly like a harmonic frequency, caused usually by a signal diffracting between two objects. If so, you may need to research more than one possible source of the RFI.

To illustrate my point, I once experienced the same effect in London, England, while listening to a radio station transmitting on 1.214 MHz. I could receive a weaker, but perfectly clear signal on 310 KHz, a fourth harmonic of the true signal.

Of course, I could be well clear of the mark, but I thought it was worth a mention.
____________

There seems to be a "ghost" of the central group of vertical lines at about the 0.12 to 0.17 Mhz range of your graph. To me, that looks possibly like a harmonic frequency, caused usually by a signal diffracting between two objects. If so, you may need to research more than one possible source of the RFI.

To illustrate my point, I once experienced the same effect in London, England, while listening to a radio station transmitting on 1.214 MHz. I could receive a weaker, but perfectly clear signal on 310 KHz, a fourth harmonic of the true signal...

Errrr... I didn't know that "sub"-harmonics existed!... [edit] (Note: They don't! However, you can see an interferometric effect whereby the amplitude of a signal gets modulated at the difference ferquency from the modulating/interfering signal. This is different from generating a (higher frequency) harmonic from non-perfectly-sinusoidal signals or from FFT decomposition of one signal. Corrections welcomed!) [/edit]

More seriously, you may well have had that (strong) signal breaking through on the mixer stage alternate frequency of one of the IF stages in your receiver. Was it a non-superhetrodyn receiver with only one (or no) IF stages even?

"Ghosting" as commonly seen on TV pictures where you get multipath reflection is from the same frequency signal arriving via multiple paths. You get the same signal but displaced in time. Hence, for an analogue modulated TV picture that is scanned across the screen horizontally, you get multiple identical pictures overlayed but slightly displaced horizontally.

If what you notice are indeed "ghosts" but time delayed, then that would suggest multipath pickup. The "multipath" part could be ANY different route, whether external or just different routes through the pickup path in the affected equipment.

It could just be that the interference source produces that repeating pattern and that it "just looks like that".

Who knows?

Until Eric tracks it down.

It would be interesting to see what more of the WUs show and how consistent/repeatable it is, and so on...

Much better to cure the source rather than to try to remove the symptoms.