September 28, 2012
Chris and Lisa
All equipment has been removed from the Turbulence Tower. Three sonics, two Licors, five TRHs, barometer, 4-comp, dsm, ALL cables (except eth), batteries, charger.
Things that we kept there were all TRH booms, barometer boom, three sonics booms, CSAT electronic plates (3), 4-comp boom WITH mounting plate and deer stand.
Gear that is in EOL: Five TRHs, two sonic booms, three CSAT electronic plates, 4-comp, barometer, three batteries, power cable and splitters.…

August 17, 2012
10:00am - 1:30pm MDT
Three batteries installed at Turbulence Tower. Four component was installed and working. New TRH 2m top plate. New 2m sonic (CSAT0536) was installed to replace odd sonic.

August 15, 2012
10:30am - 3:00pm
Putting freshly calibrated sonics (CSATs) on turbulence tower with Licors, TRHs and barometer. The four-component radiation did not get mounted due to a mount missing. System was check with just a charger powering instruments but then shut off due to missing batteries. Everything looks good but the 2m sonic.
Guy tensions were checked.
Remember: Batteries, radiation mounting plate, radiation instrument, boom and electronics box.
45m: CSAT0537,…

Aug 23, 11:30 am
Installed a new kernel with one small difference in the PC104 interrupt handling. If the PC104 interrupt hander is called, and it sees no bits set in the pending value, then it goes ahead and call handlers for all unmasked PC104 interrupts (which in manitou's case is just one, IRQ 3).
This logic was also in the patched 2.6.16 kernel. It counted these "spurious" interrupts and from time to time, complained about them. The new code doesn't log any complaints.…

The data system was re-installed on Aug 15. It is the same hardware (CPU, serial cards, usb disk, enclosure and interface panels) as before.
The Linux kernel has been upgraded, from 2.6.16 to 2.6.35. This kernel has PPS support, so an extra patch was not needed to get the PPS from the GPS. It also does not have the bug where executables could not be run from compact flash. Therefore they are not copied to ram disk at bootup.…

June 26, 2012
11:00am - 2:05pm MDT
Removal of all sensors on the Turbulence Tower today. Sonics, Licors, TRHs, pressure, four-component, batteries and dsm. Cables have remained on tower. Kept beacon on also.
Since we left cables on tower we had to mark cables on the dsm side so we know what instrument it was and which port. This was our organization codes using colored paper clips, zip ties and electrical tape. All levels have a sonic, TRH and Licor which are bundled together.…

Gordon Jun 27, 2012
After removing the sensors and data system, we're testing the set up back in Boulder, to see if we can duplicate the problem of not being able to sample 5 licors.
In setting up the test, noticed that the fan on TRH012 does not run, but the TRH does output data. Checking the data archive, that unit is data id 1,420, which was the 30 meter TRH. Will check with Chris in case he noticed whether the fan was running when he removed the sensor.…

The entire turbulence tower was down from May 31 until June 8. During this time, we were able to ping the datalogger but unable to log in. Richard Oakes from the USFS cycled the power on the datalogger at about 2pm on June 8, which restored access to the system.
All sensors came happily back to life, with the exception of the 2m LiCor that Gordon was able to resuscitate via sending ASCII commands to the sensor.…

The turbulence tower data had not been updating since March 17. A site visit by Ned found that the network issue preventing communication was associated with the 5-port Linksys/Cisco switch located in the waterproof power enclosure at the tower. Cycling the power on the switch brought back the network.

Who: Ned, Christopher
When: December 15, 2011. Arrived 9:51am, Departed 12:00pm
Removed Licor (sn 1176) from 15m.
Note, by Gordon, Jul 13, 2012: There is no sn1176. I think this must have been unit sn1167, which according to the log entry of 4/12/2011, was installed at 15m.
When removing this sensor due to bad chopper motor, it was not running. Rebooted sensor and heard the chopper motor in the head and it sounded bad.…

15 m Li-COR (sn1176) dropped out of service on November 6, 2011.
Note, by Gordon, Jul 13, 2012: There is no sn1176. I think this must have been unit sn1167, which according to the log entry of 4/12/2011, was installed at 15m.
The diagnostic value dropped to 215 indicating that Bit 5 signaled that the instrument's chopper motor failed. There was an indication a number of hours earlier that the chopper motor was beginning to fail, see:
http://www.eol.ucar.…

Gordon, Dec 9 3:15 pm
Since Dec 1 22:50 UTC the DSM log file (/var/log/isfs/adam.log) has error messages that the GPS pseudo-terminal device, /var/tmp/gps_pty0, does not exist. Apparently the tee_tty process died at that time.
tee_tty reads the GPS ASCII messages on /dev/ttyS3, and writes them to two pseudo-terminals /dev/gps0 (read by NTP) and /var/tmp/gps_pty0 (read by dsm).
I'm not sure why tee_tty died, but decided that this was an opportunity to upgrade the NIDAS software on this system.…

The "spurious" IRQ messages are annoying when one is logged onto the system console at the tower. They are logged at a priority of "warning". They should have a lower priority, like debug, or info, unless the number/sec is above some threshold. Until that change is made, I'll disable them on the console, by setting the "-c 4" option in klogd, which then suppresses messages of warning and lower, which are numeric values 4-8. 3-0 are priorities from error to emerg.…

Oct 14, 2011
Arrived: 9am MDT
Departed: 11am MDT
ChrisG and SteveS are on site to diagnose the sick licor 7500 at 16m and add the two licors that were at the CWEX11 project.
Chris could not hear the motor spinning on the 16m Licor 7500. A power cycle did not bring it back. While removing it, when the unit bumped against something, the motor started up, but was much more noisy than usual. It will be brought back for repair.
Installed SN 1167 at 16 meters, and 0813 at 7 meters.…

The data from the Licor at 16 meters went bad on Oct 2, while our system was not reachable over the network.
http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/2011/10/02/co2_20111002.png http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/2011/10/02/co2_20111002.png
The ldiag averages went from the typical 248-255 range to around 216.
http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/2011/10/02/licor_20111002.png http://www.eol.ucar.…

We have not been able to reach the turbulence tower data system since Sep 29.
We could ping the RAL server in the seatainer from Boulder. Andy Gaydos logged into the server and could not ping the data system: 192.168.100.202. So either the data system or the network between the seatainer and the tower was down.
On Oct 6 I was able to ssh to the DSM. So somebody fixed the problem, or perhaps a fiber<->copper transceiver reset itself.</->…

Monitoring the accuracy of the system clock on a real-time data acquisition system provides useful information about the performance of the system. Hence this long discussion.
Data System Clock, NTP and GPS
The data system at the Manitou Forest Observatory (aka, the DSM) uses a GPS receiver and the NTP (Network Time Protocol) software to set the system clock, which, in addition to the normal uses of a system clock, is used to time-tag the data samples.…

The 5 minute statistics of data from the turbulence tower, for the entire dataset, Jul 8 2009 to the present have been recomputed, and written to the NetCDF files. The new values incorporate the following changes:
In each 20 Hz sample from a CSAT3 sonic, if any of the CSAT diagnostic bits are non-zero, then the values for that sample (u,v,w,tc) are marked as missing, and not added to the 5 minute statistics.…

Arrived: 10:12am MDT
Departed: 1:45pm MDT
Ned and I did a couple of chores on this visit. First, we fixed the 43M sonic. It seemed to be the port on the Emerald Card. Switched 43M sonic from port16 to port20. Ned updated the xml. Second, we replaced two 15M cable from 43M Licor with a 30M cable. When we did this we noticed interrupts on the dsm. Having unplugged the battery charger for the laptop to talk with dsm,…

Ned noticed a drop out of data from 43 meters on Jul 10.
http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/20110711/plots_all.shtml http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/20110711/plots_all.shtml
I logged into the manitou data system on Jul 13, and tried to resurrect it.
I tried rserial, minicom and powering the sonic off and on:
eio 16 0
eio 16 1
rs 16
Got nutt'in
The licor is working.…

RH at 43m has been bad for a while. Looking back through the plots, looks like it failed the night of Apr 25 during a period of near 100% RH.
http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/20110425/trh_20110425.png http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/20110425/trh_20110425.png
trh_20110425.png

The plot of GPSdiff_max for May 22, 2011 shows a maximum value of around 25 seconds between 11:05 and 17:00 UTC, indicating something went haywire with the data collection during those times.
Looking at the downloaded data, it appears that data stopped coming in on the Diamond serial ports at 11:05, and resumed at 17:00. There is not a large gap in data from the 3 sensors that are sampled on the Viper serial ports, but they show large time tag delta-Ts, followed by many small delta-Ts,…

Arrived: 10:30am MDT
Departed: 11:20am MDT
Kurt and I installed three LiCors on turbulence tower at heights of 2M, 15M and 45M. TRHs were installed on all levels. New GPS was added and old Garmin was removed.
LiCor heights:
2M-1166
15M-1163
45M-1164

Apr 12, 11:32 am
The output format and reporting rate of the 3 Licor 7500s that were re-installed today had been reset during the calibration procedure to generate verbose output, with labels. To save archive space, we configure the units for a terse output, without labels, at 10 Hz.
To setup the Licors for the format that our data system expects, do the following from the data system login prompt:
shut down the data process
adn
run minicom on the Licor port (ttyS2=2m, ttyS7=7m, ttyS11=16m,…

Calibration process for LiCors 7500s. I calibrated all five LiCors. Calibration included post cal, desiccant change, spot check, zero/span. I used N2 tank which went through a scrubber of Ascarite II and Drierite to give the zero for both CO2 and H20.
Post-Cal conditions were checking zeros, span and dew point. Spans included for CO2 were a bottle of 394.611ppm and 378.867ppm. For H2O I used the Thunder Scientific, humidity chamber, set at 25C at 80%RH. Dew point was equal to 21.31C.…

The TRH sensors went through a post cal check before re-calibration.
Here are the results.
Temperature: Tested over the range of -30 to 30. It has been almost 1 year since these sensors were calibrated Beachon_040111.jpg
Relative Humidity: Tested over 10% to 90% at temperatures 25, 10, -3 Beachon_033011_RH.jpg

So sorry. This is a redundant log entry. First entry was in the main Manitou Log entry page.
Attendees: Chris, Kurt and Ned.
Arrived at 11:25am
Departed at 1:15pm
Removed LiCors and TRHs at all heights. The LiCors were at the following heights.
2m: s/n 1163
7m: s/n 1166
15m: s/n 1167
45m: s/n 1164
We unplugged main power to all LiCors in the battery boxes. Replaced both outside GFI outlets. Both outlets were blown. Power back to both outlets. Beacon back on.

Installed the latest version of nidas (revision 5771M) today, with the new process running at 19:49 UTC.
The new nidas has some improvements in the serial handling efficiency. Don't see any effect on the number of "spurious interrupts" though.
Also restarted ntp daemon. Added a "server ral" entry in /etc/ntp.conf so that we can compare our local GPS time source with the ral server.
ntpq -p shows good agreement (-8.…

The Licor 7500 at 2m has not been reporting for a while.
Looking at /proc/tty/driver/serial for port 2 indicates that no characters have been received since the last reboot. As of Jul 18 the system uptime is 19 days.
Did "rs 2" to talk to the port. After sending carriage a return the 7500 responds with
^M(Error (Received TRUE))\n
So, it is alive and responding, just not sending data. Sent it a configuration command, setting the BW and RS232 parameters,…

The turbulence tower data system clock was behind by 1 second from May 4 to Jun 11. If sub-second absolute time-tag accuracy of the data samples is a not a concern, then you can ignore this log message. Otherwise, here are the gory details...
The data system was replaced on the turbulence tower on May 4.
The new system is running a new version (4.2.6p1) of the NTP (network time protocol) service. NTP on the data system is configured to use the attached GPS (Garmin 25-HVS) as a reference clock.…

The 7m TRH has been sick since about Jun 23. It looks like a bad RS232 connection. The data is good for periods of time, then garbage.
Here's a bit of data where it was in a and out:
data_dump -A -i 1,220 manitou_20100627_120000.dat | more
2010 06 27 13:44:10.4165 22.59 20 \xfc\xfe1\xfe~3f\xf04984\xe02929\r\n
2010 06 27 13:44:12.0258 1.609 26 \xfc\xfe10.\xf08\xe088\xee33 4985 2929\r\n
2010 06 27 13:44:13.6407 1.615 29 T\xde\xf8?\xfe10.08 88.…

On May 13, two sapflow sensors were installed on each of 12 trees approx. 50 m SE of the turbulence tower. Heat flux data will be downloaded at approx. 2-week intervals, and anyone interested in the data should contact Jia Hu (jiahu@ucar.edu). At some point, heat fluxes will be converted to sapflow data and available to help constrain water flux measurements made at the turbulence tower. Sometime in the next couple of weeks,…

16m TRH has been missing since May 21, 2010 and acting up since as early as May 11.
Upon arrival, there was no fan operating on the 16m TRH. Determined that the coupling between the 5 and 15m cables going to the 16m TRH had taken on water and/or was corroded. Replaced 15m cable (upper section) of the cabling to the 16m TRH due to the corrosion. Bottom 5m cable should be replaced too. Kurt taped the current coupling and inserted a drip loop. Should be ok for a while.…

Karl Schwenz, Chris Golubieski, Gordon Maclean
Objectives:
Diagnose and fix reason that system has been offline for last few days (no data in Boulder after April 27).
Replace data system box in order to take the old box back to Boulder for a tuneup.
Replace top TRHs which have not been working well. Add new radiation shields.
Replace barometer with external unit.
Make anti-climb door easier to latch.
Arrived at site, May 4 10:30.
Chris climbed tower and replaced the TRH transducer/PIC units,…

TRH at 16 meters has had sporadic problems since Apr 8 07:00 MDT.
Over Apr 10,11 and 12 it seems to come alive in the warmest part of the day,
(at temps over 10C) and then fails as the temp falls in the evening. See plot below.
Here is a dump of some recent data (times in GMT).
Looks like the processor is getting reset, printing out its calibration
information when it boots. Perhaps the power is dropping out?
Corroded connection?
data_dump -i 1,320 -A manitou_20100411_000000.…

On Feb 4, logged into system from Boulder to verify the serial numbers of the licors:
port
SN
assumed height
2
1163
2m. Moved from port 14, 30m on Feb 2
7
1166
7m
20 (was 17)
1167
16m
14
now disconnected
11
1164
43m
This agrees with the configuration previous to Feb 2, and the changes of Feb 2: move of the 30m unit (SN1163) to 2m, and the move of the cable from the 16m unit (SN1167) from port 17 to port 20 on the new data system box.…

Feb 2, 2010: Patton, Golubieski, Maclean on site at 10:30 MST
Planned tasks:
Move Licor 7500 from 30 meters to 2 meters
Investigate why data system can't sample 4 7500's at 20 Hz
Measure guy tensions
Noticed a high-pitch sound from a TRH fan. Chris determined it was the 16m TRH. The propeller had come off the shaft, and was not repairable. Swapped the 16m and 2m TRHs, so that it will be easy for someone to replace the low unit.…

The 2m licor connected to /dev/ttyS2 was removed today, and the licors at 16 and 43 meters were reconnected to ports /dev/ttyS17 and /dev/ttyS11 by a BEACHON helper.
By checking the serial numbers, with the (Coef ?) command, and comparing against the earlier LICOR log entry, we could determine remotely what unit is connected to what port. Note that the licors at 16 and 43 meters are switched from the previous configuration.…

18Nov09 Site Visit: CG, KK, JM
- Purpose: Install 5 Licor 7500's
- Power: We rearranged the power / batteries to accomodate the licors. A high current/trickle/equalize battery charger was connected directly to the battery chain and not through the solar/battery charge by using the last 'external battery' amp connector.…

Here are my notes from this week on the first servicing we've ever done on our Li7500s.
Note that S/N 0813 and 1166 came back from LiCor this summer after being repaired (from lightning damage), so no service was performed on them.
The following all was done using LiCor's software (installed on the Aspire). Before service:
S/N
Last Cal
CO2 zero
CO2 span
H2O zero
H2O span
AGC
0813
Didn't read
1163
24 Oct 06
0.8697
0.9968
0.8468
0.9946
49%
1164
24 Oct 06
0.8883
1.017
0.8601
0.…

Nov 9, 12:30 MST.
Per Steve Oncley, increased the sampling rate of the Licor 7500 at 43 meters.
It is the only unit connected right now.
Commands to change the sampling rate:
# shut down data process
adn
# adn powers off port 17, power it back on.
eio 17 1
# start minicom
# The default baud rate of minicom on that port is 9600, which is what the 7500
# uses after it is sent a break. To set it to 9600 explicitly, use ctrl-A pe.
# Use ctrl-A minicom commands to turn on local echo (e),…

On the Oct 15 visit, Kurt noticed that the inlet of the barometer
pressure port had been taped over, though the tape had peeled back
and was no longer covering the inlet hole. Pressed the tape
back down, but it probably won't stay.
It was probably taped on Oct 1 when the TRHs were removed in anticipation
of a controlled burn.
So since Oct 1, the pressure data should not be trusted.

Oct 14. Arrived on site 10:55 MDT.
Kurt Knudson and Gordon Maclean (EOL) and Fabian Guerrero and Armando Cisneros from CISL/NETS.
Installed DC power supply in the power/network enclosure at turbulence tower. It is charging one battery for the beacon and two batteries for the data system and sensors.
Removed solar panels and other batteries from the site.
Fabian and Armando terminated and tested the fibers, installed fiber patch panels
at the tower enclosure and in the seatainer,…

Paul Painter from Green Mountain Falls Electric completed the installation of the electrical enclosure at the turbulence tower, and the connecting of the power cables at either end on October 7. So, line power should now be available at the turbulence tower.
The service should be 5 kVa, there should be two circuits each with outlets inside the enclosure and water proof outlets outside the enclosure.

Mostly Sunny. Cold and breezy.
Karl and I inspected "ding" in tower. Nothing was resolved of what could've done this dent. There's not paint/led/copper transfer for a bullet/arrow. It almost looks like somone tried to hang something over wrenched it. ??
TRH's sensors were removed and taken back to NCAR. Some of them were VERY dusty. I kept shields and cables all plugged in and at their heights. However, I did remove all TRH port fuses.…

The 16 meter TRH did not survive the power outages on Sep 16.
It was running fine up to the 03:20 am dropout but has just
given a few sporadic points since.
I logged into the manitou system and tried rserial to port 11
(rs 11), but got nothing back.

The data system at manitou died at around 03:20 am MDT on Sep 16,
and and came back at 8:30 am. It died again at 10:34 pm Sep 16 and
came up at 8:30 the next morning. Looks like we're not
getting enough solar charging.
It has stayed up since Chris G swapped batteries on Sep 18.

Stopped by the 45m tower and noticed a fire line that has been scratched around the tower and anchors. The line comes about five feet from anchors and approx. seven feet from solar panels. Looked a little close if the burn gets out-of-hand. Also swapped NDAQ and beacon batteries. The batteries on beacon were around 13V while the NDAQ were at 11V.…

TRH015, installed at 2m on Aug 12 18:51 UTC also reports
incorrect humidities. Its cutoff is around 45%. Above 45%
the reported humidity is anti-correlated with the 4 other
sensors - looking more like a temperature.
In this plot, the night of Aug 11-12 shows the problem with
unit 013, and the next night shows the problem with 015:
http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/20090812/trh_20090812.png

RH at 2m went walkabout on about Jul 30 02:30 MDT.
After that time it disagrees with the other RH's above ~60%. This looks like what we
saw in ASP09.
This may have happened during a rain event - the sonic diagnostic
flags were high during that time.
http://www.eol.ucar.edu/isf/projects/BEACHON_SRM/isfs/qcdata/plots/20090730/trh_20090730.png

Jul 17, 2009
Installed AP24, 802.11 access point on walkup tower, just above the 11th section. Ethernet IP: 192.168.100.202, Wireless IP: 192.168.12.251.
Borrowed 100 ft white cat 5 cable in server seatainer for the run from the network switch in the seatainer up the tower.
Installed a Patton Model 570 ethernet surge protector in the seatainer. I believe it was installed
between the white cable coming from the tower and the AP24 power injector.…