June 9, 2011

Present:

Absent:

CFHT

JAC

UH88

VLBA

Minutes

Current Network Status
During the last three months, networks out of Hilo have been plagued by
Oceanic fiber breaks and hardware problems at the Hilo SOB (State Office
Building). Pui Hin has discussed with UH-ITS the need for backup for the
SOB as well as the Oceanic fiber and the idea of using one of our HT
(Hawaiian Telcom) GE circuits. According to Chris Zane, using HT fiber
for backup is one of the options being considered. Also, the fiber to be
installed between Hilo and HP will continue to Mauna Lani along the Saddle
to create an INET ring. CFHT, IfA Hilo, and HP will all be on that INET
ring.

Here's an account of what happened:

On March 13 -- starting around 12:15pm -- a fiber break between Honokaa and
Hilo brought down both the INET and the SX path out of Hilo. Traffic was
rerouted via the OC3 microwave. Apparently, we were still relying on the
temporary fiber that was installed after the March 8 fiber break and the
temporary fiber cable was accidentally "trimmed" by state tree trimmers.

The SX through CFHT was restored around 2:45pm and the INET path was back up
around 3:10pm.

On March 17 Oceanic respliced the fiber that was broken on March 8 between
12:01 and 4:00. Resplicing was done strand by strand and so each site took
a short hit as it was taken off the temporarily fiber for the resplicing.

On April 7 the LuxN wave equipment at the Hilo SOB failed which brought down
the INET path for Hilo but the SX was ok. Traffic between Hilo and Oahu,
and the commodity traffic to the Internet were rerouted over the microwave.

On April 9 -- starting at about 4:00am -- SONNET equipment at Puu Kilea on
Maui also failed which resulted in the failure of the microwave in addition
to the loss of INET since April 7. The result was a complete interruption
of traffic between Hilo and Oahu including the commodity traffic to the
Internet. The microwave was restored at 5:20am.

On April 11 the LuxN equipment at the Hilo SOB was temporarily restored.

On April 25 -- starting around 9:30am -- the LuxN equipment at the Hilo SOB
once again started to malfunction causing Oahu and Internet bound traffic to
flap between the GE and the microwave path. The SX path was not affected.
Around 2:00pm, the management card was pulled from the LuxN box which kept
things stable until April 28.

On April 28 -- around 5:30am -- the LuxN box became unstable once again and
this time UH-ITS rerouted all Oahu and Internet bound traffic to the
microwave. The LuxN hardware at the SOB was finally replaced at around
9:15am.

On May 18 there was another Oceanic fiber break, this time near Hilo. Once
again connection to CFHT and both the SX and INET paths from Hilo were cut.
All traffic out of Hilo was rerouted through the microwave. Connection to
the SX via CFHT was restored at 2:55pm. The INET was restored at 6:00pm.

On May 31 -- at about 6:00pm -- the LuxN box at the SOB acted up again.
The INET was down until 8:00pm. The SX was not affected.

On June 1 the same thing happened and the INET was down between 4:41am and
6:24am.

On June 3 a firmware update on the LuxN box at the SOB seems to have
restored stability to the network connections out of Hilo.

FY 11/12 Budget
The Cisco maintenance budget has been increased by $3000 in anticipation of
additional equipment to support 10Gb upgrade at the MKOCN Hilo Network HUB.
The electricity budget has also been increased by $2000. There is a small
decrease in salary, due to an over-estimate last year. The budget this year
will be $93,294 vs $89,474 last year. However the rate per observatory will
be lower than last year -- $7,275 vs $8,002 after taking into consideration
the fund balance.

Brief Updates of Various Projects
VLBA - Fiber Connectivity to MKOCN

Despite earlier concerns, the project was approved by the Office of Mauna
Kea Management in late May. Although some of the conduits between the VLBA
and the Summit Communication Room (where the new VLBA fiber meets up with
the MKOCN fiber) do cut into the Natural Area Reserve, there are no handholes
in those areas so there will be no need for HT to access the area. Also HT
found an empty subduct that traverses the Puu which can be pulled out and
replaced with a new subduct system without having to walk along the Puu.

Fiber and subduct installation started on 6/1 and is expected to be ready
for testing on 7/6 with in-service date of 7/8.

Improvement of MK Summit Conduit Infrastructure

For the VLBA project, HT will be installing a 4-way multi-path subduct
product called Futurepath. Pui Hin was hoping initially to take advantage
of the pull and have HT install a 2-way system for future use by the MKOCN.
However she later backed out of it due to time constraints by the contract
between UH-ITS and the Naval Observatory, the cost quoted by HT ($12,500 for
approximately 1800' of conduit), and the fact that these subducts do not
come with pull strings. Instead, she will continue to persue an as-built
drawing of the current conduit system from HT and work with Bob McLaren on a
plan to provision for subduct space in the future.

Follow up from the Last Meeting
Documents pertaining to the Wireless Policy on Mauna Kea

In response to requests from several observatories, Pui Hin distributed
documents pertaining to the Mauna Kea wireless policy. The documents
include:

. A memo on the restriction of mobile and cellular telephone use within the
Mauna Kea Science Reserve, dated March 8, 1994.

. A memo covering the Policy on radio transmitters in the Mauna Kea Science
Reserve, dated April 2, 2001.

. A request from the radio telescopes to refrain from the use of wireless
and microwave technology on the summit of Mauna Kea.

Road-condition Mailing List

Pui Hin looked into the report that the road-condition mailing list was
heavily spammed but saw no evidence of it except for a very brief period in
February. Pui Hin speculated that perhaps one of the recipients had his/her
computer hacked at that time. Jonathan Chock who reported the problem also
confirmed that the problem seems to have dissipated.

The issues with IRTF -- that their users were not receiving -- is due to
configuration issues: that there were too many levels of aliases and include
files. The problem was resolved once the configurations were simplified.

Network Rack Security at HP

Pui Hin has made a request to Stewart Hunter to lock the door of the network
room. (This has happened since the last meeting - thanks to Stewart Hunter)

Weather System at HP

For the replacement/upgrade of the weather display system at HP, Pui Hin
suggested using a Google TV of some form that would need no maintenance:
a TV that comes with Google TV or a plain TV with a Logitech Revue box.
Of the two, the Logitech Revue box is preferable because the keyboard and
mouse combo that comes with it is more standard than the Google TV currently
available. However, Simon Chan -- who has experience with the Logitech
Revue -- pointed out concerns with the Logitech Revue related to certain
Java streaming and Flash Video issues. In light of this, Pui Hin will look
further into whether weather displays are affected by these problems before
deciding whether we need to go back to a regular PC and a big monitor. As
for the size of the screen, it was agreed that a TV with a screen size
between 32" and 37" will probably work best. Some of the weather pages look
almost too stretched on screens 40" and bigger.

Next meeting
September 8, Thursday, 10:00am at the JAC Base Facility, Hilo.