Permanent video monitoring for continuous improvement

NCSC – Building the Competition System

Update 05/14/15: I added a few more details on the potential staffing requirements.

The system we wanted to build is shown in the following diagram. We wanted to push HD video feeds from 9 IP cameras and distribute their video along three independent paths. One path continuously archived all 9 HD video streams. A second path carried 9 live HD video feeds to the 9 judging rooms. The third path converted and uploaded the 9 feeds as discrete, labeled HD video clips to a Web platform for offline viewing.

To achieve this, we had to deal with several potential bottlenecks:

The most serious concern was that we had to move 18 separate HD streams over a standard campus LAN and we had to display the video for judges on 9 standard classroom computers. These computers were used “as is”. There was no opportunity to re-image them prior to the competition. The LAN was nominally a solid Gigabit system, but we had to take it on faith. As it happened, the competition occurred over spring break and there were indications that the University IT staff took the opportunity to “reconfigure” the network while the competition was underway.

The second concern was that we had 30 minutes to capture and ingest 9 twenty-minute HD video streams, transcode and meta-tag them and push them up to a web platform in the cloud. With 9 new sessions every 30 minutes from 8am to 11pm, we could not afford to fall behind.

The two relevant technologies (Bosch and Dartfish) had more or less complementary strengths and weaknesses.

The Bosch system is a mature, extremely well-engineered surveillance technology. As such, it has exceptional video compression and communication stability that starts at the cameras. The Dinion cameras can broadcast 4 independent HD streams simultaneously. Our configuration meant that each camera would supply video to up to 3 different locations. The video to the judging rooms was initially configured to be broadcast from the archive server, but we had the option of taking it directly from the cameras if the server began to struggle. As it happened, the server was fine.

The Dartfish desktop TeamPro software is very good at accepting video from Onvif-standard IP cameras. That meant that they could pull HD video directly from the Bosch cameras. The fact that the cameras can broadcast multiple streams makes that easy. The Dartfish desktop software can publish to the Dartfish.TV channel with relative ease. Doing it in real time would be challenge, but it would be doable if we had enough desktops and enough people to man them.

So far, so good.

The New Challenge

The NCSC organizers added two big requirements for 2014. The previous year (2013), everything was done by trained operators who manned PCs with the Dartfish TeamPro software in each judging room. There was no attempt to push the results to the Web until after the entire competition was over. With 9 rooms and an 15 hour day, it still required a sizeable staff.

Our new goal was to do synchronized web upload and do it with far fewer people. Ideally, they wanted a totally hands-free system. We knew that wasn’t going to happen, but we were under constant pressure to reduce headcount and use student volunteers instead of trained operators.

Oh … and the organizers didn’t want to spend any more money than the cost of Bosch cameras and servers … plus the free Dartfish licenses that Dartfish donated for the duration of the competition. … Our solution had to entail no extra cost.

Our Initial Hope

Our first idea was to use the Bosch system for much of the work. It is so well engineered that we were confident it could move all of the video around to where it needed to go … except up to the Web. Typical of a surveillance system, it stores the video archive in a proprietary file format designed to prevent tampering. Even though all of the video we needed was on the Bosch server in glorious HD, there was no way to export it in real time. The Bosch server can stream low quality, real-time video through its built-in web server, but not meta-tagged HD video.This is likely a manifestation of the fact that the surveillance industry builds technology for surveillance … not session archiving and video sharing.

Our Second Hope

The second idea was to grab video feeds directly from the Bosch cameras, then use Dartfish software to convert and upload it. This almost worked. The problem is that there aren’t very many tools that are designed to efficiently pull HD video feeds from IP cameras … except for surveillance systems … and we already ruled those out. We downloaded and tried at least 20 different stand-alone video recorders … and none of them could perform reliably at this level. We ran them on multiple PCs with core i7s and SSD drives … still no luck.

Our Last Hope

More than half of the solution came in the form of a beta copy of Dartfish’s new OnAir software. It is still not released to the public, but OnAir will watch a designated set of file folders and push any new video files it finds to a designated location on the Dartfish.TV server. To be conservative, we allocated 3 core i7, SSD laptops to the task … 3 competition sessions each. The photo shows what it looks like when all three were running. Each item is a video file and each green area beside it is an indicator that it was successfully imported, converted and uploaded … totally hands free.

The OnAir took all of the labor out of uploading video. Even better, it could read the file names and generate metatags automatically. So we were remarkably close to a system that looked like this:

We just needed one thing to make a really sweet system. We needed the item shown in red … a way to capture feeds from nine Onvif-standard IP cameras and save them as MP4 video files to a network drive. Should be easy. We had dozens of computer lab computers to do it with.

Oh … wait … we were running out of time … and we had literally no budget left.

What we Did

When we ran out of time, we installed 9 Dartfish TeamPro licenses on 9 lab computers and used them to capture the IP camera video as MP4 files. We arranged the computers in banks so one person could trigger recording on 3 or 4 computers when each competition session began. The lab computers were on the same floor as the competition rooms, so the operators could literally hear the countdown for each 20 minute session.

This configuration worked like a charm. It captured 240+ 20 minute HD video clips the first day and nearly 350 overall. The average time to get them online and viewable in full HD (like the following video of the National RunnerUp, Ted Kirk) was 13 minutes. In other words, the competitor’s video was online for sponsors and recruiters to view less than 15 minutes after the end of their session.

How Close We Came

We met all of the performance criteria. Every session was recorded, judged and published. The fragile campus network crashed (not our software … an entire subnet) and caused recording errors. We quickly retrieved the sessions from the backup Bosch server and pushed them back into the workflow.

All of the work was done by just a few volunteers, with two DeeperPoint personnel (author included) as operations and recovery experts. We were called on several times, but never for a failure in the system we installed. Mostly we were called because other parts of the campus network and competition systems had LAN or PC problems. Not our problem, but we tried to help as best we could.

If we had been able to fully automate the missing IP camera recording task, we could have run the entire competition with two, or conceivably one, operator. The actual workload could be handled by one person, but the long days and intense activity would demand a second person for relief.

Last Thoughts

Our goal of setting up a super sweet, low cost, high performance system came within an eyelash of reality. We had to over-design that one step of grabbing the IP camera feeds and storing them on a hard drive. It shouldn’t take 9 Dartfish licenses to pull it off. Hopefully, that will never happen again. Once that piece is in place, the system we built can compete with anything … if you want:

The highest practical quality of video recordings and files of live performance activities

Video for Operations Blog

DeeperPoint has been exploring applications of video (including the Dartfish video software technology) for analyzing, visualizing and coaching business operations ... especially in manufacturing, retail and logistics.
The following posts touch on different insights and observations from our explorations.