FBI's Sentinel Project: 5 Lessons Learned

Agency used agile development and private sector know-how to finish its long-delayed digital case management system.

American Red Cross Social Media Command Center

(click image for larger view and for slideshow)

After six years of development, the FBI says its next-gen digital case management system, Sentinel, is finally up and running. FBI agents can now use the system to manage records electronically, with document templates, drop-down menus, and many other PC-like features.

Sentinel had been a case study in federal IT projects gone awry--missed deadlines, budget overruns, feature shortcomings, and a benchmark test last October that pooped out. The FBI Inspector General, in a 2010 report on Sentinel, cited "significant issues and concerns." FBI director Robert Mueller faced a grilling from Congress on how, when, and at what cost this all-important project would be completed.

Keep in mind that Sentinel has roots in an earlier IT project failure, the so-called Virtual Case File system. The agency pulled the plug on that effort in 2005 after pouring $170 million into it. So this week's announcement that Sentinel, as of July 1, became available to all FBI employees is a major achievement. Mueller, in a written statement, called it "an important step forward" for the FBI.

Whether the system will work as advertised and be accepted by the agency's rank and file remains to be seen. I've been following the Sentinel project closely for the past few years. Here are five takeaways.

1. Private sector expertise is valuable. The first step in Sentinel's turnaround was the recruitment of a private sector IT executive, Chad Fulgham, to oversee it. Mueller brought in Fulgham, a former senior VP of IT with brokerage firm Lehman Bros., as CIO in December 2008. Mueller said Fulgham's business experience would "fit well" with the FBI's needs. It wasn't long before Fulgham hired Jeff Johnson, also a former Lehman Bros. technologist, who is now the FBI's CTO. (Fulgham left the FBI in April 2012. More on that below.)

2. Agile development gets things done. The next big shift in strategy was Fulgham's decision in September 2010 to wrest control of the project from prime contractor Lockheed Martin and use agile development to accelerate software deliverables. The thinking was that a hands-on, incremental approach would be faster because functionality would be developed, and adjustments made, in two-week "sprints." The FBI missed its target date for finishing that work--September 2011--but it credits the agile methodology with ultimately getting the job done.

3. Commercial software plays an important role. Sentinel is based in part on commercial software, a fact that's often overlooked because of all the custom coding and systems integration involved. Under the hood are EMC's Documentum document management software, Oracle databases, IBM's WebSphere middleware, Microsoft's SharePoint, and Entrust's PKI technology. Critics who say that Sentinel would have gone more smoothly if only it had been based on off-the-shelf software seem unaware that, in fact, it is.

4. Agile development is cheaper, too. Sentinel came in under its $451 million budget. The caveat is that the FBI's original cost estimate for Sentinel was $425 million, but that was before Fulgham and Johnson took over, and they stayed within the budget they were given. The Inspector General might quibble with how the FBI accounts for the total project cost, having pointed out in the past that its tally didn't reflect the agency's staff costs. But the FBI wasn't forced to go to Congress with its hand out. Agile development wasn't only faster, but also cheaper.

5. Don't deploy new software on old hardware. The FBI learned that lesson the hard way in October when the system, during a four-hour test involving 743 users, suffered two outages. The agency made the mistake of running the test on legacy hardware, which it was forced to upgrade prior to a broader rollout. That caused a delay just as the IT team was approaching the finish line.

Johnson offered the first public demonstration of Sentinel this week at FBI headquarters. (Here's my review of the system from a demo I was given a few months earlier.)

Fulgham wasn't around to participate in the unveiling. He left the FBI this spring to take a job with Lockheed Martin--yes, the same company he elbowed aside in 2010 when choosing the agile development path. In his new role, Fulgham is a VP with Lockheed Martin's information systems and global solutions division, where he works with U.S. and international defense agencies.

In a phone call this week, Fulgham said he knew he would take heat for going to work for the same company that, as the FBI's CIO, he forced into a backup role on Sentinel. But he insists that Lockheed continued to be a key partner on the project, as well as on other FBI initiatives, such as its Next Generation Identification system.

Sentinel is "arguably the most important application at the FBI," and agile development turned out to be the right way to complete it, Fulgham told me this week. There's a lot of talk about how federal agencies can benefit from the best IT practices of the private sector. Sentinel's turnaround is the latest chapter in that play book.

The Office of Management and Budget demands that federal agencies tap into a more efficient IT delivery model. The new Shared Services Mandate issue of InformationWeek Government explains how they're doing it. Also in this issue: Uncle Sam should develop an IT savings dashboard that shows the returns on its multibillion-dollar IT investment. (Free registration required.)

ISentinel is online. Great. but I don't agree with what's said in this article too much. Commercial software playing an important role? Sort of. Documentum is not a big piece of it and only helps FBI get its archive certification 50-whatever. It's not used for primary document storage. Private sector experience helping? Perhaps. I think kicking out Lockheed had more impact than hiring Fulgam. If anything it says something about how hard it is to coordinate a team of 300 people like the original program.

This is good news. Maybe next time the FBI will file lawsuits against the financial companies who held this country hostage because it seems like when the economy tanked, it was all planned, and seems like if it would have been stopped, it would have had been a big case; possibly to test Sherman Antitrust.

Secondly, my neighbor, who recently moved, and was a special investigator at The FBI wasn't very technically savvy. Agile Project Methodology works very well, which we used at other companies. But when my neighbor yelled at me for hacking into the FBI when I installed Firefox (FBI doesn't use Internet Explorer) I was a little concerned about the technical capabilities within the FBI. Sharepoint is a wise choice. It will likely have a vulnerability sometime soon, which might be patched with the right IT and change management teams.

When I asked for a job at the FBI, at first, he said I'd have to move to Silicon Valley, so I considered this for a while.

When I asked him again, my neighbor said I had to be Catholic. Which was weird words to hear come out of the mouth of a FBI director-of-sorts, who investigates discrimination issues to the Supreme Court, including the recent Costco court cases where someone decided to challenge being passed up on a promotion to the US Supreme Court, through his office.

I really like Costco, and wondered why he recommended that case through his office. Anyways, he's taken some time off from work lately, likely because of back issues. I caught him picking up heavy boulders in his front lawn, and took some videotape of it. When I asked him if he needed any help, he said No, and was looking forward to having next week off. Which he took off, after his back surgeon wrote a strong prescription for the backpain he was having.

I don't understand him sometimes. Costco is a great place to shop, but in order to keep costs low, you have to have productive employees, who make customer service a priority.

I was talking to my former neighbor, who is a Director at the FBI about this and other issues at the FBI. He investigates labor issues to the Supreme Court, including the recent issue at CostCo, where a bunch of women weren't doing their job and passed up on promotions. It's a sad story for sure, but you'll have to read the supreme court case documents, which will likely be available on Sentinel in a few months when someone figures out how to use the program.

Anyways, James wanted a new computer, so as a security-minded individual, I installed FireFox on it, and when I did, he thought I hacked into the FBI and threatened to make a phone call.

Apparently they use FireFox because it's more secure, and rightfully so. Google had its sourcecode stolen with Internet Explorer, and it created some work for the FBI (not labor related) Anyways, by installing the free program (not pirated), with that icon on the desktop, he thought I hacked into the FBI.

Anyways, it seems that a lot of idiots at the FBI need retraining, or possibly to retire, even this guy who runs a "Special Investigations Team" in Labor, or perhaps he needs a team to help him figure out how the hell to use technology.

Based on this article, it seems the FBI can't secure itself, Agile methodology probably works, until someone finds a vulnerability in Documentum, or Sharepoint. This shouldn't take long, and it will likely go unpatched for a while, just long enough for some technical gumshoe.

I applied for a job at the FBI but they said I had to be Catholic. Which I think is odd to hear coming from the mouth of a man who investigates discrimination issues at the FBI. Since then, I saw him picking up 200-lb heavy boulders in his front yard. When I asked him if he needed any help, he said no. Later, he said his back hurt and he took a lot of time from work, probably meaning that the FBI couldn't file new lawsuits relating to labor issues in Finance or New York. Which is great news for The Financial Industry, who held the American Public at gunpoint, and forced a number of legislative changes including passing of a bill worth $787 Billion Dollars ($787,000,000.00) to recapitalize banks which apparently were a little short on money in the vault.

So if you work at the FBI, I've created an exercise for you; a game of sorts- It's called "Find The Guy Who Doesnt Like Costco, And Reccomended A Supreme Court Case" I've given you a profile. So find that person, because he's likely going to retire soon.