July 22, 2016: Military procurement bureaucrats are historically the source of epic misconduct and incompetence. This sort of thing goes back thousands of years and continues today even in the most technically advanced nations. An example of this, a dispute between SOCOM (Special Operations Command) and U.S. Army procurement officials, over a about computer software has become a classic example. It has gotten so bad that the publisher of the software preferred by the troops (Palantir) is suing army procurement officials who block the use of their software.

Palantir is the creator of a superior intel analysis system favored by troops allowed to use it. Unlike the troops, who have to follow orders and back off, Palantir refuses to quietly let the army spend billions on a less efficient software system that still does not work. In June 2016 Palantir Technologies told the army it was taking the dispute to court. This could get really ugly because there has already been a lot of testimony from troops who had used Palantir but were then forced to switch to the less effective (and not yet complete) DCGS (Distributed Common Ground System). Worse for the army a confidential Government Accountability Office investigation sided with Palantir.

Troops in combat zones and especially SOCOM prefer to use an intelligence database management system called Palantir. But many senior people in the army intelligence and procurement bureaucracies insist on using another system (DCGS). To further complicate matters it wasn’t even the army who initially created the mess.

It all began when the U.S. Air Force developed a data mining and analysis system that, when adapted for army use (as DCGS), turned out to be more expensive and less effective than commercial products (like Palantir). A 2012 government investigation reported the problems in great detail. But senior army commanders and Department of Defense procurement bureaucrats continued to block the use of commercial products the troops preferred. SOCOM troops have been complaining that since 2010 a superior system (Palantir) they have been using since 2009 is becoming more difficult to obtain because of more aggressive interference from the procurement bureaucracy and contractor lobbyists. SOCOM was ordered back into Iraq during 2014 and one of their assignments was to collect intelligence on what was going on there. SOCOM preferred Palantir but many procurement officials interfered with doing that.

The basic problem was that the army system (DCGS) was cobbled together on the fly, in the midst of a war and has not aged well. Several investigations, in response to growing complaints from the troops, found that the army refused to recognize the problems with DCGS or get them fixed, or allow cheaper and more capable commercial software (like Palantir) to be used instead of DCGS. After 2010 complaints from users and maintainers of DCGS got louder (as in more politicians receiving emails about it). Some of the troops asked for specific commercial systems that were more robust, powerful, and easier to use commercial data mining and predictive analysis software. The army complained that these commercial systems were expensive and required a lot of effort and money to integrate into DCGS. The troops insisted that this was not so and that commercial products like Palantir would save lives. Army bean counters insisted that it was probably only a few dozen lives at most and the additional money needed has to be taken from somewhere else, which might also cost lives in combat. But SOCOM and other organizations point out that they have been able to sneak Palantir into service in some areas and have lots of proof that Palantir outperformed DCGS in combat conditions. But even SOCOM was blocked from getting Palantir even though Palantir is officially approved for army use and SOCOM is supposed to be able to buy whatever they need, even if it is not on the “approved” list.

The DCGS controversy also involves professional pride, as the army techs and managers have spent years building DCGS and are confident they can match any commercial products and do it cheaper. But that is rarely the case, as the army simply can't hire the best software engineers and project managers. When it comes to complex software systems, things go better if you keep an eye on the commercial side. If there is something there that does what you need done and does it faster, better, and more reliably it's worth paying the commercial price.