With the September attack on the U.S. consulate in Benghazi, Libya, that resulted in the death of U.S. Ambassador Christopher Stevens and three other Americans, Obama administration officials have fought a slew of accusations blaming them for not foreseeing the attack, or not acting in time to protect personnel.

One counter-argument has focused on the basic assertion that there were no clear indications that an attack would occur, a point reiterated recently by the U.S. government's intelligence czar. “If people do not emit or discuss their behavior, it's hard to find out what they are going to do,” Director of National Intelligence Jim Clapper said at the U.S. Geospatial Intelligence Foundation's annual forum in Orlando, Fla., in October.

Yet at the same conference where Clapper made his remarks, companies selling software to the defense and intelligence community were trying to do just that: help anticipate attacks such as the one that took place in Benghazi. “There were indications that there would be a protest in front of the U.S. [Consulate] in Libya at 4 p.m. the day before it took place,” claims Andrew Doumitt, the vice president of business development for TerraGo Technologies, a company that makes software that allows users to sift through multiple data sources, including social media, based on specific geographic areas.

In fact, TerraGo's software is, in many ways, custom-suited to looking at something like a potential attack on a U.S. base or embassy because it can trawl through millions of social media postings and flag information based on a specific location. “Let's say that we wanted to do a buffer zone around the U.S. Embassy in Sana'a, [Yemen]” says Doumitt. “We can set up an alert; we can start by seeing what's being written about in this area.”

The software may not indicate exactly which events may spiral out of control, such as what was alleged at one point in Benghazi, but it can provide a red flag. “You essentially have a way to monitor a place against social media, news and blogs; you can much more specifically react and monitor a whole lot more sources for trigger terms,” says Doumitt.

More than just a novelty, such software is part of a growing government and private sector market for data-mining that combines open-source information with more traditional data collection. The defense and intelligence community is increasingly using such tools, and In-Q-Tel, the CIA's venture capital arm, has invested in a number of companies working on these tools, including TerraGo.

While such software may have once been seen as a niche area, the wars in Afghanistan and Iraq, where the U.S. military is trying to disrupt insurgent networks and locate roadside bombs, has moved this work into high demand. It has also sparked a public battle over the Pentagon's investment in the Distributed Common Ground System-Army (DCGS-A), the military system used for combining and sorting through intelligence collected in the field.

At the center of the battle is Palantir, a Palo Alto, Calif.-based company that has built software programs that look for hidden connections in data. Palantir, which also received financial backing from In-Q-Tel, has quickly moved into a marketplace once dominated by traditional defense companies.

While the U.S. Army has provided Palantir software to a limited number of analysts in the field to help track improvised explosive device (IED) networks, it has favored sticking with its homegrown system, the DCGS-A. Supporters of the Silicon Valley-based Palantir have argued its software offers superior capabilities.

Jonathan Percy, vice president for homeland security and cyber at Overwatch, an operating unit of Textron Systems, calls claims of Palantir's capabilities “nonsense,” arguing that Palantir can only do “5%” of the total mission performed by DCGS-A. Overwatch, which builds the data analysis tools for DCGS-A, has been ramping up for its own battle with Palantir, including a recent move into homeland security and law enforcement markets.

“What they're trying to do is set themselves up,” says Percy of Palantir. “They're trying to force the Army to buy Palantir and just get rid of the investment.”

But Palantir has also found allies in Congress; Rep. Duncan Hunter (R-Calif.) has accused the Army of altering a test report that gave a positive review of Palantir's software compared with DCGS-A. Congress over the summer launched an investigation into the Army's handling of Palantir.

These sorts of data-mining tools are moving well beyond the current war in Afghanistan and are of interest in areas like Africa, where the U.S. military is tracking terrorist and insurgent networks. There, too, social media is a central focus, particularly in areas where traditional sources of intelligence, such as airborne sensors, may be in short supply or prohibited.

“Our sensors are a scarce resource,” says Tony Frazier, a senior vice president at GeoEye, a commercial satellite company, which also has an analytics unit. “If you want to take eyes off of Afghanistan, Iran or wherever the key hot spot is, then you need to be able to tap into a broader source of content.”

Frazier says GeoEye already is working on an Africa-focused project that is culling social media data to help get information that might otherwise only be collected through traditional signals intelligence. He declined to offer further details.

Of course, such analytic work is drawing on far more than social media: Agencies, whether military or law enforcement, can use their own proprietary databases, news articles, or even data from classified sources. While much of GeoEye's analytics work for Special Operations Forces and the intelligence community is classified, the company does openly discuss some aspects of its simulations, such as looking abroad at production sites for IEDs, or in the U.S. for methamphetamine labs.

The company also recently released a geospatial analysis of Al Shabaab, the Somalia-based wing of Al Qaeda. Using the company's Signature Analyst program, GeoEye says it was able to identify areas not previously recognized as high-threat where Al Shabaab might emerge.

The ability to use analytics to make predictions—whether about IED production sites or locations of future terrorist attacks—may sound attractive, particularly when looking at a catastrophic event like Benghazi, but the reality is that this sort of forecasting still falls far short of being a crystal ball. GeoEye claims it had 66% accuracy for predicting attacks for one government client.

In that case, the benefit is really in helping someone know where to look rather than in predicting a discrete event, according to James Anderson of GeoEye's analytics division. “Sixty-six percent by itself sounds not-that-much-better than a coin toss, but when you are eliminating 98% of your terrain that's a very significant reduction,” he says.

Lockheed Martin is also offering its own program for analyzing social media, called LM Wisdom, which it touts as “transforming Internet chatter into usable intelligence.” Lockheed started Wisdom about five years ago, but at the time was collecting information primarily from news sources. The company has more recently moved into collecting from social media, according to Ollie Luba, Lockheed principal for open-source intelligence integration.

Meantime, private companies increasingly are also concerned about what social media postings on Twitter, Facebook or LinkedIn might mean. “We have some commercial clients where we're looking at their security postures,” says Luba. “We're looking at when organizations are starting to protest against them. You can pick up some of those early protests right away. We start watching that and looking at the trends to see if it could be something or not.”

Lockheed Martin is also the prime contractor for a program sponsored by the Defense Advanced Research Projects Agency called the Integrated Crisis Early Warning System, which uses news media to forecast specific events, such as revolutions or political instability. “Probably our next step in rolling out some of those analytics will be taking in social media,” says Luba.

How much precision can these sort of analytic capabilities provide the defense and intelligence community? One problem of any software program is that it is only as good as the data it is being fed, a lesson learned from an October Senate report that blasted the Homeland Security Department's fusion centers, which are supposed to collect and analyze domestic intelligence to help identify potential terrorist plots. The report said those centers had “not produced useful intelligence to support federal counterterrorism efforts,” quoting one official as saying it produced “a bunch of crap.”

Indeed, when it comes to fusing data, whether in the U.S. or abroad, the ultimate question is whether millions of daily Tweets and Facebook postings can be transformed into useful intelligence. “Posting something on Facebook is not in and of itself evidence,” Secretary of State Hillary Clinton said in response to revelations that a militant group took credit for Benghazi just hours after the attack. “It just underscores how fluid the reporting was at the time and continued for some time to be.”

In the end, it could be that the work of companies like TerraGo may prove that the truth lies somewhere in the middle. While Facebook and other forms of social media would not have predicted the Benghazi attack, it could have—if monitored and analyzed in real-time—provided early warnings that something was brewing and that perhaps more resources were needed in the area.

Doumitt, for his part, avoids extravagant claims. He says that data, like the Facebook posting linked to the U.S. Consulate prior to the Benghazi attack, would simply have indicated plans for a large demonstration. “It's not magic.”