NASA to put science face on scary DARPA project

Space agency wants to use secretive organisation’s ‘Deep Web’ search tool Memex

Photo by: NASA JPL-Caltech
Photo by: NASA JPL-Caltech

Updated Jul 28, 2015

Researchers from NASA’s Jet Propulsion Laboratory (JPL) wish to use the Defense Advanced Research Projects Agency’s (DARPA) “Deep Web” search tool Memex to “harness its benefits,” according to a press release published on the space agency’s web site at the weekend.

“We're developing next-generation search technologies that understand people, places, things and the connections between them," said Chris Mattmann, JPL’s principal investigator on Memex.

In early 2014 DARPA announced the development of Memex, a hypothetical data storing device that was first proposed in Vannevar Bush’s 1945 article titled “As We May Think.” Bush was the director of US Office of Scientific Research and Development during the Second World War.

According to the article, Memex - a portmanteau of Memory and Index - would encompass all the data that an individual encounters or produces such as books, conversations and communications. The device would also be "mechanized so that it may be consulted with exceeding speed and flexibility."

Likewise, Darpa’s Memex will search not only standard content but also images, videos, pop-up ads, forms, scripts and all other forms of information available online and their interconnections.

“We're augmenting Web crawlers to behave like browsers - in other words, executing scripts and reading ads in ways that you would when you usually go online. This information is normally not catalogued by search engines,” Mattmann said.

Deep Web, a term first coined by Mike Bergman in the 2000s, refers to content which search engines can’t access and constitutes 95 percent of the internet, according to experts.

NASA and other scientific institutions which forms some of the 24 groups taking part in the open-source Memex project want to use its powerful algorithms to enhance scientific research such as space missions or oceanography and climate research, fields in which enormous and diverse amounts of data are produced.

“We are developing open source, free, mature products and then enhancing them using DARPA investment and easily transitioning them via our roles to the scientific community,” Mattmann said.