In the past, the piles of data such tools collect would have had only fleeting significance to government, and would have been mainly used as a warning sign of illicit activity or threats or dismissed as mere "noise." The ability to collect and analyze the information by connecting the dots changes that equation, Hunt said. "We fundamentally try to collect everything and hang onto it forever," he said.
Such sentiments are not new, but the scope of such programs are. The ability to store and retrieve terabytes of data was not an issue when the legal justification for the government surveillance was established back in the telegraph era. Telegraphers used "pen registers" to log incoming and outgoing messages, and the Supreme Court has long held that communications records are open books for law enforcement.
And there is a obvious need for law enforcement to access data, as writer David Simon, who termed the NSA disclosures a "faux scandal," noted. "There has always been a stage before the wiretap, a preliminary process involving the capture, retention and analysis of raw data. It has been so for decades now in this country," he wrote in his personal blog.
But surveillance on a massive scale turns the use of an individual's "metadata" into "another thing entirely," says Harry Surden, a University of Colorado–Boulder Law School associate professor and a former computer programmer who has published research on the topic. The biggest fundamental change may be the hardest to notice: The linking of data networks that leap across multiple signal carriers in even a simple telephone call. Such metadata paints a picture even if the messages themselves are not read.
"That's what we are seeing now," Surden says. "The structures that kept data apart historically are breaking down in all kinds of dimensions. Government is using it more and more, and they will keep doing it because tech advances have made it more feasible and less expensive."
Privacy advocate and Silicon Valley entrepreneur Mark Weinstein says the high-tech industry ushered in the loss of privacy by compiling immense amounts of user data that "the government can just come in and grab." The Guardian newspaper journalist Glenn Greenwald, using information from Snowden, reported that the NSA got access to data through telecoms, Internet providers and social networks, all of which have publicly defended users' privacy rights but complied with government demands for cooperation.
"Democracy depends on privacy," Weinstein says. "We live in an imperfect world with ethnic cleansing and religious and political persecution. Democracy and privacy have protected us from unscrupulous government."
The Obama administration has argued persuasively that it needs enhanced technology to fight terrorism. But the work needs to be done in a transparent and open environment, Schneier says. Without oversight, the potential grows for overzealous officials to engage in the kind of moves seen in the IRS's profiling of Tea Party organizations and other political groups.
While the IRS used only basic spreadsheet screening of computer data in those cases, the agency has launched a huge rollout of new data-mining technology and used it in error-prone robo-audits that yielded mixed results. Still it is pressing forward with much more use of technology without publicly unveiling how it is used.