The National Security Agency is winning its long-running secret war on encryption, using supercomputers, technical trickery, court orders and behind-the-scenes persuasion to undermine the major tools protecting the privacy of everyday communications in the Internet age, according to newly disclosed documents.
The agency has circumvented or cracked much of the encryption, or digital scrambling, that guards global commerce and banking systems, protects sensitive data such as trade secrets and medical records, and automatically secures the emails, Web searches, Internet chats and phone calls of Americans and others around the world, the documents show.
Many users assume or have been assured by Internet companies that their data are safe from prying eyes, including those of the government, and the NSA wants to keep it that way. The agency treats its recent successes in deciphering protected information as among its most closely guarded secrets, restricted to those cleared for a highly classified program code-named Bullrun, according to the documents, provided by Edward Snowden, the former NSA contractor.
Beginning in 2000, as encryption tools were gradually blanketing the Web, the NSA invested billions of dollars in a clandestine campaign to preserve its ability to eavesdrop. Having lost a public battle in the 1990s to insert its own "back door" in all encryption, it set out to accomplish the same goal by stealth.
The agency, according to the documents and interviews with industry officials, deployed custom-built, superfast computers to break codes, and began collaborating with technology companies in the United States and abroad to build entry points into their products. The documents do not identify which companies have participated.
The NSA hacked into target computers to snare messages before they were encrypted. In some cases, companies say they were coerced by the government into handing over their master encryption keys or building in a back door. And the agency used its influence as the world's most experienced code-maker to covertly introduce weaknesses into the encryption standards followed by hardware and software developers around the world.
"For the past decade, NSA has led an aggressive, multipronged effort to break widely used Internet encryption technologies," said a 2010 memo describing a briefing about NSA accomplishments for employees of its British counterpart, Government Communications Headquarters, or GCHQ. "Cryptanalytic capabilities are now coming online. Vast amounts of encrypted Internet data which have up till now been discarded are now exploitable."
An intelligence budget document makes clear that the effort is still going strong.
"We are investing in groundbreaking cryptanalytic capabilities to defeat adversarial cryptography and exploit Internet traffic," the director of national intelligence, James Clapper, wrote in his budget request for the current year.
The NSA's success in defeating many of the privacy protections offered by encryption does not change the rules that prohibit the deliberate targeting of Americans' emails or phone calls without a warrant. But it shows that the agency, which was sharply rebuked by a federal judge in 2011 for violating the rules and misleading the Foreign Intelligence Surveillance Court, cannot necessarily be restrained by privacy technology. NSA rules permit the agency to store any encrypted communication, domestic or foreign, for as long as the agency is trying to decrypt it or analyze its technical features.
The NSA, which has specialized in code-breaking since its creation in 1952, sees that task as essential to its mission. If it cannot decipher the messages of foreign adversaries, the United States will be at serious risk, agency officials say.
Relying on encryption
Some experts say the NSA's campaign to bypass and weaken communications security may have serious unintended consequences. They say the agency is working at cross-purposes with its other major mission, apart from eavesdropping: ensuring the security of U.S. communications.
Some of the agency's most intensive efforts have focused on the encryption in universal use in the United States, including the protection used on smartphones. Many Americans, often without realizing it, rely on such protection every time they send an email or buy something online.
For at least three years, one document says, Britain's GCHQ, almost certainly in close collaboration with the NSA, has been looking for ways into protected traffic of the most popular Internet companies: Google, Yahoo, Facebook and Microsoft's Hotmail. By 2012, GCHQ had developed "new access opportunities" into Google's systems, according to the document.
"The risk is that when you build a back door into systems, you're not the only one to exploit it," said Matthew Green, a cryptography researcher at Johns Hopkins University. "Those back doors could work against U.S. communications, too."
Paul Kocher, a leading cryptographer who helped design the SSL protocol, which provides communication security over the Internet, recalled how the NSA lost the heated national debate in the 1990s about inserting into all encryption a government back door called the Clipper Chip.
"And they went and did it anyway, without telling anyone," Kocher said. He said he understood the agency's mission but was concerned about the danger of allowing it unbridled access to private information.
"The intelligence community has worried about 'going dark' forever, but today they are conducting instant, total invasion of privacy with limited effort," he said. "This is the golden age of spying."
The documents are among more than 50,000 shared by The Guardian with The New York Times and ProPublica, the nonprofit news organization. They focus primarily on GCHQ but include thousands either from or about the NSA.
Intelligence officials asked The Times and ProPublica not to publish this article, saying that it might prompt foreign targets to switch to new forms of encryption or communications that would be harder to collect or read. The news organizations removed some specific facts but decided to publish the article because of the value of a public debate about government actions that weaken the most powerful tools for protecting the privacy of Americans and others.
The full extent of the NSA's decoding capabilities is known only to a limited group of top analysts.
The files show that the agency is still stymied by some encryption, as Snowden suggested in a question-and-answer session on The Guardian's website in June.
"Properly implemented strong crypto systems are one of the few things that you can rely on," he said, though cautioning that the NSA often bypasses the encryption altogether by targeting the computers at one end or the other and grabbing text before it is encrypted or after it is decrypted.
Ties to Internet companies
Because strong encryption can be so effective, classified NSA documents make clear, the agency's success depends on working with Internet companies by getting their voluntary collaboration, forcing their cooperation with court orders or surreptitiously stealing their encryption keys or altering their software or hardware.
According to an intelligence budget document leaked by Snowden, the NSA spends more than $250 million a year on its Sigint Enabling Project, which "actively engages the U.S. and foreign IT industries to covertly influence and/or overtly leverage their commercial products' designs" to make them "exploitable." Sigint is the abbreviation for signals intelligence, the technical term for electronic eavesdropping.
By this year, the Sigint Enabling Project had found ways inside some of the encryption chips that scramble information for businesses and governments, either by working with chipmakers to insert back doors or by surreptitiously exploiting existing security flaws, according to the documents.
The 2013 NSA budget request highlights "partnerships with major telecommunications carriers to shape the global network to benefit other collection accesses" that is, to allow more eavesdropping.
Since Snowden's disclosures ignited criticism of overreach and privacy infringements by the NSA, U.S. technology companies have faced scrutiny from customers and the public over what some see as too cozy a relationship with the government. In response, some companies have begun to push back against what they describe as government bullying.
Google, Yahoo and Facebook have pressed for permission to reveal more about the government's secret requests for cooperation. One small email encryption company, Lavabit, shut down rather than comply with the agency's demands for what it considered confidential customer information; another, Silent Circle, ended its email service rather than face similar demands.
In effect, facing the NSA's relentless advance, the companies surrendered.