Special Report: Inside the West’s failed fight against China’s ‘Cloud Hopper’ hackers

[June 26, 2019]  By Jack Stubbs, Joseph Menn and Christopher Bing

LONDON (Reuters) - Hacked by suspected Chinese cyber spies five times from 2014 to 2017, security staff at Swedish telecoms equipment giant Ericsson had taken to naming their response efforts after different types of wine.

Pinot Noir began in September 2016. After successfully repelling a wave of attacks a year earlier, Ericsson discovered the intruders were back. And this time, the company’s cybersecurity team could see exactly how they got in: through a connection to information-technology services supplier Hewlett Packard Enterprise.

Teams of hackers connected to the Chinese Ministry of State Security had penetrated HPE’s cloud computing service and used it as a launch pad to attack customers, plundering reams of corporate and government secrets for years in what U.S. prosecutors say was an effort to boost Chinese economic interests.

The hacking campaign, known as “Cloud Hopper,” was the subject of a U.S. indictment in December that accused two Chinese nationals of identity theft and fraud. Prosecutors described an elaborate operation that victimized multiple Western companies but stopped short of naming them. A Reuters report at the time identified two: Hewlett Packard Enterprise and IBM.

Yet the campaign ensnared at least six more major technology firms, touching five of the world’s 10 biggest tech service providers.

Also compromised by Cloud Hopper, Reuters has found: Fujitsu, Tata Consultancy Services, NTT Data, Dimension Data, Computer Sciences Corporation and DXC Technology. HPE spun-off its services arm in a merger with Computer Sciences Corporation in 2017 to create DXC.

Waves of hacking victims emanate from those six plus HPE and IBM: their clients. Ericsson, which competes with Chinese firms in the strategically critical mobile telecoms business, is one. Others include travel reservation system Sabre, the American leader in managing plane bookings, and the largest shipbuilder for the U.S. Navy, Huntington Ingalls Industries, which builds America’s nuclear submarines at a Virginia shipyard.

“This was the theft of industrial or commercial secrets for the purpose of advancing an economy,” said former Australian National Cyber Security Adviser Alastair MacGibbon. “The lifeblood of a company.”

Reuters was unable to determine the full extent of the damage done by the campaign, and many victims are unsure of exactly what information was stolen.

Yet the Cloud Hopper attacks carry worrying lessons for government officials and technology companies struggling to manage security threats. Chinese hackers, including a group known as APT10, were able to continue the attacks in the face of a counter-offensive by top security specialists and despite a 2015 U.S.-China pact to refrain from economic espionage.

The corporate and government response to the attacks was undermined as service providers withheld information from hacked clients, out of concern over legal liability and bad publicity, records and interviews show. That failure, intelligence officials say, calls into question Western institutions’ ability to share information in the way needed to defend against elaborate cyber invasions. Even now, many victims may not be aware they were hit.

The campaign also highlights the security vulnerabilities inherent in cloud computing, an increasingly popular practice in which companies contract with outside vendors for remote computer services and data storage.

“For those that thought the cloud was a panacea, I would say you haven’t been paying attention,” said Mike Rogers, former director of the U.S. National Security Agency.

Reuters interviewed 30 people involved in the Cloud Hopper investigations, including Western government officials, current and former company executives and private security researchers. Reporters also reviewed hundreds of pages of internal company documents, court filings and corporate intelligence briefings.

HPE “worked diligently for our customers to mitigate this attack and protect their information,” said spokesman Adam Bauer. “We remain vigilant in our efforts to protect against the evolving threats of cyber-crimes committed by state actors.”

A spokesman for DXC, the services arm spun off by HPE in 2017, said the company put “robust security measures in place” to protect itself and customers. “Since the inception of DXC Technology, neither the company nor any DXC customer whose environment is under our control have experienced a material impact caused by APT10 or any other threat actor,” the spokesman said.

NTT Data, Dimension Data, Tata Consultancy Services, Fujitsu and IBM declined to comment. IBM has previously said it has no evidence sensitive corporate data was compromised by the attacks.

The Chinese government has denied all accusations of involvement in hacking. The Chinese Foreign Ministry said Beijing opposed cyber-enabled industrial espionage. “The Chinese government has never in any form participated in or supported any person to carry out the theft of commercial secrets,” it said in a statement to Reuters.

BREAK-INS AND EVICTIONS

For security staff at Hewlett Packard Enterprise, the Ericsson situation was just one dark cloud in a gathering storm, according to internal documents and 10 people with knowledge of the matter.

For years, the company’s predecessor, technology giant Hewlett Packard, didn’t even know it had been hacked. It first found malicious code stored on a company server in 2012. The company called in outside experts, who found infections dating to at least January 2010.

Hewlett Packard security staff fought back, tracking the intruders, shoring up defenses and executing a carefully planned expulsion to simultaneously knock out all of the hackers’ known footholds. But the attackers returned, beginning a cycle that continued for at least five years.

The intruders stayed a step ahead. They would grab reams of data before planned eviction efforts by HP engineers. Repeatedly, they took whole directories of credentials, a brazen act netting them the ability to impersonate hundreds of employees.

The hackers knew exactly where to retrieve the most sensitive data and littered their code with expletives and taunts. One hacking tool contained the message “FUCK ANY AV” – referencing their victims’ reliance on anti-virus software. The name of a malicious domain used in the wider campaign appeared to mock U.S. intelligence: “nsa.mefound.com”

Then things got worse, documents show.

After a 2015 tip-off from the U.S. Federal Bureau of Investigation about infected computers communicating with an external server, HPE combined three probes it had underway into one effort called Tripleplay. Up to 122 HPE-managed systems and 102 systems designated to be spun out into the new DXC operation had been compromised, a late 2016 presentation to executives showed.

An internal chart from mid-2017 helped top brass keep track of investigations codenamed for customers. Rubus dealt with Finnish conglomerate Valmet. Silver Scale was Brazilian mining giant Vale. Greenxmass was Swedish manufacturer SKF, and Oculus covered Ericsson.

Projects Kronos and Echo related to former Swiss biotech firm Syngenta, which was taken over by state-owned Chinese chemicals conglomerate ChemChina in 2017 – during the same period as the HPE investigation into Chinese attacks on its network.

Ericsson said it does not comment on specific cybersecurity incidents. “Our priority is always to ensure that our customers are protected,” a spokesman said. “While there have been attacks on our enterprise network, we have found no evidence in any of our extensive investigations that Ericsson’s infrastructure has ever been used as part of a successful attack on one of our customers.”

A spokesman for SKF said: "We are aware of the breach that took place in conjunction with the ‘Cloud Hopper’ attack against HPE … Our investigations into the breach have not found that any commercially sensitive information was accessed."

Syngenta and Valmet declined to comment. A spokesman for Vale declined to comment on specific questions about the attacks but said the company adopts “the best practices in the industry” to improve network security.

‘DRUNKEN BURGLARS’

The companies were battling a skilled adversary, said Rob Joyce, a senior adviser to the U.S. National Security Agency. The hacking was “high leverage and hard to defend against,” he said.

According to Western officials, the attackers were multiple Chinese government-backed hacking groups. The most feared was known as APT10 and directed by the Ministry of State Security, U.S. prosecutors say. National security experts say the Chinese intelligence service is comparable to the U.S. Central Intelligence Agency, capable of pursuing both electronic and human spying operations.

An image of Zhang Shilong, a member of a hacking group in China, appears on a U.S. Federal Bureau of Investigation (FBI) poster released by the FBI, December 21, 2018. Federal Bureau of Investigation/Handout via REUTERS/File Photo

Two of APT10’s alleged members, Zhu Hua and Zhang Shilong, were indicted in December by the United States on charges of conspiracy to commit computer intrusions, wire fraud and aggravated identity theft. In the unlikely event they are ever extradited and convicted, the two men would face up to 27 years in an American jail.

Reuters was unable to reach Zhu, Zhang or lawyers representing the men for comment. China’s Foreign Ministry said the charges were “warrantless accusations” and it urged the United States to “withdraw the so-called lawsuits against Chinese personnel, so as to avoid causing serious harm to bilateral relations.”

The U.S. Justice Department called the Chinese denials “ritualistic and bogus.”

“The Chinese Government uses its own intelligence services to conduct this activity and refuses to cooperate with any investigation into thefts of intellectual property emanating from its companies or its citizens,” DOJ Assistant Attorney General John Demers told Reuters.

APT10 often attacked a service provider’s system by “spear-phishing” – sending company employees emails designed to trick them into revealing their passwords or installing malware. Once through the door, the hackers moved through the company’s systems searching for customer data and, most importantly, the “jump servers” – computers on the network which acted as a bridge to client systems.

After the attackers “hopped” from a service provider’s network into a client system, their behavior varied, which suggests the attacks were conducted by multiple teams with different skill levels and tasks, say those aware of the operation. Some intruders resembled “drunken burglars,” said one source, getting lost in the labyrinth of corporate systems and appearing to grab files at random.

HOTELS AND SUBMARINES

It’s impossible to say how many companies were breached through the service provider that originated as part of Hewlett Packard, then became Hewlett Packard Enterprise and is now known as DXC.

The HPE operation had hundreds of customers. Armed with stolen corporate credentials, the attackers could do almost anything the service providers could. Many of the compromised machines served multiple HPE customers, documents show.

One nightmare situation involved client Sabre Corp, which provides reservation systems for tens of thousands of hotels around the world. It also has a comprehensive system for booking air travel, working with hundreds of airlines and 1,500 airports.

A thorough penetration at Sabre could have exposed a goldmine of information, investigators said, if China was able to track where corporate executives or U.S. government officials were traveling. That would open the door to in-person approaches, physical surveillance or attempts at installing digital tracking tools on their devices.

In 2015, investigators found that at least four HP machines dedicated to Sabre were tunneling large amounts of data to an external server. The Sabre breach was long-running and intractable, said two former HPE employees.

HP management only grudgingly allowed its own defenders the investigation access they needed and cautioned against telling Sabre everything, the former employees said. “Limiting knowledge to the customer was key,” one said. “It was incredibly frustrating. We had all these skills and capabilities to bring to bear, and we were just not allowed to do that.”

“The security of HPE customer data is always our top priority,” an HPE spokesman said.

Sabre said it had disclosed a cybersecurity incident involving servers managed by an unnamed third party in 2015. Media reports at the time said the hackers were linked to the Chinese government but did not name HP.

A Sabre spokeswoman said an investigation of the breach “concluded with the important finding that there was no loss of traveler data, including no unauthorized access to or acquisition of sensitive protected information, such as payment card data or personally identifiable information.” The spokeswoman declined to comment on whether any non-traveler data was compromised.

UNINVITED GUESTS

The threat also reached into the U.S. defense industry.

In early 2017, HPE analysts saw evidence that Huntington Ingalls Industries, a significant client and the largest U.S. military shipbuilder, had been penetrated by the Chinese hackers, two sources said. Computer systems owned by a subsidiary of Huntington Ingalls were connecting to a foreign server controlled by APT10.

During a private briefing with HPE staff, Huntington Ingalls executives voiced concern the hackers could have accessed data from its biggest operation, the Newport News, Va., shipyard where it builds nuclear-powered submarines, said a person familiar with the discussions. It’s not clear whether any data was stolen.

Huntington Ingalls is “confident that there was no breach of any HII data” via DXC or HPE, a spokeswoman said.

Another target was Ericsson, which has been racing against China's Huawei Technologies to build infrastructure for 5G networks expected to underpin future hyper-connected societies. The hacking at Ericsson was persistent and pervasive, said people with knowledge of the matter.

Logs were modified and some files were deleted. The uninvited guests rummaged through internal systems, searching for documents containing certain strings of characters. Some of the malware found on Ericsson servers was signed with digital certificates stolen from big technology companies, making it look like the code was legitimate so it would go unnoticed.

Like many Cloud Hopper victims, Ericsson could not always tell what data was being targeted. Sometimes, the attackers appeared to seek out project management information, such as schedules and timeframes. Another time they went after product manuals, some of which were already publicly available.

“The reality is that most organizations are facing cybersecurity challenges on a daily basis, including Ericsson,” Chief Security Officer Pär Gunnarsson said in a statement to Reuters, declining to discuss specific incidents. “In our industry, and across industries, we would all benefit from a higher degree of transparency on these issues.”

WHITE WOLF

In December 2018, after struggling to contain the threat for years, the U.S. government named the hackers from APT10 – Advanced Persistent Threat 10 – as agents of China’s Ministry of State Security. The public attribution garnered widespread international support: Germany, New Zealand, Canada, Britain, Australia and other allies all issued statements backing the U.S. allegations against China.

Even so, much of Cloud Hopper’s activity has been deliberately kept from public view, often at the urging of corporate victims.

In an effort to keep information under wraps, security staff at the affected managed service providers were often barred from speaking even to other employees not specifically added to the inquiries.

In 2016, HPE’s office of general counsel for global functions issued a memo about an investigation codenamed White Wolf. “Preserving confidentiality of this project and associated activity is critical,” the memo warned, stating without elaboration that the effort “is a sensitive matter.” Outside the project, it said, “do not share any information about White Wolf, its effect on HPE, or the activities HPE is taking.”

The secrecy was not unique to HPE. Even when the government alerted technology service providers, the companies would not always pass on warnings to clients, Jeanette Manfra, a senior cybersecurity official with the U.S. Department of Homeland Security, told Reuters.

“We asked them to notify their customers,” Manfra said. “We can’t force their hand.”

(Additional reporting by Gao Liangping, Cate Cadell and Ben Blanchard in Beijing. Editing by Ronnie Greene and Jonathan Weber)

[© 2019 Thomson Reuters. All rights reserved.]

Copyright 2019 Reuters. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.  Thompson Reuters is solely responsible for this content.