WASHINGTON: US government agencies were ordered to scour their networks for malware and disconnect potentially compromised servers after authorities learned that the Treasury and Commerce departments were hacked in a monthslong global cyberespionage campaign discovered when a prominent cybersecurity firm learned it had been breached.
In a rare emergency directive issued late Sunday, the Department of Homeland Security's cybersecurity arm warned of an "unacceptable risk" to the executive branch from a feared large-scale penetration of US government agencies that could date back to mid-year or earlier.
"This can turn into one of the most impactful espionage campaigns on record," said cybersecurity expert Dmitri Alperovitch.
The hacked cybersecurity company, FireEye, would not say who it suspected -- many experts believe the operation is Russian given the careful tradecraft -- and noted that foreign governments and major corporations were also compromised.
News of the hacks, first reported by Reuters, came less than a week after FireEye disclosed that nation-state hackers had broken into its network and stolen the company's own hacking tools.
The apparent conduit for the Treasury and Commerce Department hacks -- and the FireEye compromise -- is a hugely popular piece of server software called SolarWinds.
It is used by hundreds of thousands of organizations globally, including most Fortune 500 companies and multiple U.S. federal agencies that will now be scrambling to patch up their networks, said Alperovitch, the former chief technical officer of the cybersecurity firm CrowdStrike.
The DHS directive -- only the fifth since they were created in 2015 -- said U.S. agencies should immediately disconnect or power down any machines running the impacted SolarWinds software.
FireEye, without naming any specific targets, said in a blog post that its investigation into the hack of its own network had identified "a global campaign" targeting governments and the private sector that, beginning in the spring, had slipped malware into a SolarWinds software update.
Neither the company nor the US government publicly identified Russian state-backed hackers as responsible.
The malware gave the hackers remote access to victims' networks, and Alperovitch said SolarWinds grants "God-mode" access to a network, making everything visible.
"We anticipate this will be a very large event when all the information comes to light," said John Hultquist, director of threat analysis at FireEye.
"The actor is operating stealthily, but we are certainly still finding targets that they manage to operate in." On its website, SolarWinds says it has 300,000 customers worldwide, including all five branches of the US military, the Pentagon, the State Department, NASA, the National Security Agency, the Department of Justice and the White House.
It says the 10 leading US telecommunications companies and top five U.S. accounting firms are also among customers.
FireEye said it had confirmed infections in North America, Europe, Asia and the Middle East, including in the health care and oil and gas industry -- and had been informing affected customers around the world in the past few days.
It's customers include federal, state and local governments and top global corporations.
It said that malware that rode the SolarWinds update did not seed self-propagating malware -- like the NotPetya malware blamed on Russia that caused more than $10 billion in damage globally -- and that any actual infiltration of an infected organization required "meticulous planning and manual interaction."
That means it's a good bet only a subset of infected organizations were being spied on by the hackers.
Nation-states have their cyberespionage priorities, which include COVID-19 vaccine development.
On Sunday, Russia's US embassy described as "unfounded" in a post on its Facebook page the "attempts of the U.S. media to blame Russia for hacker attackes on US governmental bodies."
The Treasury Department referred requests for comment to the National Security Council, whose spokesman, John Ullyot, said the government was "taking all necessary steps to identify and remedy any possible issues related to this situation."
The government's Cybersecurity and Infrastructure Security Agency said it was working with other agencies to help "identify and mitigate any potential compromises." The FBI said it was engaged in a response but declined to comment further.
President Donald Trump last month fired the director of CISA, Chris Krebs, after Krebs vouched for the integrity of the presidential election and disputed Trump's claims of widespread electoral fraud.
In a tweet Sunday, Krebs said "hacks of this type take exceptional tradecraft and time," adding that he believed that its impact was only beginning to be understood.
Federal agencies have long been attractive targets for foreign hackers looking to gain insight into American government personnel and policymaking.
Hackers linked to Russia, for instance, were able to break into the State Department's email system in 2014, infecting it so thoroughly that it had to be cut off from the internet while experts worked to eliminate the infestation.
A year later, a hack at the US government's personnel office blamed on China compromised the personal information of some 22 million current, former and prospective federal employees, including highly sensitive data such as background investigations.