Blog & Archives

Salt Typhoon Proves There Is No 'Safe Lawful Access'

(The result of some Summer weekend thinking; I would love feedback.)

The 2024 Salt Typhoon cyber espionage campaign marks a watershed moment in state-sponsored cyber operations, delivering a clear answer to a long-standing debate in the security community: there is no safe form of "lawful access" or "good guys only" backdoor.

This operation, attributed with high confidence to China's Ministry of State Security (MSS), proved that mandated government access capabilities, like those required in the United States by the Communications Assistance for Law Enforcement Act (CALEA), are not just susceptible to compromise but are in fact high-value targets that become powerful weapons in the hands of adversaries.

How They Got In: Common Flaws, Not CALEA Backdoors

It's crucial to understand how Salt Typhoon worked here, in two parts.

The initial breach of at least nine major U.S. telecommunications service providers, including Verizon, AT&T, and Lumen Technologies, was not due to a direct vulnerability in CALEA infrastructure itself. Salt Typhoon's initial breach did not rely on a single, exotic flaw. Instead, the attackers executed a wide-net strategy, exploiting a range of common and often long-known security failures to find available entry points.

They weaponized known software bugs ("N-days") in public-facing network equipment like routers, firewalls, and VPNs from major vendors including Cisco, Ivanti, and Fortinet. They also used the notorious ProxyLogon vulnerability in Microsoft Exchange servers to break into corporate networks. Critically, a primary entry method was even simpler and one we all know too well: logging in with legitimate, stolen user credentials, bypassing the need to exploit a software flaw altogether.

This multi-faceted approach shows that the path to the intercept systems—wiretapping systems, effectively—was paved by conventional hacking techniques exploiting general security failures, rather than a singular CALEA-specific flaw. Of course, that's never the end of the story...

The Ultimate Prize: CALEA's Dual-Use Capabilities

Once inside the networks, Salt Typhoon's operational focus shifted dramatically. Their mission was deliberate and intelligence-driven: to locate, access, and control the CALEA-compliant intercept infrastructure. These systems were not incidental victims or another step in the exploit chain; the intercept systems were the ultimate prize.

Salt Typhoon successfully leveraged the built-in surveillance functionalities of the CALEA architecture for its own counterintelligence purposes. This "dual-use" capability meant that the very architecture designed to provide government access became a powerful tool for malicious state-level espionage.

The compromise of CALEA systems allowed Salt Typhoon to:

  • Access U.S. law enforcement wiretap requests.
  • Gain access to call metadata.
  • In some cases, obtain the content of communications, including actual audio of phone calls, involving high-profile political and government targets.

This disastrous outcome was predicted... a loooooong time ago. The 2004-2005 "Athens Affair," where unknown actors hijacked Vodafone Greece's intercept system to spy on over 100 government officials, was a stark early warning. Furthermore, academic and independent research laid the dangers bare. Work by Tom Cross in 2010, for example, demonstrated that a common CALEA implementation was susceptible to brute-force credential discovery (via SNMPv3) and lacked reliable audit trails, allowing an attacker to issue intercept requests undetected. Research from Professor Matt Blaze and his students in 2009 showed that due to poor design, a wiretap target could easily overwhelm the system's signaling channel, effectively blinding law enforcement's surveillance efforts.

A Catastrophic and Definitive Policy Failure

The intelligence value of compromising intercept systems is unparalleled for a foreign counterintelligence service like China's MSS. It provides a direct, real-time window into a rival nation's most sensitive law enforcement investigations and foreign intelligence collection priorities.

By accessing CALEA systems, the MSS could identify U.S. government targets of interest, gain insight into ongoing investigations, and understand U.S. intelligence tradecraft. This wasn't just about stealing data; it was about gaining "meta-intelligence"—intelligence about the intelligence process itself. The result is a systemic compromise of the U.S. national security apparatus and a catastrophic intelligence loss, forcing agencies to assume their methods and sources have been exposed.

The Salt Typhoon campaign is more than a technical failure; it's a stark policy failure. It provides the definitive, empirical answer that the premise of a "good guys only" backdoor is wrong and dangerous. This vulnerability is inherent to the policy of mandated access, creating a trust inversion where a system designed for privileged access becomes a powerful tool for any adversary who compromises the host network.

The long-running debate is over. The campaign unequivocally demonstrates that any mandated "backdoor" is a vulnerability that will inevitably be discovered and exploited by capable adversaries.

Alex Halderman's DC Council Testimony

Prof. Alex Halderman testifying in front of the DC Council on 10/08/2010 I captured video from today's DC Council Hearing of The Committee on Government Operations and The Environment.

Prof. Alex Halderman (Michigan), Susannah Goodman (Common Cause), Jeremy Epstein and Pamela Smith (Verified Voting) testified by yielding all their time to Halderman to speak to the technical challenges involved with internet voting and specifically the recent compromise of the DC Digital Vote-By-Mail pilot project that Alex' team was able to achieve. Alex' team included two of Alex' PhD students, Scott Wolchok and Eric Wustrow, Dawn Isabel (Michigan's Ethical Hacker) and Nadia Heninger, a PhD student from Princeton; I was an adviser to Alex' team.

This video is in raw form, so it's very big (318.2 MB):

http://dl.dropbox.com/u/8173121/DCCouncil-hearing-panel3-20101008.mov

(Here is a smaller version (153.1MB).)

I've done some paraphrasing of the key technical bits below.

Personally, as an adviser to Alex' team and as someone who was afraid that there would be no serious attack mounted during the test period, I couldn't have imagined a more successful demonstration of the technical challenges involved with fielding and defending an internet voting system. I have some thoughts that I'm writing up about what this test tells us from a policy perspective, but don't expect that very soon.

(I want to apologize for the beginning of the video capture where I'm looking at my twitter client without realizing that I'm capturing that over the video. Oh well.)


Interesting bits from the video

  • Cheh: "Basically, you're a hacker, is that what we're to understand?"
  • Alex: "No, I'm a professor of computer science."

Key new insights from Halderman, starting 9:01:

  • Other attacks did go undetected.
  • The Michigan team had been controlling and monitoring the routers and switches connected to the pilot network from the beginning.
  • Access was easily achieved because a default master password was left unchanged, which one can look up in the owner's manual. This was a 4-letter password.
  • The team could watch in real-time as system administrators configured and tested the equipment.
  • They Could also watch staffers on camera as the team found that security cameras in the data center were on the same network as the testbed.
  • 10:13: Alex passes out pictures from the cameras taken of people in the data center.
  • The team could observe these system administrators as they entered passwords on the system as well as watch them on camera.
  • This network-based attack amounts to a separate, second way to steal votes, etc. in a real election.
  • 11:20: It became clear that the team was not the only ones trying to attack.
  • While they were in control, they observed other attack attempts originating from Iran and China, attempting to guess the same default master password.
  • They defended the network by blocking these attacks, adding firewall rules and changing the default password.
  • Cheh: So, you changed the password of the BOEE system?
  • Alex: Yes, of the pilot system.
  • Alex does not feel that these were part of a targeted attack against the BOEE.

....

  • 12:40: All these things could be fixed, but it's vastly more difficult to create a secure internet voting system.
  • The crux is that there is no independent record of the vote.

...

  • 14:05: it will probably be decades, if ever, before we can perform voting over the internet safely.
  • Web security is a terribly hard problem.

....

  • 14:34: Alex later examined the data they had collected, files left around on the server, and one thing was incredibly shocking.
  • They had tested the file upload portion to make sure that files either too small or too large were not allowed.
  • These files look like they were just files laying around on some BOEE computer.
  • Some were simple single-page PDFs. One was the installation file for a Macintosh development tool.
  • 16:16: One of the files, which Alex has with him (he pulls out a cardboard box and takes out a large document, looking like two reams of paper).
  • This file was a 937-page PDF document... it appears to be the 937 invitation letters that each voter was sent to participate in this election.
  • They examined the file metadata; the author of the file is Paul Stenbjorn.
  • It appears that this may be the real thing.
  • Alex found the document on the testbed server, a system that the BOEE invited people to break into and that the team did break into.
  • We have no way of knowing who else has access to this.
  • The PINs in these documents are the most critical secrets to protect these votes.
  • If the digital ballot return had been used, a criminal could have used these to cast a vote for each voter and prevent them from voting.
  • Why was this file on the testbed system?
  • Alex is deeply concerned that the BOEE does not take security seriously and that it fails to appreciate the security challenges that are faced by any internet voting system.

Permalinks for California Bills?

I've worked with California legislation and law for almost a decade, and one consistently frustrating thing is that it seems impossible to link to bills.

For example, Gov. Schwarzenegger signed SB 1404 yesterday, which is a landmark bill to require voting system manufacturers to report to the CA Secretary of State any known flaws and defects in their products. To find this bill, one would normally do the following:

  1. Open http://leginfo.ca.gov/
  2. Click on "Bill Information".
  3. Select "Senate" in the drop-down and type "1404" in the text field and submit.
  4. This is the "splash page" for a legislative bill with all the associated information... including past versions, bill status.
  5. Woo!

However, how does one link to that page? It's a POST query, which means the terms are hidden and there's no obvious way to link to this page. However, after asking a friend who works there, this seems to be the goods, at least for current bills:

http://leginfo.ca.gov/cgi-bin/postquery?bill_number=sb_1404&sess=CUR

Sigh. Citability, anyone?

Checking a Web Page for Updates

I really like http://watchthatpage.com/ for tracking changes to web pages, but it's mostly useful for changes on the order of a day.

Today, due to the DC Digital Vote By Mail pilot project, I find myself wanting to monitor changes on the order of 5 minutes or so. (DC has a limited number of testing credentials that it will issue, and I don't know when that link will go live!)

So, this is a perfect job for a simple shell script.

First, it would be useful to have a generic script that took a web address as a URI and then compared the new version to an old version. So, I put this in a file called checkpagechange, made it executable and stuck it in /usr/bin:

#!/bin/bash curl -s $@ > /tmp/new.html diff -u /tmp/old.html /tmp/new.html mv /tmp/new.html /tmp/old.html

This uses curl to grab the page and save it to /tmp/new.html, then uses diff to compare this version to an old version and then moves the new version to the old version's location.

Then one can do checkpagechange http://foo.bar and it will print to the screen any changes. Of course, the first time old.html doesn't exist or is something from another page.

To accomplish my goal for today, I can just put this last command in a loop for the URI for which I'm interested. That is, I create a small specific shell script to use this in a loop. Save the following to something like checkdc.sh and make it executable:

#!/bin/bash while [ 1 ] do echo date checkpagechange http://www.dcboee.us/DVM/ sleep 300 done

This is a infinite loop that first outputs the date and time to the screen, then uses the previous script to output any changes in the web page and then goes dormant for 5 minutes (300 seconds).

You can run this in a terminal window and place it off to the side so that just the first few letters of the date are visible... when that changes, voila!

E-voting and Direct Democracy

(This post was originally published on VoxPopuLII, a publication of the Legal Information Institute at Cornell University Law School.)

Voting BoothsIn this post, I'd like to connect a specific area of my expertise—electronic voting (e-voting)—to issues of interest to the legal information community. Namely, I'll talk about how new computerized methods of voting might affect elements of direct democracy: that is, ballot questions, including referenda and recall. Since some readers may be unfamiliar with issues related to electronic voting, I'll spend the first two parts of this post giving some background on electronic voting and internet voting. I'll then discuss how ballot questions change the calculus of e-voting in subtle ways.

Background on E-voting

The images of officials from 2000 closely scrutinizing punchcard ballots during the U.S. presidential election tend to give theofficial scrutiny mistaken impression that if we could just fix the outdated technology we used to cast ballots, a similar dispute wouldn't happen again. However, elections are about "people, processes, and technology"; focusing on just one of those elements disregards the fact that elections are complex systems. Since 2000, the system of election administration in the United States has seen massive reform, with a lot of attention paid to issues of voting technology.

Follow up:

In the years after 2000, this system that had mostly "just worked" in previous decades was now seen as having endemic, fundamental problems. During the turn ofTammany Vote the 20th century, frauds involving ballot box-stuffing, vote-buying, and coercion were the major policy concern and the principal focus of reform. In contrast, at the turn of the 21st century, the prevalence of close, contentious contests—e.g., see this example of an analysis of New Jersey elections—often put the winning margin well within the "error" or "noise" level associated with ballot casting methods.

In 2002, Congress passed the Help America Vote Act (HAVA), which provided the first federal funding for election administration, created the Election Assistance Commission (EAC) and established the first federal requirements for voting systems, provisional balloting, and statewide voter registration databases. As my colleague Aaron Burstein and I argue in an article currently in preparation, in terms of advancing the state-of-the-art in voting technology, HAVA conspicuously focused on providing funds that had to be spent quickly on types of voting systems that were then available on the market or soon would be available. The systems on the market at the time were invariably of a specific type: "Direct Recording Electronic" (DRE) voting machines, in which the record of a voter's vote is kept entirely in digital form.

In the years since the passage of HAVA, computer science, usability, and information systems researchers have highlighted a number of shortcomings with this species of voting equipment. Three principal critiques voiced by this community are:

  • There is no proper way to do a recount on these systems. That is, if a race is close and a candidate calls for a recount, in most cases this will mean simply rerunning the software that added up all the digital votes; the exact same number would result. DREs do not keep a record that captures the voter's intent; rather, these systems "collapse" voter intent into a digital representation kept in digital memory. In other types of systems, such as optical scan systems—where voters fill in bubbles on paper ballots which are then scanned in for counting—the voter's marks are directly preserved with the ballot. In a traditional recount with non-DRE systems, election staffers interpret these marks made by voters and come up with a count based on how a trained human would interpret ballots. This is not possible with DRE voting systems and lever machines, which do not preserve individual records of voter intent.

  • There is no way to know if the software that runs DREs is correctly recording votes, and we've seen numerous cases of software errors, including errors that have resulted in lost ballots. However, the addition of a "voter-verified paper record" (VVPR)—that is, an independent record that the voter can verify before casting his or her vote—alleviates not only this problem of recounting records that show voter intent, but also the myriad of problems associated with software flaws and "malware" (malicious software) in these machines. If voters check these records and agree that the records reflect how they want to vote, this renders the paper records "independent" of the system's software, and the records can safely be audited and/or recounted if there do turn out to be software-based problems.

  • In a number of state-level technical reviews of voting systems, of which I have been a part in California and Ohio, we have found serious vulnerabilities in each voting system we examined. These findings leave little confidence in the equipment that was purchased by election officials in the wake of the 2000 election. Moreover, this was a clear indication that the systems for certifying this equipment at the state and federal level had serious shortcomings that have allowed sub-standard systems into the field.

Now, in 2010, many states have passed laws requiring auditable voting systems, and increasing numbers of election officials are moving from DRE-based systems to optical scan systems. Despite these reforms which have, in my opinion, moved e-voting in the right direction, the specter of internet voting looms large.

Internet VotingInternet Voting

During public talks I am often asked, "When will we vote over the internet?" People have an intuitive feeling that since they're doing so much online, it makes sense to vote online, too. However, we need to recognize what kinds of activities the internet is good for, and voting is perhaps the last thing we want to happen online.

Things that we do online now that require high security, such as banking, are not anonymous processes; there is a named record associated with each transaction. Yet the secret ballot is a very important part of removing coercion and vote-buying from possibly corrupting influences on the vote. (See this superb article by Allison Hayward: "Bentham & Ballots: Tradeoffs between Secrecy and Accountability in How We Vote".)

Moreover, banks and other online establishments can purchase insurance to contain the risk of losses due to online fraud (although there are some indications that even this is becoming more difficult due to the increased sophistication and magnitude of online banking fraud). But there is still no firm that offers insurance for computer intrusions and attacks, or simply just errors, because it is very difficult to estimate the magnitude and likelihood of such losses. The "value" of a vote is very different from the value of currency: the value of your vote doesn't just matter to you as a voter; it also matters to other voters. ("Vote dilution," for example, is when processes conspire to render one voter's vote more or less effective than another's.) Also, it can be very hard to estimate the fitness of a given piece of software; said another way, we haven't yet figured out how to write impervious or bug-free software.

Finally, as I mention above, the voting systems that the market has responded with in recent years leave a lot to be desired in terms of security, usability, and reliability. Internet voting essentially takes systems like those and adds the complications of sending voted electronic ballots over the public internet from users' personal computers—neither of which are reliable or secure—with no VVPR.

We are far from the day in which highly secure processes can happen over the public internet from users' computing devices. We will have to make significant technical advances in the security of personal computing devices and in network security before we can be sure that internet votes can be cast in a manner that approaches the privacy and security afforded by polling place voting.

Unfortunately, most designs for internet voting systems are un-auditable. Since these systems lack a paper trail, it is impossible to tell whether the voted ballot contents received at election headquarters correspond with what the voter intended to vote. The answer here would seem to be cryptographic voting systems, where the role of a paper trail is played by cryptographically secure records that can be transmitted over the network. Systems of this type have become increasingly more sophisticated, easy to use, and easy to understand, and have even been used in a binding municipal election here in the U.S.

E-voting and Direct Democracy

Elections don't just elect people in the U.S.; in many states, voters vote on elements of direct democracy, specifically ballot referenda and recall questions. However, we should be even more concerned about opportunities to game these kinds of contests -- and, equivalently, about how errors introduced by ballot casting methods for ballot questions could affect how we govern -- than we are about the risks of voting fraud in candidate races.

It's difficult to compare the importance of candidate elections to that of ballot questions. Certainly, ballot questions can be as simple as asking the voters to approve of city ordinances, such as increasing the amount of square footage for single-family homes. And, of course, on even-numbered years divisible by four, we elect the President of the United States, which unequivocally changes how our entire country is governed and operates. In between these two extremes are elections that many people don't vote on, from judicial elections to highly contentious ballot propositions (like Proposition 8 in California), or transportation tax bonds that can result in hundreds of millions of dollars for local firms.

Can we compare the risks involved with candidate elections and ballot questions? In some sense, being able to bound the risk of fraud or error causing the election of the wrong candidate is similar to that resulting in "electing" the wrong decision in a ballot question; it's equally difficult to compare the relative importance of elected contests and to decide on some level of likelihood that a contest runs a high risk of being targeted for attack or Voter Pollmight be especially sensitive to errors in the count. Polling may help, but it's far from perfect. However, ballot questions have one aspect that should make this process a bit easier: rather than having the considerable uncertainty of what policies a potential candidate may institute once elected, ballot measures are concrete policy proposals or actions where we know very well what will happen if they are passed. This would seem to make ballot questions more attractive to attack; the uncertainty involved with what candidates may do is not present, so the net benefit of a successful attack, all other things being equal, should be larger.

Are there special risks involved with ballot questions that we should be concerned about in the face of electronic voting methods? Certainly. First, ballot propositions are invariably at the end of the ballot; hence, they're referred to as "down-ticket" contests. Post-election auditing, where a subset of ballot records are hand-counted as a check against the electronic results, often doesn't include ballot questions. To be certain, states like California require post-election auditing of all contests on the ballot. But there are many states that do not do comprehensive election auditing; they either don't do any auditing at all or focus their auditing attention on top-ticket contests on the ballot (for more, see Sections 1 and 2 of: "Implementing Risk-Limiting Post-Election Audits in California").

While we have seen little evidence of fraud using newer computerized voting systems compared to the massive record of paper ballot fraud in our country's past, this should serve as little comfort. Just as in finance, where "past results are no indication of future performance," adversarial security is similar. That we haven't seen much evidence of computer fraud involving voting systems doesn't mean it isn't happening and doesn't mean it can't happen. Multi-million dollar ballot questions and constitutional amendments are exactly the kinds of law-making activities in which I expect to see the first evidence of outright computerized election hacking. This rings especially true if we start using the public internet for casting ballots. While foreign interests or hackers out of the reach of US law enforcement might certainly be interested in top-ticket candidate contests, the opportunities to affect state and local law as well as economic interests embodied in ballot questions would seem to be especially attractive.

Where Should We Go From Here?

To be sure, there is a lot of momentum behind moving parts of our elections processes online. In some cases, such as online voter registration, the security and reliability risks are small and the net benefits are particularly high. However, I can't say the same about internet voting, especially in the sense that elements of direct democracy may be particularly attractive to powerful foreign interests and parties outside our collective jurisdiction. The recently passed Military and Overseas Voter soldier voteEmpowerment (MOVE) Act has been interpreted to allow states to experiment with online ballot casting, and the relevant agencies charged with implementing the law—the Department of Defense's Federal Voting Assistance Program (FVAP), the EAC, and the National Institute of Standards and Testing (NIST)—have collectively interpreted the MOVE Act as requiring them to institute standards and pilot programs for internet voting for military and overseas voters. I'm on record as disagreeing with this interpretation, but I can understand that they feel limited-scale pilot projects are appropriate. I predict that the first incontrovertible evidence of computerized vote manipulation will be associated with military and overseas internet voting efforts, and it's not hard to imagine a down-ticket ballot question as being the focus of such an attack.

Should we re-think our forays into computerized voting? Definitely not. In my opinion, this is more a question of responsible uses of technology in elections than a black or white decision about using computerized voting systems or not. There is much good that stems from the use of computerized voting systems, including improved accessibility for the disabled and voters who don't speak English, improved usability of ballots on-screen versus what can be accomplished on paper, and the speed and accuracy of computerized vote counts on election night. However, these voting systems must be recountable and auditable, and those audits must be conducted after each election in such a way that we limit the risk of an incorrect candidate or ballot measure being certified as the winner.

In contrast to the beginning of the past decade, when election officials were swimming in federal money for the purchase of equipment and trying to spend these funds before a looming deadline, what we really need is regular commitments of federal funding to improve local election administration. With a sustained source of federal funds to budget and plan for technology upgrades, the market will be stable, rather than going through the upheaval of mergers and dissolutions we have recently seen. Elections are perhaps the most poorly funded of all of the critical elements of democracy in the U.S., and we get what we pay for.

joe-hallJoseph Lorenzo Hall is a postdoctoral researcher at the UC Berkeley School of Information and a visiting postdoctoral fellow at the Princeton Center for Information Technology Policy. His Ph.D. thesis examined electronic voting as a critical case study in the transparency of digital government systems.

VoxPopuLII is edited by Judith Pratt. Editor in chief is Robert Richards.