Post Mortem Meeting — Ethereum Classic (ETC) — January 2019 Reorg Attacks

First post-mortem meeting of the ETC community, Jan 9th, 2019

Summary

Today we conducted a call on Discord at 5PM UTC to go over the main points about the recent reorg attacks on ETC, analyze the community reaction, and diagnose possible solutions and next steps.

Below is a timeline of significant activity and an outline of what was discussed.

A Timeline of Significant Activity

As it may be seen in the image below, the miner identified as “Private Pool 0x3ccc8f74” on GasTracker.io started its first attack on January 5th of 2019 through 6th and lasted 7 hours. Then, the same miner performed a second attack on January 6th through 7th that lasted 12 hours.

 

The first indication of a problem (see image below), which was interpreted by the community at the time as a local problem of the service provider, arose when the wallet and exchange Coinbase announced that it had disabled deposits and withdrawals due to “unstable network conditions” on the ETC blockchain.

 

The next day, January 6th, Twitter user @Pierre_Rochard asked me, @TokenHash, an ETC community member, if there was a deep reorg on Ethereum Classic the day before.

 

After this interaction, many other members of the ETC community started to check in different sources what was the status of the network.

In parallel, on January 6th, another ETC community member, and leader of the ETC community in China, Roy Zou, noticed that there was chatter in China about a miner with more than 50% hash power in the ETC network. See image below.

 

On January 7th, Mark Nesbitt Security Engineer at Coinbase, posted a blog post with a full explanation of their view of the attacks. Below is the image they used to describe it.

 

The full Coinbase blog post can be found here:

https://blog.coinbase.com/ethereum-classic-etc-is-currently-being-51-attacked-33be13ce32de

The resulting actions detected by Coinbase in the attacks were that ETC 219,500 were stolen through double spends on block reorgs in 15 transactions.

On January 9th the security firm SlowMist confirmed the information provided by Coinbase and provided more details about specific accounts of the victims, which were several exchanges, and the accounts used by the attacker. The report may be found here:

https://medium.com/@slowmist/the-analysis-of-etc-51-attack-from-slowmist-team-728596d76ead

Outline of What Was Discussed in the ETC Post-Mortem Meeting

We were more than 20 members of the community, the meeting was led by Zach Belford, @Belfordz, ETC lead developer, and included members from the IOHK and ETC Labs Core teams, other volunteer ETC developers, the ETC Cooperative, other members of the community at large, and one journalist from CoinDesk.

The day before we had consulted with the founder of ProgPow, Kristy-Leigh Minehan, @OhGodAGirl ‏, about possible alternatives for ETC including integrating ProgPoW.

Below is a list of the topics we touched.

1. A high-level summary of what happened

Which services and customers were affected?

• Exchanges
• Pool payouts?
• Dapp users — https://dappdirect.net/ — tx dropping is possible

How long was the issue?

• The issue was from 1/5/2019 to 1/7/2019
• Included at least 15 double spend transactions and at least ETC 219,500 stolen
• Coinbase discovered reorg on block #7245623
• January 04, 2019 Beijing time — block #7249343 — *potentially ongoing*

Who was involved in the response?

• Roy Zou ETC Consortium
• Coinness
• IOHK
• ETC Co-op
• Donald McIntyre
• ETC Labs

How did we ultimately fix the problem?

• We have not
• Recommend increase confirmations: We suggest 2,500 to 5,000 blocks to achieve the same guarantees as 1 to 2 Bitcoin blocks; and more if dealing with large transactions. Essentially waiting until the cost to mine blocks to reorg is greater than the cost of what you’ve received
• Dissemination of accurate information
• Research mitigation techniques

2. A root cause analysis

What were the origins of failure?

• No root cause internally in the system. External factors: ETC, as any PoW blockchain is vulnerable to a 51% attack by miners, who can initiate reorgs enabling double spend transaction
• Malicious ‘bad actor’ miner
• Insufficient hash rate given the number of confirmations users were typically waiting
• In summary: ETC is still a relatively small PoW blockchain with a mining algorithm that is compatible with larger chains such as Ethereum so attackers can rent hash power on NiceHash to reorg the blockchain

Why do we think this happened?

• We don’t know right now

3. Steps Taken to Diagnose, Assess, and Resolve

What actions were taken?

• Check GasTracker, examine blocks, look at miner distribution
• Blockscout reorg tracker
• Mining info from https://investoon.com/mining_pools/etc
• Coordinated analysis through ETC Communications and Social Media Channels (Discord, Twitter)
• Analysis to confirm double spends — Coinbase & Gate.io
• Raising confirms — started in Discord, some then mentioned on Twitter

4. Learnings

What went well?

• Community came together pretty fast in Discord
• Shout out to GasTracker, Blockscout, and Etherhub! (they got hammered and never went down)
• Coordination was handled well, Roy Zou was in contact with a security firm Slow Mist from the start
• The ETC community didn’t try to minimize/downplay the situation
• Tooling for detection are being conceived already
• The network worked as designed and didn’t break
• This post-mortem, our commitment to improvement

What didn’t go well?

• No security alert nor contact information in the ETC community
• No alert system — exchanges, miners, wallet operators and other stakeholders had no contact points to initiate alerts or receive feedback from the developers or community
• Unable to quickly confirm victims of double spend
• Blockscout reorg data was spotty
• Retroactive security of the network, no preemptive systems exist
• Echo’ing non facts and giving energy to conspiracy theories

5. Next Steps:

We wrote a list of options and priorities for the ETC community and developers to debate, write ECIPs if necessary, and implement over the near future:

1. Create a monitoring and alert system for ETC so attacks may be seen faster to alert the ecosystem.
2. Continue the recommendation for users and operators to use a significantly higher confirmation time, 2500 to 5000 depending on the type of transaction.
3. Study the possibility of implementing deep-reorg-protection.
4. ECIP1043: The purpose of this ECIP is to set a limit on the maximum size of the DAG to its initial state and no longer increase it on an epoch schedule.
5. We will not apply any changes hastily without proper research and analysis and will continue to whether the storm.
6. The above + wait for ETH to change to ProgPoW or PoS; making ETC the dominant chain in that PoW niche.
7. Change PoW algo to be dominant in a new PoW niche and minimize NiceHash renting attacks.
8. Two of the options under analysis are ProgPoW and Keccak256 (ECIP1049).
9. We will not reorg the chain or revert the events on chain under any circumstance.

We communicated the above list to the community at large. See image below.