On the night of Sept. 22, a user sent an ATOM transfer.
By the next morning, the transaction was still stuck in a pending state. The private key had not been lost, and there was no signing issue in the wallet. A fresh check showed that multiple public RPC endpoints were all reporting the same thing: Cosmos Hub had stopped at block height 33,086,740. No new blocks were being produced, so there was nowhere for the transaction to be included. Only after Cosmos Hub resumed block production about a day later did the transfer finally go through.
For ordinary users, the episode offered a direct lesson in how blockchain consensus actually works. A public chain may not have a central operator with a literal off switch, but that does not mean it cannot stop.
In this case, the pause on Cosmos Hub exposed mechanics that usually stay hidden under the surface. The chain did not stop because a company pressed a button. It stopped because enough validators were no longer participating in consensus.
What caused Cosmos Hub to stop producing blocks
One point needs to be separated at the outset: Cosmos Hub was not the system directly attacked.
The incident began on Neutron. On Sept. 22, a Neutron governance proposal titled AIATO: AI Agent Takeover passed. The attacker exploited chain-level governance privileges and used native privileged instructions in the wasmd framework to change the contract admin of applications including Astroport and Drop to an address under the attacker’s control.
This was not a standard smart contract bug or a protocol flaw in the usual sense. A simpler way to frame it is that the applications had their own locks, but Neutron’s chain governance also held a higher-level master key. Once the attacker controlled the governance outcome, that key could be used to reassign admins, migrate contracts, and move assets held inside them.
Cosmos Hub became involved when the stolen assets began moving across chains. According to a post-incident review from Cosmos Labs, the attacker had already routed part of the assets to several networks before Neutron stopped running. That included about 1.7 million ATOM sent into Cosmos Hub, where the funds began to be swapped through interchain liquidity.
That distinction matters. Cosmos Hub itself was not directly exploited, and ordinary Hub users did not lose funds because of the Neutron issue. But the attacker’s ATOM had already entered the Hub. To stop the remaining ATOM from moving out, some Cosmos Hub validators shut down their nodes.
By around 19:18 SGT on Sept. 22, the validators that had stopped represented more than one-third of total voting power. At that point, Cosmos Hub could no longer form new blocks and stalled at height 33,086,740.
This is the key part of the story. There was no company-operated pause button, and the network was not first halted through an onchain governance vote. The chain stopped because enough validators ceased taking part in consensus.
How the network came back
The recovery process is just as important as the halt itself.
About four hours after the chain stopped, validators received a full recovery plan. The proposal called for a one-time state modification at the halted height, moving the remaining ATOM in the attacker’s address into a multisig managed jointly by community validators.
Cosmos Labs then prepared a Gaia v28.3.0 patch based on the plan validators had already agreed to, tested it, and distributed it to validators. This version of Gaia was designed to execute a one-time state change at a specified recovery height, transferring 1,227,121 ATOM from the attacker address into a 4-of-6 multisig controlled by Nansen, Keplr, Enigma, Silknodes, Kiln, and Polkachu.
By early Sept. 23, validators that had confirmed installation of v28.3.0 represented more than 67% of total voting power. Cosmos Hub then coordinated a restart at 12:00 UTC that day. Roughly six minutes later, the one-time state change executed at block height 33,086,741, and normal block production resumed.
In practical terms, the sequence had two stages. Validators first caused the network to lose liveness. Then more than two-thirds of voting power accepted a new state transition rule, and that rule became part of the canonical state after the restart.
That leads to the obvious question: if this is a decentralized public chain, why can more than one-third of voting power stop it, while bringing it back requires a larger group of validators to accept and run the same software? The answer sits inside the word consensus.
Consensus does not mean a chain can never stop
One of the most common misunderstandings in crypto is to treat decentralization as if it guarantees permanent uptime.
Consensus mechanisms solve a narrower problem. They define how many independent nodes can agree on transaction ordering and ledger state without a central bookkeeper. Different chains solve that problem in different ways.
Bitcoin’s classic model is Proof of Work. Miners compete with hash power to produce blocks. When two valid branches briefly exist at the same time, nodes follow the branch with the most accumulated work.
That means Bitcoin does not have a clean moment where a block becomes permanently finalized after a 67% vote. Its finality is probabilistic. The more blocks that are built on top of a transaction, the more expensive it becomes to reorganize it. That is why Bitcoin transactions have long been said to be safer after six confirmations.
Even so, that does not mean Bitcoin’s state is absolutely unchangeable under every condition. In theory, if the broader ecosystem accepted a new client and a new set of consensus rules, a hard fork could make a state transition valid even if it had been invalid under the old rules. The real obstacle is coordination. Who can persuade enough miners, full nodes, exchanges, wallets, and users to adopt that new rule set? The threshold is extremely high. A development team cannot decide consensus rules for the whole Bitcoin network, and miners or exchanges cannot easily do it either. When Binance lost 7,000 BTC in a hack, there were suggestions that CZ contact major miners to intervene. Nothing came of it.
Ethereum offers a different reference point. After moving to Proof of Stake, Ethereum now uses Gasper, which combines Casper FFG and LMD-GHOST. One part of the system determines which chain the network should follow. Another part gives blocks actual finality.
When validators representing at least two-thirds of staked ETH agree on the relevant checkpoint, blocks can move toward finalization. If more than one-third of stake fails to vote correctly for an extended period, the network can temporarily lose finality. Ethereum also includes an inactivity leak, which gradually reduces the effective weight of offline validators when finalization is stalled for too long, giving the network a path to recover finality over time.
Changing that outcome still requires changes to protocol rules and client software. The 2016 DAO incident remains the clearest example. The Ethereum community ultimately adopted a hard fork that executed a special state change at block 1,920,000. At the time, the Ethereum Foundation explicitly described it as an irregular state change, moving the affected ETH into a recovery contract. Not everyone accepted that decision. A portion of miners and community members refused to upgrade and continued maintaining the original state, which led to the split between ETH and Ethereum Classic, or ETC.
Cosmos Hub is different again. It uses CometBFT, a more typical BFT-style consensus system. For a block to be committed, it needs Commit votes from more than two-thirds of voting power. The advantage is clear finality. Once a block is committed with enough validator weight, there is no need to wait for more blocks the way PoW systems rely on probability for confidence.
The trade-off is just as clear. If one-third or more of voting power stops providing the votes needed for Commit, the remaining validators cannot reach the threshold on their own. In that situation, the safest option is to stop producing blocks. From a distributed systems perspective, the short halt on Cosmos Hub was not mysterious at all. Once a group of validators with enough voting power stopped participating, the protocol followed its own rules and chose to lose availability rather than confirm new blocks without sufficient agreement.
This maps directly to two concepts that are often blurred together:
- Safety: the system must not let different nodes finalize conflicting states at the same time;
- Liveness: the network must keep moving forward and continue processing new transactions.
In BFT systems, a pause can be the price paid to preserve safety when too few participants remain in consensus. Put plainly, the ledger would rather stop than let the remaining participants write incompatible versions of history.
From Bitcoin to Solana, the failure boundaries are not the same
Cosmos is not the first network to force this issue into the open. Public chain history includes several incidents that look different on the surface but revolve around the same core question: what happens when distributed participants can no longer agree on the correct state?
Bitcoin faced a classic chain split in 2013. Bitcoin 0.8 had switched its underlying database from Berkeley DB to LevelDB. A block with a large number of transaction inputs then appeared. Newer nodes processed it without trouble, but some older nodes treated it as invalid because of Berkeley DB lock limits.
The result was awkward and serious at the same time. Everyone was still running Bitcoin, but old and new clients gave different answers to the question of whether that block was valid. The network split into two chains. The 0.8 side at one point had about 60% of hash power, so the issue could not quickly resolve itself through ordinary mining competition. Large mining pools eventually coordinated a return to the older version, restoring more hash power to the old-rule side and allowing the network to converge again. Bitcoin later reviewed the incident in BIP 50.
The DAO incident in 2016 pushed the debate further. As noted above, Ethereum adopted a hard fork at block 1,920,000 to execute a special state change and move the affected ETH into a recovery contract. But not everyone agreed with that response. Some miners and community members rejected the state change and continued with the original rules, which is how Ethereum Classic persisted. The episode showed that code consensus is not the whole story. Social consensus also matters, and if agreement breaks down badly enough, one chain can become two.
Solana’s 2021 outage showed a different path to failure. In September that year, a flood of bot transactions hit the network, exhausting validator memory and causing large numbers of nodes to crash. The network could no longer maintain agreement on the current state and stopped confirming new blocks for about 17 hours. Validators later coordinated a restart.
Placed side by side, these incidents are not the same event repeated in different forms:
- Bitcoin in 2013 involved different clients enforcing different validity rules;
- Solana in 2021 involved a large number of validators losing the ability to keep participating in consensus, which cost the network liveness;
- The DAO fork on Ethereum centered on whether a community should actively alter state through new protocol rules;
- This Cosmos Hub episode added another layer: validators first coordinated a deliberate loss of liveness to stop attack-related assets from moving, then a sufficiently large share of voting power accepted new software and a recovery state so the network could converge again.
That is why reducing all of these cases to slogans such as “blockchains can be switched off” or “decentralization is fake” misses the point. Consensus is not a machine that never breaks. It is a decentralized rule set that answers harder questions: who decides the correct chain when disagreement appears, how much participation is needed for finality, whether the network should keep running or stop during a fault, and what kind of collective action can change the rules in extreme conditions.
Private key control is not the same as consensus control
The Cosmos Hub halt leaves behind a question that goes beyond whether the chain should have been stopped.
The phrase “not your keys, not your coins” still holds. But it speaks to control over assets. If you hold your own private keys, a wallet, exchange, or other third party cannot normally sign a transfer on your behalf.
That principle depends on another condition: the blockchain you are using must still be able to process the signed transaction.
On the day Cosmos Hub stopped producing blocks, users still held their private keys, and their assets did not vanish. But even a correctly signed transaction had no new block available to accept it. The recovery process also showed something else. If enough consensus participants accept a new state rule, the onchain state of a specific account can change even without a signature from the original address.
That does not invalidate “not your keys, not your coins.” It does show that private key sovereignty and control over base-layer consensus are not the same thing.
The same distinction applies to wallets. A wallet can keep private keys and signing authority in the user’s hands. It can detect chain-level anomalies more quickly, display transaction status more accurately, build redundancy across RPC endpoints and nodes, and re-check final transaction outcomes after the network recovers.
What a wallet cannot do is restore consensus for a public chain, guarantee that the underlying network will never be interrupted, or ensure that onchain rules and state will never change at the consensus layer.
A mature decentralized system may not need to promise that nothing can ever be changed. What it does need is clarity around the boundaries: who can pause consensus, how much weight is required, and under what conditions emergency intervention is allowed.
Real decentralization does not mean a system will never face accidents. It means that when an accident does happen, people can still see who made the decision, under which rules, and with how much consensus the ledger moved forward.