Hardware Failure: IncogniteeAHP CC1 Incident Post-Mortem

Incognitee on Asset Hub Polkadot is in an irrecoverable state since October 3rd, 2025. In this post-mortem, we’ll explain what happened, why the damage is irrecoverable, and how we plan to recover user funds.

Scope

IncogniteeAHP CC1 is our first beta-sidechain on Asset Hub Polkadot, and it shields the following assets: DOT, ETH, WBTC, USDC, USDT, EURC, PEPE. IncogniteeTEER is unaffected and is still live for the TEER token.

Timeline

On October 3rd 2025, at (09:37am UTC), Incognitee was gracefully shut down for planned maintenance, which should have been just a BIOS and microcode update to get the latest security patches for Intel SGX. The maintenance window was announced by us to end no later than 1:00pm UTC the same day. At 2:25pm UTC, we still hadn’t gotten any update from our cloud service provider (OVH), thus reaching out several times over the following 70h (including the weekend).

On October 4th (4:31pm UTC), we attempted to spin up Incognitee again, and realized that:

1. SGX was disabled in the BIOS
2. The serial number of the motherboard had changed
3. Incognitee could no longer unseal its data, but failed with SGX_ERROR_MAC_MISMATCH

On October 6th (12:38pm UTC), OVH finally informed us that they had replaced the mainboard due to damage. This intervention had not been approved by us, and they only informed us about it after we shared the observation that the mainboard’s serial number had changed.

Over the next couple of days, we tried to find out if the CPU was still the same and if the old mainboard could be recovered.

By October 10th (12:27pm UTC), OVH’s final statement concluded that:

1. The CPU was still the same
2. The mainboard was irrecoverable

Damage Assessment

Incognitee uses Intel SGX’s sealed data feature to persist:

1. The private ed25519 key of its wallet on Asset Hub, which is a proxy to the vault account that holds the pool of all funds shielded to Incognitee.
2. The private RSA3072 shielding key used to encrypt requests (like transactions) to the validateer
3. The symmetric AES128 encryption key used to encrypt the current state of the sidechain when stored to disk
4. The last few sidechain blocks
5. The state of the light-client DBs for Asset Hub Polkadot, and Integritee Network on Polkadot

Incognitee Beta Setup

During Incognitee’s beta phase, we used Intel’s MRSIGNER sealing policy, which allows us (the software “vendor”) to update the enclave and still unseal the data as long as we stay on the same machine and sign the enclave with our vendor key (read this for context on this design decision).

IncogniteeAHP CC1 runs on a single machine, a scalable multiplatform with one Intel XEON Gold 6526Y CPU. Although Integritee’s SDK demonstrated multi-validateer setup already back in 2022, we had observed forks in real-world multi-validateer setups on distributed nodes, and for Incognitee beta, we decided to keep the setup easy to debug and take the risks that come with missing redundancy.

Why IncogniteeAHP CC1 can’t be recovered

On older SGX machines, the sealing key is derived from a secret fused into the CPU by Intel, which is unique for each CPU. This means sealed data can only be unsealed by the same CPU instance and only by the same enclave (or an enclave signed by the same vendor as we use MRSIGNER policy)

On a multiplatform like ours, the sealing key is the result of a key agreement among all CPUs on the same platform, and it is stored in the mainboard’s NVRAM. This means sealed data can only be recovered by the same platform, which is the same CPU on the same mainboard.

As OVH informed us that the mainboard suffered irrecoverable damage, we have to conclude that there is no chance we can recover the sealed storage.

What is lost

• Access to funds: The private key of the single proxy account to the vault holding all assets shielded to Incognitee
• Information about ownership: Balances held by user accounts on incognitee. We have no way to know the balance of our users’ accounts, and we don’t even know how many accounts there were (which is what Incognitee is all about).

The total amount of assets locked in Incognitee’s vault is:

• 5,800.575 DOT
• 1,031.2 USDC
• 111.56 USDT
• 22.6 EURC
• 0.502 ETH
• 0.000195 WBTC
• 1,939,999.999 PEPE

At the time of writing, the market value of all locked assets is $17,692.

Could this have been prevented?

Unfortunately, the short answer is yes. But it’s more complicated.

Our single-validateer setup came with the known risk of total loss of access and information in the occurrence of irrecoverable hardware issues. On the other hand, a multi-validateer setup came with other risks of lower impact but higher likelihood; temporary state ambiguities due to forks, which we had not yet implemented all necessary mitigations for to ensure smooth operation. For the sake of testing the user interest in our Incognitee product, we decided to take the bounded risk, as we had to limit shielding amounts for legal reasons anyway.

In hindsight, there is one more thing we could have done, which would’ve partially spared us this awkward situation without risking forks: we could’ve run a kind of snapshotting in a master-slave setup by spinning up a second validateer at regular intervals, getting secrets and state provisioned to this backup platform, and shutting it down before it attempts to produce blocks itself. This would’ve allowed us to roll back to the latest snapshot, which would still be better than the situation we’re in now. This shall be a learning for the future (and others using our SDK).

Options for the recovery of funds

With the private key of the only proxy gone, only Polkadot OpenGov has the power to move assets out of the vault account. This, however, needs a root track referendum to be approved by DOT holders. We are able to prove to the community that the lost private key had been generated by a genuine Intel SGX setup and never left the enclave (subject to Intel SGX’s threat model). We are, of course, unable to prove the absence of the possibility that this enclave will ever be operational again, as we have to rely on the information we got from OVH.

As the information about ownership is lost as well, we could only rely on our users’ voluntary claims of ownership. Even if we can assess the plausibility of such claims, we are unable to prove the contrary (with the exception of someone claiming more than the entire vault holdings, of course).

Claim your balance

If you held a balance on Incognitee in one of the affected assets and are willing to trade your privacy for the possibility of a refund, please reach out to us at hello@integritee.network. You will have to provide as much proof for your claim as you can (e.g, sign your message with the wallet you used for shielding and optionally provide accounts you used for unshielding if you like).

We will not disclose the information you provide publicly unless we have your explicit approval to do so. However, any refund through an on-chain balance transfer may leak linkable information. We will be collecting claims until November 30th, 2025 (12:00pm UTC).

• • •

About Integritee

Integritee is the most scalable, privacy-enabling network with a Parachain on Kusama and Polkadot. Our SDK solution combines the security and trust of Polkadot, the scalability of second-layer Sidechains, and the confidentiality of Trusted Execution Environments (TEE), special-purpose hardware based on Intel Software Guard Extensions (SGX) technology, inside which computations run securely, confidentially, and verifiably.

Community & Social Media:
Join Integritee on Discord | Telegram | Twitter | Medium | Youtube | LinkedIn | Website

Products:
L2 Sidechains | Trusted Off-chain Workers | Teeracle | Attesteer | Securitee | Incognitee

Integritee Network:
Governance | Explorer | Mainnet | Github

TEER on Exchanges:
Kraken | Gate | Basilisk