Skip to content
Quantix Security Labs

A compromised upgrade multisig and the delay that stopped it

Two of three signing keys for Obsidian Rollup’s upgrade multisig lived on the same host. How an upgrade of the withdrawal portal was scheduled with them, detected within seconds and cancelled inside the delay.
CriticalKey managementIncident response · 10 min read
Illustration: two signing keys on one shared host, a malicious upgrade held at a timelock ring before the withdrawal portal.

The case at a glance

Obsidian Rollup

Ethereum · Layer 2Security profileResearch case · Quantix Protocol Research

During the case
BBB75.2
Current profile
A83.0
To containment
49 min
To verified fix
37 h 46 min
  1. 03Problem

    At 02:14 UTC on 14 September 2026 the rule for upgrades to unapproved implementations raised a critical alert: a timelock operation had been scheduled to point the Portal proxy at a contract deployed eighteen minutes earlier by a single-use address, with bytecode that matched no release.

    Read section
  2. 06Vulnerability

    CriticalKey management

    The contracts behaved as designed. The failure was operational: a multisig threshold counts keys, not machines, and two of three keys shared a host that build jobs could reach.

    Read section
  3. 08Remediation

    1. 01Pause withdrawal finalisation and cancel the operation
    2. 02Rotate the exposed owners to hardware-backed keys held by separate people
    3. 03Raise the multisig threshold to 3-of-5

    and 2 more steps

    Read section
  4. 09Outcome

    The malicious upgrade never executed and no funds left the portal. Withdrawal finalisation was paused for about a day and a half and resumed after re-verification.

    Risk Index BBB → A, +7.8 points

    Read section

Background

Obsidian Rollup settles to Ethereum. Its Portal holds deposited assets and finalises withdrawals once a dispute game has resolved. The portal is an upgradeable proxy; upgrades are scheduled by a 2-of-3 upgrade multisig through an UpgradeTimelock, and a 4-of-7 security council holds the canceller role. A guardian can pause withdrawal finalisation.

The research assessment had already flagged the upgrade path (OR-02) and moved routine upgrades behind a timelock. What it could not see was where the multisig’s keys were kept.

Architecture

Portal, OutputOracle and DisputeGame are proxies administered by a proxy admin owned by the UpgradeTimelock. Whoever can schedule a timelock operation can schedule an upgrade of the contract that releases funds; the delay and the canceller are the only checks after that.

The case covers the governance path around the portal, not the proof system.

Figure 1Obsidian Rollup: components and paths involved in this case

Obsidian Rollup: architecture relevant to this case8 elements and 7 connections. Highlighted components are directly involved in the case; dashed red arrows are attack paths, dotted brass arrows are control paths.signsexecTransaction (2 sigs)schedule()cancel(id)proxy upgradeoutput rootsgame resultCI-connected signinghostheld 2 of 3 owner keysUnauthorised key userfresh deployer addressUpgrade multisig2-of-3UpgradeTimelock48 h delay at the timeSecurity council4-of-7 cancellerPortalDeposits, withdrawalproving and finalisationOutputOracleStores proposed L2 staterootsDisputeGameBisection fault proofs overdisputed outputs
Protocol componentInvolved in this caseExternal partyAttack pathControl path

Elements:

  • CI-connected signing host (external party): held 2 of 3 owner keys
  • Unauthorised key user (adversary): fresh deployer address
  • Upgrade multisig (control, involved in this case): 2-of-3
  • UpgradeTimelock (control, involved in this case): 48 h delay at the time
  • Security council (external party): 4-of-7 canceller
  • Portal (protocol component, involved in this case): Deposits, withdrawal proving and finalisation
  • OutputOracle (protocol component): Stores proposed L2 state roots
  • DisputeGame (protocol component): Bisection fault proofs over disputed outputs

Connections:

  • CI-connected signing host to Upgrade multisig: signs (data or call flow)
  • Unauthorised key user to Upgrade multisig: execTransaction (2 sigs) (attack path)
  • Upgrade multisig to UpgradeTimelock: schedule() (attack path)
  • Security council to UpgradeTimelock: cancel(id) (control path)
  • UpgradeTimelock to Portal: proxy upgrade (control path)
  • Portal to OutputOracle: output roots (data or call flow)
  • Portal to DisputeGame: game result (data or call flow)

Components in the Obsidian Rollup profile

  • BatchInbox

    Receives compressed batches from the sequencer

  • OutputOracle

    Stores proposed L2 state roots

  • DisputeGame

    Bisection fault proofs over disputed outputs

  • Portal(involved in this case)

    Deposits, withdrawal proving and finalisation

  • Sequencer (Go)

    Orders transactions and builds batches

Security challenge

At 02:14 UTC on 14 September 2026 the rule for upgrades to unapproved implementations raised a critical alert: a timelock operation had been scheduled to point the Portal proxy at a contract deployed eighteen minutes earlier by a single-use address, with bytecode that matched no release.

The operation would become executable 48 hours later. The response had to confirm the operation was unauthorised, protect withdrawals in the meantime, cancel it, and find out how two multisig signatures had been produced outside the change window.

Attack surface

The response mapped every key and process that could cause an upgrade.

Attack surface examined in Obsidian Rollup
SurfaceExposureControl
Upgrade multisig ownersTwo of three keys on the same CI-connected signing host3-of-5 with hardware-backed keys, one per keyholder
Signing hostReachable from build jobsNo upgrade keys on CI-reachable systems
Implementation codeAny address could be scheduled as the new implementationApproved-release hash check in monitoring (R-01)
Timelock delay48 hours, shorter than the 7-day withdrawal window8 days for withdrawal-path contracts
Canceller and guardianSecurity council 4-of-7; guardian pauseUnchanged; both used within hours

Technical investigation

The scheduled implementation was decompiled within the first hour. Its withdrawal finalisation skipped the dispute-game check and paid any withdrawal submitted by one hard-coded address. Executed, it would have let that address withdraw the portal’s balance.

The multisig transaction carried valid signatures from owners one and three. Both keys sat on the same signing host, which build jobs could reach, so the 2-of-3 threshold had in practice become one compromised machine. The second owner, held separately, had not signed.

The guardian paused withdrawal finalisation 48 minutes after the alert as a precaution. The security council cancelled the operation 3 hours 26 minutes after it was scheduled, with more than 44 hours of the delay still to run, and both exposed owners were replaced that morning.

Finalisation resumed the next day once the state roots for the paused period had been re-verified. No other operation had been scheduled with the exposed keys.

Figure 2Malicious upgrade scheduled, detected and cancelled

Malicious upgrade scheduled, detected and cancelledUnauthorised keyuserUpgradeTimelockMonitoring (R-01)Guardian and councilPortal proxy1. CREATE implementation (fresh deployer)bytecode matches no release2. schedule(upgrade(Portal, newImpl), 172800 s)2-of-3 multisig3. CallScheduled4. critical: unapproved implementation5. guardian: pause finalisation6. council: cancel(id)upgrade never executes
Ordinary stepAttack stepDefensive step
  1. Unauthorised key user (adversary) to Portal proxy: CREATE implementation (fresh deployer). bytecode matches no release (attack step)
  2. Unauthorised key user (adversary) to UpgradeTimelock: schedule(upgrade(Portal, newImpl), 172800 s). 2-of-3 multisig (attack step)
  3. UpgradeTimelock (on-chain) to Monitoring (R-01): CallScheduled (normal step)
  4. Monitoring (R-01) (off-chain) to Guardian and council: critical: unapproved implementation (defensive step)
  5. Guardian and council (participant) to Portal proxy: guardian: pause finalisation (defensive step)
  6. Guardian and council (participant) to UpgradeTimelock: council: cancel(id). upgrade never executes (defensive step)
Response against the 48-hour upgrade delay. Times from the monitoring record; remaining delay is computed from the scheduled timestamp.
StepTime (UTC)Since schedulingDelay remaining
Malicious implementation deployed2026-09-14 01:56−18 min—
Upgrade multisig signs outside the change window2026-09-14 02:13−18 s—
Upgrade scheduled; R-01 alert2026-09-14 02:140 s48 h
Guardian pauses withdrawal finalisation2026-09-14 03:0249 min47 h 11 min
Security council cancels the operation2026-09-14 05:403 h 26 min44 h 34 min
Operation would have become executable2026-09-16 02:1448 h0 s
Monitoring events recorded during the incident, with the rule that matched and its threshold
Time (UTC)RuleObservationMeasuredAlert
2026-09-14 01:56:30R-16Activity above baselineContract deployed by a fresh, single-use deployerMedium4 (threshold 3 × 30-day median)Alert
2026-09-14 02:13:51R-02Privileged role changeUpgrade multisig executed outside the change windowHighCondition matchedAlert
2026-09-14 02:14:09R-01Upgrade to unapproved implementationPortal upgrade scheduled to an unapproved implementationCriticalCondition matchedAlert
2026-09-14 03:02:44R-15Pause state changedWithdrawal finalisation paused by the guardianMediumCondition matchedAlert
2026-09-14 05:40:18R-08Governance lifecycleSecurity council cancels the scheduled upgradeLowCondition matchedAlert
2026-09-14 09:25:52R-02Privileged role changeUpgrade multisig owners rotatedHighRecorded for the timelineLogged
2026-09-15 16:00:07R-15Pause state changedWithdrawal finalisation resumedMediumCondition matchedAlert
2026-09-16 12:00:00R-09Protocol parameter changePortal upgrade delay raised from 48 hours to 8 daysMediumRecorded for the timelineLogged

Timeline

  1. Exploitation

    Unreleased Portal implementation deployed by a fresh address

  2. Upgrade multisig executes outside the change window

  3. Detection

    Portal upgrade scheduled; R-01 critical alert

  4. Containment

    Guardian pauses withdrawal finalisation

  5. Security council cancels the operation

  6. Fix

    Exposed owners rotated; threshold 3-of-5

  7. Verification

    Finalisation resumed after state roots re-verified

  8. Portal upgrade delay raised to 8 days

Root cause

The contracts behaved as designed. The failure was operational: a multisig threshold counts keys, not machines, and two of three keys shared a host that build jobs could reach. A threshold of two had collapsed to a single point of compromise.

The delay was also shorter than the withdrawal window, so had the operation gone unnoticed, users could not have exited before it executed — the condition OR-02 had warned about.

Class
Key management
Severity
Critical
Components involved
Portal
Findings
OR-02
signing-service configuration (pre-incident)text
# keys served by the release signing host- id: upgrade-owner-1          # Upgrade multisig owner[0]  backend: local-file          # same host  reachable_from: [ci-runners] # build jobs can call the signer- id: upgrade-owner-2          # owner[1]  backend: hardware            # held separately  reachable_from: []- id: upgrade-owner-3          # owner[2]  backend: local-file          # same host as owner[0]  reachable_from: [ci-runners]threshold: 2-of-3

Risk evaluation

Operational Security drops sharply while two owner keys are exposed, and Governance Risk reflects that one compromised host could start an upgrade of the fund-holding contract. No code finding is open in this window: the contracts behaved as designed.

75.2BBBModerate risk
During the caseBBB75.2
83.0AModerate risk
Current profileA83.0
CodeSecurityOperationalSecurityGovernanceRiskEconomicSecurityInfrastructureResilienceTransparencyDuring the case, Code Security: 86During the case, Operational Security: 50During the case, Governance Risk: 64During the case, Economic Security: 84During the case, Infrastructure Resilience: 82During the case, Transparency: 90Current profile, Code Security: 86Current profile, Operational Security: 80Current profile, Governance Risk: 76Current profile, Economic Security: 84Current profile, Infrastructure Resilience: 82Current profile, Transparency: 90
During the caseCurrent profile

The score moves from 75.2 (BBB) to 83.0 (A), a change of +7.8 points. The largest single contribution is Operational Security: restoring it from 50.0 to 80.0 alone moves the score from 75.2 to 81.2.

Basis for the case state: Two owner keys exposed on one host; upgrade delay shorter than the withdrawal window.
ComponentWeightRangeDuring caseTodayScore points
Code Security30%86.086.00.0
Operational Security20%50.080.0+6.0
Governance Risk15%64.076.0+1.8
Economic Security15%84.084.00.0
Infrastructure Resilience10%82.082.00.0
Transparency10%90.090.00.0
Quantix Risk Index100%75.283.0+7.8
During the caseCurrent profileBasis for the case state: Two owner keys exposed on one host; upgrade delay shorter than the withdrawal window.

Scores use the six weighted components of the Quantix Risk Index. Open findings reduce Code Security by a fixed penalty per severity; the current profile is the protocol’s entry in Quantix Intelligence.

Mitigation strategy

Both exposed owners were replaced with hardware-backed keys held by separate people, and the threshold rose to 3-of-5. No upgrade key remains on any system that build jobs can reach; the release pipeline now produces an unsigned proposal for keyholders to review.

The delay for upgrades to withdrawal-path contracts was raised from 48 hours to 8 days, longer than the 7-day withdrawal window, so users can always exit before an upgrade they disagree with takes effect.

  1. 01Pause withdrawal finalisation and cancel the operation
  2. 02Rotate the exposed owners to hardware-backed keys held by separate people
  3. 03Raise the multisig threshold to 3-of-5
  4. 04Remove every upgrade key from CI-reachable systems
  5. 05Raise the withdrawal-path upgrade delay above the withdrawal window
Upgrade path configuration (after the incident)diff
-upgradeMultisig.threshold = 2 of 3      // owners 0 and 2 on one signing host+upgradeMultisig.threshold = 3 of 5      // hardware-backed keys, one per keyholder -UpgradeTimelock.delay(Portal) = 172800  // 48 hours+UpgradeTimelock.delay(Portal) = 691200  // 8 days, longer than the 7-day withdrawal window -release pipeline: signs and submits schedule()+release pipeline: prepares an unsigned proposal for keyholder review

Security outcome

The malicious upgrade never executed and no funds left the portal. Withdrawal finalisation was paused for about a day and a half and resumed after re-verification.

The incident completed the open part of OR-02: withdrawal-path upgrades now wait longer than the withdrawal window.

Detection to containment
49 min
Detection to first fix
7 h 12 min
Detection to verified fix
37 h 46 min
Risk Index change
BBB → A (+7.8)

Lessons learned

  1. 01

    A timelock buys time only if someone watches the queue.

  2. 02

    A multisig threshold counts keys; count the machines and people behind them as well.

  3. 03

    Keep signing keys off systems that run build jobs or third-party code.

  4. 04

    Make the upgrade delay longer than the time users need to exit.

Have a problem like this one?

Tell us what you are building and where it runs. We will reply with the review plan we would use for your architecture.