Optimism - All Sepolia OP-Stack chains: intermittent safe head stalls caused by L1 Sepolia Pectra deposit issue – Incident details

All systems operational

All Sepolia OP-Stack chains: intermittent safe head stalls caused by L1 Sepolia Pectra deposit issue

Resolved
Partial outage
Started 28 days agoLasted about 7 hours

Affected

Sepolia

Partial outage from 7:30 AM to 11:16 AM, Degraded performance from 7:30 AM to 11:16 AM, Partial outage from 11:16 AM to 2:38 PM, Degraded performance from 11:16 AM to 2:38 PM, Operational from 12:58 PM to 2:38 PM

Bridge UI

Partial outage from 7:30 AM to 2:38 PM, Operational from 12:58 PM to 2:38 PM

Deposits

Partial outage from 7:30 AM to 2:38 PM, Operational from 12:58 PM to 2:38 PM

Withdrawals

Partial outage from 7:30 AM to 2:38 PM, Operational from 12:58 PM to 2:38 PM

Transaction Sequencing

Degraded performance from 7:30 AM to 2:38 PM, Operational from 12:58 PM to 2:38 PM

Batch Submission

Partial outage from 7:30 AM to 2:38 PM, Operational from 12:58 PM to 2:38 PM

Updates
  • Resolved
    Resolved

    L1 Sepolia testnet has been updated and is stable again.

    For any L1 execution-engine replicas, please update to the releases in the previous status-update, if not already updated.

    Batch-submission and syncing of L2 testnets should be stable now that the L1 testnet is stable again.

  • Update
    Update

    L1 Sepolia Testnet is expected to update in 1 hour, at 2 PM UTC, to new L1 Execution-Layer Client releases.

    It is strongly recommended to update to these releases on Sepolia testnet.

    No action is needed on other L1 networks.

    Geth: https://github.com/ethereum/go-ethereum/releases/tag/v1.15.5

    Nethermind: https://github.com/NethermindEth/nethermind/releases/tag/1.31.4

    Reth: https://github.com/paradigmxyz/reth/releases/tag/v1.2.2

    Erigon: https://github.com/erigontech/erigon/releases/tag/v3.0.0-rc2

    Sepolia L1 nodes that do not update are expected to fall out of sync, which may prevent batch-submission and stops L2 nodes from syncing.

  • Monitoring
    Monitoring

    The L1 Execution-layer clients are testing and soon publishing releases that fix the identified L1 issue. Official L1 client release communications will follow.

    L1 Go-ethereum already has a ready release: https://github.com/ethereum/go-ethereum/releases/tag/v1.15.5

    L1 testnet node operators are strongly recommended to upgrade to these releases.

    The beacon-chain validators of the L1 Sepolia network are expected to deploy the fix in coordination around 2 PM UTC.

    After L1 node upgrades and the 2PM UTC deployment the L2 batch-submission issue is expected to resolve.

  • Identified
    Identified

    All OP-Stack chains on Sepolia experience intermittent safe head stalls caused by an issue regarding deposits processing on L1 Sepolia. This is an ongoing situation since the Sepolia Pectra activation at 7:29 UTC and we are monitoring it while client teams work on fixing Sepolia.