Optimism - Unsafe Head Stall on Mainnet – Incident details

All systems operational

Unsafe Head Stall on Mainnet

Resolved
Major outage
Started 10 months agoLasted about 2 hours

Affected

Mainnet

Major outage from 5:04 AM to 7:17 AM

Transaction Sequencing

Major outage from 5:04 AM to 7:17 AM

Node Sync

Major outage from 5:04 AM to 7:17 AM

Updates
  • Resolved
    Resolved

    Our team has resolved the unsafe head stall.

    If you're a node operator and your nodes are still stuck, try restarting op-node and op-geth. If you continue to experience an issue, please let us know on Discord.

    This issue was in no way related to tomorrow's planned pause of OP Mainnet.

  • Monitoring
    Monitoring
    The fix has been fully rolled out, and the network is fully operational. We will continue monitoring the situation in case any additional instability is observed. No action is required from users or node operators.
  • Update
    Update
    We are continuing to roll out a potential fix for the unsafe head stall. At this time, the fix has rolled out across our infrastructure and we are waiting for the sequencer to come back online. No action is required from users or node operators. We will continue to update this page.
  • Update
    Update
    We are continuing to roll out a potential fix for the unsafe head stall. At this time no action is required from users or node operators. We will continue to update this page as the fix rolls out.
  • Identified
    Identified
    We have identified a potential root cause of the unsafe head stall. At this time the network is functional, however until the fix is fully rolled out the possibility of continued instability remains. We will continue to update this page as the fix rolls out.
  • Investigating
    Investigating

    We are reopening this incident after reports of additional instability.

  • Resolved
    Resolved
    Our team has resolved the unsafe head stall. If you continue to experience an issue, please let us know on Discord.
  • Monitoring
    Monitoring
    We have observed both the safe and unsafe parts of the chain advancing. At this time the network is fully operational. We will continue monitoring the situation.
  • Identified
    Identified
    We have identified the cause of the unsafe head stall and have applied a fix. At this time node operators may observe a single-block reorg of the unsafe chain. Chain sync should be back to normal at this time.
  • Investigating
    Investigating
    We are investigating reports of an unsafe head stall on OP Mainnet.