Optimism - Unsafe Head Stall on Mainnet – Incident details

Unsafe Head Stall on Mainnet

ResolvedMajor outage2 hours 13 minutes

Affected

Mainnet

Major outage from 5:04 AM to 7:17 AM

Transaction Sequencing

Major outage from 5:04 AM to 7:17 AM

Node Sync

Major outage from 5:04 AM to 7:17 AM

  • ResolvedUTC

    Our team has resolved the unsafe head stall.

    If you're a node operator and your nodes are still stuck, try restarting op-node and op-geth. If you continue to experience an issue, please let us know on Discord.

    This issue was in no way related to tomorrow's planned pause of OP Mainnet.

  • MonitoringUTC
    The fix has been fully rolled out, and the network is fully operational. We will continue monitoring the situation in case any additional instability is observed. No action is required from users or node operators.
  • UpdateUTC
    We are continuing to roll out a potential fix for the unsafe head stall. At this time, the fix has rolled out across our infrastructure and we are waiting for the sequencer to come back online. No action is required from users or node operators. We will continue to update this page.
  • UpdateUTC
    We are continuing to roll out a potential fix for the unsafe head stall. At this time no action is required from users or node operators. We will continue to update this page as the fix rolls out.
  • IdentifiedUTC
    We have identified a potential root cause of the unsafe head stall. At this time the network is functional, however until the fix is fully rolled out the possibility of continued instability remains. We will continue to update this page as the fix rolls out.
  • InvestigatingUTC

    We are reopening this incident after reports of additional instability.

  • ResolvedUTC
    Our team has resolved the unsafe head stall. If you continue to experience an issue, please let us know on Discord.
  • MonitoringUTC
    We have observed both the safe and unsafe parts of the chain advancing. At this time the network is fully operational. We will continue monitoring the situation.
  • IdentifiedUTC
    We have identified the cause of the unsafe head stall and have applied a fix. At this time node operators may observe a single-block reorg of the unsafe chain. Chain sync should be back to normal at this time.
  • InvestigatingUTC
    We are investigating reports of an unsafe head stall on OP Mainnet.