All systems operational

Show current status

Resolved
Instability on Optimism Goerli

Started
March 14, 2023 at 10:46 PM
Status
Resolved after about 9 hours

Impact

Major outage
Affected
Goerli
Deposits
Gateway
Public API
Transaction Sequencing
Withdrawals
  • Resolved
    March 15, 2023 at 8:06 AM

    This incident has been resolved as of 3/15 already.

  • Resolved
    March 15, 2023 at 8:05 AM

    We are resolving this incident. Please watch our Twitter for updates once we have published a fixed op-node release and node operators can remove the temporary flag.

  • Monitoring
    March 14, 2023 at 11:47 PM

    We have publicly shared the immediate fix to solve the issue for node operators on Optimism Goerli. Major services and infrastructure that operate Optimism Goerli nodes are slowly starting to resume operations as they add the specified flag.

    We will post an update when we have published a fixed op-node release and node operators can remove the temporary flag.

  • Identified
    March 14, 2023 at 11:16 PM

    The sequencer is confirmed to be up and processing transactions, deposits, and withdrawals safely. All nodes serving the public RPC endpoint have been updated with the below flag.

  • Identified
    March 14, 2023 at 11:09 PM

    Our team has identified the cause of the Optimism Goerli node halts.

    With the L1 Goerli network upgrading to Shapella, a change during Shanghai was made to how the block hash is derived. We have checks inside the op-node to recompute the block hash to ensure your L1 RPC is being honest - these started failing with the Shanghai updates.
    This will prevent any nodes being operated on Optimism Goerli from continuing to produce new blocks. The sequencer is confirmed to be up and processing transactions, deposits, and withdrawals safely.

    If you operate nodes on Optimism Goerli:

    To immediately fix this issue, please add a flag:
    --l1.trustrpc=true
    or
    OP_NODE_L1_TRUST_RPC="true"
    to disable these block hash checks.

    We will be pushing a new release of the op-node in the coming days with a fix. After upgrading to that new release you will be able to remove the above flag.

  • Investigating
    March 14, 2023 at 10:46 PM

    We are investigating report of instability on Optimism Goerli.