Goerli
Sepolia
Notice history
Jul 2023
- ResolvedResolved
OP Goerli is no longer experiencing node halts, which were caused by the L1 Goerli testnet reorg. As shared in the previous status, full node operators do need to take one of the following actions:
- Delete Geth’s datadir, then replace it with the fully-synced archival database available from https://datadirs.optimism.io/goerli-bedrock-archival-2023-07-29.tar.zst.
- Make a call to
eth_getblockbynumberwith finalized as the block tag. Then, roll the full node back to that block value + 1 usingdebug_setHead. - Resync the full node from a backup prior to block 12572500, or from genesis.
This issue is resolved and we will be sharing a full post-mortem when ready.
- MonitoringMonitoring
We have identified the root cause of this issue.
The L1 Goerli testnet experienced a large reorg of approximately 18 blocks. In addition to the large reorg, a significant number of blocks were missed, resulting in one large reorg and many missing blocks. This caused a large (>256 block) reorg on OP Goerli. Currently, users operating full nodes on OP Goerli will experience a node halt. These nodes will not sync new blocks from the OP Goerli network, and may return inconsistent data over RPC. Users operating archival nodes on OP Goerli are not affected and we do not expect this issue to occur in any capacity on OP Mainnet.
To resolve this issue, full node operators need to perform one of these actions:
- Delete Geth’s datadir, then replace it with the fully-synced archival database available from https://datadirs.optimism.io/goerli-bedrock-archival-2023-07-29.tar.zst.
- Make a call to
eth_getblockbynumberwith finalized as the block tag. Then, roll the full node back to that block value + 1 usingdebug_setHead. - Resync the full node from a backup prior to block 12572500, or from genesis.
We apologize for the inconvenience and will share a full post-mortem when ready.
- InvestigatingInvestigating
Our team is investigating reports of issues with full node sync. At this time, users syncing full nodes may experience degraded performance.
We apologize for the inconvenience and will share an update once we have more information.
Jun 2023
- CompletedJune 06, 2023 at 10:48 PMCompletedJune 06, 2023 at 10:48 PM
Much of our our core OP Mainnet infra is back online! Continue to track https://oplabs.notion.site/External-OP-Mainnet-Bedrock-Upgrade-Infra-Status-Tracker-07d7c9c0916243b8b743b74af77d37f9?pvs=4 for updates on other key infrastructure in the OP Mainnet ecosystem.
The Bridge UI has been unpaused. Deposits and Withdrawals can now be executed on OP Mainnet from: https://app.optimism.io/bridge/
- UpdateJune 06, 2023 at 9:52 PMIn progressJune 06, 2023 at 9:52 PM
The public endpoint is back live: https://mainnet.optimism.io/
This should unblock most wallets to begin working again.
- UpdateJune 06, 2023 at 7:38 PMIn progressJune 06, 2023 at 7:38 PM
For Archival node operators: While we recommend using the
legacy-getharchival datadir we upload once it's ready, given the size of the archival data disk and the time it'll take for it to become available (up to 24 hours), we recommend preserving your legacy archival nodes and routing archival traffic to them if possible. To check if your legacy archival node is able to serve traffic, please do the following:
Verify that the head block number is105235062
Verify that the head block hash is0x21a168dfa5e727926063a28ba16fd5ee84c814e847c81a699c7a0ea551e4ca50
If both of these criteria are true, then your legacy archival node is on the same block as the archival datadir and can be used to serve user traffic.You can set
--rollup.historicalrpc <l2geth-rpc>onop-gethand it will forward requests for historical data it can't handle tol2geth. - UpdateJune 06, 2023 at 7:10 PMIn progressJune 06, 2023 at 7:10 PM
For node operators:
We have received reports that the full node Bedrock datadir is taking a long time to download. Since it is available on GCS, you can use parallel downloader tools likearia2to make it faster. This method has seen download speeds in excess of 100MB/s. Some recommendations:- Run the download on a high-network box in the cloud.
- Use aria2 or the gcloud storage cp command to download the file in parallel.
- UpdateJune 06, 2023 at 6:50 PMIn progressJune 06, 2023 at 6:50 PM
The Bedrock sequencer has started up.
Images, and data directories for node operators can be found here. Key external OP Mainnet infrastructure is starting to come back online. View the OP Mainnet Infra Tracker to follow along.
- UpdateJune 06, 2023 at 6:26 PMIn progressJune 06, 2023 at 6:26 PM
L1 smart contract upgrades have been successfully implemented. The portal has been unpaused, and deposits/withdrawals have been re-enabled.
- UpdateJune 06, 2023 at 6:20 PMIn progressJune 06, 2023 at 6:20 PM
The migrated (Bedrock) datadir required to spin up
op-gethhas been successfully hosted and is now ready for distribution.Full Node Operators are now fully unblocked to spin up.
Archival Node Operators will need to wait for the archival datadir, which may take an additional 5-10 hours.The datadir can be downloaded from the link found in the OP Mainnet docs: https://community.optimism.io/docs/useful-tools/networks/#links
- UpdateJune 06, 2023 at 5:44 PMIn progressJune 06, 2023 at 5:44 PM
Node Operators: Docker images are now available in our community hub docs.
op-nodeversion: https://github.com/ethereum-optimism/optimism/releases/tag/op-node%2Fv1.1.0 (Docker Images are in the release notes)op-gethversion: https://github.com/ethereum-optimism/op-geth/releases/tag/v1.101105.2 (Docker Images are in the release notes)
Please await the upload of the Migrated Datadir before you can fully spin up your OP Mainnet Bedrock nodes. - UpdateJune 06, 2023 at 4:36 PMIn progressJune 06, 2023 at 4:36 PM
We are now starting to execute the L2 portion of the migration. This is the longest step in the migration process, and could take up to 1.5 hours. During this phase of the migration, ETH balances and withdrawals will be migrated. Successful completion of this stage will conclude the L2 portion of the migration, and we will perform all related checks before moving on to subsequent phases.
- In progressJune 06, 2023 at 4:00 PMIn progressJune 06, 2023 at 4:00 PM
The migration has started. Legacy network deposits and withdrawals have been paused.
Transaction acceptance on the legacy sequencer has been halted.
- PlannedJune 06, 2023 at 4:00 PMPlannedJune 06, 2023 at 4:00 PM
The OP Mainnet Bedrock Upgrade is expected to run for 2-4 hours. Find all of the details here:
https://optimism.io/bedrock-upgrade
Watch for updates on this status page for:- Images for node operators
- Datadirs for both full node and archival node operators
- Notifications on when deposits, withdrawals, and the sequencer are back live.
May 2023