On April 27th, between 9:03 PM and 9:26 PM ET, 1Password experienced a brief service outage. This was not a security incident, and customer data was not affected in any way.
After completing a planned maintenance, our service received an unexpected spike in sync requests from client devices to the servers. During the outage, users erroneously received a message indicating that their Secret Key or password had changed.
Our mission is to help people safeguard their most important information. 1Password is designed to protect your information, with local copies of vault data always available on your devices – even without a connection to the 1Password service or the internet itself. As a result, your passwords and other vault items remain safe and sound.
We’re deeply sorry for any inconvenience this outage may have caused and appreciate your patience during our investigation. Service has been fully restored, and we can now share further details about what happened and how we’re working to avoid similar situations in the future.
What happened?
On April 27th, our scheduled maintenance involved migration work for several of our backend databases. After the migration work was complete, we received an unexpected spike in sync requests from devices to our servers and instead of correctly responding to those requests, we responded with a sign-in rejection.
Our US servers returned an error code that was interpreted on our client applications incorrectly. The client applications displayed an incorrect message stating: “Your Secret Key or password was recently changed. Enter your new account details to continue." In reality, neither the Secret Key or password had changed. This affected user sessions in our US environment between 9:03 PM and 9:26 PM ET.
What did we do?
We closely monitored the service health for the duration of this event, and by 9:26 PM ET on April 27th, the traffic in our US environment had returned to normal with no additional failed sign-in attempts. By April 28th, there were no additional erroneous messages, and we were able to confirm that the fixes were working as expected.
What happens next?
We care deeply about our customers, their data, and their experience, so we take any service disruption like this very seriously.
As part of our plan to avoid similar incidents in the future, our immediate next steps are to spend more time analyzing the data we collected and ensure we fully understand the underlying cause of this incident. We will use this analysis to refine our migration process and error handling and ensure that we properly plan for these scenarios in the future.
We take the integrity of your data and the stability of our systems very seriously and will continue to work hard every day to earn the trust you’ve placed in us.
Pedro Canahuati
Chief Technology Officer
Tweet about this post