Surviving a RADOS outage

Hi ~okeanos users,

Recently ~okeanos service faced some disruptions, as we informed you on a previous blog post. Let's shed some light on what caused them.

The ~okeanos storage backend (Archipelago) is backed by Ceph, an open-source distributed software-defined storage solution. On Friday, September 9, at 9 p.m., Object Storage Daemons (OSDs) of the RADOS cluster became unstable resulting in the cluster erroneously marking OSDs as down. This incident led to I/O freeze and cluster malfunctioning. In order to ensure data integrity, we had to resort to special handling and tooling and managed to make the cluster fully functional again on Tuesday, September 13.

If you are interested in more technical details, you may find an analysis of the incident, described by our NOC team, in this blog post.

We would like to thank you for your support on handling this unexpected outage. The lessons learned from this incident will be used to better tune and monitor the service, which will help towards our commitment for high levels of service availability.

the ~okeanos team

Scheduled maintenance/upgrade operation

We'd like to inform you about scheduled maintenance/upgrade for ~okeanos service between 09:00 EET and 11:00 EET on Wednesday, October 19.

Your VMs and Pithos+ files should remain unaffected. Maintenance is expected to cause minor disruptions to the ~okeanos service. Astakos, Cyclades and Pithos+ Web UIs and their APIs will be unavailable for a short time during the upgrade.

No loss of connectivity is expected for the VMs.

Thanks for your understanding,
the ~okeanos team

Upgrading ~okeanos, take a sneak peek!

Hello ~okeanos users,

We are happy to announce that on Wednesday, September 21 2016 between 09:00 and 11:00 EET we will be doing a major upgrade to the ~okeanos service, with a new version of the Synnefo software.

During the upgrade Astakos, Cyclades and Pithos+ Web UIs and their APIs will be unavailable. Your VMs and Pithos+ files should remain unaffected.

After the upgrade, you will be able to enjoy more robust services with several improvements and additions across ~okeanos, as well as some pithos+ performance enhancements.

Last but not least, don't forget to check out our new unified policy for handling resources & quotas that will apply from now on!

the ~okeanos team


Dear ~okeanos users,

From Friday September 9, at 9 p.m., we have observed some functionality problems that affect a number of Virtual Machines as well as Pithos+.
We are working hard in order to resolve the issue as soon as possible.

We apologise for the inconvenience,
the ~okeanos team