forums

Stephen Crawley

The Ceph cluster (QLD-Ceph which stores the NeCTAR Volume Storage
data) has sustained a hardware failure. Due to insufficient storage
capacity to cope with a node failure, QLD-Ceph is now running in
degraded mode and is likely to go into read-only mode very soon. When this happens:

Instances that are booted from a volume in in the QRIScloud AZ will lock up.

Instances that have a volume attached and mounted will experience hard file system errors.

If you
have critical data on your volume that you have not backed up, you
should do this NOW.

If you no longer need your QRIScloud NeCTAR volume, could they please contact QRIScloud Support (07 3346 4202) urgently.

For more information and updates, please refer to the QRIScloud announcement: https://qriscloud.zendesk.com/hc/en-us/articles/207711663

UPDATE - The outage is now happening. All affected NeCTAR instances are currently paused and will be un-paused when it is is safe to do so. Instance owners have been contacted individually.

FINAL UPDATE - The outage is now over. Please refer to the link above for the full history.