New post about 28-29 oct tilserv downtime

This commit is contained in:
Arya Kiran 2022-10-29 09:01:20 +05:30
parent 1830cf0c02
commit d7d656c953
No known key found for this signature in database
GPG Key ID: 3A6EA2D0EE314EEF
2 changed files with 23 additions and 0 deletions

View File

@ -0,0 +1,21 @@
---
title: Explanation on ~vern's downtime on 28-29 oct
date: Mon, 28 Oct 2022
author: ~vern team
---
Hello ~vern members,
We recently had downtime of the tilserv caused due to cleaning which accidentally disconnected the power cable
I started having issues at 19:30 IST (14:00 UTC) when my ssh connection to the server freezed.
Since cobra wasn't at home, we had to wait until 11:50 PM IST (6:20 UTC) when she could check on the server and confirm our assumption.
But, due to unrelated issues, the pubnixvm and website were down. This took until 7 AM IST (1:30 AM UTC) to fix since I was not online at the time.
I am planning on making a script that informs us when the tilserv goes on battery power to help prevent-this.
We hope this does not happen again.
~aryak

View File

@ -16,4 +16,6 @@ I estimate that the migration will take up around 6 hours in total (including ba
We apologize for this downtime but if we do not do it now it would be a pain to do it in the future due to more storage use etc.
UPDATE: Since we had [another downtime this week](/blog/29oct-tilserv-downtime), the zfs migration has been rescheduled to next sunday (6th november).
~aryak