As many/most of you have probably noticed, the site had an extended period of downtime, starting at about 12:00pm EDT yesterday and running until about 12:45pm EDT today.
At this point, we're still determining exactly what went wrong. As I get updates, I'll post them here.
--kurt
The problem with political jokes is that they get elected
To clear up any confusion, this downtime was limited to forums.gentoo.org and rsync5.us.gentoo.org. It did not, in any way, affect www.gentoo.org or any of the source mirrors.
--kurt
The problem with political jokes is that they get elected
My inital analysis shows that it was a kernel bug. At 14:36:31, July 7th, a kernel error was logged. Shortly after the inital, another was logged, and ~ 20 seconds later mysql restarted and logged an error. The kernel error was logged several more times, and mysqld restarted once more in an attempt to stay alive.
Klieber notified me, and at that time it had appeared as if only mysqld had died. I tried shutting services, to no avail. I then attempted to reboot the server, and it never came back.
Klieber swung by the data center earlier this morning and says that server was locked up at the console. I imagine that the kernel "oops" struck again in the shutdown sequence. Hard reboot restored services.
In light of the crash, their appears to be no data loss, and the server is back up running fine. The kernel will get replaced soon. :)