symlink | dunelab | remote console

A large chunk of Dunelab is hosted at my workplace (which is generous enough to let me tinker with things in free time), scattered around several buildings, and therefore somewhat difficult to physically access outside work hours. The rest is at home and therefore out-of-reach when I am at work.

This has led to me building all kinds of redundancy into the systems so that they could be (mostly) repaired remotely, without having to drag my arse to whichever building they're in: Redundant network interfaces when possible, remote power-cycle, serial console to troubleshoot kernel issues or reset networking.

Ember, Wind, and Dust are "proper" servers so they have HP iLO providing power control, full KVM redirection (if I have an HTML5-capable browser) and a virtual serial port through IPMI SOL (ipmiconsole or ipmitool sol).
Myth₂ happens to have a CPU with Intel AMT, which also provides remote power-cycle and serial console using amtterm, although not for the firmware stage. (In theory AMT also supports KVM console redirection, but accessing it through MeshCommander is a bit of a pain.)
Mirage, Horizon and Endless are just regular desktops without any such features. They do however have a physical RS232 serial port – which doesn't let me see firmware messages, but does work with GRUB (and of course Linux) which is enough to repair network misconfiguration and even troubleshoot kernel issues.
In addition, I have added the EFI Shell to Horizon's GRUB menu and made it load the EFI serial port drivers so in a pinch I could use that to tweak EFI variables and such.

Currently both Horizon and Mirage have their ttyS0 attached to the nearby Cisco routers' AUX ports while Endless is connected to FrostGW through a pl2303 USB serial adapter. For power-cycling I have a bunch of TP-Link's Tapo P110 "smart power socket" devices whose protocol is just open enough that I've built my own shell tools and CGI scripts around it.
The Cisco routers, vice versa, have their own serial console ports connected to 2xserial PCI cards on Horizon and Mirage. (Used to be USB serial adapters connected to Dune, but the knockoff pl2303 "Cisco-to-USB" cables I had were inconvenient and tickled bugs in the Linux driver – which I suspect may be fixed as of 7.2, but at the time I'd needed to pass them to a dedicated NetBSD virtual machine, which was a pain.)
As much as possible, ~/bin/rcons acts as a wrapper for the various means of accessing a console, e.g. rcons mirage will telnet to port 2005 of a Cisco (where line 5 is the AUX port) and rcons myth will invoke amtterm.