Note to self. pfSense full + SSD = Not Good.
Had a bit of a kerfuffle over the long weekend. Was working remotely on a firewall trying to get NAT through VPN working properly, the firewall didn’t appear to work as it should so I decided to reboot it to make sure all routes are clear. Turns out it was a bad idea. The firewall failed to come back up which resulted in a 100km/drive to the data center to investigate the issue.
When I got there, I quickly plugged in a monitor to the 1U server acting as the firewall and was greeted by a nice “Boot Disk Failure” error on the screen. Good thing I brought a spare firewall, one of my modified Watchguard x750e’s. A quick swap and few minutes later the new firewall was running a restore script on the Watchguard (thanks to a config backup).
Back at home a little digging showed that non-embedded version of pfSense do not play nicely with SSD’s. Especially if RRD is installed. Turns out that RRD writes out new traffic images to physical disk every minute. Multiply that by 8 months and that’s a lot of writes to an SSD that doesn’t support write leveling.
Lesson learned. I bought a couple more Watchguard x750e’s from eBay. One to serve as a replacement at another data center, which is also running installed pfSense on a PC/SSD combo and another Firebox as a warm spare.