Dtyjlui

I have an Ubuntu system which will mostly be used as a headless appliance, and needs to recover from abrupt power loss gracefully and start operating normally.

I understand overlayfs is the way to go, but I still need to log and store data, including system logs such as crashes and journaling.

If I understand correctly, I can't simply set up a directory under root and make that read/write - I need a separate partition with a separate file system for the logging and data collection so if it does become corrupted I can still boot into root, detect the situation, and fsck the data partition.

1. Is my understanding correct? Are there other options I should consider?


2. Ubuntu is already installed and working (pre-built systems), how can I accomplish this without a full reinstallation?


3. Is looking up journal and crash logging sufficient, or are there other system logging processes that are also generally preserved in these types of situations?


4. Is there a guide for this? The guides I've seen cover overlayfs generally, or implementing it without logging specifically, but I haven't seen a guide or script that does overlayfs with logging with good detail, leaving me to search out each part separately.

These systems use desktop ubuntu, and the graphical interface will be used occasionally, however I'd like to be able to automate configuration and installation eventually, so while graphical tools are ok, command line configuration is preferred.

1. Using an Ubuntu LiveUSB, run gparted to shrink the existing ext4 partition and create a new partition, here called "data-rw" in both name and label.




2. Reboot.

That partition will automatically be mounted on boot, probably at /media/user/data_rw. Now that we have a second drive, let's move the log files to it. Nearly all the logging is done to /var/log, so let's make a new directory on the second drive and bind it there.

3. 
   

   Make log directory on new partition, sync existing files with users and permissions, and bind in fstab so it's automatic on boot:

   
   
   

   sudo mkdir /media/user/data_rw/log
sudo rsync -a --include '*/' --exclude '*' /var/log/ /media/user/data_rw/log/
sudo nano /etc/fstab

   
   

Add these lines to the end (alter the device name as needed):

/dev/sda3 /media/user/data_rw ext4 rw,nosuid,nodev,relatime,data=ordered 0 2  

/media/user/data_rw/log /var/log none rw,bind 0 0  

4. We'd like to force a disk check for the writeable partition, with automated error correction. In theory, this should work, but it's not clear that it is in my case so consider looking into this further. Edit /etc/default/grub and find the line with GRUB_CMDLINE_LINUX_DEFAULT, then add fsck.mode=force fsck.repair=yes to that line. Do sudo update-grub to write the changes to the bootloader




5. Install overlayfs, sudo apt-get install overlayroot and edit /etc/overlayroot.conf changing the overlayroot option to overlayroot="tmpfs:swap=1,recurse=0" This will cause overlayroot to cover only the root drive, and nothing else, leaving our data_rw drive alone.

Now reboot, and you should find that adding a file to ~/ will disappear on next boot but adding a file to /media/user/data_rw or /var/log will persist across boots.

Using journalctl you should find that logs persist across boots as well, and if your system crashes you should find crash logs in /var/log even after a reboot.

If you need to change the protected filesystem, reboot and when grub comes up press e to edit the grub startup script. On the linux line add overlayroot=disabled to the end. This will work until reboot. You can add a new grub bootloader option if this is frequently needed.

Since writes are never made to the read-only root partition, it should never show errors and should boot back up just fine after abrupt power removal. The data_rw partition, however, may have unfinalized writes to it. The fsck should repair that, but in the worst case scenario, the drive won't mount on boot, the bind won't occur, and the logs will be written to the overlay, and not be preserved. However, the system will still keep running and if this is a critical need you can detect this condition using scripts or software, and take action.

You can either write directly to /media/user/data_rw or you can use the systemd logging service to capture your logs.

Much of this answer is possible due to the following resources:

https://spin.atomicobject.com/2015/03/10/protecting-ubuntu-root-filesystem/

https://askubuntu.com/a/763645/12100

How to make mount --bind permanent?

Automatically force fsck -fy when encountering "UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY."