May 16 2021
MySQL / PostgreSQL on iSCSI Fail to Start at Boot
You are hosting Mysql or PostgreSQL data directory on iSCSI disks but the service fails to start at server’s boot. The service does not find the directory. However, you can start the service manually if you log on the server once SSH is available.
These are logs for Mariadb but they would be similar for Mysql:
mariadbd[795]: 0 [Note] /usr/sbin/mysqld (mysqld 10.5.9-MariaDB-1:10.5.9+maria~buster-log) starting as process 795 ...
mariadbd[795]: 0 [Warning] Can't create test file /opt/db/data/database_server.lower-test
mariadbd[795]: #007/usr/sbin/mysqld: Cannot change dir to '/var/lib/mysql/data/' (Errcode: 2 "No such file or directory")
mariadbd[795]: 0 [ERROR] Aborting
systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: mariadb.service: Failed with result 'exit-code'.
systemd[1]: Failed to start MariaDB 10.5.9 database server.
You may also get logs similar to these for PostgreSQL. I stored PostgreSQL on a XFS partition, hosted on LVM for a more flexible disk space management, such as adding space while the filesystem is mounted. This is probably the main reason why iSCSI disks are popular.
systemd: mounting /var/lib/pgsql
starting PostgreSQL database server
sd 2:0:0:0: [sdb] attached SCSI disk
xfs (dm-4): Mounting V4 Filesystem
postgresql-check-db-dir: "/var/lib/pgsql/data" is missing or empty
postgresql.service: control process exited, code=exited status=1
Failed to start PostgreSQL database server.
The database starting manually after the boot indicates there’s most likely a problem with the services boot order. Databases should start after iscsi disks are made available. You can solve this issue adding “After=remote-fs.target” in the service systemd file such as /usr/lib/systemd/system/postgresql-9.5.service for PosgreSQL for instance. This is a way to manage service precedence and dependencies.
Note you may lose these changes next time the package is upgraded. Systemd lets you create an extra file where you can define your own settings. It will never modify it since this is your own file.
Just create /etc/systemd/system/mariadb.service.d/override.conf as follow for Mariadb, it is the same process for Mysql or PostgreSQL:
[Service]
Environment="UMASK_DIR=0750"
[Unit]
After=remote-fs.target
In this file, I also changed the default data directory permissions so users in Mysql group can walk into it.
Run systemctl daemon-reload so it takes the new settings into account and reboot the server. The database service should now start.