One of our production web application runs on a dedicated node in DO (Digital Ocean) cloud server. This web application has following services running.

  • Elixir Phoenix API server
  • React.js client server
  • Redis caching server
  • Bots (Slack)

Few days back, Digital Ocean triggered an urgent maintenance for Meltdown and Spectre Mitigations resulting into a reboot of our PROD server. DO did alert us in advance by sending an email notification with schedules. But seems we missed these schedules accidentally. Thus, our PROD web application was suddenly down after this auto reboot / restart by DO forced maintenance. We realised that we forgot to install auto startup scripts in our PROD server on DO to automatically restart our API server, Client server, Redis caching server and Bots. We immediately installed "system" auto startup scripts to manage all these web application services and tested against few forced reboots. This resulted in improved uptime of our PROD app in case of reboots or restarts of entire DO server. Following are the details and snippets for these auto startup scripts using systemd

What is systemd?
Systemd provides a standard process for controlling what programs run when a Linux system boots up. Here is a link to nice article about what is Systemd and why should we care about it? This article sheds a lot of light into systemd and its pros and cons.

Anyways, we won't debate about systemd in this article but we will make use of it for our auto startup scripts.

Here are various scripts

React.js Client App
File: /etc/systemd/system/app_client.service

[Unit]
Description=AppName Client
After=network.target
Requires=redis.service

[Service]
Restart=on-failure
EnvironmentFile=/root/.env
WorkingDirectory=/path_to/app_folder/app_client
ExecStart=/path_to/app_folder/app_client/pm2 serve build/ 3001 --spa

[Install]
WantedBy=multi-user.target

Phoenix Server App
File: /etc/systemd/system/app_server.service

[Unit]
Description=AppName Server
After=network.target
Requires=redis.service

[Service]
Restart=on-failure
Environment=HOME=/root
EnvironmentFile=/root/.env
WorkingDirectory=/path_to/app_folder/server_app
ExecStart=/usr/local/bin/mix phoenix.server

[Install]
WantedBy=multi-user.target

Redis
File: /etc/systemd/system/redis.service

Description=Redis Datastore Server
After=network.target

[Service]
Restart=on-failure
PIDFile=/var/run/redis_6379.pid
WorkingDirectory=/root/redis-stable/
ExecStart=/usr/local/bin/redis-server /root/redis-stable/redis.conf
ExecReload=/bin/kill -USR2 $MAINPID

[Install]
WantedBy=multi-user.target

Thats it. All these scripts once stored on the above mentioned File paths will ensure that our services (i.e. API server, Client, Redis, Others) starts automatically on a system reboot.

Also, we can now use journalctl to attach to these services and check the console logs or service logs or service output.
Example: journalctl -u app_client -f

Its now pretty easy to even restart these services using service app_server restart

Other useful commands we can now use with journalctl

journalctl -u app_server.service --since today
journalctl -u app_client.service --since 09:00 --until "1 hour ago"
journalctl -u redis.service --since "2016-11-10 12:00" --until "2016-11-10 13:00"