One of our production web application runs on a dedicated node in DO (Digital Ocean) cloud server. This web application has following services running.
- Elixir Phoenix API server
- React.js client server
- Redis caching server
- Bots (Slack)
Few days back, Digital Ocean triggered an urgent maintenance for Meltdown and Spectre Mitigations resulting into a reboot of our PROD server. DO did alert us in advance by sending an email notification with schedules. But seems we missed these schedules accidentally. Thus, our PROD web application was suddenly down after this auto reboot / restart by DO forced maintenance. We realised that we forgot to install auto startup scripts in our PROD server on DO to automatically restart our API server, Client server, Redis caching server and Bots. We immediately installed "system" auto startup scripts to manage all these web application services and tested against few forced reboots. This resulted in improved uptime of our PROD app in case of reboots or restarts of entire DO server. Following are the details and snippets for these auto startup scripts using systemd
Systemd provides a standard process for controlling what programs run when a Linux system boots up. Here is a link to nice article about what is Systemd and why should we care about it? This article sheds a lot of light into systemd and its pros and cons.
Anyways, we won't debate about systemd in this article but we will make use of it for our auto startup scripts.
Here are various scripts
React.js Client App
[Unit] Description=AppName Client After=network.target Requires=redis.service [Service] Restart=on-failure EnvironmentFile=/root/.env WorkingDirectory=/path_to/app_folder/app_client ExecStart=/root/.yarn/bin/yarn prod-server [Install] WantedBy=multi-user.target
Phoenix Server App
[Unit] Description=AppName Server After=network.target Requires=redis.service [Service] Restart=on-failure Environment=HOME=/root EnvironmentFile=/root/.env WorkingDirectory=/path_to/app_folder/server_app ExecStart=/usr/local/bin/mix phoenix.server [Install] WantedBy=multi-user.target
Description=Redis Datastore Server After=network.target [Service] Restart=on-failure PIDFile=/var/run/redis_6379.pid WorkingDirectory=/root/redis-stable/ ExecStart=/usr/local/bin/redis-server /root/redis-stable/redis.conf ExecReload=/bin/kill -USR2 $MAINPID [Install] WantedBy=multi-user.target
Thats it, all these scripts once stored on the above mention
File paths will ensure that our services (i.e. API server, Client, Redis, Others) start automatically on a system reboot.
Also, we can now use
journalctl to attach to these services and check the console logs or service logs or service output.
journalctl -u app_client -f
Its now pretty easy to even restart these services using
service app_server restart
Other useful commands we can now use with
journalctl -u app_server.service --since today journalctl -u app_client.service --since 09:00 --until "1 hour ago" journalctl -u redis.service --since "2016-11-10 12:00" --until "2016-11-10 13:00"
Subscribe to Engineering At Kiprosh
Get the latest posts delivered right to your inbox