TL;DR

I went from a hard to reproduce, hand configured server for hosting a Rails app to a nicely automated setup using Ansible. The repo is here and you can scope the README for easy to reproduce steps.

The long version AKA Story time

So you have this old Rails app. You know the one. It has been dutifully doing the right thing behind the scenes since 2014. Doesn’t really need any maintenance, but your friend’s business.. needs it around. Doesn’t depend on it, really, but like. It can’t go away.

You put it on a Linode which you configured by hand and it’s been working just fine.

Fast forward, now it’s 2020. You’ve really been meaning to make an update or two. Maybe go from Rails 3 to 6? Where does the time go? How did I deploy this again?

It’s time to get serious. A quick DDG for “ansible rails” turns up a contender.

It works, until it doesn’t

Major credit is due to this repo for getting me started. It doesn’t not work out of the box, which was weird, but I suppose you get what you pay for. This fork was my savior in a time of great need though. This person fixed some of the errors that I was feeling frozen about. I tend to get stuck in this mode where if code is in a repo on github somewhere, then it must be right and I must be misunderstanding it. Nothing could be further from the truth! Thank you github.com/noahgibbs for the great reminder that there is a fork button for a reason, and that I needed to be the arbiter of my own destiny.

Configuration

I started by opening up app-vars.yml and getting to work. App name, github repo, etc. It’s all pretty straightforward. Something to keep in mind is that SSL/Certbot setup requires that you already have your DNS correctly pointed at whatever new machine you’re trying to deploy to. The certbot role will make real certificates. I tested using a subdomain at first, and then when it all worked I moved the real one over. Both had real certificates generated which is fine, but keep in mind Let’s Encrypt will rate limit you at some point. I never had a problem with it.

You can tell me. I’ll put it in the vault.

Provisioning the machine went fine. I had to update a dependency to a newer version to make Ubuntu 20.04 happy, but that was no problem. I had a machine with a deployer user that had ssh and sudo configured, good to go.

I got a little tripped up by the fact that both Ansible and Rails each have a mechanism for encrypted secrets. Rails calls them credentials, Ansible, a vault.

Since I was updating an old Rails app I hadn’t encountered the new (circa 5.2) Rails credentials functionality and I wasn’t sure what to put where. There is now a master.key which is used to encrypt/decrypt. This gets created the first time you run rails credentials:edit. This file has a key in it that Rails uses to encrypt/decrypt sensitive information for you. It stores the encrypted information in a file called credentials.yml.enc. This credentials file will be in config/ and potentially even config/<environment>. You can then check in the encrypted file which keeps it all in once place. Anyone with a key can view/edit. You DO NOT check the key into your repo, but share it directly with team using something like 1Password to keep it safe. Since we’re trying to automate our deployment, we need to get the contents of this file onto our server so when Rails boots up in production mode it can read the secret information and go on its merry way.

This is where Ansible’s vault comes in. It’s essentially the same mechanism, a key and an encrypted file you can share publicly. Using the ansible command we create a new vault e.g. ansible-vault create group_vars/all/vault.yml and we’re off to the races. Inside of this file we put our production postgres password, and our rails master key. I was confused at first and put the secret_key_base as the master key. Whoops! Provide the string here that is found in the file master.key so that Ansible can securely pass it along when running Rails in production.

You can use the ansible-vault command by specifying a key file in ansible.cfg or by passing the flag --ask-vault-pass. If you use a key file DO NOT check it into source control! Same as the Rails one. Secrets abound.

Two for the price of two

Why do we need these two mechanisms that are essentially doing the same thing and making your life more complicated? Great question! It’s really just a necessary evil to gain the automated delight that is having Ansible around.

Rails needs a way to have secrets. So far so good. Ansible needs to know the secrets, but since Ansible isn’t a trustworthy human we also have to protect ourselves from Ansible knowing the secrets.

I think it’s worth it, but it definitely confused me at first. Up to you!

Deploy!

Our deploy playbook uses Ansistrano to coordinate deploys. This is a pair of Ansible playbooks that mimic the functionality of Capistrano and do a good job of it. The name feels a little.. not perfect, but it works!

I didn’t have any idea how it would find these playbooks because I hadn’t installed them, but I was impressed it could figure it out. I was wrong to be impressed because it had no idea where they were. Naturally. The simple fix was to use Ansible Galaxy to install them myself. I didn’t know Ansible Galaxy existed when I started this project so I actually got a bit stuck here!

$ ansible-galaxy install ansistrano.deploy ansistrano.rollback

Super easy, time to deploy!

Database.yml

I followed the instructions this far and started getting database access problems. My code was there, my database config was there, what’s going on?

I realized that the playbook was copying our production database config to the server, but not linking it to Rails. To fix this I added a symlink step to the after_cleanup.yml that Ansistrano runs after deploying.

- name: symlink database.yml
  file:
    src: "/config/database.yml"
    dest: "/config/database.yml"
    owner: ""
    group: ""
    state: link
    force: yes

Voila! The database is off and running.

So close

The final thing that tripped me up was that I had a vendor folder in my Rails root holding (you guessed it) vendored JavaScript. This problem probably doesn’t exist if you are using webpacker, but the deploy playbook by default was creating a shared vendor folder for gems, and then symlinking right over the top of my already existing vendor folder. An easy fix was to just call the folder something else. I am a creative force so naturally I chose the name: gemvendor.

Success

After all of this my app worked great and I was able to push up changes and re-run the deploy script just like I wanted. I could also destroy this box on Linode, grab a new one, and have a new server running my app in about 20 minutes. The initial build out takes a while because we build Ruby from scratch AND install some gems that have to build some C. Follow up deploys are fast.

I have updated the repo for my app with step by step instructions for using the updated Ansible playbooks that are contained in there. At some point I will perhaps split them into their own repo but for now I only use them in this one app so that is where they live.

Happy deploying!

Alternative title: Hosting your own Jekyll blog with Ansible, Nginx, and Ubuntu 20.04 LTS

Not as fun though.

It seems fitting that my first post would be a meta post about how it all works. We must talk about our creations, yes? If a tree falls in a forest and no one is around to hear it, does it make a sound?

Let us observe as much as we can, I say! If for no other reason than to give the servers running our simulation a run for their money. Or credits, or whatever.

TL;DR

This is a static website generated with Jekyll, hosted on a Linode, running Ubuntu 20.04 LTS. Provisioned and deployed with Ansible. Toss in an SSL cert from Let’s Encrypt. Even a dash of GoAccess to watch all of those delicious visitors in real time? Sold!

Why? (In which I justify my existence - meaning is created! not an absolute!)

To fend off the inevitable heat death of the universe with toil. With work. For a purpose beyond ourselves!

Nah, it’s just fun. I enjoy control and knowing how things work. Heroku and Netlify are cool options. There are many others. If you don’t care about doing this yourself I super don’t blame you. If you do, let’s dive in!

Why! (Actually answering the question)

Linode: I have been using Linode for years. They aren’t Amazon, Google, or any other major nightmare of a company. I consider that a Good Thing on the whole. Though I am a citizen on planet earth, we’re all doing our best. I use them to host a few web things, including this site. A 5$ VPS hosting a static site can get pretty well hammered and be fine. If I ever write something that so many people care about that this thing falls down I will consider it 100% a victory only. I challenge you, dear readers, to make me eat these words.

Jekyll: Hugo can generate sites insanely fast and I genuinely enjoy Go. That said.. I don’t know how you can get the templating system so wrong. I am fully willing to admit that I am just not smart enough to use it, but.. I fought Hugo for hours, and still didn’t get what I wanted. I switched to Jekyll and everything did what I expected almost instantly. Old dog new tricks? I dunno, man. Do what makes you happy … if you are lucky enough to have any idea what that is.

NGINX: NGINX runs the internet. It runs my little piece of the internet as well. What else are you going to use.. Apache? Something new that people haven’t hammered on as much? Something that wasn’t written in C by a russian? Good luck with that.

Ubuntu: Straightforward and stable. I’m young enough that I caught the Ubuntu train before the Debian one got to me. I used Arch for tinkering around, and thought it was very cool when I took three days to build out a Gentoo install on an old laptop. Servers need to just work. This seemed like the most boring choice. But like.. sexy boring.

Let’s Encrypt: A non-profit that’s only goal is to make the web a safer place by providing free SSL certs. Sign me up. Have I missed any good reasons I should skeptical here? Let’s me just hop on twi.. NOPE. Happiness is fleeting and this one, I think I’ll keep this one for myself.

GoAccess: We’ve got to stop giving all the data to Google only to have them treat us badly in return. Do no evil. Psh. I’m trying to de-google my life lately and it’s going.. ok. It turns out having a little self respect is harder than it looks. This beautiful (and free!) software parses the logs you’ve already got to give you pretty results in the terminal OR it will generate a nice (live updating!) HTML page. Also written by one guy in C. Patterns emerge? Seriously though. Needed a websocket server.. wrote a websocket server. May we all find our passions.

How?

Potentially this is the part that anyone might actually find helpful. Most of this work is based on the work already done here. There are some thing straight up broken there, so when I got stuck, I found inspiration in a fork here. Standing on the shoulders of giants and all that!

Preprovision

First and foremost! Make sure you check out app-vars.yml and set the variables there correctly. They are all set for my site (obviously) but should easily work for you.

Pay special attention to nginx_https_enabled. It defaults to true because I’m not messing around. You can turn it off if your domain isn’t quite ready to point to this machine, or like.. you hate privacy and seeing good prevail over evil.

Now you want to get a server from Linode. Or anywhere. It has to have your SSH key for the root user (Hey there Eagle Eye. Don’t get mad. We’ll get that disabled! Thanks for caring and know that I see you.). Also an IP address. Put that IP address in your inventory file(s) and hold onto your butts. You could probably get away with password authentication somehow but then you’d be back to fighting for the wrong side. Don’t give in.

These steps are carried out through this playbook.

Run ansible-playbook -i inventories/preprovision.ini preprovision.yml.

This phase gets some basics going. Installs a user that isn’t root, get an ssh key in place for that user, etc. The reason we use a whole inventory file for this stage is that it sets the user to root. The production inventory uses the deployer user (or whatever you called it in app-vars!). You don’t need to have two separate inventory files for this, but it’s nice to not have to think about providing the correct user when running the playbooks.

Provision

The meat and potatoes of our situation is handled here. You can reference this playbook here.

Run ansible-playbook -i inventories/production.ini provision.yml.

• Harden SSH (Phew!)
• Install and configure NGINX
• Install and run certbot to get Let’s Encrypt certs
• Setup a cron job for certbot to refresh the certs
• Setup UFW to only allow ports 80, 443, and 22 (80 to helpfully redirect people to 443, 22 for ssh)
• Install goaccess (beautiful data!)

Deploy

This playbook is dead simple because deploying a static site is so easy.

All it does is make sure there is a folder in the right place, and then rsyncs our static site to it.

You have to remember to build the site yourself even. Maybe I should build that in. Alas.

JEKYLL_ENV=production bundle exec jekyll build

ansible-playbook -i inventories/production.ini deploy.yml

If all went well you should be able to surf on over to your domain and see your beautiful posts. Like this one.

If you’re into it, Check your google page speed! Do an SSL labs test!

Bask in the glorious results!

GoAccess Bonus

If you’re still here you get a fun bonus.

I was really excited to find out about GoAccess and while I’m no expert, I wanted to pass along a fun little setup to get the real time HTML going through an SSH tunnel.

I wasn’t super keen to share the logs of my blog publicly, also opening up another port in the firewall doesn’t overwhelm me with excitement.

What I do is fire up an SSH tunnel like so!

ssh -L 8000:localhost:8000 -L 7890:localhost:7890 deployer@192.81.130.222

For anyone not as familiar with SSH, this essentially means that when you access ports 8000, and 7980 on your personal machine, they will (through your secure SSH connection) access those ports on your server. So useful!

So, then on the remote machine I use tmux to run:

sudo goaccess /var/log/nginx/access.log -o report.html --log-format=COMBINED --real-time-html

This fires up goaccess pointed at our logs. Then ctrl-b c to make a new tmux tab and fire up a tiny webserver.

python3 -m http.server

This starts a webserver at port 8000 to share the html file that goaccess generates. You can use anything you want for this but python3 is there and it’s easy.

Go ahead and navigate to http://localhost:8000/report.html on your local machine and the realtime html pops up! Beauty.

Any web server will work, but trying to host it through SSL and opening ports gave me fits. The SSH tunnel really makes it nice.

Happy hosting to you all and thank you for reading. 👨🏼‍💻🎉