very nomagic

Multiple DNS resolvers in OS X

Mon, 06 Jul 2015 23:41:00 +0000

Lately I’ve been trying out consul and I love some of its core concepts. One of them is service discovery, which is provided through either an HTTP API or a DNS interface.

The DNS interface works well, but it’s hard to try out on your own laptop. Sure, you can dig @<consul_server_ip> -p 8600 but anything else turns out to be difficult.

My first try was to use curl’s --dns-servers option. The documentation reads:

--dns-servers <ip-address,ip-address>
       Set  the  list  of  DNS  servers to be used instead of the system
       default.  The list of IP addresses should be separated with commas. Port
       numbers may also optionally be given as :<port-number> after each IP
       address.

       This option requires that libcurl was built with a resolver backend that
       supports this operation. The c-ares backend is the only such one.
       (Added in 7.33.0)

Simple enough, right? (#1)

Unfortunately, at the time of writing OS X’s curl isn’t compiled with c-ares resolver. So let’s compile our own curl to bake in the support needed:

$ brew install curl --with-c-ares

Simple enough, right? (#2)

Unfortunately, curl times out using --dns-servers <consul_server_ip>:8600. A quick tcpdump shows requests going out to DNS standard port 53, so something’s up.

I had a quick look at curl’s source code, following into c-ares source code and found this gem:

The port option is currently ignored by c-ares internals
and the standard port is always used.

Aw, blërg!

As usual with The Internets, someone else already had a solution for me, so I just had to brew edit curl, add the following patch and brew reinstall curl --with-c-ares:

diff --git a/Library/Formula/c-ares.rb b/Library/Formula/c-ares.rb
index 960521c..366fa16 100644
--- a/Library/Formula/c-ares.rb
+++ b/Library/Formula/c-ares.rb
@@ -6,11 +6,9 @@ class CAres < Formula
   url 'http://c-ares.haxx.se/download/c-ares-1.10.0.tar.gz'
   sha1 'e44e6575d5af99cb3a38461486e1ee8b49810eb5'

-  bottle do
-    cellar :any
-    sha1 "aa711a345bac4780f2e7737c212c1fb5f7862de8" => :yosemite
-    sha1 "c6851c662552524fa92e341869a23ea72dbc4375" => :mavericks
-    sha1 "27494a19ac612daedeb55356e911328771f94b19" => :mountain_lion
+  patch do
+    url "https://github.com/bagder/c-ares/pull/19.patch"
+    sha256 "99ef83d196fa550f2c46335abd63d825ba8650d686d7713e774579385d7c8998"
   end

   def install

Note: remember to rollback that change after compiling, so that you don’t get merge conflicts next time this formula is updated!

(sidenote: how cool is GitHub?? Just adding a .diff or .patch gives you exactly what you want!)

So, after all this work, curl should work with consul’s DNS interface. But in practice, you just enabled curl to use alternate DNS servers, not your whole system. Wouldn’t it be great to use your browser to access a web server that consul knows about?

This is usually where /etc/resolv.conf comes into play. Being OS X though, things aren’t as simple; as far as I could tell, half of the standard *nix CLI tools have this notice on their man pages:

Mac OS X NOTICE
       The host command does not use the host name and address resolution or
       the DNS query routing mechanisms used by other processes running on Mac
       OS X.  The results of name or address queries printed by host may differ
       from those found by other processes that use the Mac OS X native name
       and address resolution mechanisms.  The results of DNS queries may also
       differ from queries that use the Mac OS X DNS routing library.

Well, that sucks. But it got me curious as to what exactly is this “Mac OS X native name and address resolution mechanisms”.

man 5 resolver is quite interesting in that regard. It suggests the possibility of different DNS configurations for specific domains, so I tried it by creating this file:

$ cat /etc/resolver/dc1.consul
domain dc1.consul
port 8600
nameserver <consul_server_1>.8600
nameserver <consul_server_2>.8600
nameserver <consul_server_3>.8600

Some of these configs are redundant, namely defining explicitly a domain when the file name should be enough and defining the port in every nameserver when the default port was changed before. This was made to make clear what should be happening here.

Anyway, I also found out a nice little command that lets you check your current DNS configurations (which resolvers you have defined, in which order are they configured, which domains do they resolve): scutil --dns

Assuming everything went ok, you should see your custom resolver there.

Another way you can test this now is to run:

$ dscacheutil -q host -a name webserver.service.dc1.consul
name: webserver.service.dc1.consul
ip_address: <webserver_ip_address>

In other news, you can now use Consul domains directly in your browsers! Sadly, none of the major browsers support RFC 2782 SRV lookups, so you’ll still have to add the port if your webserver is running on a non-standard port.

Say 'No' to sshpass

Tue, 16 Jun 2015 00:00:00 +0000

December 3, 2019: updated post to work on macOS 10.14

I recently started fiddling with Ansible. I’m in a position where I see what all the fuss is about, but its quirks still nag me; one of which is the requirement to use sshpass for when you don’t have your SSH keys in place.

I really don’t like sshpass - mostly because of security concerns - but the end goal of SSH automation is still worth pursuing, I think.

So, this exercise started by being stubborn in believing you could mostly do what sshpass does with plain vanilla ssh! In fact, I would argue that this might be A Better Way™, but you’re free to disagree. =)

I want to store my passwords in an OS X keychain and have them read straight to ssh, so first we’ll create a secure keychain for this purpose:

# create a new keychain
$ security create-keychain -P test.keychain
# have it lock on sleep or after 5min
$ security set-keychain-settings -lu -t 300 test.keychain

OS X command line tools for system management seem to be an after-thought, as I accidentally messed up my keychain search index while researching for this post and could only recover by using the GUI Keychain Access app (that might be a story for a later post).

In any case, securely adding a new password to the keychain doesn’t seem possible through the CLI as the tool mandates inserting the password as a command line argument. The hack I came up with for not having the password stored in the clear on the shell history file was to write it elsewhere (anywhere you can type text), copying to the clipboard (I know, I know -_- ) and having the shell read from the clipboard:

$ security add-generic-password -a <username> -s ldap -w $(pbpaste) \
           test.keychain

The other way you could go about it would be to use the Keychain Access app and create the password item there.

Having done that, we now have to coerce ssh to use the password in the keychain. This was a lot harder than I previously thought, as ssh tries very hard to force you to type the password interactively, for security reasons. Having said that, here is what the man page states:

SSH_ASKPASS           If ssh needs a passphrase, it will read the passphrase
                      from the current terminal if it was run from a terminal.
                      If ssh does not have a terminal associated with it but
                      DISPLAY and SSH_ASKPASS are set, it will execute the
                      program specified by SSH_ASKPASS and open an X11 window
                      to read the passphrase. (...)

This means we can use SSH_ASKPASS environment variable to pipe a password into ssh, as long as:

ssh does not have a terminal associated with it;
there is a DISPLAY environment variable set.

This was made so that X11 password prompts could be used with ssh. As we’re on OS X, this is kind of irrelevant.

Oh well.

The hard part here is tricking ssh to run without an associated terminal and, after several failed attempts, I had to resort to The Internets. Luckily, I’m not the first person to have had this idea so sample code was readily available. In fact, the linked post has almost everything you need to do ssh automation; it just needed a little OS X love to work. I’ve set up a github repository with the code so that this is easily reproduceable. You just need to clone the repo and follow the instructions to install notty.

The fun part is that SSH_ASKPASS just needs to point to an executable that outputs the password to stdout. Of course that would be lame and terribly insecure, so we just need to write a script that, with your permission, grabs your password from the keychain.

Place the following in ~/bin/askpass:

#!/usr/bin/env bash
/usr/bin/security find-generic-password -a <username> -s ldap -w \
                  test.keychain

and make it executable:

$ chmod u+x ~/bin/askpass

The parameters you use here are the same you used when creating your generic-password item earlier.

We now have the foundations to passwordless (sort of) ssh and can try it with a server with password authentication:

$ DISPLAY=:99 SSH_ASKPASS="~/bin/askpass" notty ssh -q <server> uptime

If everything went well, a keychain prompt should appear asking for the keychain password. After that you’ll feel the sweet bliss of realizing you just had to type a password so that you don’t need to type another. ^_^’

Most of the pieces are now in place to replace sshpass (that was the point of the exercise, remember?). As ansible is hard-coded to require sshpass for password-based authentication and to disallow password authentication when not using -k, we need to fool it into using our SSH_ASKPASS setup.

Our cool sshpass replacement (place it /usr/local/bin/sshpass):

#!/usr/bin/env bash

export DISPLAY=:99
export SSH_ASKPASS="$HOME/bin/askpass"

[[ $1 == -d* ]] && shift
notty $@

ansible uses the -d flag to tell sshpass which file descriptor to read the password from. As we don’t care about that, we just ignore it and use the rest of the generated command directly.

$ cat /tmp/a
server0[1:3]

$ ansible -i /tmp/a all -m ping -k
SSH password:
server01 | success >> {
    "changed": false,
    "ping": "pong"
}

server02 | success >> {
    "changed": false,
    "ping": "pong"
}

server03 | success >> {
    "changed": false,
    "ping": "pong"
}

Success!

This, of course, is not ideal as ansible prompts you for a password anyway and then our replacement sshpass disregards that entirely. Fixing this requires patching lib/ansible/plugins/connections/ssh.py, which is a lot uglier than to type gibberish on the ansible prompt.

And that concludes our exercise for now :)

Thanks to @kintoandar for all the help with Ansible, and for pushing me to write this post!