blog.oddbit.com

Things I Made: Fighting words generator

Sat, 11 Oct 2025 00:00:00 +0000

If you’ve ever seen the 1960’s live-action Batman TV show, you’ll know that one of the defining characteristics of the show was its use of comic-book style words overlaying the action scenes, a technique pioneered in comic books by Roy Crane:

It was Crane who pioneered the use of onomatopoeic sound effects in comics, adding “bam,” “pow” and “wham” to what had previously been an almost entirely visual vocabulary. Crane had fun with this, tossing in an occasional “ker-splash” or “lickety-wop” along with what would become the more standard effects. Words as well as images became vehicles for carrying along his increasingly fast-paced storylines.

Things I Made: Slow terminal emulation

Fri, 10 Oct 2025 00:00:00 +0000

Ah, the good old days: when computers were chunky, the Internet was a dream of the future, and you could make a cup of coffee while waiting for a screenful of text to display. If you miss that as much as I do, let me introduce you to Slow, a low bit rate emulator that lets you travel back in time to those simpler days.

Slow lets you run commands with a reduced output character rate. For example, we can ask for the date at speeds of 50, 75, and 110 bps:

Applying custom configuration to Nginx Gateway Fabric

Fri, 17 Nov 2023 00:00:00 +0000

In this post, we take a look at how to apply custom Nginx configuration directives when you’re using the NGINX Gateway Fabric.

What’s the NGINX Gateway Fabric?

The NGINX Gateway Fabric is an implementation of the Kubernetes Gateway API.

What’s the Gateway API?

The Gateway API is an evolution of the Ingress API; it aims to provide a flexible mechanism for managing north/south network traffic (that is, traffic entering or exiting your Kubernetes cluster), with additional work to support east/west traffic (traffic between pods in your cluster).

Processing deeply nested JSON with jq streams

Thu, 27 Jul 2023 00:00:00 +0000

I recently found myself wanting to perform a few transformations on a large OpenAPI schema. In particular, I wanted to take the schema available from the /openapi/v2 endpoint of a Kubernetes server and minimize it by (a) extracting a subset of the definitions and (b) removing all the description attributes.

The first task is relatively easy, since everything of interest exists at the same level in the schema. If I want one or more specific definitions, I can simply ask for those by key. For example, if I want the definition of a DeploymentConfig object, I can run:

Managing containers with Pytest fixtures

Sat, 15 Jul 2023 00:00:00 +0000

A software fixture “sets up a system for the software testing process by initializing it, thereby satisfying any preconditions the system may have”. They allow us to perform setup and teardown tasks, provide state or set up services required for our tests, and perform other initialization tasks. In this article, we’re going to explore how to use fixtures in Pytest to create and tear down containers as part of a test run.

NAT between identical networks using VRF

Sun, 19 Feb 2023 00:00:00 +0000

Last week, Oskar Stenberg asked on Unix & Linux if it were possible to configure connectivity between two networks, both using the same address range, without involving network namespaces. That is, given this high level view of the network…

…can we set things up so that hosts on the “inner” network can communicate with hosts on the “outer” network using the range 192.168.3.0/24, and similarly for communication in the other direction?

Simple error handling in C

Fri, 17 Feb 2023 00:00:00 +0000

Overview

I was recently working with someone else’s C source and I wanted to add some basic error checking without mucking up the code with a bunch of if statements and calls to perror. I ended up implementing a simple must function that checks the return value of an expression, and exits with an error if the return value is less than 0. You use it like this:

must(fd = open("textfile.txt", O_RDONLY));

Or:

A review of the Garmin Fenix 6(x)

Wed, 15 Feb 2023 00:00:00 +0000

I’ve been using a Garmin Fenix 6x for a couple of weeks and thought it might be interesting to put together a short review.

Is it really a smartwatch?

I think it’s a misnomer to call the Fenix a “smartwatch”. I would call it a highly capable fitness tracker. That’s not a knock on the product; I really like it so far, but pretty much everything it does is centered around either fitness tracking or navigation. If you browse around the “Connect IQ” store, mostly you’ll find (a) watch faces, (b) fitness apps, and (c) navigation apps. It’s not able to control your phone (for the most part; there are some apps available that offer remote camera control and some other limited features); you can’t check your email on it, or send text messages, and you’ll never find a watch version of any major smartphone app.

Packet, packet, who's got the packet?

Tue, 14 Feb 2023 00:00:00 +0000

In this question, August Vrubel has some C code that sets up a tun interface and then injects a packet, but the packet seemed to disappear into the ether. In this post, I’d like to take a slightly extended look at my answer because I think it’s a great opportunity for learning a bit more about performing network diagnostics.

The original code looked like this:

c original sendpacket.c



#include <arpa/inet.h>
#include <fcntl.h>
#include <linux/if.h>
#include <linux/if_tun.h>
#include <netinet/in.h>
#include <poll.h>
#include <stdio.h>
#include <string.h>
#include <sys/ioctl.h>
#include <sys/socket.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

static int tunAlloc(void) {
 int fd;
 struct ifreq ifr = {.ifr_name = "tun0", .ifr_flags = IFF_TUN | IFF_NO_PI};

 fd = open("/dev/net/tun", O_RDWR);
 ioctl(fd, TUNSETIFF, (void *)&ifr);
 ioctl(fd, TUNSETOWNER, geteuid());
 return fd;
}

// this is a test
static void bringInterfaceUp(void) {
 int sock;
 struct sockaddr_in addr = {.sin_family = AF_INET};
 struct ifreq ifr = {.ifr_name = "tun0"};

 inet_aton("172.30.0.1", &addr.sin_addr);
 memcpy(&ifr.ifr_addr, &addr, sizeof(struct sockaddr));

 sock = socket(AF_INET, SOCK_DGRAM, 0);
 ioctl(sock, SIOCSIFADDR, &ifr);
 ioctl(sock, SIOCGIFFLAGS, &ifr);
 ifr.ifr_flags |= IFF_UP | IFF_RUNNING;
 ioctl(sock, SIOCSIFFLAGS, &ifr);
 close(sock);
}

static void emitPacket(int tap_fd) {
 unsigned char packet[] = {
 0x45, 0x00, 0x00, 0x3c, 0xd8, 0x6f, 0x40, 0x00, 0x3f, 0x06, 0x08, 0x91,
 172, 30, 0, 1, 192, 168, 255, 8, 0xa2, 0x9a, 0x27, 0x11,
 0x80, 0x0b, 0x63, 0x79, 0x00, 0x00, 0x00, 0x00, 0xa0, 0x02, 0xfa, 0xf0,
 0x89, 0xd8, 0x00, 0x00, 0x02, 0x04, 0x05, 0xb4, 0x04, 0x02, 0x08, 0x0a,
 0x5b, 0x76, 0x5f, 0xd4, 0x00, 0x00, 0x00, 0x00, 0x01, 0x03, 0x03, 0x07};

 write(tap_fd, packet, sizeof(packet));
}

int main() {
 int tap_fd;

 tap_fd = tunAlloc();
 bringInterfaceUp();
 emitPacket(tap_fd);
 close(tap_fd);

 return 0;
}

A problem with the original code is that it creates the interface, sends the packet, and tears down the interface with no delays, making it very difficult to inspect the interface configuration, perform packet captures, or otherwise figure out what’s going on.

Setting up an IPv6 VLAN

Wed, 16 Nov 2022 00:00:00 +0000

My internet service provider (FIOS) doesn’t yet (sad face) offer IPv6 capable service, so I’ve set up an IPv6 tunnel using the Hurricane Electric tunnel broker. I want to provide IPv6 connectivity to multiple systems in my house, but not to all systems in my house ¹. In order to meet those requirements, I’m going to set up the tunnel on the router, and then expose connectivity over an IPv6-only VLAN. In this post, we’ll walk through the steps necessary to set that up.

Using KeyOxide

Sun, 13 Nov 2022 00:00:00 +0000

In today’s post, we look at KeyOxide, a service that allows you to cryptographically assert ownership of online resources using your GPG key. Some aspects of the service are less than obvious; in response to some questions I saw on Mastodon I though I would put together a short guide to making use of the service.

We’re going to look at the following high-level tasks:

Delete GitHub workflow runs using the gh cli

Thu, 22 Sep 2022 00:00:00 +0000

Hello, future me. This is for you next time you want to do this.

When setting up the CI for a project I will sometimes end up with a tremendous clutter of workflow runs. Sometimes they have embarrassing mistakes. Who wants to show that to people? I was trying to figure out how to bulk delete workflow runs from the CLI, and I came up with something that works:

gh run list --json databaseId -q '.[].databaseId' |
 xargs -IID gh api \
 "repos/$(gh repo view --json nameWithOwner -q .nameWithOwner)/actions/runs/ID" \
 -X DELETE

This will delete all (well, up to 20, or whatever you set in --limit) your workflow runs. You can add flags to gh run list to filter runs by workflow or by triggering user.

Kubernetes, connection timeouts, and the importance of labels

Sat, 10 Sep 2022 00:00:00 +0000

We are working with an application that produces resource utilization reports for clients of our OpenShift-based cloud environments. The developers working with the application have been reporting mysterious issues concerning connection timeouts between the application and the database (a MariaDB instance). For a long time we had only high-level verbal descriptions of the problem (“I’m seeing a lot of connection timeouts!”) and a variety of unsubstantiated theories (from multiple sources) about the cause. Absent a solid reproducer of the behavior in question, we looked at other aspects of our infrastructure:

Directing different ports to different containers with Traefik

Mon, 20 Jun 2022 00:00:00 +0000

This post is mostly for myself: I find the Traefik documentation hard to navigate, so having figured this out in response to a question on Stack Overflow, I’m putting it here to help it stick in my head.

The question asks essentially how to perform port-based routing of requests to containers, so that a request for http://example.com goes to one container while a request for http://example.com:9090 goes to a different container.

Creating entrypoints

A default Traefik configuration will already have a listener on port 80, but if we want to accept connections on port 9090 we need to create a new listener: what Traefik calls an entrypoint. We do this using the --entrypoints.<name>.address option. For example, --entrypoints.ep1.address=80 creates an entrypoint named ep1 on port 80, while --entrypoints.ep2.address=9090 creates an entrypoint named ep2 on port 9090. Those names are important because we’ll use them for mapping containers to the appropriate listener later on.

Udev rules for CH340 serial devices

Sun, 13 Feb 2022 00:00:00 +0000

I like to fiddle with Micropython, particularly on the Wemos D1 Mini, because these are such a neat form factor. Unfortunately, they have a cheap CH340 serial adapter on board, which means that from the perspective of Linux these devices are all functionally identical – there’s no way to identify one device from another. This by itself would be a manageable problem, except that the device names assigned to these devices aren’t constant: depending on the order in which they get plugged in (and the order in which they are detected at boot), a device might be /dev/ttyUSB0 one day and /dev/ttyUSB2 another day.

A pair of userscripts for cleaning up Stack Exchange sites

Sun, 05 Sep 2021 00:00:00 +0000

I’ve been a regular visitor to Stack Overflow and other Stack Exchange sites over the years, and while I’ve mostly enjoyed the experience, I’ve been frustrated by the lack of control I have over what questions I see. I’m not really interested in looking at questions that have already been closed, or that have a negative score, but there’s no native facility for filtering questions like this.

I finally spent the time learning just enough JavaScript ~~~to hurt myself~~~ to put together a pair of scripts that let me present the questions that way I want:

Kubernetes External Secrets

Fri, 03 Sep 2021 00:00:00 +0000

At $JOB we maintain the configuration for our OpenShift clusters in a public git repository. Changes in the git repository are applied automatically using ArgoCD and Kustomize. This works great, but the public nature of the repository means we need to find a secure solution for managing secrets (such as passwords and other credentials necessary for authenticating to external services). In particular, we need a solution that permits our public repository to be the source of truth for our cluster configuration, without compromising our credentials.

Connecting OpenShift to an External Ceph Cluster

Mon, 23 Aug 2021 00:00:00 +0000

Red Hat’s OpenShift Data Foundation (formerly “OpenShift Container Storage”, or “OCS”) allows you to either (a) automatically set up a Ceph cluster as an application running on your OpenShift cluster, or (b) connect your OpenShift cluster to an externally managed Ceph cluster. While setting up Ceph as an OpenShift application is a relatively polished experienced, connecting to an external cluster still has some rough edges.

NB I am not a Ceph expert. If you read this and think I’ve made a mistake with respect to permissions or anything else, please feel free to leave a comment and I will update the article as necessary. In particular, I think it may be possible to further restrict the mgr permissions shown in this article and I’m interested in feedback on that topic.

Creating a VXLAN overlay network with Open vSwitch

Sat, 17 Apr 2021 00:00:00 +0000

In this post, we’ll walk through the process of getting virtual machines on two different hosts to communicate over an overlay network created using the support for VXLAN in Open vSwitch (or OVS).

The test environment

For this post, I’ll be working with two systems:

node0.ovs.virt at address 192.168.122.107
node1.ovs.virt at address 192.168.122.174

These hosts are running CentOS 8, although once we get past the package installs the instructions will be similar for other distributions.

Getting started with KSOPS

Tue, 09 Mar 2021 00:00:00 +0000

Kustomize is a tool for assembling Kubernetes manifests from a collection of files. We’re making extensive use of Kustomize in the operate-first project. In order to keep secrets stored in our configuration repositories, we’re using the KSOPS plugin, which enables Kustomize to use sops to encrypt/files using GPG.

In this post, I’d like to walk through the steps necessary to get everything up and running.

Set up GPG

We encrypt files using GPG, so the first step is making sure that you have a GPG keypair and that your public key is published where other people can find it.

Tools for writing about Git

Sat, 27 Feb 2021 00:00:00 +0000

I sometimes find myself writing articles or documentation about git, so I put together a couple of terrible hacks for generating reproducible histories and pretty graphs of those histories.

git synth

The git synth command reads a YAML description of a repository and executes the necessary commands to reproduce that history. It allows you set the name and email address of the author and committer as well as static date, so you every time you generate the repository you can identical commit ids.

File reorganization

Wed, 24 Feb 2021 00:00:00 +0000

This is just a note that I’ve substantially changed how the post sources are organized. I’ve tried to ensure that I preserve all the existing links, but if you spot something missing please feel free to leave a comment on this post.

Editing a commit message without git rebase

Thu, 18 Feb 2021 00:00:00 +0000

While working on a pull request I will make liberal use of git rebase to clean up a series of commits: squashing typos, re-ordering changes for logical clarity, and so forth. But there are some times when all I want to do is change a commit message somewhere down the stack, and I was wondering if I had any options for doing that without reaching for git rebase.

It turns out the answer is “yes”, as long as you have a linear history.

Object storage with OpenShift Container Storage

Wed, 10 Feb 2021 00:00:00 +0000

OpenShift Container Storage (OCS) from Red Hat deploys Ceph in your OpenShift cluster (or allows you to integrate with an external Ceph cluster). In addition to the file- and block- based volume services provided by Ceph, OCS includes two S3-api compatible object storage implementations.

The first option is the Ceph Object Gateway (radosgw), Ceph’s native object storage interface. The second option called the “Multicloud Object Gateway”, which is in fact a piece of software named Noobaa, a storage abstraction layer that was acquired by Red Hat in 2018. In this article I’d like to demonstrate how to take advantage of these storage options.

Remediating poor PyPi performance with DevPi

Mon, 08 Feb 2021 00:00:00 +0000

Performance of the primary PyPi service has been so bad lately that it’s become very disruptive. Tasks that used to take a few seconds will now churn along for 15-20 minutes or longer before completing, which is incredibly frustrating.

I first went looking to see if there was a PyPi mirror infrastructure, like we see with CPAN for Perl or CTAN for Tex (and similarly for most Linux distributions). There is apparently no such beast,

symtool: a tool for interacting with your SYM-1

Sat, 06 Feb 2021 00:00:00 +0000

The SYM-1 is a 6502-based single-board computer produced by Synertek Systems Corp in the mid 1970’s. I’ve had one floating around in a box for many, many years, and after a recent foray into the world of 6502 assembly language programming I decided to pull it out, dust it off, and see if it still works.

The board I have has a whopping 8KB of memory, and in addition to the standard SUPERMON monitor it has the expansion ROMs for the Synertek BASIC interpreter (yet another Microsoft BASIC) and RAE (the “Resident Assembler Editor”). One interacts with the board either through the onboard hex keypad and six-digit display, or via a serial connection at 4800bps (or lower).

To sleep or not to sleep?

Fri, 18 Dec 2020 00:00:00 +0000

Let’s say you have a couple of sensors attached to an ESP8266 running MicroPython. You’d like to sample them at different frequencies (say, one every 60 seconds and one every five minutes), and you’d like to do it as efficiently as possible in terms of power consumption. What are your options?

If we don’t care about power efficiency, the simplest solution is probably a loop like this:

import machine

lastrun_1 = 0
lastrun_2 = 0

while True:
 now = time.time()

 if (lastrun_1 == 0) or (now - lastrun_1 >= 60):
 read_sensor_1()
 lastrun_1 = now
 if (lastrun_2 == 0) or (now - lastrun_2 >= 300):
 read_sensor_2()
 lastrun_2 = now

 machine.idle()

If we were only reading a single sensor (or multiple sensors at the same interval), we could drop the loop and juse use the ESP8266’s deep sleep mode (assuming we have wired things properly):

Animating a map of Covid in the Northeast US

Sun, 13 Dec 2020 00:00:00 +0000

I recently put together a short animation showing the spread of Covid throughout the Northeast United States:

I thought it might be interesting to walk through the process I used to create the video. The steps described in this article aren’t exactly what I used (I was dealing with data in a PostGIS database, and in the interests of simplicity I wanted instructions that can be accomplished with just QGIS), but they end up in the same place.

A note about running gpgv

Mon, 05 Oct 2020 00:00:00 +0000

I found the following error from gpgv to be a little opaque:

gpgv: unknown type of key resource 'trustedkeys.kbx'
gpgv: keyblock resource '/home/lars/.gnupg/trustedkeys.kbx': General error
gpgv: Can't check signature: No public key

It turns out that’s gpg-speak for “your trustedkeys.kbx keyring doesn’t exist”. That took longer to figure out than I care to admit. To get a key from your regular public keyring into your trusted keyring, you can run something like the following:

Installing metallb on OpenShift with Kustomize

Sun, 27 Sep 2020 00:00:00 +0000

Out of the box, OpenShift (4.x) on bare metal doesn’t come with any integrated load balancer support (when installed in a cloud environment, OpenShift typically makes use of the load balancing features available from the cloud provider). Fortunately, there are third party solutions available that are designed to work in bare metal environments. MetalLB is a popular choice, but requires some minor fiddling to get it to run properly on OpenShift.

Vortex Core Keyboard Review

Sat, 26 Sep 2020 00:00:00 +0000

I’ve had my eye on the Vortex Core keyboard for a few months now, and this past week I finally broke down and bought one (with Cherry MX Brown switches). The Vortex Core is a 40% keyboard, which means it consists primarily of letter keys, a few lonely bits of punctuation, and several modifier keys to activate different layers on the keyboard.

Physical impressions

It’s a really cute keyboard. I’m a big fan of MX brown switches, and this keyboard is really a joy to type on, at least when you’re working primarily with the alpha keys. I’m still figuring out where some of the punctuation is, and with a few exceptions I haven’t yet spent time trying to remap things into more convenient positions.

Building multi-architecture images with GitHub Actions

Fri, 25 Sep 2020 00:00:00 +0000

At work we have a cluster of IBM Power 9 systems running OpenShift. The problem with this environment is that nobody runs Power 9 on their desktop, and Docker Hub only offers automatic build support for the x86 architecture. This means there’s no convenient options for building Power 9 Docker images…or so I thought.

It turns out that Docker provides GitHub actions that make the process of producing multi-architecture images quite simple.

OpenShift and CNV: MAC address management in CNV 2.4

Mon, 10 Aug 2020 00:00:00 +0000

This is part of a series of posts about my experience working with OpenShift and CNV. In this post, I’ll look at how the recently released CNV 2.4 resolves some issues in managing virtual machines that are attached directly to local layer 2 networks

In an earlier post, I discussed some issues around the management of virtual machine MAC addresses in CNV 2.3: in particular, that virtual machines are assigned a random MAC address not just at creation time but every time they boot. CNV 2.4 (re-)introduces MAC address pools to alleviate these issues. The high level description reads:

OpenShift and CNV: Exposing virtualized services

Thu, 30 Jul 2020 01:00:00 +0000

This is the second in a series of posts about my experience working with OpenShift and CNV. In this post, I’ll be taking a look at how to expose services on a virtual machine once you’ve git it up and running.

TL;DR

Networking seems to be a weak area for CNV right now. Out of the box, your options for exposing a service on a virtual machine on a public address at a well known port are slim.

OpenShift and CNV: Installer network requirements

Thu, 30 Jul 2020 00:00:00 +0000

This is the first in a series of posts about my experience working with OpenShift and CNV (“Container Native Virtualization”, a technology that allows you to use OpenShift to manage virtualized workloads in addition to the containerized workloads for which OpenShift is known). In this post, I’ll be taking a look at the installation experience, and in particular at how restrictions in our local environment interacted with the network requirements of the installer.

You can't get an N95 mask: Now what?

Tue, 28 Jul 2020 00:00:00 +0000

[This is a guest post by my partner Alexandra van Geel.]

Disclaimer: I am not an expert, just a private individual summarizing available information. Please correct me if I’ve gotten something wrong.

TL;DR

I suggest: (a) the Vogmask valveless mask or (b) the O2 Canada Curve Respirator. If (b), make sure you cut some an extra filter into two small circles and put the material in the respirator to cover the exhale valves.

Grove Beginner Kit for Arduino (part 2): First look

Sun, 07 Jun 2020 00:00:00 +0000

The folks at Seeed Studio were kind enough to send me a Grove Beginner Kit for Arduino for review. That’s a mouthful of a name for a compact little kit!

The Grove Beginner Kit for Arduino (henceforth “the Kit”, because ain’t nobody got time to type that out more than a few times in a single article) is about 8.5 x 5 x 1 inches. Closed, you could fit two of them on a piece of 8.5x11 paper with a little room leftover.

Grove Beginner Kit for Arduino (part 1)

Wed, 15 Apr 2020 00:00:00 +0000

The folks at Seeed Studio have just released the Grove Beginner Kit for Arduino, and they asked if I would be willing to take a look at it in exchange for a free kit. At first glance it reminds me of the Radio Shack (remember when they were cool?) electronics kit I had when I was a kid – but somewhat more advanced. I’m excited to take a closer look, but given shipping these days means it’s probably a month away at least.

Some thoughts on Mechanical Keyboards

Wed, 15 Apr 2020 00:00:00 +0000

Since we’re all stuck in the house and working from home these days, I’ve had to make some changes to my home office. One change in particular was requested by my wife, who now shares our rather small home office space with me: after a week or so of calls with me clattering away on my old Das Keyboard 3 Professional in the background, she asked if I could get something that was maybe a little bit quieter.

I see you have the machine that goes ping...

Fri, 20 Mar 2020 00:00:00 +0000

We’re all looking for ways to keep ourselves occupied these days, and for me that means leaping at the chance to turn a small problem into a slightly ridiculous electronics project. For reasons that I won’t go into here I wanted to generate an alert when a certain WiFi BSSID becomes visible. A simple solution to this problem would have been a few lines of shell script to send me an email…but this article isn’t about simple solutions!

A passwordless serial console for your Raspberry Pi

Mon, 24 Feb 2020 00:00:00 +0000

legendre on #raspbian asked:

How can i config rasp lite to open a shell on the serial uart on boot? Params are 1200-8-N-1 Dont want login running, just straight to sh

In this article, we’ll walk through one way of implementing this configuration.

Activate the serial port

Raspberry Pi OS automatically starts a getty on the serial port if one is available. You should see an agetty process associated with your serial port when you run ps -ef. For example:

Configuring Open vSwitch with nmcli

Sat, 15 Feb 2020 00:00:00 +0000

I recently acquired a managed switch for my home office in order to segment a few devices off onto their own isolated vlan. As part of this, I want to expose these vlans on my desktop using Open vSwitch (OVS), and I wanted to implement the configuration using NetworkManager rather than either relying on the legacy /etc/sysconfig/network-scripts scripts or rolling my own set of services. These are my notes in case I ever have to do this again.

How long is a cold spell in Boston?

Thu, 23 Jan 2020 00:00:00 +0000

We’ve had some wacky weather recently. In the space of a week, the temperature went from a high of about 75°F to a low around 15°F. This got me to thinking about what constitutes “normal” weather here in the Boston area, and in particular, how common it is to have a string of consecutive days in which the high temperature stays below freezing. While this was an interesting question in itself, it also seemed like a great opportunity to learn a little about Pandas, the Python data analysis framework.

Snarl: A tool for literate blogging

Wed, 15 Jan 2020 00:00:00 +0000

Literate programming is a programming paradigm introduced by Donald Knuth in which a program is combined with its documentation to form a single document. Tools are then used to extract the documentation for viewing or typesetting or to extract the program code so it can be compiled and/or run. While I have never been very enthusiastic about literate programming as a development methodology, I was recently inspired to explore these ideas as they relate to the sort of technical writing I do for this blog.

OVN and DHCP: A minimal example

Thu, 19 Dec 2019 00:00:00 +0000

Introduction

A long time ago, I wrote an article all about OpenStack Neutron (which at that time was called Quantum). That served as an excellent reference for a number of years, but if you’ve deployed a recent version of OpenStack you may have noticed that the network architecture looks completely different. The network namespaces previously used to implement routers and dhcp servers are gone (along with iptables rules and other features), and have been replaced by OVN (“Open Virtual Network”). What is OVN? How does it work? In this article, I’d like to explore a minimal OVN installation to help answer these questions.

TM-V71A and Linux, part 1: Programming mode

Thu, 03 Oct 2019 00:00:00 +0000

I recently acquired my Technician amateur radio license, and like many folks my first radio purchase was a Baofeng UV-5R. Due to its low cost, this is a very popular radio, and there is excellent open source software available for programming it in the form of the CHIRP project. After futzing around with the UV-5R for a while, I wanted to get something a little nicer for use at home, so I purchased a Kenwood TM-V71A. CHIRP claims to have support for this radio as well, but it turns out it’s not very good: it uses a “live” connection so every time you edit a channel it tries to update the radio. This result in a slow and flaky UI, especially when trying to make bulk changes (like relocating a block of channels). I ended up using Kenwood’s official MCP-2A software running on a Windows guest on my system, which works but isn’t ideal. I decided to learn more about how the radio interacts with the computer to see if I could improve the situation.

Avoid rebase hell: squashing without rebasing

Mon, 17 Jun 2019 00:00:00 +0000

You’re working on a pull request. You’ve been working on a pull request for a while, and due to lack of sleep or inebriation you’ve been merging changes into your feature branch rather than rebasing. You now have a pull request that looks like this (I’ve marked merge commits with the text [merge]):

7e181479 Adds methods for widget sales
0487162 [merge] Merge remote-tracking branch 'origin/master' into my_feature
76ee81c [merge] Merge branch 'my_feature' of https://github.com/my_user_name/widgets into my_feature
981aab4 Adds api for the widget service.
b048836 Includes fixes suggested by reviewer.
3dd0c22 adds changes requested by reviewer
5891db2 [merge] fixing merge conflicts
2e226e4 fixes suggestions given by the reviewer
da1e85c Adds gadget related API spec
c555cc1 Adds doodad related API spec
e5beb3e Adds api for update and delete of widgets
c43bade Adds api for creating widgets
deaa962 Adds all get methods for listing widgets
9de79ab Adds api for showing a widget and simple data model
8288ab1 Adds api framework for widget service

You know that’s a mess, so you try to fix it by running git rebase -i master and squashing everything together…and you find yourself stuck in an endless maze of merge conflicts. There has to be a better way!

Git Etiquette: Commit messages and pull requests

Fri, 14 Jun 2019 00:00:00 +0000

Always work on a branch (never commit on master)

When working with an upstream codebase, always make your changes on a feature branch rather than your local master branch. This will make it easier to keep your local master branch current with respect to upstream, and can help avoid situations in which you accidentally overwrite your local changes or introduce unnecessary merge commits into your history.

Rebase instead of merge

If you need to incorporate changes from the upstream master branch in the feature branch on which you are currently doing, bring in those changes using git rebase rather than git merge. This process will generally start by ensuring that your local copy of the upstream master is current:

Running Keystone with Docker Compose

Fri, 07 Jun 2019 00:00:00 +0000

In this article, we will look at what is necessary to run OpenStack’s Keystone service (and the requisite database server) in containers using Docker Compose.

Running MariaDB

The standard mariadb docker image can be configured via a number of environment variables. It also benefits from persistent volume storage, since in most situations you don’t want to lose your data when you remove a container. A simple docker command line for starting MariaDB might look something like:

A DIY CPAP Battery Box

Tue, 14 May 2019 00:00:00 +0000

A year or so ago I was diagnosed with sleep apnea, and since them I’ve been sleeping with a CPAP. This past weekend, I joined my daughter on a scout camping trip to a campground without readily accessible electricity. This would be the first time I found myself in this situation, and as the date approached, I realized I was going to have to build or buy some sort of battery solution for my CPAP. I opted for the “build” option because it seemed like a fun project.

Unpacking a Python regular expression

Tue, 07 May 2019 10:00:00 +0000

I recently answered a question from Harsha Nalore on StackOverflow that involved using Ansible to extract the output of a command sent to a BigIP device of some sort. My solution – which I claim to be functional, but probably not optimal – involved writing an Ansible filter module to parse the output. That filter made use of a complex-looking regular expression. Harsha asked for some details on that regular expression works, and the existing StackOverflow answer didn’t really seem the write place for that: so, here we are.

New comment system

Tue, 07 May 2019 09:00:00 +0000

As long as I’m switching site generators, it seems like a good idea to refresh the comment system as well. I’ve been using Disqus for a while, since when I started it was one of the only games in town. There are now alternatives of different sorts, and one in particular caught my eye: Utterances uses GitHub issues for storing comments, which seems like a fantastic idea.

That means that comments will finally be stored in the same place as the blog content, which I think is a happy state of affairs.

New static site generator

Mon, 06 May 2019 00:00:00 +0000

I’ve switched my static site generator from Pelican to Hugo. I’ve tried to ensure that all the old links continue to work correctly, but if you notice anything missing or otherwise not working as intended, please let me know by opening an issue. Thanks!

Adding support for privilege escalation to Ansible's docker connection driver

Fri, 26 Apr 2019 00:00:00 +0000

Update 2019-05-09 Pull request #55816 has merged, so you can now use sudo with the docker connection driver even when sudo is configured to require a password.

I often use Docker to test out Ansible playbooks. While normally that works great, I recently ran into an unexpected problem with privilege escalation. Given a simple playbook like this:

---
- hosts: all
 gather_facts: false
 become: true
 tasks:
 - ping:

And an inventory like this:

Writing Ansible filter plugins

Thu, 25 Apr 2019 00:00:00 +0000

I often see questions from people who are attemping to perform complex text transformations in their Ansible playbooks. While I am a huge fan of Ansible, data transformation is not one of its strong points. For example, this past week someone asked a question on Stack Overflow in which they were attempting to convert the output of the keytool command into a list of dictionaries. The output of the keytool -list -v command looks something like this:

Docker build learns about secrets and ssh agent forwarding

Sun, 24 Feb 2019 00:00:00 +0000

A common problem for folks working with Docker is accessing resources which require authentication during the image build step. A particularly common use case is getting access to private git repositories using ssh key-based authentication. Until recently there hasn’t been a great solution:

you can embed secrets in your image, but now you can’t share the image with anybody.
you can use build arguments, but this requires passing in an unenecrypted private key on the docker build command line, which is suboptimal for a number of reasons
you can perform all the steps requiring authentication at runtime, but this can needlessly complicate your container startup process.

With Docker 18.09, there are some experimental features available that makes this much easier. You can read the official announcement here, but I wanted to highlight the support for ssh agent forwarding and private keys.

In which I PEBKAC so you don't have to

Mon, 11 Feb 2019 00:00:00 +0000

Say you have a simple bit of code:

#include <avr/io.h>
#include <util/delay.h> 

#define LED_BUILTIN _BV(PORTB5)

int main(void) 
{
 DDRB |= LED_BUILTIN;

 while (1)
 {
 PORTB |= LED_BUILTIN; // turn on led
 _delay_ms(1000); // delay 1s

 PORTB &= ~LED_BUILTIN; // turn off led
 _delay_ms(1000); // delay 1s
 } 
}

You have a Makefile that compiles that into an object (.o) file like this:

avr-gcc -mmcu=atmega328p -DF_CPU=16000000 -Os -c blink.c

If you were to forget to set the device type when compiling your .c file into an object file (.o), you would get a warning:

ATOMIC_BLOCK magic in avr-libc

Fri, 01 Feb 2019 00:00:00 +0000

The AVR C library, avr-libc, provide an ATOMIC_BLOCK macro that you can use to wrap critical sections of your code to ensure that interrupts are disabled while the code executes. At high level, the ATOMIC_BLOCK macro (when called using ATOMIC_FORCEON) does something like this:

cli();

...your code here...

seti();

But it’s more than that. If you read the documentation for the macro, it says:

Creates a block of code that is guaranteed to be executed atomically. Upon entering the block the Global Interrupt Status flag in SREG is disabled, and re-enabled upon exiting the block from any exit path.

AVR micro-optimization: Avr-gcc and --short-enums

Mon, 28 Jan 2019 00:00:00 +0000

How big is an enum?

I noticed something odd while browsing through the assembly output of some AVR C code I wrote recently. In the code, I have the following expression:

int main() {
 setup();

 while (state != STATE_QUIT) {
 loop();
 }
}

Here, state is a variable of type enum STATE, which looks something like this (not exactly like this; there are actually 19 possible values but I didn’t want to clutter this post with unnecessary code listings):

AVR micro-optimization: Losing malloc

Mon, 28 Jan 2019 00:00:00 +0000

Pssst! Hey…hey, buddy, wanna get an extra KB for cheap?

When I write OO-style code in C, I usually start with something like the following, in which I use malloc() to allocate memory for a variable of a particular type, perform some initialization actions, and then return it to the caller:

Button *button_new(uint8_t pin, uint8_t poll_freq) {
 Button *button = (Button *)malloc(sizeof(Button));
 // do some initialization stuff

 return button;
}

And when initially writing pipower, that’s exactly what I did. But while thinking about it after the fact, I realized the following:

Debugging attiny85 code, part 1: simavr and gdb

Tue, 22 Jan 2019 00:00:00 +0000

In a case of awful timing, after my recent project involving some attiny85 programming I finally got around to learning how to use simavr and gdb to help debug my AVR code. It was too late for me (and I will never get the time back that I spent debugging things with an LED and lots of re-flashing), but maybe this will help someone else!

I’ve split this into three posts:

Part 1: Using GDB

Debugging attiny85 code, part 2: Automating GDB with scripts

Tue, 22 Jan 2019 00:00:00 +0000

This is the second of three posts about using gdb and simavr to debug AVR code. The complete series is:

Part 1: Using GDB

A walkthrough of using GDB to manually inspect the behavior of our code.
Part 2: Automating GDB with scripts

Creating GDB scripts to automatically test the behavior of our code.
Part 3: Tracing with simavr

Using simavr to collect information about the state of microcontroller pins while our code is running.

Debugging attiny85 code, part 3: Tracing with simavr

Tue, 22 Jan 2019 00:00:00 +0000

This is the third of three posts about using gdb and simavr to debug AVR code. The complete series is:

Part 1: Using GDB

A walkthrough of using GDB to manually inspect the behavior of our code.
Part 2: Automating GDB with scripts

Creating GDB scripts to automatically test the behavior of our code.
Part 3: Tracing with simavr

Using simavr to collect information about the state of microcontroller pins while our code is running.

PiPower: A Raspberry Pi UPS

Sat, 19 Jan 2019 00:00:00 +0000

I have a Raspberry Pi running RetroPie hooked up to a television. It’s powered from a USB port on the TV, which is convenient, but it means that whenever we shut off the TV we’re pulling the plug on the Pi. While there haven’t been any problems so far, this is a classic recipe for filesystem problems or data loss at some point. I started looking into UPS options to alleviate this issue. I wanted something with the following features:

Integrating Bitwarden with Ansible

Fri, 19 Oct 2018 00:00:00 +0000

Bitwarden is a password management service (like LastPass or 1Password). It’s unique in that it is built entirely on open source software. In addition to the the web UI and mobile apps that you would expect, Bitwarden also provides a command-line tool for interacting with the your password store.

At $WORK(-ish) we’re looking into Bitwarden because we want a password sharing and management solution that was better than dropping files into directories on remote hosts or sharing things over Slack. At the same time, we are also thinking about bringing more automation to our operational environment, possibly by making more extensive use of Ansible. It looked like all the pieces were available to use Bitwarden as a credential storage mechanism for Ansible playbooks, so I set out to write a lookup plugin to implement the integration…

Systemd unit for managing USB gadgets

Fri, 19 Oct 2018 00:00:00 +0000

The Pi Zero (and Zero W) have support for acting as a USB gadget: that means that they can be configured to act as a USB device – like a serial port, an ethernet interface, a mass storage device, etc.

There are two different ways of configuring this support. The first only allows you to configure a single type of gadget at a time, and boils down to:

Enable the dwc2 overlay in /boot/config.txt
Reboot.
modprobe g_serial

This process is more fully documented here.

Configuring a static address for wlan0 on Raspbian Stretch

Thu, 14 Jun 2018 00:00:00 +0000

Recent releases of Raspbian have adopted the use of dhcpcd to manage both dynamic and static interface configuration. If you would prefer to use the traditional /etc/network/interfaces mechanism instead, follow these steps.

First, disable dhcpcd and wpa_supplicant.

 systemctl disable --now dhdpcd wpa_supplicant

You will need a wpa_supplicant configuration for wlan0 in /etc/wpa_supplicant/wpa_supplicant-wlan0.conf.

If you already have an appropriate configuration in /etc/wpa_supplicant/wpa_supplicant.conf, you can just symlink the file:
```
 cd /etc/wpa_supplicant
 ln -s wpa_supplicant.conf wpa_supplicant-wlan0.conf
```
Enable the wpa_supplicant service for wlan0:

Using a TM1637 LED module with CircuitPython

Thu, 03 May 2018 00:00:00 +0000

CircuitPython is “an education friendly open source derivative of MicroPython”. MicroPython is a port of Python to microcontroller environments; it can run on boards with very few resources such as the ESP8266. I’ve recently started experimenting with CircuitPython on a Wemos D1 mini, which is a small form-factor ESP8266 board.

I had previously been using Mike Causer’s micropython-tm1637 for MicroPython to drive a 4 digit LED display. I was hoping to get the same code working under CircuitPython, but when I tried to build an image that included the tm1637 module I ran into:

Multiple 1-Wire Buses on the Raspberry Pi

Tue, 27 Mar 2018 00:00:00 +0000

The DS18B20 is a popular temperature sensor that uses the 1-Wire protocol for communication. Recent versions of the Linux kernel include a kernel driver for this protocol, making it relatively convenient to connect one or more of these devices to a Raspberry Pi or similar device. 1-Wire devices can be daisy chained, so it is possible to connect several devices to your Pi using only a single GPIO pin, and you’ll find many articles out there that describe how to do so.

Using Docker macvlan networks

Mon, 12 Mar 2018 00:00:00 +0000

A question that crops up regularly on #docker is “How do I attach a container directly to my local network?” One possible answer to that question is the macvlan network type, which lets you create “clones” of a physical interface on your host and use that to attach containers directly to your local network. For the most part it works great, but it does come with some minor caveats and limitations. I would like to explore those here.

Listening for connections on all ports/any port

Tue, 27 Feb 2018 00:00:00 +0000

On IRC – and other online communities – it is common to use a “pastebin” service to share snippets of code, logs, and other material, rather than pasting them directly into a conversation. These services will typically return a URL that you can share with others so that they can see the content in their browser.

One of my favorite pastebin services is termbin.com, because it works from the command line using tools you probably already have installed. Termbin runs the fiche service, which listens for TCP connections on port 9999, reads any content that you provide, and then returns a URL. For example, if I wanted to share my iptables configuration with someone I could just run:

Grouping aggregation queries in Gnocchi 4.0.x

Mon, 26 Feb 2018 00:00:00 +0000

In this article, we’re going to ask Gnocchi (the OpenStack telemetry storage service) how much memory was used, on average, over the course of each day by each project in an OpenStack environment.

Environment

I’m working with an OpenStack “Pike” deployment, which means I have Gnocchi 4.0.x. More recent versions of Gnocchi (4.1.x and later) have a new aggregation API called dynamic aggregates, but that isn’t available in 4.0.x so in this article we’ll be using the legacy /v1/aggregations API.

Listing iptables rules with line numbers

Thu, 08 Feb 2018 00:00:00 +0000

You can list iptables rules with rule numbers using the --line-numbers option, but this only works in list (-L) mode. I find it much more convenient to view rules using the output from iptables -S or iptables-save.

You can augment the output from these commands with rule numbers with the following awk script:

#!/bin/awk -f

state == 0 && /^-A/ {state=1; chain=$2; counter=1; printf "\n"}
state == 1 && $2 != chain {chain=$2; counter=1; printf "\n"}
!/^-A/ {state=0}
state == 1 {printf "[%03d] %s\n", counter++, $0}
state == 0 {print}

This will produce output along the lines of:

Pelican and theme update

Fri, 26 Jan 2018 00:00:00 +0000

I’ve just refreshed the version of Pelican used to generate this blog, along with the associated themes and plugins. It all seems to be working, but if you spot a problem feel free to drop me a line.

Fun with devicemapper snapshots

Thu, 25 Jan 2018 00:00:00 +0000

I find myself working with Raspbian disk images fairly often. A typical workflow is:

Download the disk image.
Mount the filesystem somewhere to check something.
Make some changes or install packages just to check something else.
Crap I’ve made changes.

…at which point I need to fetch a new copy of the image next time I want to start fresh.

Sure, I could just make a copy of the image and work from there, but what fun is that? This seemed like a perfect opportunity to learn more about the device mapper and in particular how the snapshot target works.

Safely restarting an OpenStack server with Ansible

Wed, 24 Jan 2018 00:00:00 +0000

The other day on #ansible, someone was looking for a way to safely shut down a Nova server, wait for it to stop, and then start it up again using the openstack cli. The first part seemed easy:

- hosts: myserver
 tasks:
 - name: shut down the server
 command: poweroff
 become: true

…but that will actually fail with the following result:

TASK [shut down server] *************************************
fatal: [myserver]: UNREACHABLE! => {"changed": false, "msg":
"Failed to connect to the host via ssh: Shared connection to
10.0.0.103 closed.\r\n", "unreachable": true}

This happens because running poweroff immediately closes Ansible’s ssh connection. The workaround here is to use a “fire-and-forget” asynchronous task:

Some notes on PWM on the Raspberry Pi

Tue, 26 Sep 2017 00:00:00 +0000

I was recently working on a project in which I wanted to drive a simple piezo buzzer attached to a GPIO pin on a Raspberry Pi. I was already using the RPi.GPIO module in my project so that seemed like a logical place to start, but I ran into a few issues.

You drive a piezo buzzer by generating a PWM signal with the appropriate frequency. The RPi.GPIO module implements PWM via software, which is tricky on a non-realtime system. It’s difficult to get the timing completely accurate, which results in sounds that are a little wobbly at best. Since I’m simply generating tones with a buzzer (rather than, say, controlling a servo) this is mostly just an annoyance.

Ansible for Infrastructure Testing

Wed, 02 Aug 2017 00:00:00 +0000

At $JOB we often find ourselves at customer sites where we see the same set of basic problems that we have previously encountered elsewhere (“your clocks aren’t in sync” or “your filesystem is full” or “you haven’t installed a critical update”, etc). We would like a simple tool that could be run either by the customer or by our own engineers to test for and report on these common issues. Fundamentally, we want something that acts like a typical code test suite, but for infrastructure.

Better bulk filtering for Gmail

Fri, 07 Jul 2017 00:00:00 +0000

I use Gmail extensively for my personal email, and recently my workplace has been migrated over to Gmail as well. I find that for my work email I rely much more extensively on filters and labels to organize things (like zillions of internal and upstream mailing lists), and that has posed some challenges. While Gmail is in general fairly snappy, attempting to apply an action to thousands of messages (for example, trying to mark 16000 messages as “read”, or applying a new filter to all your existing messages) results in a very poor experience: it is not possible to interact with Gmail (in the same tab) while the action is running, and frequently actions will timeout.

OpenStack, Containers, and Logging

Wed, 14 Jun 2017 00:00:00 +0000

I’ve been thinking about logging in the context of OpenStack and containerized service deployments. I’d like to lay out some of my thoughts on this topic and see if people think I am talking crazy or not.

There are effectively three different mechanisms that an application can use to emit log messages:

Via some logging-specific API, such as the legacy syslog API
By writing a byte stream to stdout/stderr
By writing a byte stream to a file

A substantial advantage to the first mechanism (using a logging API) is that the application is logging messages rather than bytes. This means that if you log a message containing embedded newlines (e.g., python or java tracebacks), you can collect that as a single message rather than having to impose some sort of structure on the byte stream after the fact in order to reconstruct those message.

FAA Cannot Require Drone Registration

Thu, 25 May 2017 00:00:00 +0000

This is now old news if you’re already following the drone industry, but if you’re not, I’d like to highlight a recent decision made by the US Court of Appeals regarding the FAA’s drone registration requirements.

To place this in context, back in 2015 the FAA established a new set of regulations (the “Registration Rule”) requiring anyone with a UAV (“unmanned aerial vehicle”, or “drone”) weighing between 0.5 and 55 lbs to register with the FAA. Unfortunately, the 2012 FAA Modernization and Reform Act says says (in section 336(a)):

Making sure your Gerrit changes aren't broken

Sun, 22 Jan 2017 00:00:00 +0000

It’s a bit of an embarrassment when you submit a review to Gerrit only to have it fail CI checks immediately because of something as simple as a syntax error or pep8 failure that you should have caught yourself before submitting…but you forgot to run your validations before submitting the change.

In many cases you can alleviate this through the use of the git pre-commit hook, which will run every time you commit changes locally. You can have the hook run tox or whatever tool your project uses for validation on every commit. This works okay for simple cases, but if the validation takes more than a couple of seconds the delay can be disruptive to the flow of your work.

Exploring YAQL Expressions

Thu, 11 Aug 2016 00:00:00 +0000

The Newton release of Heat adds support for a yaql intrinsic function, which allows you to evaluate yaql expressions in your Heat templates. Unfortunately, the existing yaql documentation is somewhat limited, and does not offer examples of many of yaql’s more advanced features.

I am working on a Fluentd composable service for TripleO. I want to allow each service to specify a logging source configuration fragment, for example:

parameters:
 NovaAPILoggingSource:
 type: json
 description: Fluentd logging configuration for nova-api.
 default:
 tag: openstack.nova.api
 type: tail
 format: |
 /(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}.\d+) (?<pid>\d+) (?<priority>\S+) (?<message>.*)/
 path: /var/log/nova/nova-api.log
 pos_file: /var/run/fluentd/openstack.nova.api.pos

This generally works, but several parts of this fragment are going to be the same across all OpenStack services. I wanted to reduce the above to just the unique attributes, which would look something like:

Connecting another vm to your tripleo-quickstart deployment

Thu, 19 May 2016 00:00:00 +0000

Let’s say that you have set up an environment using tripleo-quickstart and you would like to add another virtual machine to the mix that has both “external” connectivity (“external” in quotes because I am using it in the same way as the quickstart does w/r/t the undercloud) and connectivity to the overcloud nodes. How would you go about setting that up?

For a concrete example, let’s presume you have deployed an environment using the default tripleo-quickstart configuration, which looks like this:

A collection of git tips

Fri, 19 Feb 2016 00:00:00 +0000

This is a small collection of simple git tips and tricks I use to make my life easier.

Quickly amend an existing commit with new files

I have this alias in place that will amend the current commit while automatically re-using the existing commit message:

alias.fix=commit --amend -C HEAD

With this in place, fixing a review becomes:

$ vim some/file/somewhere
$ git add -u
$ git fix

Which I find much more convenient than git commit --amend, following by saving the commit message.

Deploying an HA OpenStack development environment with tripleo-quickstart

Fri, 19 Feb 2016 00:00:00 +0000

In this article I would like to introduce tripleo-quickstart, a tool that will automatically provision a virtual environment and then use TripleO to deploy an HA OpenStack on top of it.

Introducing Tripleo-Quickstart

The goal of the Tripleo-Quickstart project is to replace the instack-virt-setup tool for quickly setting up virtual TripleO environments, and to ultimately become the tool used by both developers and upstream CI for this purpose. The project is a set of Ansible playbooks that will take care of:

Gruf gets superpowers

Fri, 19 Feb 2016 00:00:00 +0000

In my last article article I introduced Gruf, a command line tool for interacting with Gerrit. Since then, Gruf has gained a few important new features.

Caching

Gruf will now by default cache results for five minutes. This avoids repeatedly querying the server for the same information when you’re just displaying it with different templates (for example, if you run a gruf query open here followed by a gruf -t patches query open here).

Gruf, a Gerrit command line utility

Tue, 16 Feb 2016 00:00:00 +0000

(See also the followup to this article.)

I’ve recently started spending more time interacting with Gerrit, the code review tool used both by OpenStack, at review.openstack.org, and by a variety of other open source projects at GerritForge’s GitHub-linked review.gerrithub.io. I went looking for command line tools and was largely disappointed with what I found. Many of the solutions out there assume that you’re regularly interacting with a single Gerrit instance, and that’s seldom the case: more often, the Gerrit server in use varies from project to project.
I also found that many of the tools were opinionated in what sort of output they would produce.

A systemd-nspawn connection driver for Ansible

Mon, 08 Feb 2016 00:00:00 +0000

I wrote earlier about systemd-nspawn, and how it can take much of the fiddly work out of setting up functional chroot environments. I’m a regular Ansible user, and I wanted to be able to apply some of those techniques to my playbooks.

Ansible already has a chroot module, of course, but for some situations – such as targeting an emulated chroot environment – that just means a lot of extra work. Using systemd-nspawn makes this trivial.

Folding long lines in Ansible inventory files

Sun, 07 Feb 2016 00:00:00 +0000

If you have an Ansible inventory file that includes lots of per host variables, it’s not unusual for lines to get long enough that they become unwieldly, particularly if you want to discuss them in an email or write about them in some context (e.g., a blog post).

I’ve just submitted pull request #14359 to Ansible which implements support for folding long lines using the INI-format convention of using indent to mark extended logical lines.

Systemd-nspawn for fun and...well, mostly for fun

Sun, 07 Feb 2016 00:00:00 +0000

systemd-nspawn has been called “chroot on steroids”, but if you think of it as Docker with a slightly different target you wouldn’t be far wrong, either. It can be used to spawn containers on your host, and has a variety of options for configuring the containerized environment through the use of private networking, bind mounts, capability controls, and a variety of other facilities that give you flexible container management.

There are many different ways in which it can be used. I’m going to focus on one that’s a bit of a corner use case that I find particularly interesting. In this article we’re going to explore how we can use systemd-nspawn to spawn lightweight containers for architectures other than that of our host system.

Installing pyspatialite on Fedora

Tue, 17 Nov 2015 00:00:00 +0000

If you should find yourself wanting to install pyspatialite on Fedora – perhaps because you want to use the Processing plugin for QGIS – you will first need to install the following dependencies:

gcc
python-devel
sqlite-devel
geos-devel
proj-devel
python-pip
redhat-rpm-config

After which you can install pyspatialite using pip by running:

CFLAGS=-I/usr/include pip install pyspatialite

At this point, you should be able to use the “Processing” plugin.

Ansible 2.0: New OpenStack modules

Mon, 26 Oct 2015 00:00:00 +0000

This is the second in a loose sequence of articles looking at new features in Ansible 2.0. In the previous article I looked at the Docker connection driver. In this article, I would like to provide an overview of the new-and-much-improved suite of modules for interacting with an OpenStack environment, and provide a few examples of their use.

In versions of Ansible prior to 2.0, there was a small collection of OpenStack modules. There was the minimum necessary to boot a Nova instance:

Automatic git cache

Mon, 19 Oct 2015 00:00:00 +0000

This post is in response to a comment someone made on irc earlier today:

[I] would really like a git lookaside cache which operated on an upstream repo, but pulled objects locally when they’re available

In this post I present a proof-of-concept solution to this request. Please note that thisand isn’t something that has actually been used or tested anywhere!

If you access a git repository via ssh, it’s easy to provide a wrapper for git operations via the command= option in an authorized_keys file. We can take advantage of this to update a a local “cache” repository prior to responding to a clone/pull/etc. operation.

Stupid Ansible Tricks: Running a role from the command line

Mon, 19 Oct 2015 00:00:00 +0000

When writing Ansible roles I occasionally want a way to just run a role from the command line, without having to muck about with a playbook. I’ve seen similar requests on the mailing lists and on irc.

I’ve thrown together a quick wrapper that will allow you (and me!) to do exactly that, called ansible-role. The --help output looks like this:

usage: ansible-role [-h] [--verbose] [--gather] [--no-gather]
 [--extra-vars EXTRA_VARS] [-i INVENTORY] [--hosts HOSTS]
 [--sudo] [--become] [--user USER]
 role

positional arguments:
 role

optional arguments:
 -h, --help show this help message and exit
 --verbose, -v
 --gather, -g
 --no-gather, -G
 --extra-vars EXTRA_VARS, -e EXTRA_VARS

Inventory:
 -i INVENTORY, --inventory INVENTORY
 --hosts HOSTS, -H HOSTS

Identity:
 --sudo, -s
 --become, -b
 --user USER, -u USER

Example

If you have a role roles/testrole that contains the following in tasks/main.yml:

Bootstrapping Ansible on Fedora 23

Thu, 15 Oct 2015 00:00:00 +0000

If you’ve tried running Ansible against a Fedora 23 system, you may have run into the following problem:

fatal: [myserver]: FAILED! => {"changed": false, "failed": true,
"msg": "/bin/sh: /usr/bin/python: No such file or directory\r\n",
"parsed": false}

Fedora has recently made the switch to only including Python 3 on the base system (at least for the cloud variant), while Ansible still requires Python 2. With Fedora 23, Python 3 is available as /usr/bin/python3, and /usr/bin/python is only available if you have installed the Python 2 interpreter.

Ansible 2.0: The Docker connection driver

Tue, 13 Oct 2015 00:00:00 +0000

As the release of Ansible 2.0 draws closer, I’d like to take a look at some of the new features that are coming down the pipe. In this post, we’ll look at the docker connection driver.

A “connection driver” is the mechanism by which Ansible connects to your target hosts. These days it uses ssh by default (which relies on the OpenSSH command line client for connectivity), and it also offers the Paramiko library as an alternative ssh implementation (this was in fact the default driver in earlier versions of Ansible). Alternative drivers offered by recent versions of ansible included the winrm driver, for accessing Windows hosts, the fireball driver, a (deprecated) driver that used 0mq for communication, and jail, a driver for connecting to FreeBSD jails.

Running NTP in a Container

Fri, 09 Oct 2015 00:00:00 +0000

Someone asked on IRC about running ntpd in a container on Atomic, so I’ve put together a small example. We’ll start with a very simple Dockerfile:

FROM alpine
RUN apk update
RUN apk add openntpd
ENTRYPOINT ["ntpd"]

I’m using the alpine image as my starting point because it’s very small, which makes this whole process go a little faster. I’m installing the openntpd package, which provides the ntpd binary.

By setting an ENTRYPOINT here, the ntpd binary will be started by default, and any arguments passed to docker run after the image name will be passed to ntpd.

Migrating Cinder volumes between OpenStack environments using shared NFS storage

Tue, 29 Sep 2015 00:00:00 +0000

Many of the upgrade guides for OpenStack focus on in-place upgrades to your OpenStack environment. Some organizations may opt for a less risky (but more hardware intensive) option of setting up a parallel environment, and then migrating data into the new environment. In this article, we look at how to use Cinder backups with a shared NFS volume to facilitate the migration of Cinder volumes between two different OpenStack environments.

Provider external networks (in an appropriate amount of detail)

Thu, 13 Aug 2015 00:00:00 +0000

In Quantum in Too Much Detail, I discussed the architecture of a Neutron deployment in detail. Since that article was published, Neutron gained the ability to handle multiple external networks with a single L3 agent. While I wrote about that back in 2014, I covered the configuration side of it in much more detail than I discussed the underlying network architecture. This post addresses the architecture side.

The players

This document describes the architecture that results from a particular OpenStack configuration, specifically:

In which we are amazed it doesn't all fall apart

Sun, 26 Jul 2015 00:00:00 +0000

So, the Kilo release notes say:

nova-manage migrate-flavor-data

But nova-manage says:

nova-manage db migrate_flavor_data

But that says:

Missing arguments: max_number

And the help says:

usage: nova-manage db migrate_flavor_data [-h]
 [--max-number <number>]

Which indicates that –max-number is optional, but whatever, so you try:

nova-manage db migrate_flavor_data --max-number 100

And that says:

Missing arguments: max_number

So just for kicks you try:

nova-manage db migrate_flavor_data --max_number 100

And that says:

Mapping local users to Kerberos principals with SSSD

Thu, 16 Jul 2015 00:00:00 +0000

I work for an organization that follows the common model of assigning people systematically generated user ids. Like most technically inclined employees of this organization, I have local accounts on my workstation that don’t bear any relation to the generated account ids. For the most part this isn’t a problem, except that our organization uses Kerberos to authenticate access to a variety of resources (such as the mailserver and a variety of web applications).

OpenStack Networking without DHCP

Fri, 26 Jun 2015 00:00:00 +0000

In an OpenStack environment, cloud-init generally fetches information from the metadata service provided by Nova. It also has support for reading this information from a configuration drive, which under OpenStack means a virtual CD-ROM device attached to your instance containing the same information that would normally be available via the metadata service.

It is possible to generate your network configuration from this configuration drive, rather than relying on the DHCP server provided by your OpenStack environment. In order to do this you will need to make the following changes to your Nova configuration:

Heat-kubernetes Demo with Autoscaling

Fri, 19 Jun 2015 00:00:00 +0000

Next week is the Red Hat Summit in Boston, and I’ll be taking part in a Project Atomic presentation in which I will discuss various (well, two) options for deploying Atomic into an OpenStack environment, focusing on my heat-kubernetes templates.

As part of that presentation, I’ve put together a short demonstration video:

This shows off the autoscaling behavior available with recent versions of these templates (and also serves as a very brief introduction to working with Kubernetes).

Teach git about GIT_SSL_CIPHER_LIST

Fri, 08 May 2015 00:00:00 +0000

Someone named hithard on StackOverflow was trying to clone a git repository via https, and was running into an odd error: “Cannot communicate securely with peer: no common encryption algorithm(s).”. This was due to the fact that the server (openhatch.org) was configured to use a cipher suite that was not supported by default in the underlying SSL library (which could be either OpenSSL or NSS, depending on how git was built).

Many applications allow the user to configure an explicit list of ciphers to consider when negotiating a secure connection. For example, curl has the CURLOPT_SSL_CIPHER_LIST option. This turns out to be especially relevant because git relies on libcurl for all of its http operations, which means all we need to do is (a) create a new configuration option for git, and then (b) pass that value through to libcurl.

Suggestions for the Docker MAINTAINER directive

Mon, 27 Apr 2015 00:00:00 +0000

Because nobody asked for it, this is my opinion on the use of the MAINTAINER directive in your Dockerfiles.

The documentation says simply:

The MAINTAINER instruction allows you to set the Author field of the generated images.

Many people end up putting the name and email address of an actual person here. I think this is ultimately a bad idea, and does a disservice both to members of a project that produce Docker images and to people consuming those images.

Using tools badly: time shifting git commits with Workinghours

Fri, 10 Apr 2015 00:00:00 +0000

This is a terrible hack. If you are easily offended by bad ideas implemented poorly, move along!

You are working on a wonderful open source project…but you are not supposed to be working on that project! You’re supposed to be doing your real work! Unfortunately, your extra-curricular activity is well documented in the git history of your project for all to see:

And now your boss knows why the TPS reports are late. You need workinghours, a terrible utility for doing awful things to your repository history. Workinghours will programatically time shift your git commits so that they appear to have happened within specified time intervals (for example, “between 7PM and midnight”).

Booting cloud images with libvirt

Tue, 10 Mar 2015 00:00:00 +0000

Most major distributions now provide “cloud-enabled” images designed for use in cloud environments like OpenStack and AWS. These images are usually differentiated by (a) being relatively small, and (b) running cloud-init at boot to perform initial system configuration tasks using metadata provided by the cloud environment.

Because of their small size and support for automatic configuration (including such useful tasks as provisioning ssh keys), these images are attractive for use outside of a cloud environment. Unfortunately, when people first try to boot them they are met with frustration as first the image takes forever to boot as it tries to contact a non-existent metadata service, and then when it finally does boot they are unable to log in because the images typically only support key-based login.

Diagnosing problems with an OpenStack deployment

Mon, 09 Mar 2015 00:00:00 +0000

I recently had the chance to help a colleague debug some problems in his OpenStack installation. The environment was unique because it was booting virtualized aarch64 instances, which at the time did not have any PCI bus support…which in turn precluded things like graphic consoles (i.e., VNC or SPICE consoles) for the Nova instances.

This post began life as an email summarizing the various configuration changes we made on the systems to get things up and running. After writing it, I decided it presented an interesting summary of some common (and maybe not-so-common) issues, so I am posting it here in the hopes that other folks will find it interesting.

Converting hexadecimal ip addresses to dotted quads with Bash

Sun, 08 Mar 2015 00:00:00 +0000

This is another post that is primarily for my own benefit for the next time I forget how to do this.

I wanted to read routing information directly from /proc/net/route using bash, because you never know what may or may not be available in the minimal environment of a Docker container (for example, the iproute package is not installed by default in the Fedora Docker images). The contents of /proc/net/route looks something like:

Visualizing Pacemaker resource constraints

Tue, 24 Feb 2015 00:00:00 +0000

If a picture is worth a thousand words, then code that generates pictures from words is worth…uh, anyway, I wrote a script that produces dot output from Pacemaker start and colocation constraints:

https://github.com/larsks/pacemaker-tools/

You can pass this output to graphviz to create visualizations of your Pacemaker resource constraints.

The graph-constraints.py script in that repository consumes the output of cibadmin -Q and can produce output for either start constraints (-S, the default) or colocation constraints (-C).

Stupid Pacemaker XML tricks

Thu, 19 Feb 2015 00:00:00 +0000

I’ve recently spent some time working with Pacemaker, and ended up with an interesting collection of XPath snippets that I am publishing here for your use and/or amusement.

Check if there are any inactive resources

pcs status xml |
 xmllint --xpath '//resource[@active="false"]' - >&/dev/null &&
 echo "There are inactive resources"

This selects any resource (//resource) in the output of pcs status xml that has the attribute active set to false. If there are no matches to this query, xmllint exits with an error code.

Unpacking Docker images with Undocker

Fri, 13 Feb 2015 00:00:00 +0000

In some ways, the most exciting thing about Docker isn’t the ability to start containers. That’s been around for a long time in various forms, such as LXC or OpenVZ. What Docker brought to the party was a convenient method of building and distributing the filesystems necessary for running containers. Suddenly, it was easy to build a containerized service and to share it with other people.

I was taking a closer at the systemd-nspawn command, which it seems has been developing it’s own set of container-related superpowers recently, including a number of options for setting up the network environment of a container. Like Docker, systemd-nspawn needs a filesystem on which to operate, but unlike Docker, there is no convenient distribution mechanism and no ecosystem of existing images. In fact, the official documentation seems to assume that you’ll be building your own from scratch. Ain’t nobody got time for that…

Installing nova-docker with devstack

Wed, 11 Feb 2015 00:00:00 +0000

This is a long-form response to this question, and describes how to get the nova-docker driver up running with devstack under Ubuntu 14.04 (Trusty). I wrote a similar post for Fedora 21, although that one was using the RDO Juno packages, while this one is using devstack and the upstream sources.

Getting started

We’ll be using the Ubuntu 14.04 cloud image (because my test environment runs on OpenStack).

First, let’s install a few prerequisites:

$ sudo apt-get update
$ sudo apt-get -y install git git-review python-pip python-dev

And generally make sure things are up-to-date:

External networking for Kubernetes services

Tue, 10 Feb 2015 00:00:00 +0000

I have recently started running some “real” services (that is, “services being consumed by someone other than myself”) on top of Kubernetes (running on bare metal), which means I suddenly had to confront the question of how to provide external access to Kubernetes hosted services. Kubernetes provides two solutions to this problem, neither of which is particularly attractive out of the box:

There is a field createExternalLoadBalancer that can be set in a service description. This is meant to integrate with load balancers provided by your local cloud environment, but at the moment there is only support for this when running under GCE.

Installing nova-docker on Fedora 21/RDO Juno

Fri, 06 Feb 2015 00:00:00 +0000

This post comes about indirectly by a request on IRC in #rdo for help getting nova-docker installed on Fedora 21. I ran through the process from start to finish and decided to write everything down for posterity.

Getting started

I started with the Fedora 21 Cloud Image, because I’m installing onto OpenStack and the cloud images include some features that are useful in this environment.

We’ll be using OpenStack packages from the RDO Juno repository. Because there is often some skew between the RDO packages and the current Fedora selinux policy, we’re going to start by putting SELinux into permissive mode (sorry, Dan):

Creating minimal Docker images from dynamically linked ELF binaries

Thu, 05 Feb 2015 00:00:00 +0000

In this post, we’ll look at a method for building minimal Docker images for dynamically linked ELF binaries, and then at a tool for automating this process.

It is tempting, when creating a simple Docker image, to start with one of the images provided by the major distributions. For example, if you need an image that provides tcpdump for use on your Atomic host, you might do something like:

FROM fedora
RUN yum -y install tcpdump

And while this will work, you end up consuming 250MB for tcpdump. In theory, the layering mechanism that Docker uses to build images will reduce the practical impact of this (because other images based on the fedora image will share the common layers), but in practice the size is noticeable, especially if you often find yourself pulling this image into a fresh environment with no established cache.

Filtering libvirt XML in Nova

Thu, 05 Feb 2015 00:00:00 +0000

I saw a request from a customer float by the other day regarding the ability to filter the XML used to create Nova instances in libvirt. The customer effectively wanted to blacklist a variety of devices (and device types). The consensus seems to be “you can’t do this right now and upstream is unlikely to accept patches that implement this behavior”, but it sounded like an interesting problem, so…

https://github.com/larsks/nova/tree/feature/xmlfilter

This is a fork of Nova (Juno) that includes support for an extensible filtering mechanism that is applied to the generated XML before it gets passed to libvirt.

Docker vs. PrivateTmp

Sun, 18 Jan 2015 00:00:00 +0000

While working with Docker the other day, I ran into an undesirable interaction between Docker and systemd services that utilize the PrivateTmp directive.

The PrivateTmp directive, if true, “sets up a new file system namespace for the executed processes and mounts private /tmp and /var/tmp directories inside it that is not shared by processes outside of the namespace”. This is a great idea from a security perspective, but can cause some unanticipated consequences.

The problem in a nutshell

Start a Docker container:

Running nova-libvirt and nova-docker on the same host

Sat, 17 Jan 2015 00:00:00 +0000

I regularly use OpenStack on my laptop with libvirt as my hypervisor. I was interested in experimenting with recent versions of the nova-docker driver, but I didn’t have a spare system available on which to run the driver, and I use my regular nova-compute service often enough that I didn’t want to simply disable it temporarily in favor of nova-docker.

NB As pointed out by gustavo in the comments, running two neutron-openvswitch-agents on the same host – as suggested in this article – is going to lead to nothing but sadness and doom. So kids, don’t try this at home. I’m leaving the article here because I think it still has some interesting bits.

Building a minimal web server for testing Kubernetes

Sun, 04 Jan 2015 00:00:00 +0000

I have recently been doing some work with Kubernetes, and wanted to put together a minimal image with which I could test service and pod deployment. Size in this case was critical: I wanted something that would download quickly when initially deployed, because I am often setting up and tearing down Kubernetes as part of my testing (and some of my test environments have poor external bandwidth).

Building thttpd

My go-to minimal webserver is thttpd. For the normal case, building the software is a simple matter of ./configure followed by make. This gets you a dynamically linked binary; using ldd you could build a Docker image containing only the necessary shared libraries:

Accessing the serial console of your Nova servers

Mon, 22 Dec 2014 00:00:00 +0000

One of the new features available in the Juno release of OpenStack is support for serial console access to your Nova servers. This post looks into how to configure the serial console feature and then how to access the serial consoles of your Nova servers.

Configuring serial console support

In previous release of OpenStack, read-only access to the serial console of your servers was available through the os-getConsoleOutput server action (exposed via nova console-log on the command line). Most cloud-specific Linux images are configured with a command line that includes something like console=tty0 console=ttyS0,115200n81, which ensures that kernel output and other messages are available on the serial console. This is a useful mechanism for diagnosing problems in the event that you do not have network access to a server.

Cloud-init and the case of the changing hostname

Wed, 10 Dec 2014 00:00:00 +0000

Setting the stage

I ran into a problem earlier this week deploying RDO Icehouse under RHEL 6. My target systems were a set of libvirt guests deployed from the RHEL 6 KVM guest image, which includes cloud-init in order to support automatic configuration in cloud environments. I take advantage of this when using libvirt by attaching a configuration drive so that I can pass in ssh keys and a user-data script.

Starting systemd services without blocking

Tue, 02 Dec 2014 00:00:00 +0000

Recently, I’ve been playing around with Fedora Atomic and Kubernetes. I ran into a frustrating problem in which I would attempt to start a service from within a script launched by cloud-init, only to have have systemctl block indefinitely because the service I was attempting to start was dependent on cloud-init finishing first.

It turns out that systemctl has a flag meant exactly for this situation:

 --no-block
 Do not synchronously wait for the requested operation to finish. If
 this is not specified, the job will be verified, enqueued and
 systemctl will wait until it is completed. By passing this
 argument, it is only verified and enqueued.

Replacing systemctl start <service> with systemctl start --no-block <service> has solved that particular problem.

Fedora Atomic, OpenStack, and Kubernetes (oh my)

Mon, 24 Nov 2014 00:00:00 +0000

While experimenting with Fedora Atomic, I was looking for an elegant way to automatically deploy Atomic into an OpenStack environment and then automatically schedule some Docker containers on the Atomic host. This post describes my solution.

Like many other cloud-targeted distributions, Fedora Atomic runs cloud-init when the system boots. We can take advantage of this to configure the system at first boot by providing a user-data blob to Nova when we boot the instance. A user-data blob can be as simple as a shell script, and while we could arguably mash everything into a single script it wouldn’t be particularly maintainable or flexible in the face of different pod/service/etc descriptions.

Creating a Windows image for OpenStack

Sat, 15 Nov 2014 00:00:00 +0000

If you want to build a Windows image for use in your OpenStack environment, you can follow the example in the official documentation, or you can grab a Windows 2012r2 evaluation pre-built image from the nice folks at CloudBase.

The CloudBase-provided image is built using a set of scripts and configuration files that CloudBase has made available on GitHub.

The CloudBase repository is an excellent source of information, but I wanted to understand the process myself. This post describes the process I went through to establish an automated process for generating a Windows image suitable for use with OpenStack.

Building Docker images with Puppet

Wed, 22 Oct 2014 00:00:00 +0000

I like Docker, but I’m not a huge fan of using shell scripts for complex system configuration…and Dockerfiles are basically giant shell scripts.

I was curious whether or not it would be possible to use Puppet during the docker build process. As a test case, I used the ssh module included in the openstack-puppet-modules package.

I started with a manifest like this (in puppet/node.pp):

class { 'ssh': }

And a Dockerfile like this:

Docker networking with dedicated network containers

Mon, 06 Oct 2014 00:00:00 +0000

The current version of Docker has a very limited set of networking options:

bridge – connect a container to the Docker bridge
host – run the container in the global network namespace
container:xxx – connect a container to the network namespace of another container
none – do not configure any networking

If you need something more than that, you can use a tool like pipework to provision additional network interfaces inside the container, but this leads to a synchronization problem: pipework can only be used after your container is running. This means that when starting your container, you must have logic that will wait until the necessary networking is available before starting your service.

Integrating custom code with Nova using hooks

Sat, 27 Sep 2014 00:00:00 +0000

Would you like to run some custom Python code when Nova creates and destroys virtual instances on your compute hosts? This is possible using Nova’s support for hooks, but the existing documentation is somewhat short on examples, so I’ve spent some time trying to get things working.

The demo_nova_hooks repository contains a working example of the techniques discussed in this article.

What’s a hook?

A Nova “hook” is a mechanism that allows you to attach a class of your own design to a particular function or method call in Nova. Your class should define a pre method (that will be called before the method is called) and post function (that will be called after the method is called):

Stupid command line tricks: Quickly share screen captures

Tue, 23 Sep 2014 00:00:00 +0000

Sometimes you want to quickly share a screenshot with someone. Here’s my favorite mechanism, which assumes you have installed both curl and the ImageMagick suite.

$ import png:- | curl -T- -s chunk.io
http://chunk.io/f/76ea98ea081748e19de4507fde3c2c65

When you run this command, you cursor will change into crosshairs. Click on a window, and this will grab a png image of that window and send it to chunk.io using curl.

You’ll get back a URL that you can use to share the image with people.

Heat Hangout

Fri, 05 Sep 2014 00:00:00 +0000

I ran a Google Hangout this morning on Deploying with Heat. You can find the slides for the presentation on line here, and the Heat templates (as well as slide sources) are available on github.

If you have any questions about the presentation, please feel free to ping me on irc (larsks).

Visualizing Heat stacks

Tue, 02 Sep 2014 00:00:00 +0000

I spent some time today learning about Heat autoscaling groups, which are incredibly nifty but a little opaque from the Heat command line, since commands such as heat resource-list don’t recurse into nested stacks. It is possible to introspect these resources (you can pass the physical resource id of a nested stack to heat resource-list, for example)…

…but I really like visualizing things, so I wrote a quick hack called dotstack that will generate dot language output from a Heat stack. You can process this with Graphviz to produce output like this, in which graph nodes are automatically colorized by resource type:

Docker plugin bugs

Mon, 01 Sep 2014 00:00:00 +0000

This is a companion to my article on the Docker plugin for Heat.

While writing that article, I encountered a number of bugs in the Docker plugin and elsewhere. I’ve submitted patches for most of the issues I encountered:

Bugs in the Heat plugin

https://bugs.launchpad.net/heat/+bug/1364017

docker plugin fails to delete a container resource in CREATE_FAILED state.
https://bugs.launchpad.net/heat/+bug/1364041

docker plugin volumes_from parameter should be a list.
https://bugs.launchpad.net/heat/+bug/1364039

docker plugin volumes_from parameter results in an error

Annotated documentation for DockerInc::Docker::Container

Sat, 30 Aug 2014 00:00:00 +0000

This is a companion to my article on the Docker plugin for Heat.

DockerInc::Docker::Container

Properties

cmd : List

Command to run after spawning the container.

Optional property.

Example:
```
 cmd: [ 'thttpd', '-C', '/etc/thttpd.conf', '-D', '-c', '*.cgi']
```
dns : List

Set custom DNS servers.

Example:
```
 dns:
 - 8.8.8.8
 - 8.8.4.4
```
docker_endopint : String

Docker daemon endpoint. By default the local Docker daemon will be used.

Example:
```
 docker_endpoint: tcp://192.168.1.100:2375
```
env : String

Docker plugin for OpenStack Heat

Sat, 30 Aug 2014 00:00:00 +0000

I have been looking at both Docker and OpenStack recently. In my last post I talked a little about the Docker driver for Nova; in this post I’ll be taking an in-depth look at the Docker plugin for Heat, which has been available since the Icehouse release but is surprisingly under-documented.

The release announcement on the Docker blog includes an example Heat template, but it is unfortunately grossly inaccurate and has led many people astray. In particular:

Using wait conditions with Heat

Sat, 30 Aug 2014 00:00:00 +0000

This post accompanies my article on the Docker plugin for Heat.

In order for WaitCondition resources to operate correctly in Heat, you will need to make sure that that you have:

Created the necessary Heat domain and administrative user in Keystone,
Configured appropriate values in heat.conf for stack_user_domain, stack_domain_admin, and stack_domain_admin_password.
Configured an appropriate value in heat.conf for heat_waitcondition_server_url. On a single-system install this will often be pointed by default at 127.0.0.1, which, hopefully for obvious reasons, won’t be of any use to your Nova servers.
Enabled the heat-api-cfn service,
Configured your firewall to permit access to the CFN service (which runs on port 8000).

Steve Hardy has a blog post on stack domain users that goes into detail on configuring authentication for Heat and Keystone.

nova-docker and environment variables

Thu, 28 Aug 2014 00:00:00 +0000

I’ve been playing with Docker a bit recently, and decided to take a look at the nova-docker driver for OpenStack.

The nova-docker driver lets Nova, the OpenStack Compute service, spawn Docker containers instead of hypervisor-based servers. For certain workloads, this leads to better resource utilization than you would get with a hypervisor-based solution, while at the same time givin you better support for multi-tenancy and flexible networking than you get with Docker by itself.

lvcache: a tool for managing LVM caches

Sat, 16 Aug 2014 00:00:00 +0000

Until recently I had a bcache based setup on my laptop, but when forced by circumstance to reinstall everything I spent some time looking for alternatives that were less disruptive to configure on an existing system.

I came across Richard Jones’ article discussing the recent work to integrate dm-cache into LVM. Unlike bcache and unlike using dm-cache directly, the integration with LVM makes it easy to associate devices with an existing logical volume.

I have put together a small tool called lvcache that simplies the process of:

Four ways to connect a docker container to a local network

Mon, 11 Aug 2014 00:00:00 +0000

Update (2018-03-22) Since I wrote this document back in 2014, Docker has developed the macvlan network driver. That gives you a supported mechanism for direct connectivity to a local layer 2 network. I’ve written an article about working with the macvlan driver.

This article discusses four ways to make a Docker container appear on a local network. These are not suggested as practical solutions, but are meant to illustrate some of the underlying network technology available in Linux.

gpio-watch: Run scripts in response to GPIO signals

Sat, 26 Jul 2014 00:00:00 +0000

For a small project I’m working on I needed to attach a few buttons to a Raspberry Pi and have some code execute in response to the button presses.

Normally I would reach for Python for a simple project like this, but constraints of the project made it necessary to implement something in C with minimal dependencies. I didn’t want to write something that was tied closely to my project…

…so I ended up writing gpio-watch, a simple tool for connecting shell scripts (or any other executable) to GPIO events. There are a few ways to interact with GPIO on the Raspberry Pi. For the fastest possible performance, you will need to interact directly with the underlying hardware using, e.g., something like direct register access. Since I was only responding to button presses I opted to take advantage of the GPIO sysfs interface, which exposes the GPIO pins via the filesystem.

Tracking down a kernel bug with git bisect

Mon, 21 Jul 2014 00:00:00 +0000

After a recent upgrade of my Fedora 20 system to kernel 3.15.mumble, I started running into a problem (BZ 1121345) with my Docker containers. Operations such as su or runuser would fail with the singularly unhelpful System error message:

$ docker run -ti fedora /bin/bash
bash-4.2# su -c 'uptime'
su: System error

Hooking up something (like, say, socat unix-listen:/dev/log -) to /dev/log revealed that the system was logging:

Jul 19 14:31:18 su: PAM audit_log_acct_message() failed: Operation not permitted

Downgrading the kernel to 3.14 immediately resolved the problem, suggesting that this was at least partly a kernel issue. This seemed like a great opportunity to play with the git bisect command, which uses a binary search to find which commit introduced a particular problem.

Booting an instance with multiple fixed addresses

Wed, 28 May 2014 00:00:00 +0000

This article expands on my answer to Add multiple specific IPs to instance, a question posted to ask.openstack.org.

In order to serve out SSL services from an OpenStack instance, you will generally want one local ip address for each SSL virtual host you support. It is possible to create an instance with multiple fixed addresses, but there are a few complications to watch out for.

Assumptions

This article assumes that the following resources exist:

Multiple external networks with a single L3 agent

Wed, 28 May 2014 00:00:00 +0000

In the old days (so, like, last year), Neutron supported a single external network per L3 agent. You would run something like this…

$ neutron net-create external --router:external=true

…and neutron would map this to the bridge defined in external_network_bridge in /etc/neutron/l3_agent.ini. If you wanted to support more than a single external network, you would need to run multiple L3 agents, each with a unique value for external_network_bridge.

There is now a better option available.

Video: Configuring OpenStack's external bridge on a single-interface system

Tue, 27 May 2014 00:00:00 +0000

I’ve just put a video on Youtube that looks at the steps required to set up the external bridge (br-ex) on a single-interface system:

Open vSwitch and persistent MAC addresses

Fri, 23 May 2014 00:00:00 +0000

Normally I like to post solutions, but today’s post is about a vexing problem to which I have not been able to find a solution.

This started as a simple attempt to set up external connectivity on an all-in-one Icehouse install deployed on an OpenStack instance. I wanted to add eth0 to br-ex in order to model a typical method for providing external connectivity, but I ran into a very odd problem: the system would boot and work fine for a few seconds, but would then promptly lose network connectivity.

Solved: Open vSwitch and persistent MAC addresses

Fri, 23 May 2014 00:00:00 +0000

In my previous post I discussed a problem I was having setting a persistent MAC address on an OVS bridge device. It looks like the short answer is, “don’t use ip link set ...” for this purpose.

You can set the bridge MAC address via ovs-vsctl like this:

ovs-vsctl set bridge br-ex other-config:hwaddr=$MACADDR

So I’ve updated my ifconfig-br-ex to look like this:

DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
ONBOOT=yes
OVSBOOTPROTO=dhcp
OVSDHCPINTERFACES=eth0
MACADDR=fa:16:3e:ef:91:ec
OVS_EXTRA="set bridge br-ex other-config:hwaddr=$MACADDR"

The OVS_EXTRA parameter gets passed to the add-br call like this:

Sharing a terminal session with termshare

Wed, 21 May 2014 00:00:00 +0000

Termshare is a tool for sharing your terminal in a browser session. It supports both read-only and read-write sessions, and unlike many other tools it does not require any software installation on the remote side. This makes it tremendously handy for:

Streaming terminal demonstrations to a diverse audience, or
Sharing a terminal session with someone without needing to much about with ssh, tmux, screen, etc.

I’ve successfully used Termshare under both Fedora (19 and 20) and CentOS. To get started on these platforms, you’ll need to install the Go language, git for cloning the termshare repository, and mercurial to support installation of some Go libraries:

Fedora and OVS Bridge Interfaces

Tue, 20 May 2014 00:00:00 +0000

I run OpenStack on my laptop, and I’ve been chasing down a pernicious problem with OVS bridge interfaces under both F19 and F20. My OpenStack environment relies on an OVS bridge device named br-ex for external connectivity and for making services available to OpenStack instances, but after rebooting, br-ex was consistently unconfigured, which caused a variety of problems.

This is the network configuration file for br-ex on my system:

DEVICE=br-ex
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROT=static
IPADDR=192.168.200.1
NETMASK=255.255.255.0
ONBOOT=yes
NM_CONTROLLED=no
ZONE=openstack

Running ifup br-ex would also fail to configure the interface, but running ifdown br-ex; ifup br-ex would configure things appropriately.

Firewalld, NetworkManager, and OpenStack

Tue, 20 May 2014 00:00:00 +0000

These are my notes on making OpenStack play well with firewalld and NetworkManager.

NetworkManager

By default, NetworkManager attempts to start a DHCP client on every new available interface. Since booting a single instance in OpenStack can result in the creation of several virtual interfaces, this results in a lot of:

May 19 11:58:24 pk115wp-lkellogg NetworkManager[1357]: <info>
 Activation (qvb512640bd-ee) starting connection 'Wired connection 2'

You can disable this behavior by adding the following to /etc/NetworkManager/NetworkManager.conf:

Flat networks with ML2 and OpenVSwitch

Mon, 19 May 2014 00:00:00 +0000

Due to an unfortunate incident involving sleep mode and an overheated backpack I had the “opportunity” to rebuild my laptop. Since this meant reinstalling OpenStack I used this as an excuse to finally move to the ML2 network plugin for Neutron.

I was attempting to add an external network using the normal incantation:

neutron net-create external -- --router:external=true \
 --provider:network_type=flat \
 --provider:physical_network=physnet1

While this command completed successfully, I was left without any connectivity between br-int and br-ex, despite having in my /etc/neutron/plugins/ml2/ml2_conf.ini:

Extending Puppet

Wed, 16 Apr 2014 00:00:00 +0000

I wanted to learn about writing custom Puppet types and providers. The official documentation is a little sparse, but I finally stumbled upon the following series of articles by Gary Larizza that provide a great deal of insight into the process and a bunch of example code:

Multinode OpenStack with Packstack

Thu, 27 Feb 2014 00:00:00 +0000

I was the presenter for this morning’s RDO hangout, where I ran through a simple demonstration of setting up a multinode OpenStack deployment using packstack.

The slides are online here.

Here’s the video (also available on the event page):

Show OVS external-ids

Sun, 19 Jan 2014 00:00:00 +0000

This is just here as a reminder for me:

An OVS interface has a variety of attributes associated with it, including an external-id field that can be used to associate resources outside of OpenVSwitch with the interface. You can view this field with the following command:

$ ovs-vsctl --columns=name,external-ids list Interface

Which on my system, with a single virtual instance, looks like this:

# ovs-vsctl --columns=name,external-ids list Interface
.
.
.
name : "qvo519d7cc4-75"
external_ids : {attached-mac="fa:16:3e:f7:75:b0", iface-id="519d7cc4-7593-4944-af7b-4056436f2d66", iface-status=active, vm-uuid="0330b084-03db-4d42-a231-2cd6ad89515b"}
.
.
.

Note the information contained here:

Stupid OpenStack Tricks

Thu, 16 Jan 2014 00:00:00 +0000

I work with several different OpenStack installations. I usually work on the command line, sourcing in an appropriate stackrc with credentials as necessary, but occasionally I want to use the dashboard for something.

For all of the deployments with which I work, the keystone endpoint is on the same host as the dashboard. So rather than trying to remember which dashboard url I want for the environment I’m currently using on the command line, I put together this shell script:

Direct access to Nova metadata

Tue, 14 Jan 2014 00:00:00 +0000

When you boot a virtual instance under OpenStack, your instance has access to certain instance metadata via the Nova metadata service, which is canonically available at http://169.254.169.254/.

In an environment running Neutron, a request from your instance must traverse a number of steps:

From the instance to a router,
Through a NAT rule in the router namespace,
To an instance of the neutron-ns-metadata-proxy,
To the actual Nova metadata service

When there are problem accessing the metadata, it can be helpful to verify that the metadata service itself is configured correctly and returning meaningful information.

RDO Bug Triage

Mon, 13 Jan 2014 00:00:00 +0000

This Wednesday, January 15, at 14:00 UTC (that’s 9AM US/Eastern, or date -d "14:00 UTC" in your local timezone) I will be helping out with the RDO bug triage day. We’ll be trying to validate all the untriaged bugs opened against RDO.

Feel free to drop by on #rdo and help out or ask questions.

Visualizing Neutron Networking with GraphViz

Mon, 23 Dec 2013 00:00:00 +0000

I’ve put together a few tools to help gather information about your Neutron and network configuration and visualize it in different ways. All of these tools are available as part of my neutron-diag repository on GitHub.

In this post I’m going to look at a tool that will help you visualize the connectivity of network devices on your system.

mk-network-dot

There are a lot of devices involved in your Neutron network configuration. Information originating in one of your instances has two traverse at least seven network devices before seeing the light of day. Understanding how everything connects is critical if you’re trying to debug problems in your envionment.

An introduction to OpenStack Heat

Fri, 06 Dec 2013 00:00:00 +0000

Heat is a template-based orchestration mechanism for use with OpenStack. With Heat, you can deploy collections of resources – networks, servers, storage, and more – all from a single, parameterized template.

In this article I will introduce Heat templates and the heat command line client.

Writing templates

Because Heat began life as an analog of AWS CloudFormation, it supports the template formats used by the CloudFormation (CFN) tools. It also supports its own native template format, called HOT (“Heat Orchestration Templates”). In this article I will be using the HOT template syntax, which is fully specified on the OpenStack website.

A Python interface to signalfd() using FFI

Thu, 28 Nov 2013 00:00:00 +0000

I just recently learned about the signalfd(2) system call, which was introduced to the Linux kernel back in 2007:

signalfd() creates a file descriptor that can be used to accept signals targeted at the caller. This provides an alternative to the use of a signal handler or sigwaitinfo(2), and has the advantage that the file descriptor may be monitored by select(2), poll(2), and epoll(7).

The traditional asynchronous delivery mechanism can be tricky to get right, whereas this provides a convenient fd interface that integrates nicely with your existing event-based code.

Long polling with Javascript and Python

Sat, 23 Nov 2013 00:00:00 +0000

In this post I’m going to step through an example web chat system implemented in Python (with Bottle and gevent) that uses long polling to implement a simple publish/subscribe mechanism for efficiently updating connected clients.

My pubsub_example repository on GitHub has a complete project that implements the ideas discussed in this article. This project can be deployed directly on OpenShift if you want to try things out on your own. You can also try it out online at http://pubsub.example.oddbit.com/.

Sockets on OpenShift

Sat, 23 Nov 2013 00:00:00 +0000

In this article, a followup to my previous post regarding long-poll servers and Python, we investigate the code changes that were necessary to make the code work when deployed on OpenShift.

In the previous post, we implemented IO polling to watch for client disconnects at the same time we were waiting for messages on a message bus:

poll = zmq.Poller()
poll.register(subsock, zmq.POLLIN)
poll.register(rfile, zmq.POLLIN)

events = dict(poll.poll())

.
.
.

If you were to try this at home, you would find that everything worked as described…but if you were to deploy the same code to OpenShift, you would find that the problem we were trying to solve (the server holding file descriptors open after a client disconnected) would still exist.

A unified CLI for OpenStack

Fri, 22 Nov 2013 00:00:00 +0000

The python-openstackclient project, by Dean Troyer and others, is a new command line tool to replace the existing command line clients (including commands such as nova, keystone, cinder, etc).

This tool solves two problems I’ve encountered in the past:

Command line options between different command line clients are sometimes inconsistent.
The output from the legacy command line tools is not designed to be machine parse-able (and yet people do it anyway).

The new openstack CLI framework is implement using the cliff module for Python, which will help enforce a consistent interface to the various subcommands (because common options can be shared, and just having everything in the same codebase will help tremendously). Cliff also provides flexible table formatters. It includes a number of useful formatters out of the box, and can be extended via setuptools entry points.

Automatic maintenance of tag feeds

Fri, 22 Nov 2013 00:00:00 +0000

I recently added some scripts to automatically generate tag feeds for my blog when pushing new content. I’m using GitHub Pages to publish everything, so it seemed easiest to make tag generation part of a pre-push hook (new in Git 1.8.2). This hook is run automatically as part of the git push operation, so it’s the perfect place to insert generated content that must be kept in sync with posts on the blog.

Enabled blog comments

Mon, 18 Nov 2013 00:00:00 +0000

I’ve enabled blog comments using Disqus. This is something of an experiment, since (a) I’m not really happy with how Disqus is handling user registration these days and (b) I don’t know that I have the time to moderate anything. But we’ll see.

json-tools: cli for generating and filtering json

Sun, 17 Nov 2013 00:00:00 +0000

Interacting with JSON-based APIs from the command line can be difficult, and OpenStack is filled with REST APIs that consume or produce JSON. I’ve just put pair of tools for generating and filtering JSON on the command line, called collectively json-tools.

Both make use of the Python dpath module to populate or filter JSON objects.

The jsong command generates JSON on stdout. You provide /-delimited paths on the command line to represent the JSON structure. For example, if you run:

Quantum in Too Much Detail

Thu, 14 Nov 2013 00:00:00 +0000

I originally posted this article on the RDO website.

The players

This document describes the architecture that results from a particular OpenStack configuration, specifically:

Quantum networking using GRE tunnels;
A dedicated network controller;
A single instance running on a compute host

Much of the document will be relevant to other configurations, but details will vary based on your choice of layer 2 connectivity, number of running instances, and so forth.

The examples in this document were generated on a system with Quantum networking but will generally match what you see under Neutron as well, if you replace quantum by neutron in names. The OVS flow rules under Neutron are somewhat more complex and I will cover those in another post.

Moving to GitHub

Wed, 13 Nov 2013 00:00:00 +0000

This blog has been hosted on scriptogram for the past year or so. Unfortunately, while I like the publish-via-Dropbox mechanism, there have been enough problems recently that I’ve finally switched over to using GitHub Pages for hosting. I’ve been thinking about doing this for a while, but the things that finally pushed me to make the change were:

Sync problems that would prevent new posts from appearing (and that at least once caused posts to disappear).
Lack of any response to bug reports by the site maintainers.

A benefit of the publish-via-Dropbox mechanism is, of course, that I already had all the data and didn’t need to go through any sort of export process.

A random collection of OpenStack Tools

Tue, 12 Nov 2013 00:00:00 +0000

I’ve been working with OpenStack a lot recently, and I’ve ended up with a small collection of utilities that make my life easier. On the odd chance that they’ll make your life easier, too, I thought I’d hilight them here.

Crux

Crux is a tool for provisioning tenants, users, and roles in keystone. Instead of a sequence of keystone command, you can provision new tenants, users, and roles with a single comand.

Why does the Neutron documentation recommend three interfaces?

Mon, 28 Oct 2013 00:00:00 +0000

The documentation for configuring Neutron recommends that a network controller has three physical interfaces:

Before you start, set up a machine to be a dedicated network node. Dedicated network nodes should have the following NICs: the management NIC (called MGMT_INTERFACE), the data NIC (called DATA_INTERFACE), and the external NIC (called EXTERNAL_INTERFACE).

People occasionally ask, “why three interfaces? What if I only have two?”, so I wanted to provide an extended answer that might help people understand what the interfaces are for and what trade-offs are involved in using fewer interfaces.

Automatic hostname entries for libvirt domains

Fri, 04 Oct 2013 00:00:00 +0000

Have you ever wished that you could use libvirt domain names as hostnames? So that you could do something like this:

$ virt-install -n anewhost ...
$ ssh clouduser@anewhost

Since this is something that would certainly make my life convenient, I put together a small script called virt-hosts that makes this possible. You can find virt-hosts in my virt-utils GitHub repository:

https://raw.github.com/larsks/virt-utils/master/virt-hosts

Run by itself, with no options, virt-hosts will scan through your running domains for interfaces on the libvirt default network, look up those MAC addresses up in the corresponding default.leases file, and then generate a hosts file on stdout like this:

Interrupts on the PiFace

Mon, 05 Aug 2013 00:00:00 +0000

I recently acquired both a Raspberry Pi and a PiFace IO board. I had a rough time finding examples of how to read the input ports via interrupts (rather than periodically polling for values), especially for the newer versions of the PiFace python libraries.

After a little research, here’s some simple code that will print out pin names as you press the input buttons. Button 3 will cause the code to exit:

#!/usr/bin/python

import pifacecommon.core
import pifacecommon.interrupts
import os
import time

quit = False

def print_flag(event):
 print 'You pressed button', event.pin_num, '.'

def stop_listening(event):
 global quit
 quit = True

pifacecommon.core.init()

# GPIOB is the input ports, including the four buttons.
port = pifacecommon.core.GPIOB

listener = pifacecommon.interrupts.PortEventListener(port)

# set up listeners for all buttons
listener.register(0, pifacecommon.interrupts.IODIR_ON, print_flag)
listener.register(1, pifacecommon.interrupts.IODIR_ON, print_flag)
listener.register(2, pifacecommon.interrupts.IODIR_ON, print_flag)
listener.register(3, pifacecommon.interrupts.IODIR_ON, stop_listening)

# Start listening for events. This spawns a new thread.
listener.activate()

# Hang around until someone presses button 3.
while not quit:
 time.sleep(1)

print 'you pressed button 3 (quitting)'
listener.deactivate()

Generating a memberOf attribute for posixGroups

Mon, 22 Jul 2013 00:00:00 +0000

This showed up on #openstack earlier today:

2013-07-22T13:56:10 <m0zes> hello, all. I am looking to
setup keystone with an ldap backend. I need to filter
users based on group membership, in this case a
non-rfc2307 posixGroup. This means that memberOf doesn't
show up, and that the memberUid in the group is not a
dn. any thoughts on how to accomplish this?

It turns out that this is a not uncommon question, so I spent some time today working out a solution using the dynlist overlay for OpenLDAP.

Split concatenated certificates with awk

Tue, 16 Jul 2013 00:00:00 +0000

This is a short script that takes a list of concatenated certificates as input (such as a collection of CA certificates) and produces a collection of numbered files, each containing a single certificate.

#!/bin/awk -f
 
# This script expects a list of concatenated certificates on input and
# produces a collection of individual numbered files each containing
# a single certificate.
 
BEGIN {incert=0}
 
/-----BEGIN( TRUSTED)? CERTIFICATE-----/ {
certno++
certfile=sprintf("cert-%d.crt", certno)
incert=1
}
 
/-----END( TRUSTED)? CERTIFICATE-----/ {
print >> certfile
incert=0
}
 
incert==1 { print >> certfile }

Did Arch Linux eat your KVM?

Mon, 08 Apr 2013 00:00:00 +0000

A recent update to Arch Linux replaced the qemu-kvm package with an updated version of qemu. A side effect of this change is that the qemu-kvm binary is no longer available, and any libvirt guests on your system utilizing that binary will no longer operate. As is typical with Arch, there is no announcement about this incompatible change, and queries to #archlinux will be met with the knowledge, grace and decorum you would expect of that channel:

I2C on the Raspberry Pi

Tue, 12 Mar 2013 00:00:00 +0000

I’ve set up my Raspberry Pi to communicate with my Arduino via I2C. The Raspberry Pi is a 3.3v device and the Arduino is a 5v device. While in general this means that you need to use a level converter when connecting the two devices, you don’t need to use a level converter when connecting the Arduino to the Raspberry Pi via I2C.

The design of the I2C bus is such that the only device driving a voltage on the bus is the master (in this case, the Raspberry Pi), via pull-up resistors. So when “idle”, the bus is pulled to 3.3v volts by the Pi, which is perfectly safe for the Arduino (and compatible with it’s 5v signaling). To transmit data on the bus, a device brings the bus low by connecting it to ground. In other words, slave devices never drive the bus high. This means that the Raspberry Pi will never see a 5v signal from the Arduino…unless, of course, you make a mistake and accidentally digitalWrite a HIGH value on one of the Arduino’s I2C pins. So don’t do that.

Interrupt driven GPIO with Python

Fri, 08 Mar 2013 00:00:00 +0000

There are several Python libraries out there for interacting with the GPIO pins on a Raspberry Pi:

RPi.GPIO
The WiringPi bindings for Python, and
The Quick2Wire Python API (which depends on Python 3)

All of them are reasonably easy to use, but the Quick2Wire API provides a uniquely useful feature: epoll-enabled GPIO interrupts. This makes it trivial to write code that efficiently waits for and responds to things like button presses.

The following simple example waits for a button press attached to GPIO1 (but refer to the chart in this document to see exactly what that means; this is pin 12 on a Raspberry Pi v2 board) and lights an LED attached to GPIO0 when the button is pressed:

Controlling a servo with your Arduino

Thu, 07 Mar 2013 00:00:00 +0000

I’ve recently started playing with an Arduino kit I purchased a year ago (and only just now got around to unboxing). I purchased the kit from SparkFun, and it includes a motley collection of resistors, LEDs, a motor, a servo, and more.

I was fiddling around with this exercise, which uses the SoftwareServo library to control a servo. Using this library, you just pass it an angle and the library takes care of everything else, e.g. to rotate to 90 degrees you would do this:

A quote about XMLRPC

Mon, 25 Feb 2013 00:00:00 +0000

I’ve been reading up on Puppet 3 lately, and came across the following:

XMLRPC was the new hotness when development on Puppet started. Now, XMLRPC is that horrible thing with the XML and the angle brackets and the pain and sad.

(from http://somethingsinistral.net/blog/the-angry-guide-to-puppet-3/)

…which also accurately sums up my feelings when I come across yet another piece of software where someone has decided that XML (or even JSON) is a good user-facing configuration syntax.

A systemd unit for ucarp

Thu, 21 Feb 2013 00:00:00 +0000

In Fedora 17 there are still a number of services that either have not been ported over to systemd or that do not take full advantage of systemd. I’ve been investigating some IP failover solutions recently, including ucarp, which includes only a System-V style init script.

I’ve created a template service for ucarp that will let you start a specific virtual ip like this:

systemctl start ucarp@001

This will start ucarp using settings from /etc/ucarp/vip-001.conf. The unit file is on github and embedded here for your reading pleasure:

Running dhcpcd under LXC

Fri, 01 Feb 2013 00:00:00 +0000

I’ve been working with Arch Linux recently, which uses dhcpcd as its default DHCP agent. If you try booting Arch inside an LXC container, you will find that dhcpcd is unable to configure your network interfaces. Running it by hand you will first see the following error:

# dhcpcd eth0
dhcpcd[492]: version 5.6.4 starting
dhcpcd[492]: eth0: if_init: Read-only file system
dhcpcd[492]: eth0: interface not found or invalid

This happens because dhcpcd is trying to modify a sysctl value. Running dhcpcd under strace we see:

Cleaning up LXC cgroups

Mon, 28 Jan 2013 00:00:00 +0000

I spent some time today looking at systemd (44) under Fedora (17). When stopping an LXC container using lxc-stop, I would always encounter this problem:

# lxc-stop -n node0
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/systemd/node0

This prevents one from starting a new container with the same name:

# lxc-start -n node0 
lxc-start: Device or resource busy - failed to remove previous cgroup '/sys/fs/cgroup/systemd/node0'
lxc-start: failed to spawn 'node0'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/systemd/node0'

You can correct the problem manually by removing all the child cgroups underneath /sys/fs/cgroup/systemd/<container>, like this:

How do I LXC console?

Mon, 28 Jan 2013 00:00:00 +0000

It took me an unreasonably long time to boot an LXC container with working console access. For the record:

When you boot an LXC container, the console appears to be attached to a pts device. For example, when booting with the console attached to your current terminal:

# lxc-start -n node0
...
node0 login: root
Last login: Mon Jan 28 16:35:19 on tty1
[root@node0 ~]# tty
/dev/console
[root@node0 ~]# ls -l /dev/console
crw------- 1 root tty 136, 12 Jan 28 16:36 /dev/console

This is also true when you attach to a container using lxc-console:

Systemd and the case of the missing network

Mon, 28 Jan 2013 00:00:00 +0000

I was intrigued by this post on socket activated containers with systemd. The basic premise is:

systemd opens a socket on the host and listens for connections.
When a client connections, systemd spawns a new container.
The host systemd passes the connected socket to the container systemd.
Services in the container receive these sockets from the container systemd.

This is a very neat idea, since it delegates all the socket listening to the host and only spins up container and service resources when necessary.

A second look at Arch Linux

Fri, 25 Jan 2013 00:00:00 +0000

This is a followup to an earlier post about Arch Linux.

I’ve since spent a little more time working with Arch, and these are the things I like:

The base system is very small and has a very short boot time. I replaced Ubuntu on my old Eee PC with Arch and suddenly the boot time is reasonable (< 10 seconds to a text prompt, < 30 seconds to a full GUI login).

Parsing Libvirt XML with xmllint

Fri, 21 Dec 2012 00:00:00 +0000

I’ve been playing around with the LXC support in libvirt recently, and I’m trying to use a model where each LXC instance is backed by a dedicated LVM volume. This means that the process of starting an instance is:

mount the instance root filesystem if necessary
start the instance

It’s annoying to have to do this by hand. I could simply add all the LXC filesystems to /etc/fstab, but this would mean and extra step when creating and deleting each instance.

Getting the IP address of a libvirt domain

Sat, 15 Dec 2012 00:00:00 +0000

If you are starting virtual machines via libvirt, and you have attached them to the default network, there is a very simple method you can use to determine the address assigned to your running instance:

Libvirt runs dnsmasq for the default network, and saves leases in a local file (/var/lib/libvirt/dnsmasq/default.leases under RHEL).
You can get the MAC address assigned to a virtual machine by querying the domain XML description.

Putting this together gets us something along the lines of:

A first look at Arch Linux

Mon, 10 Dec 2012 00:00:00 +0000

I decided to take a look at Arch Linux this evening. It’s an interesting idea, but has a long way to go:

The installer configured the wrong root= command line into my syslinux configuration, resulting in a system that wouldn’t boot.

Update: As far as I can tell, the syslinux-install_update command doesn’t actually make any attempt to configure syslinux.cfg at all.
I tried to install libvirt and lxc, but there are unresolved library dependencies…the virsh command apparently requires libX11.so.6, but the package is missing the appropriate dependencies to pull in the necessary packages automatically.

Service discovery in the cloud using Avahi

Tue, 27 Nov 2012 00:00:00 +0000

I’m been writing a provisioning tool for OpenStack recently, and I’ve put together a demo configuration that installs a simple cluster consisting of three backend nodes and a front-end http proxy. I needed a way for the backend servers to discover the ip address of the frontend server. Since in my target environment everything would be on the same layer-2 network segment, service discovery with multicast DNS (mDNS) seemed like the way to go.

Using Oracle JDK under CentOS

Mon, 26 Nov 2012 00:00:00 +0000

I needed to replace the native OpenJDK based Java VM with the Oracle Java distribution on one of our CentOS servers. In order to do it cleanly I wanted to set up the alternatives system to handle it, but it took a while to figure out the exact syntax.

For the record (and because I will probably forget):

alternatives --install /usr/bin/java java /usr/java/latest/bin/java 2000 \
 --slave /usr/bin/keytool keytool /usr/java/latest/bin/keytool \
 --slave /usr/bin/rmiregistry rmiregistry /usr/java/latest/bin/rmiregistry

Document classification with POPFile

Thu, 08 Nov 2012 00:00:00 +0000

I recently embarked upon a quest to categorize a year’s worth of trouble tickets (around 15000 documents total). We wanted to see what sort of things are generating the most work for our helpdesk staff so that we can identify areas in which improvements would have the biggest impact. One of my colleagues took a first pass at the data by manually categorizing the tickets based on their subject. This resulted in some useful data, but in the end just over 40% of the tickets are still uncategorized.

Converting HTML to Markdown

Tue, 06 Nov 2012 00:00:00 +0000

In order to import posts from Blogger into Scriptogr.am I needed to convert all the HTML formatting into Markdown. Thankfully there are a number of tools out there that can help with this task.

MarkdownRules. This is an online service build around Markdownify. It’s a slick site with a nice API, but the backend wasn’t able to correctly render <pre> blocks. Since I’m often writing about code, my posts are filled with things like embedded XML and #include <stdio.h>, so this was a problem.

Relocating from Blogger

Tue, 06 Nov 2012 00:00:00 +0000

I’m in the process of porting over content from Blogger. This may lead to odd formatting or broken links here and there. If you spot something, please let me know.

If you came here from Google and found a broken link, try starting at the archive and see if you can spot what you were looking for.

Posting to Scriptogr.am using the API

Mon, 05 Nov 2012 00:00:00 +0000

Scriptogr.am has a very simple api that allows one to POST and DELETE articles. POSTing an article will place it in the appropriate Dropbox directory and make it available on your blog all in one step.

Here is how you could use this API via Curl:

curl \
 -d app_key=$APP_KEY \
 -d user_id=$USER_ID \
 -d name="${title:-$1}" \
 --data-urlencode text@$tmpfile \
 \
 http://scriptogr.am/api/article/post/

This assumes that you’ve registered for an application key and that you have configured the value into $APP_KEY and your Scriptogr.am user id into $USER_ID.

Private /tmp directories in Fedora

Mon, 05 Nov 2012 00:00:00 +0000

I ran into an odd problem the other day: I was testing out some configuration changes for a web application by dropping files into /tmp and pointing the application configuration at the appropriate directory. Everything worked out great when testing it by hand…but when starting up the httpd service, the application behaved as if it was unable to find any of the files in /tmp.

My first assumption was that had simply missed something obvious like file permissions or that I had a typo in my configuration, but after repeated checks and lots of testing it was obvious that something else was going on.

Automatic configuration of Windows instances in OpenStack, part 1

Sun, 04 Nov 2012 00:00:00 +0000

This is the first of two articles in which I discuss my work in getting some Windows instances up and running in our OpenStack environment. This article is primarily about problems I encountered along the way.

Motivations

Like many organizations, we have a mix of Linux and Windows in our environment. Some folks in my group felt that it would be nice to let our Windows admins take advantage of OpenStack for prototyping and sandboxing in the same ways our Linux admins can use it.

Generating random passwords in PowerShell

Sun, 04 Nov 2012 00:00:00 +0000

I was looking for PowerShell solutions for generating a random password (in order to set the Administrator password on a Windows instance provisioned in OpenStack), and found several solutions using the GeneratePassword method of System.Web.Security.Membership (documentation here), along the lines of this:

Function New-RandomComplexPassword ($length=8)
{
 $Assembly = Add-Type -AssemblyName System.Web
 $password = [System.Web.Security.Membership]::GeneratePassword($length,2)
 return $password
}

While this works, I was unhappy with the generated passwords: they were difficult to type or transcribe because they make heavy use of punctuation. For example:

Waiting for networking using PowerShell

Sun, 04 Nov 2012 00:00:00 +0000

I’ve recently been exploring the world of Windows scripting, and I ran into a small problem: I was running a script at system startup, and the script was running before the network interface (which was using DHCP) was configured.

There are a number of common solutions proposed to this problem:

Just wait for some period of time.

This can work but it’s ugly, and because it doesn’t actually verify the network state it can result in things breaking if some problem prevents Windows from pulling a valid DHCP lease.

Growing a filesystem on a virtual disk

Wed, 24 Oct 2012 00:00:00 +0000

Occasionally we will deploy a virtual instance into our KVM infrastructure and realize after the fact that we need more local disk space available. This is the process we use to expand the disk image. This process assumes the following:

You’re using legacy disk partitions. The process for LVM is similar and I will describe that in another post (it’s generally identical except for an additional pvresize thrown in and lvextend in place of resize2fs).
The partition you need to resize is the last partition on the disk.

This process will work with either a qcow2 or raw disk image. For raw images you can also run fdisk on the host, potentially saving yourself a reboot, but that’s less convenient for qcow2 format images.

Parsing XML with Awk

Mon, 10 Sep 2012 00:00:00 +0000

Recently, changes from the xmlgawk project have been integrated into GNU awk, and xmlgawk has been renamed to gawkextlib. With both a recent (post-4.0.70) gawk and gawkextlib built and installed correctly, you can write simple XML parsing scripts using gawk.

For example, let’s say you would like to generate a list of disk image files associated with a KVM virtual instance. You can use the virsh dumpxml command to get output like the following:

Markdown in your Email

Thu, 09 Aug 2012 00:00:00 +0000

I really like Markdown, a minimal markup language designed to be readable as plain text that can be rendered into structurally valid HTML. Markdown is already used on sites such as GitHub and all the StackExchange sites.

I use Markdown often enough that it’s become ingrained in my fingers, to the point that I’ve started unconsciously using Markdown syntax in my email. This isn’t particularly useful by itself, although it means that I can take a message and render it to something pretty if I decide it needs to go somewhere other than my sent mail folder.

Chasing OpenStack idle connection timeouts

Mon, 30 Jul 2012 00:00:00 +0000

The original problem

I’ve recently spent some time working on an OpenStack deployment. I ran into a problem in which the compute service would frequently stop communicating with the AMQP message broker (qpidd).

In order to gather some data on the problem, I ran the following simple test:

Wait n minutes
Run nova boot ... to create an instance
Wait a minute and see if the new instance becomes ACTIVE
If it works, delete the instance, set n = 2n and repeat

This demonstrated that communication was failing after about an hour, which correlates rather nicely with the idle connection timeout on the firewall.

Git fetch, tags, remotes, and more

Fri, 27 Jul 2012 00:00:00 +0000

I’ve been playing around with Git, Puppet, and GPG verification of our Puppet configuration repository, and these are some random facts about Git that have come to light as part of the process.

If you want to pull both changes and new tags from a remote repository, you can do this:

$ git fetch
$ git fetch --tags

Or you can do this:

$ git fetch --tags
$ git fetch

What’s the difference? git fetch will leave FETCH_HEAD pointing at the remote HEAD, whereas git fetch --tags will leave FETCH_HEAD pointing at the most recent tag.

Capturing Envoy Data

Wed, 22 Feb 2012 00:00:00 +0000

Pursuant to my last post, I’ve written a simple man-in-the-middle proxy to intercept communication between the Envoy and the Enphase servers. The code is available here.

What it does

As I detailed in my previous post, the Envoy sends data to Enphase via http POST requests. The proxy intercepts these requests, extracts the XML data from the request, and writes it to a local file (by default in /var/spool/envoy). It then forwards the request on to Enphase, and returns the reply to your Envoy.

Enphase Envoy XML Data Format

Mon, 13 Feb 2012 00:00:00 +0000

We recently installed a (photovoltaic) solar array on our house. The system uses Enphase microinverters, and includes a monitoring device called the “Envoy”. The Envoy collects data from the microinverters and sends it back to Enphase. Enphase performs monitoring services for the array and also provides access to the data collected by the Envoy product.

I’m interested in getting direct access to the data provided by the Envoy. In pursuit of that goal, I set up a man-in-the-middle proxy server on my home network to intercept the communication from the Envoy to the Enphase servers. I’m documenting the results of my exploration here in case somebody else finds the information useful.

Rate limiting made simple

Mon, 26 Dec 2011 00:00:00 +0000

I use CrashPlan as a backup service. It works and is very simple to set up, but has limited options for controlling bandwidth. In fact, if you’re running it on a headless system (e.g., a fileserver of some sort), your options are effectively “too slow” and “CONSUME EVERYTHING”. There is an open request to add time-based limitations to the application itself, but for now I’ve solved this using a very simple traffic shaping configuration. Because the learning curve for “tc” and friends is surprisingly high, I’m putting my script here in the hopes that other people might find it useful, and so that I can find it when I need to do this again someday.

Puppet, scope, and inheritance

Tue, 16 Aug 2011 00:00:00 +0000

I note this here because it wasn’t apparent to me from the Puppet documentation.

If you have a Puppet class like this:

class foo {
 File { ensure => file,
 mode => 600,
 }
}

And you use it like this:

class bar {
 include foo

 file { '/tmp/myfile': }
}

Then /tmp/myfile will not be created. But if instead you do this:

class bar inherits foo {
 file { '/tmp/myfile': }
}

It will be created with mode 0600. In other words, if you use inherits then definitions in the parent class are available in the scope of your subclass. If you include, then definitions in he included class are “below” the scope of the including class.

Fixing RPM with evil magic

Tue, 26 Jul 2011 00:00:00 +0000

Fixing rpmsign with evil magic

At my office we are developing a deployment mechanism for RPM packages. The general workflow looks like this:

You build a source rpm on your own machine.
You sign the rpm with your GPG key.
You submit the source RPM to our buildserver.
The buildserver validates your signature and then builds the package.
The buildserver signs the package using a master signing key.

The last step in that sequence represents a problem, because the rpmsign command will always, always prompt for a password and read the response from /dev/tty. This means that (a) you can’t easily provide the password on stdin, and (b) you can’t fix the problem using a passwordless key.

Installing CrashPlan under FreeBSD 8

Sun, 22 May 2011 00:00:00 +0000

This articles describes how I got CrashPlan running on my FreeBSD 8(-STABLE) system. These instructions by Kim Scarborough were my starting point, but as these were for FreeBSD 7 there were some additional steps necessary to get things working.

Install Java

I had originally thought that it might be possible to run the CrashPlan client “natively” under FreeBSD. CrashPlan is a Java application, so this seemed like a possible solution. Unfortunately, Java under FreeBSD 8 seems to be a lost cause. I finally gave up and just installed Java under Linux.

Signing data with ssh-agent

Mon, 09 May 2011 00:00:00 +0000

This is follow-up to my previous post, Converting OpenSSH public keys.

OpenSSH allows one to use an agent that acts as a proxy to your private key. When using an agent – particularly with agent forwarding enabled – this allows you to authenticate to a remote host without having to (a) repeatedly type in your password or (b) expose an unencrypted private key to remote systems.

If one is temtped to use SSH keys as authentication credentials outside of ssh, one would ideally be able to take advantage of the ssh agent for these same reasons.

Converting OpenSSH public keys

Sun, 08 May 2011 00:00:00 +0000

I’ve posted a followup to this article that discusses ssh-agent.

For reasons best left to another post, I wanted to convert an SSH public key into a PKCS#1 PEM-encoded public key. That is, I wanted to go from this:

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD7EZn/BzP26AWk/Ts2ymjpTXuXRiEWIWn
HFTilOTcuJ/P1HfOwiy4RHC1rv59Yh/E6jbTx623+OGySJWh1IS3dAEaHhcGKnJaikrBn3c
cdoNVkAAuL/YD7FMG1Z0SjtcZS6MoO8Lb9pkq6R+Ok6JQjwCEsB+OaVwP9RnVA+HSYeyCVE
0KakLCbBJcD1U2aHP4+IH4OaXhZacpb9Ueja6NNfGrv558xTgfZ+fLdJ7cpg6wU8UZnVM1B
JiUW5KFasc+2IuZR0+g/oJXaYwvW2T6XsMgipetCEtQoMAJ4zmugzHSQuFRYHw/7S6PUI2U
03glFmULvEV+qIxsVFT1ng3pj lars@tiamat.house

To this:

If you have a recent version of OpenSSH (where recent means 5.6 or later), you can just do this:

Python ctypes module

Tue, 10 Aug 2010 00:00:00 +0000

I just learned about the Python ctypes module, which is a Python module for interfacing with C code. Among other things, ctypes lets you call arbitrary functions in shared libraries. This is, from my perspective, some very cool magic. I thought I would provide a short example here, since it took me a little time to get everything working smoothly.

For this example, we’ll write a wrapper for the standard statvfs(2) function:

Importing vCard contacts into an LG 420G

Fri, 06 Aug 2010 00:00:00 +0000

Alix recently acquired an LG 420G from TracFone. She was interested in getting all of her contacts onto the phone, which at first seemed like a simple task – transfer a vCard (.vcf) file to the phone via Bluetooth, and the phone would import all the contacts. This turned out to be a great idea in theory, but in practice there was a fatal flaw – while the phone did indeed import the contacts, it only imported names and the occasional note or email address. There were no phone numbers.

Patch to gPXE dhcp command

Thu, 22 Jul 2010 00:00:00 +0000

Update: This patch has been accepted into gPXE.

I just released a patch to gPXE that modifies the dhcp command so that it can iterate over multiple interfaces. The stock dhcp command only accepts a single interface as an argument, which can be a problem if you are trying to boot on a machine with multiple interfaces. The builtin autoboot commands attempts to resolve this, but is only useful if you expect to receive appropriate boot parameters from your dhcp server.

Kerberos authenticated queries to Active Directory

Tue, 29 Jun 2010 00:00:00 +0000

There are many guides out there to help you configure your Linux system as an LDAP and Kerberos client to an Active Directory server. Most of these guides solve the problem of authentication by embedding a username and password into a configuration file somewhere on your system. While this works, it presents some problems:

If you use a common account for authentication from all of your Linux systems, a compromise on one system means updating the configuration of all of your systems.
If you don’t want to use a common account, you need to provision a new account for each computer…
…which is silly, because if you join the system to Active Directory there is already a computer object associated with the system that can be used for authentication.

This document describes how to configure a Linux system such that queries generated by nss_ldap will use either the current user’s Kerberos credentials, or, for the root user, credentials stored in a Kerberos credentials cache.

Pushing a Git repository to Subversion

Tue, 11 May 2010 00:00:00 +0000

I recently set up a git repository server (using gitosis and gitweb). Among the required features of the system was the ability to publish the git repository to a read-only Subversion repository. This sounds simple in principle but in practice proved to be a bit tricky.

Git makes an excellent Subversion client. You can use the git svn … series of commands to pull a remote Subversion repository into a local git working tree and then have all the local advantages of git forcing the central code repository to change version control software. An important aspect of this model is that:

LDAP redundancy through proxy servers

Wed, 24 Feb 2010 00:00:00 +0000

Problem 1: Failover

The problem

Many applications only allow you to configure a single LDAP server. This can lead to unnecessary service outages if your directory service infrastructure is highly available (e.g., you are running Active Directory) and your application cannot take advantage of this fact.

A solution

We can provide a level of redundancy by passing the LDAP connections through a load balancing proxy. While this makes the proxy a single point of failure, it is (a) a very simple tool and thus less prone to complex failure modes, (b) running on the same host as the web application, and (c) is completely under our control.

Apache virtual host statistics

Fri, 19 Feb 2010 00:00:00 +0000

As part of a project I’m working on I wanted to get a rough idea of the activity of the Apache virtual hosts on the system. I wasn’t able to find exactly what I wanted, so I refreshed my memory of curses to bring you vhoststats.

This tools reads an Apache log file (with support for arbitrary formats) and generates a dynamic bar chart showing the activity (in number of requests and bytes transferred) of hosts on the system. The output might look something like this (but with colors):

Merging directories with OpenLDAP's Meta backend

Tue, 16 Feb 2010 00:00:00 +0000

This document provides an example of using OpenLDAP’s meta backend to provide a unified view of two distinct LDAP directory trees. I was frustrated by the lack of simple examples available when I went looking for information on this topic, so this is my attempt to make life easier for the next person looking to do the same thing.

The particular use case that motiviated my interest in this topic was the need to configure web applications to (a) authenticate against an existing Active Directory server while (b) also allowing new accounts to be provisioned quickly and without granting any access in the AD environment. A complicating factor is that the group managing the AD server(s) was not the group implementing the web applications.

Filtering Blogger feeds

Wed, 10 Feb 2010 00:00:00 +0000

After encountering a number of problems trying to filter Blogger feeds by tag (using services like Feedrinse and Yahoo Pipes), I’ve finally put together a solution that works:

Shadow the feed with Feedburner.
Enable the Convert Format Burner, and convert your feed to RSS 2.0.
Use Yahoo Pipes to filter the feed (because Feedrinse seems to be broken).

This let me create a feed that excluded all my posts containing the fbpost tag, thus allowing me to avoid yet another postgasm in Facebook when adding new import URL to notes.

Funny usage message

Mon, 08 Feb 2010 00:00:00 +0000

I was poking around in a command shell on my Droid to see what was available. While it’s a pretty restricted environment, there’s a number of commands available in /system/bin, including dexopt.

Apparently dexopt isn’t something I’m supposed to poke at:

$ dexopt
Usage: don't use this

Hah.

MBTA realtime XML feed

Sun, 07 Feb 2010 00:00:00 +0000

The MBTA has a trial web service interface that provides access to realtime location information for select MBTA buses, as well as access to route information, arrival prediction, and other features. More information can be found here:

http://www.eot.state.ma.us/developers/realtime/

The service is provided by NextBus, which specializes in real-time location information for public transit organizations. The API (sorry, PDF) is very simple and does not require any sort of advance registration.

At the moment, the service only provides coverage for a small number of routes (39, 111, 114, 116, 117). I hope they expand the coverage of this service in the near future!

Blocking VNC with iptables

Thu, 04 Feb 2010 00:00:00 +0000

VNC clients use the RFB protocol to provide virtual display capabilities. The RFB protocol, as implemented by most clients, provides very poor authentication options. While passwords are not actually sent “in the clear”, it is possible to brute force them based on information available on the wire. The RFB 3.x protocol limits passwords to a maximum of eight characters, so the potential key space is relatively small.

It’s possible to securely connect to a remote VNC server by tunneling your connection using ssh port forwarding (or setting up some sort of SSL proxy). However, while this ameliorates the password problem, it still leaves a VNC server running that, depending on the local system configuration, may accept connections from all over the world. This leaves open the possibility that someone could brute force the password and gain access to the systsem. The problem is exacerbated if a user is running a passwordless VNC session.

NFS and the 16-group limit

Tue, 02 Feb 2010 00:00:00 +0000

I learned something new today: it appears that the underlying authorization mechanism used by NFS limits your group membership to 16 groups. From http://bit.ly/cBhU8N:

NFS is built on ONC RPC (Sun RPC). NFS depends on RPC for authentication and identification of users. Most NFS deployments use an RPC authentication flavor called AUTH_SYS (originally called AUTH_UNIX, but renamed to AUTH_SYS).

AUTH_SYS sends 3 important things:

A 32 bit numeric user identifier (what you’d see in the UNIX /etc/passwd file)

A 32 bit primary numeric group identifier (ditto)

A variable length list of up to 16 32-bit numeric supplemental group identifiers (what’d you see in the /etc/group file)

We ran into this today while diagnosing a weird permissions issue. Who knew?

Cleaning up Subversion with Git

Fri, 29 Jan 2010 00:00:00 +0000

Overview

At my office, we have a crufty Subversion repository (dating back to early 2006) that contains a jumble of unrelated projects. We would like to split this single repository up into a number of smaller repositories, each following the recommended trunk/tags/branches repository organization.

What we want to do is move a project from a path that looks like this:

.../projects/some-project-name

To a new repository using the recommended Subversion repository layout, like this:

Linux UPnP Gateway

Fri, 29 Jan 2010 00:00:00 +0000

Like many other folks out there, I have several computers in my house connected to the outside world via a Linux box acting as a NAT gateway. I often want to use application such as BitTorrent and Freenet, which require that a number of ports be forwarded from my external connection to the particular computer on which I happen to be working. It turns out there’s a protocol for this, called UPnP. From Wikipedia:

Retrieving Blogger posts by post id

Fri, 29 Jan 2010 00:00:00 +0000

I spent some time recently trying to figure out, using Google’s gdata API, how to retrieve a post from a Blogger blog if I know corresponding post id. As far as I can tell there is no obvious way of doing this, at least not using the gdata.blogger.client api, but after much nashing of teeth I came up with the following solution.

Given client, a gdata.blogger.client instance, and blog, a gdata.blogger.data.Blog instance, the following code will return a gdata.blogger.data.BlogPost instance:

Fring: How not to handle registration

Sun, 24 Jan 2010 00:00:00 +0000

I thought I’d give Fring a try after seeing some favorable reviews on other sites. If you haven’t previously heard of Fring, the following blurb from their website might be helpful:

Using your handset’s internet connection, you can interact with friends on all your favourite social networks including Skype, MSN Messenger, Google Talk, ICQ, SIP, Twitter, Yahoo! and AIM. You can listen to music with your Last.fm friends, check out what each other are up to on Facebook, receive alerts of new Google Mail and tailor make your very own fring by adding more cool experiences from fringAdd-ons