• rather than naked shell, try tmux for an organized collection of shells
• rather than local file system navigation, try fzf to fuzzily get what you want
• rather than cd, try autojump, which remembers your favorite locations
• rather than find, try fd, which has a much cleaner interface
• rather than grep, try ag, which is simpler and faster

Introduction

The built-in shell environment is tremendously productive, but “second generation” tools can make it even more so.

If you want to try out these tools, you can run

docker run -it quay.io/matsen/cozy-demo

to test them out in Docker. There is some funkiness with the display on this Docker image (tmux pane splits use hashes and q’s) but it’s enough to get the idea. See the corresponding GitHub repository if you want to mimic the configuration.

tmux

When working on a modern desktop computer, it’s easy to arrange multiple windows side by side, to switch between applications, etc. On the command line this is achieved by use of a “terminal multiplexer”. This is especially lovely for working on remote machines, where one can detach and re-attach sessions with all their attendant windows.

The two most popular terminal multiplexers are GNU Screen and tmux. This tutorial will demonstrate tmux, which is newer and more feature-rich. Note that the example commands below use the configuration described in the above-mentioned GitHub repository. This configuration is vi-keybinding oriented (emacs-style keybindings are also available).

You send commands to these programs using a “prefix” key command followed by another keystroke. In our configuration, the prefix is Ctrl-a, the most easily typed control combination. For example, hitting Ctrl-a, letting go, then typing c opens a new terminal window. We will abbreviate this combination Ctrl-a c.

First steps in tmux

As a first step, I suggest just starting up tmux by executing tmux and trying out the following commands. Open a few windows, type some commands, cycle through them, open a few panes in a single window, and move around through them.

• Ctrl-a c   New window
• Ctrl-a <Space>   Next window
• Ctrl-a NUMBER   Move to window number NUMBER
• Ctrl-a Ctrl-a   Move to the previously visited window
• Ctrl-a v   Split window into panes vertically
• Ctrl-a s   Split window into panes horizontally
• Ctrl-a <arrow>   Move between panes
• Ctrl-d   Close a pane or window (works outside of tmux)
• Ctrl-a q   Kill a pane or window

Hopefully it’s clear that the windows are complete environments which are indexed at the bottom, and the panes are sub-windows that have no associated indicator. Each pane has a shell running inside of it.

The following commands are also essential:

• Ctrl-a ?   Command help. Note that / will allow you to search.
• Ctrl-a a   Actually send a Ctrl-a to the terminal (e.g. to move to the beginning of the line when editing a shell command)

Detaching and attaching a session

Tmux can keep your window configuration and all of the running processes going, even if your terminal goes away. This is called detaching and reattaching.

The polite way to detach and reattach is to (try this!)

• Ctrl-a d   Detach
• tmux attach   Reattach

Reattaching is also very handy if you are moving between laptops. You can detach your remote session from one, log in on another, and then reattach on the second laptop. Note that if you have different size terminal windows between the two machines the session can look wonky. In that case simply use tmux attach -d, which will “steal” the session from any other login shell and redraw the session from scratch.

If your wifi drops you might just get disconnected to a remote machine, in which case you wouldn’t have the opportunity to politely detach. That’s fine. In that case, just re-connect to your remote machine, and tmux attach.

Also note that tmux attach will fail if you don’t have a session open already; if that happens just enter tmux to start a new session.

Resizing panes

• Ctrl-a <   Move vertical split left
• Ctrl-a >   Move vertical split right
• Ctrl-a +   Move horizontal split up
• Ctrl-a =   Move horizontal split down

Note: You can hit Ctrl-a once, and press the resize operator a number of times (or hold it) to do larger resizes. For example, try Ctrl-a <<<<.

Naming & finding windows

• Ctrl-a ,   Name a window
• Ctrl-a '   Present a list of windows (by name), which you can choose from by <arrow> and Enter to switch to a window
• Ctrl-a <numeric>   Switch to a window by number.

Scrolling and copy/paste in tmux

When you have lots of output, it’s nice to be able to scroll up and down through history. Pressing Ctrl-a [ will place you in scroll mode. Use can now use arrow keys, PageUp/PageDown keys, or Ctrl-u/Ctrl-d to scroll through the history. You can search up through history with ? and down with /.

From this mode, you can also press Space to enter copy-mode, arrow keys to specify a selection range, and Enter to copy the selection. To paste the selection, use Ctrl-a ].

Note: if you are running tmux 2.1 or greater (check with tmux -V) and you really like your mouse scroll wheel, you can try set -g mouse on which will drop you into tmux scroll mode when you use the scroll wheel. But I suggest keeping your fingers on the keyboard and ignoring the “rat”.

Advanced tmux tips

You can also use h, j, k, and l in place of the arrow keys, as in vim.

If you want to pipe to the tmux copy/paste buffer, try this alias:

alias tc='tmux loadb -'

With this alias I can, for example, ls /very/long/path.txt | tc and then paste rather than having to do an explicit select and copy.

There are many more commands than we have connected with command sequences. See the tmux man page for a complete list. Ctrl-a : will put you at a prompt that will allow you to execute an arbitrary command.

Using tmux with ssh

Ssh keys in a long-running tmux session require some care. I use keychain to automate ssh-agent and friends together with tmux like so:

eval `keychain --eval id_rsa` && tmux attach -d

Sometimes it seems that ssh connections inside those sessions need a little refresh. For that I use this shell command.

You can get fancy with tmux

Some people love complex tmux setups with tons of sub-panes with things running in them. I prefer to have a simple setup where things can get set up and torn down easily. If you like a fancy setup, you can use something like tmux-resurrect to maintain such setups on your local machine between reboots. There are also tmux plugin managers, etc, but just a simple configuration file is all I’ve needed.

You can also have lots of different detached sessions, give them names, etc. I used to do this, with each session corresponding to a single project. For me it ended up being too much work to manage, but perhaps it fits for you.

fzf

We all love tab completion for matching files. But sometimes it too can be painful. Consider this list of files, which is some the various posts that have appeared on fredhutch.io:

2014-04-24-command-line.md        2015-04-06-R-sequence-analysis.md
2014-04-27-terminal-multiplex.md  2015-04-23-ucsc-xena-workshop.md
2014-05-09-galaxy.md              2015-04-24-third-galaxy-101.md
2014-05-11-editing.md             2015-05-04-spring-2015-r.md
2014-05-17-git.md                 2015-05-20-spring-2015-unix-hpc.md
2014-05-20-R.md                   2015-06-03-summer-bioinfo.md
2014-07-13-toolbox.md             2015-08-26-data-center-tour.md
2014-07-16-aws.md                 2015-08-27-cloud-computing.md
2014-08-13-synapse.md             2015-12-02-galaxy-rna-seq.md
2014-09-19-R-course.md            2016-01-21-cds-git.md
2014-10-20-introductory-r-.md     2016-02-25-immunespace.md
2014-10-22-shippable.md           2016-03-01-i-heart-pandas.md
2014-10-27-bioconductor.md        2016-03-28-gizmo-brownbag.md
2014-11-03-labkey.md              2016-06-14-scicomp-unix-hpc.md
2014-11-07-intermediate-R.md      2016-07-28-galaxy-101.md
2014-12-09-hidra.md               2016-08-23-chipseq-class.md
2014-12-15-scicomp-unix-hpc.md    2016-08-31-rnaseq-class.md
2015-02-11-scicomp-unix-hpc.md    2016-09-08-inkscape.md
2015-03-09-introductory-R.md      2016-09-27-intro-r.md
2015-03-12-rollout-galaxy-101.md  2016-10-03-introbio.md
2015-04-02-april-galaxy-101.md    2017-01-05-command-line-cozy.md

to tab-complete through these files is a pain, because I actually want to choose a file by the text portion of the descriptor, which is past the date.

What we really want is fuzzy finding, which allows us to match via partial substring matches. By this I mean that arbitrary substrings of your query string get matched with arbitrary substrings of your target strings, in an ordered fashion (see below for an example). This is available as part of the GitHub web interface for finding files, and actually even in the Chrome address bar.

fzf brings fuzzy finding to the shell. For example, if I want the posts from 2015 that contain galaxy, I could type 2015galaxy, and 💥:

  2015-03-12-rollout-galaxy-101.md
  2015-04-24-third-galaxy-101.md
  2015-04-02-april-galaxy-101.md
  2014-05-09-galaxy.md
> 2015-12-02-galaxy-rna-seq.md
  5/41
> 2015galaxy

we get a list of the files that contain 201 5 and galaxy. In fact, I could have typed 15axy and gotten the same result, because it matches the same set of files. I can then type more characters to limit the search, or use the arrow keys to move between results, and then return to select. Note that there is a “false positive” here, namely 2014-05-09-galaxy.md which is from 2014 but has a 5 in the month. That’s just a natural consequence of the way fuzzy finding works.

Here are two very easy ways to get started with fzf from the shell:

• Ctrl-t drops you into the fzf fuzzy command completer from the shell prompt. For example, less Ctrl-t will allow you to find the file that you want to less.
• Ctrl-r, the usual reverse search on your history, gets replaced by the fuzzified version. This is great and allows you to be vague about what you’re looking for.

For many more uses, including vim integration, see the fzf GitHub page.

autojump

autojump is a cd replacement that learns. It keeps a record of where you have been in the past, and given multiple directories that match an argument, chooses the one that you have been in the most. The autojump command is j.

For example, I frequently visit our data directory on the shared filesystem at /fh/fast/matsen_e/data. Instead of typing out that full filename I can just type j data to cd there.

This works with partial filenames as well. In the example Docker container I have a few directories that have cats in them, and j cats will take you to /root/cats/siamese-mostpopular, which is the most popular subdirectory containing cats.

If you are in the most popular directory with a certain string, then running the same j command will take you to the second most popular. For example, repeating j cats a second time would take us to /root/cats/bengal, which is the subdirectory we’ve visited the second most. (We could have used cat here as the argument to j instead of cats but I didn’t want to confuse things with the cat shell command.)

The information about what directories you spend the most time in is stored in ~/.local/share/autojump/autojump.txt. For our example Docker image it looks like this:

50.0	/root/cats/siamese-mostpopular
30.0	/root/cats/siamese-lesspopular
40.0	/root/cats/bengal
10.0	/root/work

The left-hand column here is the popularity index and the right hand is the directory.

fd

GNU Find is incredibly powerful, but has an annoying interface. The fd command fixes that.

• fd PATTERN finds a file in the current directory whose filename matches PATTERN
• fd PATTERN -x COMMAND executes COMMAND for each file matching PATTERN

For example, fd .txt -x head -n 1 will perform head -n 1 on each file matching .txt.

The fd command has sensible defaults for the modern environment, such as ignoring .git directories and files that appear in .gitignore. See the fd GitHub page for more examples and detail.

ag

ag is a faster replacement for grep that has a nicer input and output interface.

For example, as I’m writing this I might look for instances of the word “fancy” in the repository for this blog. Using ag fancy gets me:

_posts/2019-11-05-travis.md
22:It can do all sorts of fancy pipelines, but here we'll just get it to do one simple thing: run a command in a Docker container.

_posts/2019-12-31-command-line-cozy.md
145:### You can get fancy with tmux
149:If you like a fancy setup, you can use something like [tmux-resurrect](https://github.com/tmux-plugins/tmux-resurrect) to maintain such setups on your local machine between reboots.

which shows clearly that this word appeared once in a post about Travis, and twice in this file (before I wrote this section).

In contrast, we can try the same thing using grep -r fancy

_posts/2019-11-05-travis.md:It can do all sorts of fancy pipelines, but here we'll just get it to do one simple thing: run a command in a Docker container.
_posts/2019-12-31-command-line-cozy.md:### You can get fancy with tmux
_posts/2019-12-31-command-line-cozy.md:If you like a fancy setup, you can use something like [tmux-resurrect](https://github.com/tmux-plugins/tmux-resurrect) to maintain such setups on your local machine between reboots.
_site/feed.xml:<h3 id="you-can-get-fancy-with-tmux">You can get fancy with tmux</h3>
_site/feed.xml:If you like a fancy setup, you can use something like <a href="https://github.com/tmux-plugins/tmux-resurrect">tmux-resurrect</a> to maintain such setups on your local machine between reboots.
_site/feed.xml:It can do all sorts of fancy pipelines, but here we’ll just get it to do one simple thing: run a command in a Docker container.</p>
_site/2019/11/05/travis.html:It can do all sorts of fancy pipelines, but here we’ll just get it to do one simple thing: run a command in a Docker container.</p>
_site/articles/2019/12/31/command-line-cozy.html:<h3 id="you-can-get-fancy-with-tmux">You can get fancy with tmux</h3>
_site/articles/2019/12/31/command-line-cozy.html:If you like a fancy setup, you can use something like <a href="https://github.com/tmux-plugins/tmux-resurrect">tmux-resurrect</a> to maintain such setups on your local machine between reboots.

First, things aren’t as nicely organized into sections. Second, grep looks into all of the files, including the ones that are in .gitignore. The git-ignored files in _site are auto-generated and they only add noise to my search.

ag is also very fast and has lots of nice editor integrations. See the ag GitHub page for more details.

Conclusion and future directions

We’ve looked at a few tools that can help make it easier to interact with the shell.

We haven’t considered the shell itself. If you spend a lot of time in the shell, you might consider zsh or fish.

Or you could just be happy with the defaults. Every minute you spend tweaking your configuration is a minute you aren’t doing creative and beautiful things.

Thank you to Chris Small and Connor McCoy, two previous group members who pointed me to some of these tools. Chris wrote parts of the section on tmux as part of his course material for fredhutch.io.

• you have a project on GitHub which contains a Dockerfile specifying how to install your project’s dependencies into a Docker image
• you have some means of building and testing that project within the corresponding Docker container
• you would like this build/test to happen automatically on push.

For background on how to set up the first two steps, see a previous post about Docker.

Travis

Travis is an industry-standard continuous-integration (CI) service that is nicely integrated with GitHub. It can do all sorts of fancy pipelines, but here we’ll just get it to do one simple thing: run a command in a Docker container.

To demonstrate we’ll use the same cowsay-test repository as in the previous post. This time let’s look at the .travis.yml file in that repository, which is in YAML format:

language: minimal

services:
  - docker

script:
  - docker run -t matsen/cowsay-test /bin/bash -c "cowsay hello"

Travis has all sorts of preconfigured machines to run code in a variety of languages. In the first line, we are specifying that we want to use a “minimal” machine, which isn’t designed for a specific language. This image just has a collection of standard tools, including Docker. The second block specifies that we would like to use Docker.

The third and final block specifies what command we’d like Travis to run. It runs cowsay hello via bash inside the matsen/cowsay-test Docker image from Docker Hub.

We can test this locally with a local clone of the repository. If we want to test the whole process, we can execute the

docker build -t matsen/cowsay-test .

command inside the repository, and then copy and paste the docker run command from the .travis.yml file (without the leading -):

docker run -t matsen/cowsay-test /bin/bash -c "cowsay hello"

If we want to test the version that’s on Docker Hub, we can instead do docker pull -t matsen/cowsay-test rather than the above docker build command.

To automate this process for any repository, we just need these files committed to our repository, and to:

• set up an automated build on Docker Hub as described previously
• make an account on travis-ci.org, signing in via your GitHub account
• go to https://travis-ci.org/account/repositories and find the repository you want to test
• click on the switch to activate testing of your repository
• click on the hamburger menu button (☰) in the upper right, and select “Trigger build” using the default values
• add a Travis badge (see the cowsay-test repository for an example that makes the badge a link to your build page)

Using quay.io

Docker-the-software is great and very popular. Docker-the-company is struggling somewhat and it shows in that Docker Hub is slow and sometimes unreliable. quay.io is an alternative service that we are starting to use.

If you want to try it out, fork the cowsay-test repository on GitHub then:

• make an account on quay.io
• click on “Create New Repository” on the upper right of your user page
• name your repository in the text box
• Follow the prompts, selecting
  • Public repository
  • Link to a GitHub Repository Push
  • Select your forked repository
  • your Dockerfile is /Dockerfile as per the help on the right hand side
  • your build context is just / as per the help on the right hand side
• Continue to the page for your build
• Select “Start New Build”

On the Travis side, all you need to do is to add quay.io/ to the start of your Docker image name, like so:

script:
  - docker run -t quay.io/matsen/cowsay-test /bin/bash -c "cowsay hello"

Adding Conda

Conda is a great cross-platform package manager that has a lot of traction. You can incorporate Conda as follows, which I’ll describe using the example astronaut-requests repository. This example uses the requests Python library to fetch the collection of astronauts currently in space.

We just need one more file, which is the environment.yml file. This file specifies your Conda dependencies. It’s a helpful file for folks to have in general if they wish to install your software via conda, as they can just execute

conda env create -f environment.yml

and get a Conda environment ready for them. In the example environment.yml file I show how to combine pip and conda.

The astronaut-requests Dockerfile just builds the corresponding Conda environment:

FROM continuumio/anaconda3:2019.07

COPY environment.yml .

RUN /opt/conda/bin/conda env create -f environment.yml

Here we use the continuumio/anaconda3:2019.07 base image (using a specific tag is a good idea to avoid future breakage!). We then copy the environment.yml file in from the repository, and use it to make our Conda environment (note that we need to use the full path to conda). In our case it creates a Conda environment called requests.

The astronaut-requests Travis YAML file just gets a tiny bit more complex. I’ll break it down:

• docker run -t quay.io/matsen/astronaut-requests runs commands in the astronaut-requests container hosted on quay.io as before
• -v ${PWD}:/astronauts mounts the current directory (containing the clone of the repository on Travis) as /astronauts inside the container
• /bin/bash -c "..." runs ... as a bash command inside the container
• /opt/conda/bin/conda run -n requests ... runs ... inside the requests Conda environment that we built in our Docker container
• python /astronauts/get-astronauts.py is the command we are running

If we look at the Travis page for the repository we can see a JSON blob at the bottom with the current astronauts in space at the bottom. Success!

Note: conda run is not documented as of right now on the online Conda documentation, but it’s lovely and is documented via conda run --help on your local installation.

Wrap

That’s it! In practice, you’ll want Travis to build and test your code. You can see an example of all of this working in our libsbn repository.

I emphasize that we used Docker as a build environment and that the code is brought in from the outside. If you want to build a Docker container with the code built in, you can push the resulting Docker image to a registry.

Overview

We had a delightful time riding from Seoul to Busan along the Four Rivers trail. We then took a ferry to Tsushima for another week of riding, which was delicious icing on the cake. If you want to go, here are some tips that might be useful.

Season

Air pollution in Korea is not a joke. The locals say that it’s primarily a problem in the spring (and yes, they do blame it on China). Doing some web searching I couldn’t find compelling evidence that there’s a strong seasonality, but it’s worth some research. Looking retrospectively at the pollution levels for Seoul and Busan we happened to get lucky with the smog levels for our April 2019 trip. The Koreans, including cyclists, frequently wear masks, and you might consider doing the same.

We had perfect weather: a little cold at the beginning and a little hot at the end. It seemed like we had a predominant tailwind going North to South. We had two afternoons of rain, and that was all.

Maps

We found it very useful to have offline GPS maps with the route marked on the map. Although the route is very well signed, there are inevitably times when you’ve veered off course and need to find your way back. I was quite happy with the Guru Maps app available for iOS and Android. Here are a nice set of waypoints for the trip that you can upload into Guru Maps by exporting them as a KML file. Note that you will deviate somewhat from these waypoints, say by going on the other side of the river for a stretch, but it’s still a useful handrail.

The OpenStreetMap map available through the app was remarkably accurate.

Travel

We took our bikes on Asiana Airlines, which flew them as baggage without extra charges. The bikes were partially disassembled, and then packed into cardboard boxes we got from a bike shop. (Protip: get the biggest boxes you can, and don’t take off the rear wheel. This will save you from having to take off the derailleur, fenders, etc.) We also brought along a trailer for our daughter, which has accessory wheels that allows it to pass as a stroller. This allowed us to gate-check the trailer, which was handy in lots of ways. Although it’s possible to rent bikes and panniers in Seoul, we were happy to have our own kit.

Taking your bikes around the country on the long-distance buses was straightforward. They have large cargo bays going the width of the buses that fit all of our gear easily. I would even venture to say that they would fit a tandem, though I didn’t take measurements. However, tandems would be hard in general: we rode dozens of smallish elevators with our bikes getting in and out of the airport, as well as up and down overpasses.

Arrival

We booked nights on either end of our trip at the Best Western Incheon Airport, which was willing to store our bike boxes for a month. They also have a regular shuttle to and from the airport that has cargo space big enough to take bikes in boxes. The back entrance to the hotel has a nice cement area for assembling bikes.

From the Best Western, you can bike back to the airport (along the maglev train line) in order to take the train into Seoul. The trip from the airport to Seoul is over water and not bikeable. We took the all-stop train to Cheonga International City station, from which you can ride back to the Ara West Lock start of the bike trail. You get on the bike trail behind the Cheonga International City station stop (don’t cross the parking lot). You can get your stamp book from the information center in that complex.

Seoul

Seoul was one of our favorite parts of the trip. We booked two nights ahead of time in the modest Monai guest house which is close to the trail and a subway stop to get around the city. The owner Joseph was very kind and allowed us to store our bikes inside during our stay (warn him ahead of time that you’re bringing bikes). He has fantastic English.

You will find your own delights, but we wouldn’t have wanted to miss going to the Changdeokgung palace and its secret garden. I’d say that the zillion-dish vegan meal we had at Sanchon near the palace was one of the 5 most memorable meals of my life (though it wasn’t exactly a budget meal).

Riding

The riding is mostly on well-paved bike-only trails. It’s also mostly flat, except for when it’s not: there are two biggish climbs and 100 or so shorter inclines when you leave the river. Some of these are quite steep, with grades of 15% and more, so bring your low gears.

Camping

We mostly camped. We enjoyed the campgrounds when they were available, and when they weren’t we could always find a spot down by the river. Korean fishermen also set up tents by the river making this wild camping somewhat normal. Nobody hassled us for camping, but we did still try to stay out of sight in the reeds, just in case. Some campgrounds in Tsushima were closed because April was early in their camping season.

Although we loved eating in restaurants as much as we could, having a little cooking setup increased our flexibility. Propane canisters are easy to find in the larger grocery stores. They are called “gas-u” but taking your old one or having a picture of a canister will ease your canister shopping.

We were happy to have our gravity-feed water filter. We certainly wanted a water bag to be able to transport water, and having the filter allowed us to use some water that was marked as non-potable. We didn’t hesitate to drink tap water.

Phones and electronics

Because we were depending on our phones for navigation, I borrowed quite a chunky external battery for the trip so we wouldn’t run out of juice. That wasn’t really necessary, and a small one for a charge or two would have been sufficient. Restaurants all seemed to have outlets available and don’t seem to mind you charging, and all of the campgrounds had electricity as well.

Although just offline maps would have been fine if we had stuck to the main trail, cell data allowed us to be more spontaneous and recover from issues. Specifically, cell data will allow you to use Naver maps, the excellent Korean maps app, to see where the bike routes are, and to navigate places that you learn about en route. We would have gotten stuck a couple of times without having it, and having the web available to learn more about what we were seeing enriched our experience.

I used Google Fi, and the service was flawless even in the most remote locations of Korea. The service in Tsushima, Japan, was a little spottier (but this is simply a function of the cell network rather than the service). Even though I was liberal in my data use in Korea and Japan, my Fi bill was just $5 more than usual.

Spend a night in Hahoe

Staying a night in the UNESCO world heritage Hahoe village was a truly special experience. We stayed in centuries-old house that used a traditional wood-burning ondol floor heating system. Staying there will allow you an early morning walk in the quiet mist through Hahoe.

Although it’s not directly on the North-South main line, Hahoe is definitely worth a side-trip. It’s also an easy place to start or end your trip, as there is good bus service there.

Ride it with your family

We took our 7-month-old daughter along for the trip, which made it such a special family experience. She was our “ambassador,” and we had so many more interactions with locals because of her. Our friends rode with us to Hahoe with a 3 and a 5 year old in trailers and trail-a-bike, which was truly remarkable. We were comfortable with the safety level of the trip, given that it was predominantly on bike-only paths, but you will have to make your own choice.

Both Korea and Japan felt perfectly safe in every other way. For example, we parked our bikes outside a bustling market in Busan (the Korean equivalent of LA) with all of our stuff on them and came back an hour later with everything untouched. We brought a reasonably beefy cable lock, but would have been fine with a very light-weight one.

Tsushima, Japan

Tsushima was stunning. The island is a jungly forest, interrupted only by picturesque fishing villages in every cove. Here’s the view from Mount Eboshi Observatory:

The two home-stays we did here were very special. The dinners they served us had many interesting dishes, and the raw seafood was particularly notable. The Japanese use Google Maps, and you can find home-stays by looking for the bed icon.

There are lots of ferries from the International Ferry Terminal in Busan. Getting to the ferry terminal from the end of the bike trail is not easy. You can only take your bike on the subway during the weekends, and city buses do not take bikes. We rode our bikes there, which involved a lot of riding on bumpy sidewalks and a little bit on big roads. Naver was crucial for that leg. Getting back, we worked with the super-helpful folks at the tourist information office to arrange a taxi to the bus terminal.

Not all ferry services allow you to take bikes, but DAE-A Express Shipping Co do.

Japan is much more oriented towards cash versus card. We didn’t have luck with the ATMs at banks, but the post office ATMs worked great. Get some cash where you can.

There aren’t grocery stores everywhere in Tsushima, so stock up when you have the opportunity.

Have you ever been frustrated because a software package’s installation instructions were incomplete? Or have you wanted to try out software without going through a complex installation process? Or have you wanted to execute your software on some remote machine in a defined environment?

Docker can help.

In my group, we use Docker to make sure that our code compiles properly in a defined environment and analyses are reproducible. We automatically create Docker images through Dockerfiles. This provides a clear list of dependencies which are guaranteed to work starting from a defined starting point.

Once a Docker image is built, it can be run anywhere that runs the Docker engine.

Introduction to Docker

Have you ever seen someone boot up Windows on a Mac? That process of running one complete OS on top of another OS called running a virtual machine.

Docker containers are like virtual machines except that they share guest OSs, which makes them very lightweight. However, it’s still useful to think of Docker containers as virtual machines, because they feel like their own self-contained units.

Docker images are analogous to virtual machine images, and a container is analogous to a running virtual machine. There is a whole ecosystem around Docker containers, somewhat analogous to git repositories.

Docker Hub plays the role of GitHub (shown as a “Docker registry” above).

Pulling and running a Docker image

Let’s get our feet wet. To follow along, you’ll need to install Docker, or if you are on a shared machine you’ll need to be added to the docker group.

First, note that I don’t have cowsay available on the machine running Docker:

» cowsay
zsh: command not found: cowsay

We aim to remedy this situation with Docker. First pull a Docker image:

» docker pull matsen/cowsay-test

Once that is done, start and enter the Docker container:

» docker run -it matsen/cowsay-test /bin/bash
root@5d2942f9d5e2:/test-cowsay# cowsay "Hello"
 _______
< Hello >
 -------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Note that my prompt on the host machine is » whereas that inside the container is root@5d2942f9d5e2:/test-cowsay#.

Explore around a little inside the container and note that you can’t see anything from your host machine. Also, any modifications you make inside the container don’t change things on the host machine. For the ultimate test of this, it’s fun to recursively remove everything starting at /, though of course be very careful that you are actually inside the container! In any case, when you are done you can exit out of the Docker image.

If this all seems rather abstract, think of entering your Docker container as being like connecting to a remote computer. You need to open a connection to the operating system inside the Docker container via the docker run command, just like you open a connection to your computing cluster via the ssh command. In the next section, we’ll see how you need to create a share between the Docker container and your filesystem just like you would for a remote computer over the network.

Making a directory available inside of a container

As we have seen, Docker is designed such that changes inside the container are not reflected on the host machine. That’s perfect for some use cases, if you want to keep something, you typically want it stored outside of the container. For this Docker provides volumes.

To have a directory available in a Docker container, the syntax is

-v /host/path:/container/path

where /host/path should be replaced by the directory on the host machine you wish to make available, and /container/path is where you would like this directory to appear inside of the container.

For example, say we have a ~/foo directory that we would like available inside our container. We can start a container and mount this directory like so:

» docker run -v ${HOME}/foo:/foo -it matsen/cowsay-test /bin/bash

Once we’ve mounted it we can do stuff that lives beyond our container:

root@2cdca42bc8d9:/test-cowsay# cd /foo
root@2cdca42bc8d9:/foo# cowsay "Hello foo" > output.txt
root@2cdca42bc8d9:/foo# exit
exit
» cat ~/foo/output.txt
 ___________
< Hello foo >
 -----------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Dockerfiles

Dockerfiles are complete instructions about how to build a Docker image. This makes them very valuable even for people who don’t want to use your software via Docker: they can read this document, knowing that it provides a complete set of installation instructions (especially if you are using an automated build as described below).

We will illustrate using an example cowsay-test repository that made the image we pulled above. The first line,

FROM ubuntu:trusty

states that the image should be built starting with the Trusty version of the Ubuntu Linux distribution. This next section installs cowsay using apt-get (more on apt below):

RUN apt-get update -q && \
    apt-get install -y -q --no-install-recommends \
        cowsay

Then we link cowsay to a location from which we can execute it.

RUN ln -s /usr/games/cowsay /usr/bin

These FROM and RUN commands are Docker instructions; see the Dockerfile reference for more details.

Building a Docker image

Next we’ll build a Docker image from a Dockerfile. To follow along, clone the cowsay-test repository and enter that directory. To build the Docker image, use the command

» docker build -t matsen/cowsay-test .

This builds the Docker image we used in the previous section using the Dockerfile in the current directory.

Image names

So far we’ve been working with a Docker image named matsen/cowsay-test. We used this both when we pulled from Docker Hub, and when we built it locally (using the -t flag). This image name is composed of a user name matsen, and the name cowsay-test.

Names can also have tags. For example, rocker/tidyverse:3.5 is tag 3.5 of rocker/tidyverse. The latest tag is special: it is the latest version that was built without an explicit tag version (which may be different than the latest built).

A tiny bit about apt

We’re going to interrupt our discussion of Docker briefly to describe the wonderful apt package manager that was developed by Debian and adopted by Ubuntu (a Debian derivative). Here is a FAQ with common commands. This section is only relevant for base images based eventually on Debian or Ubuntu. If you have to use an image based on Red Hat (e.g. CentOS), then look up info on yum and/or DNF.

Here’s a good apt recipe:

RUN apt-get update && apt-get install -y --no-install-recommends \
    package1 \
    package2

Replace package1 and package2 with what your image needs.

To find what you need, do the following.

1. Learn what version of Debian or Ubuntu you are using in your base image. For example, if the base image is debian:latest you can go to the corresponding Docker Hub page to learn that this is Debian stretch.
2. Run your install until it says that it’s missing some package or header file. Say it’s libpng.
3. Do a web search for debian package libpng, which will take you to a page like this, telling you that there’s a libpng-dev package. You can click on stretch to get more details on what’s up with that package in the stretch version of Debian.

Tip: packages that end with -dev are generally the ones you want, because they install libraries with the corresponding header files that you can use for compilation.

Dockerfiles: making your own

Here we’ll extend our cowsay Docker image a little by extending its powers with wget. Doing the steps above, we find that there is a wget Debian package. So we can just add it:

RUN apt-get update -q && \
    apt-get install -y -q --no-install-recommends \
        cowsay \
        wget

Now, building and running our container as above, we win! (Note that I’ve abbreviated the command line prompt to # below).

# wget --no-check-certificate -O docker.cow https://git.io/vpOr5
# cowsay -f /test-cowsay/docker.cow "w00t"
 ______
< w00t >
 ------
    \
     \
      \
                    ##         .
              ## ## ##        ==
           ## ## ## ## ##    ===
       /"""""""""""""""""\___/ ===
      {                       /  ===-
       \______ O           __/
         \    \         __/
          \____\_______/

I suggest iteratively building up your Dockerfile as follows:

1. Start with a Dockerfile that doesn’t have everything you want, but does build.
2. Build and enter the corresponding container.
3. Run the install process for your tool, until it hits some problem.
4. Install the required dependency interactively within the container.
5. If that advances your tool install process, add that dependency install to your Dockerfile.
6. Rebuild and re-enter your container and continue the process.

Tips for making your Dockerfile

• Start with the best base image you can. If your code extends some already Dockerized software tool, start with that as a base image. For R, see rocker. For conda, see miniconda.
• If you are installing dependencies via pip, R, or conda, go crazy with those, and if you have binary dependencies you may want to use apt.
• Keep it clean. Don’t add extra things, because others will be looking at your Dockerfile and using it as a guide for installing your software locally.
• Docker is very clever at caching partial execution, but this can be frustrating if something has been updated remotely. To start from scratch, use the --no-cache flag to docker build.
• If you want to “shelve” your Dockerfile and still have it work down the road without modification, make sure to use a tag. So, rather than basing your image on continuumio/miniconda, base it on continuumio/miniconda:4.7.10. That way your build won’t break when the base image is updated.

A minimal list of commands

• docker build .: build a Docker image from this directory (assuming it contains a Dockerfile)
• docker run <image>: run an image, executing the CMD line then exiting
• docker run -it <image> /bin/bash: run an image and “log into” it
• docker exec -it <container> /bin/bash: “log into” a running container
• docker ps -a: shows all of the containers and their statuses

Automated builds

Docker Hub can automatically build your Docker image every time you push to a GitHub or Bitbucket repository. To do so, follow these steps:

1. Write a Dockerfile for your project. Test it locally.
2. Make an account on Docker Hub if you don’t have one already.
3. Add a Docker automated build.
4. Add a Docker build badge to your README (see cowsay-test again, changing what’s needed in the paths).

Where to next?

Having your tool in a Docker container enables a few more next steps.

• You can set up your tool in a continuous testing framework that uses Docker, such as Travis, CircleCI or Wercker.
• You can use Amazon’s version of a compute queue, called AWS Batch, to run your tool on large data sets in the cloud.
• To run a Docker image on your local cluster, you will need to use Singularity and convert your Docker image into a Singularity image then run it under Singularity. (Cluster administrators don’t like Docker because it doesn’t have a security model.)

Resources

There is a lot more to Docker than we’ve covered in this post. Here are some good resources.

• Docker basics
• Docker installation
• Dockerfile best practices
• An introduction to Docker for reproducible research, by Carl Boettiger

Thanks

Thank you to Will DeWitt, Jason Vander Heiden, and William Lees for feedback that improved this post.