// css

# Pipelines and your Unix toolbox

Unix commands are great for manipulating data and files. They get even better when used in shell pipelines. The following are a few of my go-tos – I’ll list the commands with an example or two. While many of the commands can be used standalone, I’ll provide examples that assume the input is piped in because that’s how you’d used these commands in a pipeline. Lastly, most of these commands are pretty simple and that is by design – the Unix philosophy focuses of simple, modular code, which can be composed to perform more complex operations.

Note: if you’re using a Mac, the builtin tools shipped with macOS might behave a little differently than the most recent versions. You can get more recently compiled version of these tools by running brew install coreutils. The typically usage is then ghead instead of head, gtail instead of tail, gpaste instead of paste, etc.

Here are the commands and a one sentence description:

Please forgive the “useless use of cat”. I’m using cat to show the pipe-able versions of the piped-to commands.

Print the first 5 lines

cat somefile.txt | head -n 5


Print the last 5 lines:

cat somefile.json | tail -n 5


Print all but the first line

cat somefile.csv | tail -n +2


Send all lines of output as arguments to echo

cat somefile.xml | xargs echo


Send each line individually as an argument to echo

cat somefile.yaml | xargs -n 1 echo


Send two arguments at a time to echo

cat somefile | xargs -n 2 echo


Print only column two of each line (assumes whitespace between columns)

cat somefile | awk '{print $2}'  Print only column two of each line with , separators cat somefile | awk -F',' '{print$2}'


or

cat somefile | cut -d',' -f 2


String replace each line matching 1, 2, or 3 using regex with the letter ‘x’

cat somefile.txt | sed 's/[1-3]/x/'


Count the number of times each line appears in the file

cat somefile.txt | sort | uniq -c


Write out the intermediate result to a file in the middle of a pipeline using tee

cat somefile.txt | grep "mystring" | tee newfile.txt | wc -l


The above are a bunch of “tools” to file away for when you need them. I like to store them along with a few keywords or a sentence describing what each does for easy searching and recall.

Now, let’s consider the following example file myfile.txt:

uuid,name
237fd8a4-58fb-11e9-8bb7-bf5388556288,name2
91834624-58fb-11e9-8c01-2393beecfc80,name3
223c7438-58fc-11e9-a629-7707a76a95c5,name4
a0362ab0-58fb-11e9-8aed-1f7ca90ae318,name5
f3f153aa-58fb-11e9-a37c-3ffed25b518b,name6
09f6cdb0-58fc-11e9-9663-93ede25c7774,name7
21f4c2ec-58fc-11e9-baf6-7792114ee968,name8


We want to create files in the current folder using the names in column two, where the uuids in column one start with a “2”.

Break down the transformation into parts:

• remove the first line (the header)
• filter for lines starting with 2
• grab the second column using a , separator
• use the result to create the files with touch
tail -n +2 myfile.txt | grep "^2" | awk -F',' '{print $2}' | xargs touch  Confirm it worked: $ ls name*
name2 name4 name8


As you add new tools to your toolbox, you can plug them in to your shell one-liners to manipulate data streams.

Some other useful commands for text processing worth looking into: