Using Placeholder Templates With Xargs In The Pipeline

pipes running along a wall

Using placeholder templates with xargs gives you a lot more power than simply using xargs to append args onto the end of a command.

I previously blogged about how to use xargs to append arguments to another command in the pipeline. This post goes into a bit more detail and shows you a more powerful way of using xargs.

Basic Xargs Example

With a typical, simple xargs command, you might append one argument onto the end of an existing command in the pipeline like this:

echo "this is a basic xargs example" | xargs echo "you said:"

The above command results in the output:

you said: this is a basic xargs example

Placeholder Templates With Xargs Example

Now, here is a simple example of using a placeholder, or template in your command, and passing your argument into that.

echo "this is a basic xargs example" | xargs -I {} echo "you said: {}"

The output is exactly the same as with the basic example. However, you can now use the curly braces placeholder to move the argument placement to anywhere in your command.

You’re no longer constrained to it being on the end of the echo (or whichever command you’re using). You can also do multiple placements of the argument in your command.

For example:

echo "FOO" | xargs -I {} echo "you said: {}. Here is another usage of your sample argument: {}. And here is yet another: {}"

A Slightly More Practical Example

Enough of the simple echo examples though. How about using this for a more practical, real world example?

In the following example, we want to list a bunch of AWS S3 buckets, and then do a summary output of their total size in GiB. We cut out the bucket name using cut from the initial listing that is returned with aws s3 ls.

aws s3 ls | cut -d' ' -f3 | xargs -n1 aws s3 ls --summarize --human-readable --recursive s3://

Using xargs to append the bucket name from the pipeline looks like it would work, as we only need it right at the end of the aws s3 ls command. There is an issue though, xargs would add a space, and we want the bucket name appended to s3:// without a space.

Using The Template or Placeholder

This is where using a placeholder or template with xargs can come in handy.

aws s3 ls | cut -d' ' -f3 | xargs -n1 -I {} aws s3 ls --summarize --human-readable --recursive s3://{}

It’s also worth noting that you can change your template or placeholder token with the -I parameter. It doesn’t have to be {} as in the examples above.

In summary, your usage of xargs can be levelled up by using the -I parameter to leverage placeholder or template tokens.

Select, match and pipe output into another command with jq, grep, and xargs

select-match-and-pipe-output-with-jq-grep-xargs

This is a quick reference post if you’re looking to pipe output into another command on Linux using xargs.

The pipeline is immensley powerful and you can leverage it to act on different stages of your full command to do specific selecting, matching, and manipulation.

Say you are running an executable that outputs a bunch of JSON and you want to select certain a certain subset of this data, pattern match it, and then send that matched data into another command.

This is the perfect use case for a mixture of the jq, grep and xargs commands in the pipeline.

Practical example with xargs

Here is a practical example where you might want to list all your AWS CodePipeline pipelines, match only on certain ones, and then execute (Release Changes) on each of them.

aws codepipeline list-pipelines | jq -r '.pipelines[].name' | grep project-xyz | xargs -n1 aws codepipeline start-pipeline-execution --name

The piped command above does the following:

  • Lists all AWS CodePipelines with the command aws codepipeline list-pipelines
  • Uses jq to ‘raw’ select the name from each pipeline object in the pipelines[] array that the above command outputs
  • Sends each pipeline name into grep to match only those containing the string “project-xyz”
  • Pipes the resulting pipeline names using xargs into the command aws codepipeline start-pipeline-execution --name. The -n1 argument tells xargs to use at most max-args of 1 per command line.

Accessing network equipment via console cable from your ESX / linux server

Ever wondered how you can access your Cisco router, switch, or other network device over console cable from your ESX host / other linux machine? Obviously you’ll need a COM port on the physical hardware to start… Here’s a guide I wrote for SysAdmin-Talk. Have a read if you are interested in finding out how to achieve this. They have some other great articles and how-to’s posted up there. I have already found some extremely useful Exchange how-tos and guides and am looking forward to writing more for SysAdmin Talk!

SysAdmin Talk – Don’t Tear your Hair Out over Access to Cisco Devices

Change iPhone root SSH password

If you have jailbroken your iPhone and have SSH installed it is a very good idea to change your default root password. The default root password for the iPhone 3G is “alpine” many people know this and if you are not careful you could get someone gaining access to your phone over your service providers’ data network or over a local wifi connection.

iphone-ssh

Once SSH is installed and active login to your phone using PuTTY.  Download PuTTy here.

You will just need to specify your phone’s local wifi IP address and SSH as the connection method. When prompted, enter your username as : root and password as : alpine

Once you get a command line, type in the command “passwd” and press enter.

Enter your existing password of alpine, then specify your new root password. Be sure to keep this safe and secure! I found that after changing the root password on my phone I needed to restart it – close putty, then restart your iPhone.

uBuntu 9.04 Server edition, Jaunty Jackalope

My installation of uBuntu 8.04 desktop edition was getting quite dirty and tired so today I decided to do a clean install of 9.04 server edition (Jaunty Jackalope). So far so good – just got the terminal now and I can add features I need as I go along. I will update the post or blog on any cool new features I find.

allowing root SSH access to your ESX host

In order to be able to SSH into your ESX host server via putty you need to enable root access via SSH. By default this is disabled – we will modify a configuration file and restart the service to allow ourselves into the ESX host console remotely.

From the console (you will need to physically be at the machine, or at least via a DRAC or KVM over IP), press Alt-F1 to access the command line and login as root.

Edit the following file:

/etc/ssh/sshd_config

You can do this by typing:

nano /etc/ssh/sshd_config

Go to the line that reads “PermitRootLogin no”

and change this to read “PermitRootLogin yes”

Press Ctrl-X to exit and press the “Y” key, hitting Enter to commit the choice to save.

Now we need to restart the sshd service to enable the changes. Type:

service sshd restart

and press enter.

You will now be able to SSH in as root.

Please drop me a comment if this has helped in any way!