You are here:
  1. Home
  2. eResearch Tips
  3. Using BaSH ‘for’ loops for parameter sweeps
Dr Collin Storlie

By Dr Collin Storlie, a former QCIF eResearch Analyst at JCU.*

Analytical tools and statistical algorithms frequently have a variety of options or parameters. A basic example would be a Linear Regression Model with two parameters, slope and y-intercept.

In some cases, such as Linear Regression, algorithms (e.g. OLS) can be employed to estimate the optimal parameter values given the data and choice of model.

In other cases, we may not be able to estimate optimal parameters values, and will need to perform our own parameter sweeps and comparisons.

Here, we will demonstrate how to use the BaSH shell to generate sequences of numbers (i.e. potential parameter values), provide those values as parameters to programs, and return those results in a text-based format for further post-hoc analysis.

Paste the below three lines of code into a document, and save that document as ‘prog.sh’ within your home directory. We’ll be using ‘prog.sh’ as the ‘program’ with multiple parameters in this example:

#!/bin/bash

let avg=($1+$2)/2 #Find the average of the two input values

echo "$1 $2 $avg" >> results.dat # Concatenate the parameters values and their average

I’ll explain what each of the three lines in the above program do.

  • #!/bin/bash
    • Sets the interpreter for this script. This means whenever we invoke this script our computer will know to use the BaSH shell.
  • let avg=($1+$2)/2 #Find the average of the two input values
    • Creates a variable ‘avg’ using the BaSH program ‘let’, the value of ‘avg’ is the average value of our two input parameters (labelled $1 and $2).
  • echo "$1 $2 $avg" >> results.dat # Concatenate the parameters values and their average
    • Concatenate all three values ($1, $2, $avg) into a document ‘results.dat’.

So, from above we can see that it’s possible to provide two (or more) parameters to BaSH programs, typically using the ‘$’ notation.

We can also use the BaSH shell to create sequences of numbers to ‘loop’ across. 

Here we generate two sequences of possible parameter values labelled ‘param1values’ and ‘param2values’:

param1values=`seq 1 1 10` # Creates a sequence of integers from 1 - 10 by 1's
param2values=`seq 0 5 100` # Creates a sequence of integers from 0 - 100 by 5's

A ‘for’ loop divides a sequence of numbers into individual ‘elements’. It then performs the same operation on each element of the sequence, in the order they are encountered. Let’s use a small ‘for’ loop to demonstrate:

for letter in A B C
do
echo $letter
done

Here, the ‘for’ loop takes our sequence of input values A B C (important to separate with a space), and:

  • assesses them in order, i.e. A 1st, B 2nd, C 3rd
  • assigns them to a variable ‘letter’
  • then uses ‘echo’ to print ‘letter’ to the screen.

Note the use of the ‘$’, which must be used when recalling variables.

Your output should be:

A
B
C

‘For’ loops can be nested within one another. In these cases, every element of the inner loop will be operated on for each element of the outer loop encountered, e.g.:

for letter in A B C
do
for number in 1 2 3
do
echo $letter $number
done
done

Your output should be:

A 1
A 2
A 3
B 1
B 2
B 3
C 1
C 2
C 3

This demonstrates that for each element of the outer loop, e.g. ‘A’, the entirety of the inner loop is being executed.

Let’s apply this knowledge to use our two sequences of numbers ‘param1values’ and ‘param2values’, to provide their values as arguments to our BaSH program ‘prog.sh’.

Now using a ‘for’ loop, provide each unique combination of parameters to your program ‘prog.sh’:

for parameter1 in $param1values
do
for parameter2 in $param2values
do
bash ~/prog.sh $parameter1 $parameter2
done
done

You should now have a file called ‘results.dat’containing your unique pairs of possible parameter values, along with the output of ‘prog.sh’ which corresponds to those two values.

At this point, you could analyse your ‘results.dat’ table to determine your optimum parameter combination.

It should be noted that analytical environments such as ‘R’ and ‘Python’ also have the capacity to use ‘for’ loops.

NB: If you see an error such as…

bash: /Users/jc152199/prog.sh: Permission denied

…you likely need to add ‘executable’ permissions to your ‘prog.sh’ file. You can do so with the following BaSH command:

chmod u+x ~/prog.sh

* This material is designed to help users of *NIX systems. Windows users will need to use a Terminal program such as ‘cygwin’ in order to follow along.

This article was first published on 10/04/2018.