IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 23

Microphone

Graded Quiz 9

You can connect to Gradescope to take weekly graded quiz today during the last 15 minutes of the class.

Once you start the quiz you have 15 minutes to finish it.

You can only take this quiz today.

There is not makeup for the weekly quiz because Gradescope does not permit it.

Homework 11

I have posted homework 11 here.

It is due this Sunday at 11:59 PM

This will be the last homework assignment.

Questions

Are there any questions before I begin?

Tips and Examples

A Real World Script Example

Unix has daemons, background processes that provide a service
We can log into pe15 because an ssh daemon, sshd, is always running
This daemon listens to port 22 on pe15 and creates a login session
Unix machines run a daemon, named cron ...
that will execute commands or scripts at specific times
You create something called a cron job ...
and tell cron when to run it
When that time rolls around cron will run the job
Usually, the job is a shell script
I had a client who used a database application I created
The tables in the database were backed up every night by a cron job ...
that ran a special shell script

Here is the script

$ cat -n backup_tables.sh 
   1  #!/bin/bash
   2  #
   3  # backs up all tables in the version2 database
   4  
   5  cd $v2/sql/bak
   6  rm *.bak 2> /dev/null
   7  dir=$(date +%Y-%m-%d)
   8  rm -rf $dir
   9  mysql version2 -u root -p$1 < $v2s/scripts/backup.sql
   10  mkdir $dir
   11  cp *.bak $dir
   12  date=$(date +%Y-%m-%d)
   13  echo $date ": Table backup complete"

Line 5 of the script uses the variable v2 to go to the backup directory
v2 holds the absolute pathname of the directory containing code for this client
I had a copy of this directory on my machine, but it had a different pathname
I set the value of v2 differently on each machine
This allows me to run the same script on both machines
This is a good example of how variables can customize scripts
In line 6, the script removes the previous backup files ...
all of which have a .bak extension
```
rm *.bak 2> /dev/null
```
I have copies of these files elsewhere
These copies are just the latest backups
If I need to restore the database tables ...
I will use these copies
Notice that I redirect standard error to /dev/null
If I don't do this, and there are no files to delete ...
I'll get an error message ...
which can be confusing
Line 7 sets the value of the dir variable to today's date ...
expressed in yyyy-mm-dd format
```
dir=$(date +%Y-%m-%d)
```
I am using command substitution to do this
Line 8 removes any directory already created to hold backup files for today's date
```
rm -rf $dir
```
The script can be run more than once in a given day
So I want only the most recent backup files in the directory
Line 9 runs a mysql script that actually creates the backup
```
mysql version2 -u root -p$1 < $v2s/scripts/backup.sql
```
The backup files, with the .bak extension, are created in the backup directory
Another mysql script can restore the tables from these files
Line 10 creates a new directory whose name is today's date
```
mkdir $dir
```
This gives me an historical archive
Line 11 copies the newly created backup files into this directory
```
cp *.bak $dir
```
Line 12 creates the date variable that also contains today's date
```
date=$(date +%Y-%m-%d)
```
Again I use command substitution
I use this variable in line 13 to print a message which will go to a log file
```
echo $date ": Table backup complete"
```
In the command line of the cron job I redirect this output to a log file
Why didn't I use the dir variable defined above?
At some time in the future to change one of these values
Using two variables gives me flexibility
I should have changed the last line so it prints the message only if everything goes well
To do this, I would need to use a conditional statement and the special parameter $?

Review

Shell Scripts

Most of the rest of this course will deal with shell scripts
Shell scripts are a series of Unix commands placed in a text file
You can run a shell script like any other program
Shell scripts allow you to automate certain routine operations
Shell script programming is not like other kinds of programming
Unix commands are not uniform in the way they work
Also, the control structures used in shell scripts work differently ...
than those in programming languages
As a general rule you should only use shell script for simple tasks
When you run a shell script, your current shell creates a sub-shell ...
to run the script
You must have both read and execute permissions to run a script ...
without first typing Bash

Shell Script Control Structures

The order in which script commands are executed is called the flow of control ...
or the path of execution
Without control structures the flow of control would be simple
It would start at the top and go to the bottom
Once
Control structures allow commands in a script to be executed in a different order
There are two basic types of control structures
- Conditionals
- Loops

The Problem With Unix `if` Statements

With an if statement, a certain segment of code is executed ...
but only if some condition is met
Usually this is done by evaluating something ...
that can only be true or false
A value that can only be true or false is called a boolean value
But Unix does not have boolean values
Where another language would use a boolean value
Unix uses the exit status
The results of running a command
If the command runs without error, it returns an exit status of 0
Unix treats an exit status of 0 as true
It treats any other exit status value as false

The `if` ... `then` Construct

The most basic conditional is the if ... then construct

Which has the format

if TEST_COMMAND
then
    COMMAND_1
    COMMAND_2
    ...
fi

fi is if spelled backwards
The exit status of the command following if ...
determines what happens
If it is 0, the commands between then and fi are run
Otherwise they will be skipped over
This flowchart shows how it works

Here is a script that that uses an if statement

$ cat foo_in_dir.sh
#! /bin/bash

# prompts for a directory and creates foo.txt there

read -p 'Directory: ' dir
if cd $dir
then
    touch foo.txt
fi

`test`

What if I wanted to make sure a variable was greater than 5 ...
before I did something?
I would use an if statement
But I need to run a command to get an exit status
So what can I do?
Unix provides a command called test ...
that can be used in situations like this
test takes a boolean expression as an argument
It returns an exit status of 0 if the expression is true
Otherwise it returns 1
test is a built-in so in runs inside the script process

The `test` operators

test can be used to check for a number of different conditions
But this is not done in the way you would expect
In most computer languages you write boolean expressions ...
using the operators you learned in algebra
Operators like >, < and =
Unfortunately using test is much more complicated
test has five different categories of operators
- Operators for comparing numbers
- Operators for comparing strings
- File and directory operators
- String operators
- Logical operators
This is one reason I think you should avoid shell scripts ...
when you need to to something complicated

`test` Operators for Comparing Numbers

When comparing numbers test uses special operators

That consist of a - followed by two letters

Operator	Result
-eq	True if the numbers are equal
-ne	True if the numbers are not equal
-ge	True if the first number is greater than, or equal to, the second
-gt	True if the first number is greater than the second
-le	True if the first number is less than, or equal to, the second,
-lt	True if the first number is less than the second

To give you some examples, I need to use echo ? to see the exit status

$a=1; b=2

$ test $a -gt $b; echo $? # is a greater than b?
1

$ test $a -ge $b; echo $? # is a greater than or equal to b?
1

$ test $a -lt $b; echo $? # is a less than b?
0

$ test $a -le $b; echo $? # is a less than or equal to b?
0

$ test $a -eq $b; echo $? # is a equal to b?
1

$ test $a -ne $b; echo $? # is a not equal to b?
0

`test` String Operators

test uses different operators with strings
There are two sets of of string operators
Comparison operators that compare two strings

= True if two strings are the same

!= True if two strings are different
Operators that work on a single value

Operator Result

-n True if the string has a length greater than 0

-z True if the string has a length of 0

Operator	Result
-n	True if the string has a length greater than 0
-z	True if the string has a length of 0

Here are some examples

$ s1=foo; s2=bar

$ test $s1 = s2; echo $?
1

$ test $s1 != s2; echo $?
0

$ test -n  ""; echo $?
1

$ test -z  ""; echo $?
0

Remember, Unix is case sensitive ...

and so are the string comparison operators

$ test "foo" = "FOO"; echo $?
1

$ test "foo" != "FOO"; echo $?
0

File and Directory Operators

There are test operators that can be used on files and directories

They take a single argument that comes after the operator

Operator	Result
-d	True if the argument is the name of a directory
-e	True if the argument is the name of a file or directory
-f	True if the argument is the name of an ordinary file (i.e. not a directory)
-s	True if the argument is the name of a file or directory that is not empty
-r	True if the argument is the name of a file or directory that is readable
-w	True if the argument is the name of a file or directory that is writable
-x	True if the argument is the name of a file or directory that is executable

Here are some examples

$ ls -ld dir1 # the directory I will use for testing
drwxr-xr-x  4 glenn  staff  128 Aug  9 11:24 dir1

$ test -d dir1; echo $? # is it a directory?
0

$ test -e dir1; echo $? # is it a file or a directory?
0

$ test -f dir1; echo $? # is it an ordinary file?
1

$ test -s dir1; echo $? # does the directory have things in it?
0

$ test -r dir1; echo $? # do I have read permission?
0

$ test -w dir1; echo $? # do I have write permission?
0

$ test -x dir1; echo $? # do I have execute permission?
0

$ ls -l foo1.txt # the file I will use for testing
-rw-r--r--  1 glenn  staff  46 Aug  9 11:24 foo1.txt

$ test -e foo1.txt ; echo $? # is it a file or a directory?
0

$ test -f foo1.txt ; echo $? # is it an ordinary file?
0

$ test -s foo1.txt ; echo $? # does the file have something in it?
0

$ test -r foo1.txt ; echo $? # do I have read permission?
0

$ test -w foo1.txt ; echo $? # do I have write permission?
0

$ test -x foo1.txt ; echo $? # do I have execute permission?
1

I use the -d, -f, -r and -x options in my test scripts for Class Exercises

Logical Operators

The logical operators work on the results of other test expressions

They are

Operator	Condition Tested
-a	Logical AND, meaning both expressions must be true
-o	Logical OR, meaning either of the two expressions must be true
!	Logical NOT, meaning true becomes false and false becomes true

Examples

$ test 1 -lt 5 -a 5 -lt 10; echo $? # is 1 less than 5 AND 5 less than 10
0

$ test 5 -lt 1 -a 5 -lt 10; echo $? # is 5 less than 1 AND 5 less than 10
1

$ test 5 -lt 1 -o 5 -lt 10; echo $? # is 5 less than 1 OR 5 less than 10
0

$  test 5 -lt 1; echo $? # is 5 less than one
1

$  test !  5 -lt 1; echo $? # is 5 NOT less than 1
0

Attendance

New Material

Another Way to run `test`

To a programmer the following code looks weird
```
if cd $dir
then
    touch foo.txt
fi
```
In all computer languages I know
if is followed by a boolean expression
Which is often enclosed in parentheses

In Perl the format is

if (BOOEAN_EXPRESSION) {
   STATEMENT;
   STATEMENT;
   ...
}

To make programmers more comfortable ...
Unix provides another way of calling test
[ ] is a synonym for test

Instead of writing

if test $word1 = $word2
then
    echo $word1 and $word2 are the same
fi

I can write

if [ $word1 = $word2 ]
then
    echo $word1 and $word2 are the same
fi

It doesn't matter what form you use
The two are equivalent
There is one issue that can trip you up when using [ ]
You must always be sure there is a space between [ and what comes after it
And you must be sure there is a space before ]
If you don't you will get an error message
```
$ [ 5 -ne 6]
-bash: [: missing `]'
```
That's because Bash reads 6] as a single token ...
which it does not understand
Forgetting this caused me endless grief ...
when I first started using [ ]

The `if ... then ... else ...` Construct

There are three versions of the if statement
The if ... then statement only does something if a condition is met
What if you wanted to do one thing if the condition were met ...
but something else if it were not?

The script foo_in_dir.sh created a foo.txt file if it could

read -p 'Directory: ' dir
if cd $dir
then
    touch foo.txt
fi

But it says nothing if it cannot go to the directory
Perhaps the most widely used if statement is the if ... then ... else ...
It begins just like the if ... then statement ...
but it adds an else clause

It has the following format

if TEST_COMMAND
then
   COMMAND_1
   COMMAND_2
   ....
else
   COMMAND_A
   COMMAND_B
   ...
fi

If COMMAND returns an exit status of 0 ...
the commands between then and else are executed
Otherwise, those between else and fi are run
Here is the flowchart

Using this version of the if statement we can rewrite the script above

read -p 'Directory: ' dir
if cd $dir 2> /dev/null
then
    touch foo.txt
else
    echo Cannot cd into $dir
fi

Running this, I get

$ ./foo_in_dir_2.sh
Directory: xxx
Cannot cd into xxx

Notice that I had to redirect standard error for the call to cd

If I didn't I would get

$ ./foo_in_dir_2.sh
Directory: xxx
./foo_in_dir_2.sh: line 6: cd: xxx: No such file or directory
Cannot cd into xxx

The `if ... then ... elif ...` Construct

if statements create a branch in the flow of control
An if ... then statement creates one branch
An if ... then ... else statement creates two branches
If you need more branches ...
you need to use an if ... then ... elif statement
elif stands for "else if"
An if ... then ... elif statement is open ended

You can create as many branches as you wish

if TEST_COMMAND_1
then
    COMMAND_1
    COMMAND_2
    ...
elif TEST_COMMAND_2
then
    COMMAND_A
    COMMAND_B
    ...
...   
[else
    COMMAND_N1
    COMMAND_N2
    ...]
fi

Each elif must be followed by its own command
Notice that elif must be followed by then
Essentially, this if statement allows you to nest one if statement ...
within another if statement
The nesting can go as deep as you wish
The else clause that the end is optional
If you have an else clause, one of the branches must be run
Without an else clause ...
it is possible that none of the branches will be taken
Here is the flowchart

Let's use an if ... then ... elif statement ...
to improve foo_in_dir_2.sh

Here is the code for the script

read -p 'Directory: ' dir
if cd $dir 2> /dev/null
then
    touch foo.txt
else
    echo Cannot cd into $dir
fi

Let's change it so it first checks whether the directory exists ...
and then checks whether we have write access to the directory
Only if both conditions are true will it create foo.txt

To do this we have to add two if clauses

read -p 'Directory: ' dir
if [ ! -d $dir ] # if dir is not the name of a directory 
then
    echo There is no directory named $dir
elif [ ! -w $dir ] # if we don't have write permission on dir
then
    echo Cannot create a directory in $dir
elif cd $dir # 2> /dev/null
then
    touch foo.txt
else
    echo Cannot cd into $dir
fi

Notice that we had to use the NOT logical operator ! ...
in the first two tests
Here is another example
This script asks for three words

Then sees if any of them are the same

$ cat compare_three_words.sh
#! /bin/bash
#
# prompts for three words then sees if any match

read -p "word 1: "  word1
read -p "word 2: "  word2
read -p "word 2: "  word3
if [ $word1 = $word2 -a $word2 = $word3 ]
then
    echo "Match: words 1, 2 & 3"
elif [ $word1 = $word2 ]
then
    echo "Match: words 1 & 2"
elif [ $word1 = $word3 ]
then
    echo  "Match: words 1 & 3"
elif [ $word2 = $word3 ]
then
    echo  "Match: words 2 & 3"
else
    echo No match
fi

Now to test it

$ ./compare_three_words.sh 
word 1: foo
word 2: bar
word 2: bletch
No match

$ ./compare_three_words.sh 
word 1: foo
word 2: foo
word 2: foo
Match: words 1, 2 & 3

$ ./compare_three_words.sh 
word 1: foo
word 2: bar
word 2: bar
Match: words 2 & 3

$ ./compare_three_words.sh 
word 1: foo
word 2: foo
word 2: bletch 
Match: words 1 & 2

Exit

What do you do if a script encounters a situation ...
where it can't go any further?
Maybe the script needs to create a directory ...
but it can't
In this situation you need the exit command
exit terminates the sub-shell in which it is run
This was the command you used to close a script session

exit is a built-in

$ help exit
exit: exit [n]
    Exit the shell.
    
    Exits the shell with a status of N.  If N is omitted, the exit status
    is that of the last command executed.

Most of the time when you run exit ...
something has gone wrong
In this case you should give exit 1 as an argument
```
exit 1
```

Checking the Arguments to a Script

Some scripts need a certain number of command line arguments ...
to work properly
When you write such a script you need to check ...
that you get the arguments you need
If the script does not get the required arguments ...
it should quit
The special parameter # gives the number of command line arguments
So the first line of a script that needs arguments ...
should be an if statement using #
Here is a new version of foo_in_dir.sh that uses a command line argument ...

and checks that it got one

$ cat foo_in_dir_4.sh
#! /bin/bash

# creates foo.txt in a directory given at the command line

if [ $# -gt 0 ]
then
    dir=$1
else
    exit 1
fi
if cd $dir 2> /dev/null
then
    touch foo.txt
else
    echo Could not go to $dir 
fi

Usage Messages

The previous script exits if it does not get a command line argument
But the user has no idea if the script did or didn't work
Scripts should always print an error message when they don't work
When the script failed because it did not get a command line argument ...
it should let the users know this
In Unix, the preferred way of doing this is with a usage message
The usage message has a standard form
```
Usage:  PROGRAM_NAME  ARG1  ARG2  ...
```
You need one ARG for each argument needed
The ARGs should be words that describe what is needed
For foo_in_dir.sh DIR_NAME would be a good choice
The name of the script could change over time
So you should use the positional parameter 0 ...
to print the name of the script

Here is the improved code

$ cat ./foo_in_dir_5.sh 
#! /bin/bash

# creates foo.txt in a directory given at the command line

if [ $# -gt 0 ]
then
    dir=$1
else
    echo Usage: $0 DIR_NAME
    exit 1
fi
if cd $dir 2> /dev/null
then
    touch foo.txt
else
    echo Could not go to $dir 
fi

When this script is run without an argument we get

$ ./foo_in_dir_5.sh 
Usage: ./foo_in_dir_5.sh DIR_NAME

Better Usage Messages Using `basename`

Look carefully at the usage message above
The name of the script has . / in front of it
Because ./foo_in_dir_5.sh is what was used to run the script
That's not much of a problem when running the script ...
in the directory in which it lives
But what if I put this in my script directory, /home/ghoffman/bin/shell
This directory is included in my PATH variable
So I can run any script in this directory anyplace in the file system

If I run the script and leave out the argument I get

$ foo_in_dir_5.sh 
Usage: /Users/glenn/bin/shell/home/foo_in_dir_5.sh DIR_NAME

This makes it very hard to see the name of the script
The Unix command basename can help in ths situation
basename takes a pathname as an argument

And returns whatever it finds after the final /

$ basename /Users/glenn/bin/shell/home/foo_in_dir_5.sh
foo_in_dir_5.sh

But just replacing $0 with basename $0 does not work

$ ./foo_in_dir_6.sh 
Usage: basename ./foo_in_dir_6.sh DIR_NAME

Instead I have to use command substitution

It will replace the call to basename with the value it returns

$ cat foo_in_dir_6.sh 
#! /bin/bash

# creates foo.txt in a directory given at the command line

if [ $# -gt 0 ]
then
    dir=$1
else
    echo Usage: $(basename $0) DIR_NAME
    exit 1
fi
if cd $dir 2> /dev/null
then
    touch foo.txt
else
    echo Could not go to $dir 
fi

$ ./foo_in_dir_6.sh 
Usage: foo_in_dir_6.sh DIR_NAME

You don't need a usage message if the script does not require arguments

Incremental Development

It is easy to make a mistake when writing a script
When I write a script I do something called incremental development
The idea is simple
Don't try to write the entire script ...
all at once
Instead write a few lines of code
Then test the code by running it
If there is an error, fix it
Since you have only added a few lines
It should be easy to find the problem ...
and fix it
Once the script works, add a few more lines
Limiting what you write at each step makes is easy to spot errors
It also make it easier for you to think through what needs to be done
But it has an additional, subtle benefit

Another Benefit of Incremental Development

You can't do something ...
if you don't believe you can
Every athlete knows you can defeat yourself in your head
This is true in learning as in life
Most people believe that math is very hard
This can be a self-fulfilling prophecy
If you don't immediately succeed at math ...
it's easy to think you just can't do it
This is true of writing scripts
You have all heard that coding is hard
The first time you write programs you will make many mistakes ...
which can make you discouraged ...
and think that programming is not something you can do
Incremental development can help with this
You only write a little bit of code at each step
This makes finding errors easier
Working this way results in slow but steady progress ...
which increases you self-confidence ...
and makes it easier to code

Debugging Scripts

Like most Unix programs, Bash has options
I have never found most of these options useful
But the -x option can help with debugging
Finding where the error occurs is the first step in debugging
-x makes Bash print a command before running it

Here is an example

$ cat match_three.sh 
#! /bin/bash
#
# takes three stings as input and compares them

if [ $# -lt 3 ]
then
    echo Usage: $(basename $0)  STRING1  STRING2  STRING3
    exit 1
fi

if [ $1 = $2 -a $2 = $3 ]
then
    echo All arguments match
elif [ $1 = $2 ]
then
    echo Arguments 1 and 2 match
elif [ $1 = $3 ]
then
    echo  Arguments 1 and 3 match
elif [ $2 = $3 ]
then
    echo  Arguments 2 and 3 match
else
    echo No arguments match
fi

$ bash -x match_three.sh foo bar foo
+ '[' 3 -lt 3 ']'
+ '[' foo = bar -a bar = foo ']'
+ '[' foo = bar ']'
+ '[' foo = foo ']'
+ echo Arguments 1 and 3 match
Arguments 1 and 3 match

Run this way, Bash prints a + before each command
Notice that values from the command line appear ...
when printing the values of positional parameters

Indenting When Writing Scripts

Control structures create sections of code that run under certain conditions
In programming we call these sections code blocks
It's important to see the code blocks when reading a script
Special Unix keywords set off these code blocks
But sometimes this is not enough
It is good practice to indent all commands within a code block
Compare two versions of the same script

One without indenting

if [ $# -lt 3 ]
then
echo Usage: $(basename $0)  STRING1  STRING2  STRING3
exit 1
fi

if [ $1 = $2 -a $2 = $3 ]
then
echo All arguments match
elif [ $1 = $2 ]
then
echo Arguments 1 and 2 match
elif [ $1 = $3 ]
then
echo  Arguments 1 and 3 match
elif [ $2 = $3 ]
then
echo  Arguments 2 and 3 match
else
echo No arguments match
fi

And one with indenting

if [ $# -lt 3 ]
then
    echo Usage: $(basename $0)  STRING1  STRING2  STRING3
    exit 1
fi

if [ $1 = $2 -a $2 = $3 ]
then
    echo All arguments match
elif [ $1 = $2 ]
then
    echo Arguments 1 and 2 match
elif [ $1 = $3 ]
then
    echo  Arguments 1 and 3 match
elif [ $2 = $3 ]
then
    echo  Arguments 2 and 3 match
else
    echo No arguments match
fi

Which version would you rather read?

=	True if two strings are the same
!=	True if two strings are different

Tips and Examples

Review

New Material

Graded Quiz 9

Homework 11

Questions

Tips and Examples

A Real World Script Example

Review

Shell Scripts

Shell Script Control Structures

The Problem With Unix if Statements

The if ... then Construct

test

The test operators

test Operators for Comparing Numbers

test String Operators

File and Directory Operators

Logical Operators

Attendance

New Material

Another Way to run test

The if ... then ... else ... Construct

The if ... then ... elif ... Construct

Exit

Checking the Arguments to a Script

Usage Messages

Better Usage Messages Using basename

Incremental Development

Another Benefit of Incremental Development

Debugging Scripts

Indenting When Writing Scripts

Class Exercise

Class Quiz

The Problem With Unix `if` Statements

The `if` ... `then` Construct

`test`

The `test` operators

`test` Operators for Comparing Numbers

`test` String Operators

Another Way to run `test`

The `if ... then ... else ...` Construct

The `if ... then ... elif ...` Construct

Better Usage Messages Using `basename`