IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 22

Today's Topics

Review

Shell Modification of the Command Line
History Expansion
Alias Substitution
Brace Expansion
Tilde, ~, Expansion
Parameter and Variable Expansion
Arithmetic Expansion
Command Substitution
Word Splitting
Pathname Expansion
Process Substitution

Homework 10

Homework 10 is posted here.

As usual, it will be due this coming Sunday at 11:59 PM.

Announcement

There will be no graded quiz next week.

But there will be one the following week covering the material in the previous 3 Class Quizzes.

Review

Shell Modification of the Command Line

After you hit the Enter key ...
the keyboard device driver tty ...
gives the shell the text you have entered at the command line
The shell can now try to execute the commands that you have typed
But before the shell executes the commands ...
if first looks to see if it need to make changes ...
to the tokens on the command line
The shell actually rewrites the command line before executing it
It does this to implement features of the shell ...
like command substitution and pathname expansion
These are features that make the shell more powerful ...
but they require the shell to change what you typed on the command line ...
before executing it
There are 10 different ways in which the shell can modify the command line
The order in which the shell performs them is as follows
1. History Expansion
2. Alias Substitution
3. Brace Expansion
4. Title, ~, Expansion
5. Parameter and Variable Expansion
6. Arithmetic expansion
7. Command substitution
8. Word splitting
9. Pathname expansion
10. Process substitution

History Expansion

The first substitution bash performs is history expansion
History expansion occurs when you use the exclamation mark, ! ...

to run again a previous command using the history mechanism

    $ history 5
      540  cat output.txt 
      541  echo "Go Red Sox" > output.txt
      542  cat output.txt 
      543  echo foo
      544  history 5
      
    $ !543
    echo foo
    foo

Alias Substitution

After history expansion, bash performs alias substitution

    $ alias ll="ls -l'
    $ ll
    total 2
    lrwxrwxrwx 1 it244gh man    34 Sep  6 21:09 it244 -> /courses/it244/f12/ghoffman/it244gh
    drwxr-xr-x 2 it244gh ugrad 512 Oct 27 09:16 work

The shell replaces the name of the alias ...
with the value of the alias
Aliases allow you to run complicated commands ...
by typing only a few characters

Brace Expansion

After alias substitution, bash performs brace expansion
Braces, { }, allow you to specify several strings ...
all at once
Braces can appear with strings of characters in front ...
or behind
The braces contain strings of characters separated by commas
The shell expands a brace by creating many strings ...
one for each string contained within the braces

If I wanted to create 5 foo files I could use braces expansion as follows

    $ touch foo{1,2,3,4,5}.txt
    
    $ ls
    foo1.txt  foo2.txt  foo3.txt  foo4.txt  foo5.txt

The strings inside the braces can contain one or more characters

    $ touch {a,ab,abc}.txt
    
    $ ls
    abc.txt  ab.txt  a.txt

There must not be any unquoted spaces or tabs ...
within the braces
If there are, the expansion will not work properly

Tilde, ~, Expansion

After brace expansion, bash performs tilde expansion
Whenever bash sees a tilde, ~ , by itself ...
it substitutes the absolute pathname of your home directory
Whenever bash sees a tilde, ~ , followed by a Unix user name,
it substitutes the absolute pathname of the home directory ...
of that account
If there is no user name matching the string following the tilde ...
no expansion is performed
There are two other tilde expansions
- ~+
- ~-
When bash sees ~+ ...
it substitutes the value of the current directory
When bash sees ~- ...
it substitutes the value of the previous directory

Parameter and Variable Expansion

After tilde expansion, bash performs parameter and variable expansion

    $ echo $SHELL
    /bin/bash
    
    $ echo $?
    0

bash notices the $ in front of a string ...
and looks to see if that string is the name of a variable
If the string is a variable, bash substitutes the value of the variable ...
for the dollar sign and variable name

Arithmetic Expansion

After parameter and variable expansion, bash performs arithmetic expansion
Unix treats everything on the command line as text ...
unless told otherwise
```
    $ echo 5 + 4
    5 + 4
```
Arithmetic expansion where the text inside $(( )) is treated as an arithmetic expression ...
and the result of evaluating that expression replaces $(( )) ...
and everything inside it
```
    $ echo $(( 5 + 4 ))
    9
```
Inside the arithmetic expression itself ...
you do not have to use a dollar sign, $ ...

to get the value of a variable

    $ echo $a $b
    5 3
    
    $ echo $(( a * b))
    15

Command Substitution

After arithmetic expansion, bash performs command substitution
In command substitution, a command is run in a sub-shell ...
and the output of that command ...
replaces the command itself
Command substitution uses the following format
```
    $(COMMAND)
```
Where COMMAND is any valid Unix command
To set a variable to the current time and date ...

I could use

    $ today=$(date)
    
    $ echo $today
    Tue Oct 25 17:00:07 EDT 2011

There is an alternate format for command substitution
You can place the command to be run ...

within backticks, ` `

    $ ls -l `which bash`
    -rwxr-xr-x 1 root root 954896 2011-03-31 17:20 /bin/bash

Before running ls, bash first runs the command
```
    which bash
```
And replaces the command with the value returned by which
ls can now take /bin/bash as its argument

Word Splitting

After command substitution, Bash performs word splitting
When Bash gets a command line ...
it spits the text into tokens
Tokens are strings of characters ..
usually separated by whitespace
- Spaces
- Tabs
- Newlines (carriage returns)
I have found a small script which demonstrates word splitting ...

by showing all command arguments

$ args.sh foo bar bletch 
3 args: [foo] [bar] [bletch]

Word splitting does not work ...

when strings are quoted

$ args.sh 'foo bar bletch' 
1 args: [foo bar bletch]

$ args.sh "foo bar bletch"
1 args: [foo bar bletch]

But word splitting does work ...

with both command substituion

$ args.sh $(date)
6 args: [Wed] [Apr] [12] [16:19:17] [EDT] [2017]

and variable substitution

$ cheer="Let's go Red Sox!"

$ args.sh $cheer
4 args: [Let's] [go] [Red] [Sox!]

But there are some situations ...
where you want another character to separate tokens
You can do this using the Unix keyword variable IFS
IFS stands for Internal Field Separator
If you give IFS a value ...
say the colon : ...

Bash will be use this character to separate tokens

$ args.sh $PATH
1 args: [/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/oracle/12.1/client64/bin:/home/ghoffman/bin/shell:/home/ghoffman/bin/python/umb:/home/ghoffman/bin:/home/ghoffman/bin/shell:/home/ghoffman/bin/python/umb]

$ IFS=:

$ args.sh $PATH
15 args: [/usr/local/sbin] [/usr/local/bin] [/usr/sbin] [/usr/bin] [/sbin] [/bin] [/usr/games] [/usr/local/games] [/snap/bin] [/usr/lib/oracle/12.1/client64/bin] [/home/ghoffman/bin/shell] [/home/ghoffman/bin/python/umb] [/home/ghoffman/bin] [/home/ghoffman/bin/shell] [/home/ghoffman/bin/python/umb]

Word splitting will not be on the final

Pathname Expansion

After word splitting, bash performs pathname expansion
Pathname expansion is where you use meta-characters ...
to specify one or more pathnames
The metacharacters are used to create patterns ...
that are called ambiguous file references
The meta-characters are
- ?
- *
- [ ]

Process Substitution

After pathname expansion, bash performs process substitution
Process substitution allows you to create a file on the fly
You use a subprocess running a program to create lines of text
These lines of text can then be used at the command line ...
as if they were contained in an ordinary file
Process substitution uses the following format
```
    <(COMMAND)
```
The output of the command that appears between the parenthesis ...
is placed in a Unix structure called a named pipe
When Unix performs process substitution it creates a process to run a command ...
and sends the output of that command to a named pipe
Then Unix redirects input ...
to come from the named pipe

We can use named pipes to compare the contents of two directories

    $ diff -y <(ls -1 tia777/ce) <(ls -1 jgreen/ce)
    ce1                             ce1
    ce10                            ce10
    ce11                                  <
    ce2                             ce2
    ce3                             ce3
    ce4                             ce4
    ce5                             ce5
    ce6                             ce6
    ce7                             ce7
    ce8                             ce8
    ce9                             ce9

Here we have two ls commands ...
each running in their own subshell ...
and each one sending output ...
to its own named pipe
We can then run diff to look for differences ...
in these two "files"
Process substitution will not be on the final

New Material

Shell Scripts

Most of the rest of this course will deal with shell scripts
Shell scripts are a series of Unix commands placed in a file
You can run a shell script like any other program
Shell scripts allow you to automate certain routine operations
Much of the work in Unix system administration is done using shell scripts
Shell script programming is not like other kinds of programming
Unix commands are not uniform in the way they work ...
because each was developed separately by different people
The control structures used in shell scripts ...
are different from those in programming languages
You should only write shell scripts for simple tasks
If you need if statements or loops to write a script ...
use a scripting language like Perl or Python
But you should know how to read scripts
When you run a shell script ...
your current shell creates a sub-shell to run the script
You must have both read and execute permissions ...
to run a script without using the bash command

Shell Script Control Structures

Control structures allow commands in a script ...
to be executed in an different order
Without control structures ...
a shell script could only start at the beginning ...
and go to the end ...
once ...
which would limit what it could do
There are two basic types of control structures
- Loops
- Conditionals
Loops are constructs that repeat a number of statements ...
until some condition is reached
Conditionals are statements where different things happen ...
based on some condition ...
which is either true or false
In Bash the conditional control structure is the if statement
Shell scripts can have both loops and conditionals

The `if` ... `then` Construct

The most basic conditional is the if ... then construction which has the format

    if COMMAND
    then
        COMMAND_1
        COMMAND_2
        ...
    fi

where COMMAND is any Unix command ...
that returns an exit status
and COMMAND_1, COMMAND_2, ..., are a series of Unix commands
The most commonly used command following if is test
It is used to test the truth of some condition

Let's look at an example

    $ cat if_1.sh
    #! /bin/bash
    ## 
    ## a shell script that demonstrates the Unix if construct

    echo -n "word 1: " 
    read word1
    echo -n "word 2: " 
    read word2

    if test "$word1" = "$word2"
    then
        echo The two words match
    fi
    echo End of script
    
    $ ./if_1.sh
    word 1: foo
    word 2: foo
    The two words match
    End of script
    
    $ ./if_1.sh
    word 1: foo
    word 2: bar
    End of script

read is a utility that takes input from standard input ...
and stores that value in the variable ...
given to it as an argument
Notice that echo was used with the -n option
The -n option prevents echo from sending a newline character ...
which would move down to the next line
This allows echo print a prompt ...
for input that will be read by read
The statements that are run if the condition is true ...
lie between the then and fi keywords
then must either be on a separate line from if ...

or on the same line, but separated by a semi-colon

    $ cat if_2.sh
    #! /bin/bash
    ## 
    ## a shell script that demonstrates the Unix if construct

    echo -n "word 1: " 
    read word1
    echo -n "word 2: " 
    read word2

    if test "$word1" = "$word2" ; then
        echo The two words match
    fi
    echo End of script
    
    $ ./if_2.sh
    word 1: foo
    word 2: foo
    The two words match
    End of script

The keyword fi must close the conditional statement
If you don't you will get an error
fi is if spelled backwards

`test`

test is a command that is often used in an if statement
But while test evaluates the expression that follows ...
it does not return true or false ...
as you would expect
In Unix, everything is text ...
unless it is enclosed in double parentheses, (( )) ...
in which case the contents are treated as numbers
Most programming languages have boolean variables
A boolean variable can only have one of two values
- True
- False
But Unix does not have boolean values
So how can test return a value ...
that can be used in an if statement?
It returns a value through the status code
Every program on Unix must return a status code ...
before it finishes running
If the program runs without a hitch ...
it returns a status code of 0
If the program runs into a problem ...
it returns a status code of a number greater than 0
When you run test it evaluates an expression ...
and returns 0 if the expression is true ...
and 1 if the expression is false
To a programmer like me, this is weird
In most scripting languages 0 is false ...
and any value greater than 0 is true
But this bit of weirdness is useful when writing scripts ...
because it means we are not limited to using test ...
in an if statement
Every Unix command returns a status code ...

so we can use any Unix command in an if statement

    $ cat if_3.sh
    #! /bin/bash
    ## 
    ## a shell script that demonstrates the Unix if construct

    if cd ~ghoffman
    then
        echo was able to go to ~ghoffman
    fi
    echo End of script
    
    $ ./if_3.sh
    was able to go to ~ghoffman
    End of script

This means that a shell script could run a command ...
that might fail ...
and then take appropriate action if it does
In bash, test is a built-in, a part of the shell
test is also a stand alone program
```
    $ which test
    /usr/bin/test
```
bash will always use the built-in version of test ...
unless you specify the absolute pathname ...
of the executable file
The two versions differ slightly

The `test` operators

test understands a number of operators
The operators test for different conditions
When used with two arguments ...
the operators are placed between the arguments

Some operators work only on numbers

Operator Condition Tested

-eq Two numbers are equal

-ne Two numbers are not equal

-ge The first number is greater than, or equal to, the second

-gt The first number is greater than the second

-le The first number is less than, or equal to, the second

-lt The first number is less than the second

Operator	Condition Tested
-eq	Two numbers are equal
-ne	Two numbers are not equal
-ge	The first number is greater than, or equal to, the second
-gt	The first number is greater than the second
-le	The first number is less than, or equal to, the second
-lt	The first number is less than the second

test uses the different operators when comparing strings

Operator Condition Tested

= When placed between strings, are the two strings the same

!= When placed between strings, are the two strings not the same

Note that test uses symbols (=) when comparing strings
But letters preceded by a dash (-eq) when comparing numbers

Operator	Condition Tested
=	When placed between strings, are the two strings the same
!=	When placed between strings, are the two strings not the same

There are a couple of operators that apply only to a single string

Operator Condition Tested

-n Whether the string given as an argument has a length greater than 0

-z Whether the string given as an argument has a length of 0

In these cases, the operator comes before the string

Operator	Condition Tested
-n	Whether the string given as an argument has a length greater than 0
-z	Whether the string given as an argument has a length of 0

Other operators apply to files and directories

Operator Condition Tested

-d Whether the argument is a directory

-e Whether the argument exits as a file or directory

-f Whether the argument is an ordinary file (not a directory)

-r Whether the argument exists and is readable

-s Whether the argument exists and has a size greater than 0

-w Whether the argument exists and is writable

-x Whether the argument exists and is executable

There are two additional operators that test uses when evaluating two test expressions

Operator	Condition Tested
-d	Whether the argument is a directory
-e	Whether the argument exits as a file or directory
-f	Whether the argument is an ordinary file (not a directory)
-r	Whether the argument exists and is readable
-s	Whether the argument exists and has a size greater than 0
-w	Whether the argument exists and is writable
-x	Whether the argument exists and is executable

They are placed between the two expressions

Operator Condition Tested

-a Logical AND meaning both expressions must be true

-o Logical OR meaning either of the two expressions must be true

The exclamation mark, ! , is a negation operator
It changes the value of the logical expression that follows it
- It changes a false expression to true
- And a true expression to false
I find it very hard to remember these operators
This is why I urged you not to write anything but the simplest shell scripts
If you need to write a script that uses conditionals ...
do it in a real scripting language ...
like Perl or Python

Operator	Condition Tested
-a	Logical AND meaning both expressions must be true
-o	Logical OR meaning either of the two expressions must be true

Today's Topics

Review

New Material

Homework 10

Announcement

Review

Shell Modification of the Command Line

History Expansion

Alias Substitution

Brace Expansion

Tilde, ~, Expansion

Parameter and Variable Expansion

Arithmetic Expansion

Command Substitution

Word Splitting

Pathname Expansion

Process Substitution

New Material

Shell Scripts

Shell Script Control Structures

The if ... then Construct

test

The test operators

Attendance

Class Quiz

The `if` ... `then` Construct

`test`

The `test` operators