IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 22

Review

Shell Modification of the Command Line
History Expansion
Alias Substitution
Brace Expansion
Tilde, ~, Expansion
Parameter and Variable Expansion
Arithmetic Expansion
Command Substitution
Word Splitting
Pathname Expansion
Process Substitution

Microphone

Homework 10

Homework 10 is posted here.

As usual, it will be due this coming Sunday at 11:59 PM.

Quiz 8

Let's look at the answers to Quiz 8.

Questions

Are there any questions before I begin?

Review

Shell Modification of the Command Line

When you hit Enter or Return the command line is given to the shell
Before the shell executes the commands ...
it looks to see if it needs to change anything
The shell may actually rewrite the command line before executing it
It does this to implement features of the shell like command substitution and pathname expansion
There are 10 different ways in which the shell can modify the command line
The order in which the shell performs them is as follows
1. History Expansion
2. Alias Substitution
3. Brace Expansion
4. Title, ~, Expansion
5. Parameter and Variable Expansion
6. Arithmetic expansion
7. Command substitution
8. Word splitting
9. Pathname expansion
10. Process substitution

History Expansion

This is when you use the exclamation mark, ! to run again a previous command

$ history 5
  540  cat output.txt 
  541  echo "Go Red Sox" > output.txt
  542  cat output.txt 
  543  echo foo
  544  history 5

$ !543
echo foo
foo

Alias Substitution

After history expansion, Bash performs alias substitution

$ alias ll="ls -l"
$ ll
total 2
lrwxrwxrwx 1 it244gh man    34 Sep  6 21:09 it244 -> /courses/it244/s19/ghoffman/it244gh
drwxr-xr-x 2 it244gh ugrad 512 Oct 27 09:16 work

The shell replaces the name of the alias with the value of the alias
Aliases allow you to run complicated commands by typing only a few characters

Brace Expansion

After alias substitution, Bash performs brace expansion
Braces, { }, allow you to create several strings all at once
Braces can appear with strings of characters in front or behind
The braces contain strings of characters separated by commas
The expansion creates multiple strings
One for each string contained within the braces

If I wanted to create 5 foo files I could use braces expansion as follows

$ touch foo{1,2,3,4,5}.txt

$ ls
foo1.txt  foo2.txt  foo3.txt  foo4.txt  foo5.txt

The strings inside the braces can contain one or more characters
```
$ touch {a,ab,abc}.txt

$ ls
abc.txt  ab.txt  a.txt
```
There must not be any unquoted spaces or tabs within the braces
If there are, the expansion will not work properly

Tilde, ~, Expansion

Next, Bash performs tilde expansion
Tilde, ~ , by itself is replaced with the absolute pathname ...
of your home directory
```
$ echo ~
/home/ghoffman
```
Tilde followed by a Unix user name, is replaced with absolute pathname ...
of the home directory for that user
```
$ echo ~it244gh
/home/it244gh
```

Parameter and Variable Expansion

Parameter and variable expansion comes next
Variables and parameters have values
To get these values you put a $ before the name
Bash replaces this string with its value
```
$ echo $SHELL
/bin/bash

$ echo $?
0
```

Arithmetic Expansion

Next comes arithmetic expansion
Unix treats everything on the command line as text, unless told otherwise
```
$ echo 5 + 4
5 + 4
```
Arithmetic expansion lets you do arithmetic on the command line
You put the arithmetic expression inside two parentheses, preceded by a $
```
$ echo $(( 5 + 4 ))
9
```
You can use variables inside this construction
```
$ echo $a $b
5 3
```
$ echo $(($a * $b)) 15
You do not have to put a $ before variable names
```
$ echo $(( a * b))
15
```

Command Substitution

Next comes command substitution
You put a Unix command inside single parentheses, preceded by a $
Bash runs the command in a subshell
Then it replaces the construction with the output of the command
Command substitution uses the following format
```
$(COMMAND)
```
Where COMMAND is any valid Unix command

To set a variable to the current time and date, I could use

$ today=$(date)

$ echo $today
Tue Oct 25 17:00:07 EDT 2011

Word Splitting

Next comes word splitting
When Bash gets a command line it spits the text into tokens
Tokens are strings of characters usually separated by spaces
But there are some situations where you want another character to separate tokens
You can do this using the Unix keyword variable IFS
IFS stands for Internal Field Separator
If you give IFS a value ...
Bash will be use this character to separate tokens
When I run the script args.sh ...
it shows each of its tokenized arguments ...

within square brackets, [ ]

$ args.sh $PATH
1 args: [/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/lib/oracle/12.1/client64/bin:/home/ghoffman/bin/shell:/home/ghoffman/bin/python/umb:/home/ghoffman/bin:/home/ghoffman/bin/shell:/home/ghoffman/bin/python/umb]

$ IFS=:

$ args.sh $PATH
15 args: [/usr/local/sbin] [/usr/local/bin] [/usr/sbin] [/usr/bin] [/sbin] [/bin] [/usr/games] [/usr/local/games] [/snap/bin] [/usr/lib/oracle/12.1/client64/bin] [/home/ghoffman/bin/shell] [/home/ghoffman/bin/python/umb] [/home/ghoffman/bin] [/home/ghoffman/bin/shell] [/home/ghoffman/bin/python/umb]

Word splitting will not be on the final

Pathname Expansion

Next comes pathname expansion
Pathname expansion is where you use meta-characters ...
to specify one or more pathnames
The metacharacters are used to create patterns ...
that are called ambiguous file references
The meta-characters are
- ?
- *
- [ ]

Process Substitution

Next comes process substitution
Process substitution allows you to create a file on the fly
You use a subprocess running a program to create lines of text
These lines of text can then be used at the command line ...
as if they were contained in an ordinary file
Process substitution uses the following format
```
<(COMMAND)
```
The output of COMMAND is placed in a Unix structure ...
called a named pipe
The named pipe can be used as a command line argument ...
as if it were a filename
Process substitution will not be on the final

Attendance

New Material

Shell Scripts

Most of the rest of this course will deal with shell scripts
Shell scripts are a series of Unix commands placed in a text file
You can run a shell script like any other program
Shell scripts allow you to automate certain routine operations
Much of the work in Unix system administration is done using shell scripts
Shell script programming is not like other kinds of programming
Unix commands are not uniform in the way they work ...
because each was developed separately by different people
And each group had their own ideas of how things should work
In Unix there is no overall authority to specify standards...
as there is with Windows and MacOS
Also the control structures used in shell scripts work differently ...
than those in programming languages
They can be very confusing
As a general rule you should only use shell script for simple tasks
If you need if statements or loops you probably should not use a shell script
You should instead use a scripting language like Python
But you should know how to read shell scripts
That is why I am spending so much time on shell scripts
When you run a shell script ...
your current shell creates a sub-shell to run the script
You must have both read and execute permissions to run a script ...
without first typing Bash

Shell Script Control Structures

The order in which script commands are executed ...
is called the flow of control ...
or the path of execution
Without control structures the flow of control would be simple
It would start at the top and go to the bottom
Once
There is not a lot you can do which such scripts
Control structures allow commands in a script to be executed ...
in an different order
There are two basic types of control structures
- Conditionals
- Loops
Loops are constructs that repeat a number of statements ...
until some condition is reached
Conditionals are also called branches
Conditionals are like a fork in the road
Which fork the code takes ...
depends on some condition

The Problem With Unix `if` Statements

All computer languages that I know of have an if statement
In an if statement, a certain segment of code is executed ...
but only if some condition is met
Usually this is done by evaluating something that can only be true or false
A value that can only be true or false is called a boolean value
But Unix does not have the concept of a boolean value
Everything in Unix is text ...
unless you do something special
Where another language would use a boolean value ...
Unix uses the results of running a command
If the command runs without error ...
it returns an exit status of 0
Unix treat an exit status of 0 as true
It treats any other exit status value as false
This works ...
but it makes working with if statements more complicated
It takes some getting use to to program this way
That's one of the reasons I suggest using shell scripts only for simple tasks
When Unix was created there were no languages like Python or Perl
The only choices were C and shell scripts
You would not want to use C to automate routine system maintenance work
So shell scripts were the best choice
Today you have more options

The `if` ... `then` Construct

The most basic conditional is the if ... then construction

Which has the format

if TEST_COMMAND
then
    COMMAND_1
    COMMAND_2
    ...
fi

fi is if spelled backwards
The exit status of the command following if ...
determines what happens
If the exit status is 0 ...
the commands between then and fi are run
Otherwise they will be skipped over
This flowchart shows how it works

Here is a script that that uses an if statement

$ cat foo_in_dir.sh
#! /bin/bash

# prompts for a directory and creates foo.txt there

read -p 'Directory: ' dir
if cd $dir
then
    touch foo.txt
fi

Whenever I write an if statement
I always indent the code between then and fi
I don't have to do this ...
but it makes the code more readable
Another thing I do is put then on the same line as if
It makes the code a little shorter

To do this I must put a ; before then

if COMMAND; then
    COMMAND_1
    COMMAND_2
    ...
fi

You do not have to do this in your scripts
But I would recommend indenting the commands between then and fi

Why Use Exit Status?

Using the exit status in an if statement might seem strange
But it makes some sense when you think what scripts are used for
Scripts are a series of Unix commands
Some commands will only work if a previous command was succesful
Scripts are used to install software packages
Such a script might ask the user for the name of an install directory ...
then call cd to go to that directory
If the directory did not exists ...
you would not want to run the rest of the script

`test`

What if I wanted to make sure a variable was greater than 5 ...
before doing something in a script
I want to use an if statement
But I need to run some command ...
that will give an exit status of 0 ...
if the variable is greater than 5
Unix provides such a command in test
test takes a boolean expression as an argument
A boolean expression is something whose value ...
can only be true ot false
test returns an exit status of 0 if the expression is true ...
and returns 1 if it is not
test is a built-in
So it runs inside the script's process
To a programmer like me, using 0 for true is weird
In many computer languages 0 is false ...
and any value greater than 0 is true
To be blunt, test is a kludge
A kludge is an ugly solution to a problem
Like using duct tape and a clear plastic sheet ...
to patch a broken car window

The `test` operators

test can be used to check for a number of different conditions
But this is not done in the way you would expect
In most computer languages you write boolean expressions ...
using the operators you learned in algebra
Operators like >, < and =
Unfortunately the situation with test is much more complicated
test has five different categories of operators
- Operators for comparing numbers
- Operators for comparing strings
- File and directory operators
- String operators
- Logical operators
This is one reason you should avoid shell scripts ...
when you need to to something complicated

`test` Operators for Comparing Numbers

You might expect that comparing numbers ...
would use the same operators as in algebra ...
but you would be wrong

Instead the operators are two letters following a dash, -,

Operator	Result
-eq	True if the numbers are equal
-ne	True if the numbers are not equal
-ge	True if the first number is greater than, or equal to, the second
-gt	True if the first number is greater than the second, the second
-le	True if the first number is less than, or equal to, the second, the second
-lt	True if the first number is less than the second, the second

To give you some examples, I need to use echo ? to see the exit status

$a=1; b=2

$ test $a -gt $b; echo $? # is a greater than b?
1

$ test $a -ge $b; echo $? # is a greater than or equal to b?
1

$ test $a -lt $b; echo $? # is a less than b?
0

$ test $a -le $b; echo $? # is a less than or equal to b?
0

$ test $a -eq $b; echo $? # is a equal to b?
1

$ test $a -ne $b; echo $? # is a not equal to b?
0

`test` String Operators

test uses different operators with strings
There are two sets of of string operators
Comparison operators that compare two strings

Operator Result

= True if two strings are the same

!= True if two strings are NOT the same
And operators that work on a single value

-n True if the string has a length greater than 0

-z True if the string has a length of 0

Operator	Result
=	True if two strings are the same
!=	True if two strings are NOT the same

Here are some examples

$ s1=foo; s2=bar

$ test $s1 = $s2; echo $?
1
 
$ test $s1 != $s2; echo $?
0

$ test -n  $s2; echo $?
0

$ test -n  ""; echo $?
1

$ test -z  ""; echo $?
0

Remember, Unix is case sensitive ...

and so are the string comparison operators

$ test "foo" = "FOO"; echo $?
1

$ test "foo" != "FOO"; echo $?
0

File and Directory Operators

There are operators to test that can be used on files and directories

They take a single argument that comes after the operator

Operator	Result
-d	True if the argument is the name of a directory
-e	True if the argument is the name of a file or directory
-f	True if the argument is the name of an ordinary file (i.e. not a directory)
-s	True if the argument is the name of a file or directory that is not empty
-r	True if the argument is the name of a file or directory that is readable
-w	True if the argument is the name of a file or directory that is writable
-x	True if the argument is the name of a file or directory that is executable

Here are some examples

$ ls -ld dir1 # the directory I will use for testing
drwxr-xr-x  4 glenn  staff  128 Aug  9 11:24 dir1

$ test -d dir1; echo $? # is it a directory?
0

$ test -e dir1; echo $? # is it a file or a directory?
0

$ test -f dir1; echo $? # is it an ordinary file?
1

$ test -s dir1; echo $? # does the directory have things in it?
0

$ test -r dir1; echo $? # do I have read permission?
0

$ test -w dir1; echo $? # do I have write permission?
0

$ test -x dir1; echo $? # do I have execute permission?
0

$ ls -l foo1.txt # the file I will use for testing
-rw-r--r--  1 glenn  staff  46 Aug  9 11:24 foo1.txt

$ test -e foo1.txt ; echo $? # is it a file or a directory?
0

$ test -f foo1.txt ; echo $? # is it an ordinary file?
0

$ test -s foo1.txt ; echo $? # does the file have something in it?
0

$ test -r foo1.txt ; echo $? # do I have read permission?
0

$ test -w foo1.txt ; echo $? # do I have write permission?
0

$ test -x foo1.txt ; echo $? # do I have execute permission?
1

I use the -d, -f, -r and -x options in my test scripts for Class Exercises

Logical Operators

The logical operators work on the results of other test expressions

They are

Operator	Condition Tested
-a	Logical AND, meaning both expressions must be true
-o	Logical OR, meaning either of the two expressions must be true
!	Logical NOT, meaning true becomes false and false becomes tru

Examples

$ test 1 -lt 5 -a 5 -lt 10; echo $? # is 1 less than 5 AND 5 less than 10
0

$ test 5 -lt 1 -a 5 -lt 10; echo $? # is 5 less than 1 AND 5 less than 10
1

$ test 5 -lt 1 -o 5 -lt 10; echo $? # is 5 less than 1 OR 5 less than 10
0

$  test 5 -lt 1; echo $? # is 5 less than one
1

$  test !  5 -lt 1; echo $? # is 5 NOT less than 1
0

A Final Word on `test`

I can remember most of these operators because I use them in test scripts
But if you don't use test much it will be hard to remember them
If you try, you are likely to become very frustrated
This is why I advise people to only write simple Bash scripts
For everything else, use a scripting language like Python
I won't ask you to remember all of these operators on a quiz or final

-n	True if the string has a length greater than 0
-z	True if the string has a length of 0

Review

New Material

Homework 10

Quiz 8

Questions

Review

Shell Modification of the Command Line

History Expansion

Alias Substitution

Brace Expansion

Tilde, ~, Expansion

Parameter and Variable Expansion

Arithmetic Expansion

Command Substitution

Word Splitting

Pathname Expansion

Process Substitution

Attendance

New Material

Shell Scripts

Shell Script Control Structures

The Problem With Unix if Statements

The if ... then Construct

Why Use Exit Status?

test

The test operators

test Operators for Comparing Numbers

test String Operators

File and Directory Operators

Logical Operators

A Final Word on test

Class Exercise

Class Quiz

The Problem With Unix `if` Statements

The `if` ... `then` Construct

`test`

The `test` operators

`test` Operators for Comparing Numbers

`test` String Operators

A Final Word on `test`