IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 16

Today's Topics

Review

Pathname Expansion
The ? Meta-character
The * Meta-character
The [ and ] Meta-characters
Built-ins
Different Versions of the Shell
Ways a Shell Can Be Created
Your Login Shell
Interactive Non-login Shells
Non-interactive Shells
Creating Startup Files

New Material

Running a Startup File after a Change has been Made
Commands that are Symbols
File Descriptors
Redirecting Standard Error
Shell Scripts
Making a Shell Script Executable
Specifying Which Shell Will Run a Script
Comments in Shell Scripts

Reading Assignment

You should have read chapter 8 of Sobell, The Bourne Again Shell.

Homework 8

I have posted Homework 8 here.

As usual, it will be due next Sunday at 11:59 PM.

Mid-term

Let's review the answers to the Mid-term.

Your Current Standing in This Course

If you want to know your grade as of this moment, send me an email.

In performing this calculation, I will assume that your score on the final exam will be the same as that of you mid-term score.

I can do this only for your grade as of the mid-term.

I can't do this for your grade later in the semester.

Review

Pathname Expansion

Pathname expansion allows you to specify a file or directory ...
without typing the full name
It also allows you to specify more than one file or directory ...
with a single string of characters
Pathname expansion uses characters with special meaning to the shell
These special characters are called meta-characters
Meta-characters are also sometimes called wildcards
They allow you to specify a pattern
When the shell sees meta-characters on the command line ...
it replaces the pattern with a sorted list ...
of all pathnames that match the pattern
The shell then runs this altered command line
The pattern is called an ambiguous file reference
You can use as many meta-characters as you want to form a pattern
Pathname expansion is different from pathname completion ...
which you get by hitting Tab

The ? Meta-character

The question mark, ?, meta-character stands for a single instance of any character
? can be used with any command ...
even those that don't normally deal with files
```
    $ echo dir?
    dir1 dir2 dir3 dir4
```
The ? meta-character does not match a leading period in a filename
You must explicitly enter a leading period, . ...
when specifying an "invisible" file

The * Meta-character

An asterisk, * , will match any number of characters in a pathname
It will even match no characters
* can be used with any command ...

even those that don't normally deal with files

    $ echo dir*
    dir dir1 dir10 dir2 dir3 dir4

* cannot be used to match the initial period in a hidden filename

The [ and ] Meta-characters

The square brackets, [ and ], are also meta-characters
They work somewhat like ?
They only match a single character in a pathname ...
but the pathname character must match one of the characters ...
within the brackets
No matter how many characters are within the bracket ...
the pattern can match only a single character
You can use the bracket meta-characters with any program
You can use a range to avoid listing all characters
A range is specified by listing the first and last characters of a sequence ...
separated by a dash, -
The sequence is specified in alphabetical order
The square brackets provide another shortcut
If you insert an exclamation mark, ! , or a caret, ^ ...
immediately after the opening bracket ...
the shell will match any single character ...
that is NOT included within the brackets

Built-ins

Not all commands can be found on disk as executable files
Some are actually written as part of the shell
Such commands are called built-ins
When you run a built-in ...
the shell does not have to create a new process ...
when you run these programs
Instead the command runs in the same process as the shell
This makes execution faster
You can use the help built-in command ...

like you would use man for normal commands

$ help cd
cd: cd [-L|[-P [-e]] [-@]] [dir]
    Change the shell working directory.
    
    Change the current directory to DIR.  The default DIR is the value of the
    HOME shell variable.
    ...

Different Versions of the Shell

The shell we have been using is Bash
Bash stands for Bourne again shell
The original Bourne shell was written by Steve Bourne ...
at AT&T's Bell Laboratories
The original Bourne shell
also called the sh shell ...
is still with us
Many important scripts were written using the sh shell
Those scripts are still in use to setup and maintain Unix
There are subtle differences between different shells
It is always best to run a script in the shell ...
in which it was originally written
A standard exists for how shells should run on Unix
It was created by Portable Application Standards Committee ...
of the IEEE (Institute of Electrical and Electronics Engineers )
It is POSIX (Portable Operating System Interface) 1003.2
The GNU community is working on making Bash fully compliant with POSIX
Until then you can run Bash with the --posix option
This will make Bash run in a way more compatible with the POSIX standard

Ways a Shell Can Be Created

There are three ways a user can create a shell
- Login shell
- Interactive non-login shell
- Non-interactive shell
There are subtle differences between these three types of shells

Your Login Shell

The login shell is the shell you get ...
after your password has been accepted
Each login session has one, and only one, login shell
Your default shell version you run is set ...
when your account is created
The absolute pathname of this shell is contained in the variable SHELL
On users3 the default login shell version is Bash
When your login shell starts up ...
it runs the commands found in /etc/profile
This is a file customized by the system administrator ...
for all users
You can create your own customizations in a startup file ...
in you home directory
The file must have one of these names
- .bash_profile
- .bash_login
- .profile
We will use .bash_profile

Interactive Non-login Shells

You can run another shell from your login shell

This is called a sub-shell

    $ ps
      PID TTY          TIME CMD
    12778 pts/1    00:00:00 bash
    12969 pts/1    00:00:00 ps
    
    $ bash
    
    $ ps
      PID TTY          TIME CMD
    12778 pts/1    00:00:00 bash
    12970 pts/1    00:00:00 bash
    12973 pts/1    00:00:00 ps
    
    $

Notice that the first entry for bash ...
has the same process ID each time you run ps
This is your login shell
The second bash process is the sub-shell
So your login shell is still running ...
but you are talking to a sub-shell ...
and the login shell is sleeping ...
waiting for the sub-shell to finish
This sub-shell is the second type of shell
It is not a login shell
It is an interactive non-login shell
An interactive non-login shell is a shell that you create ...
without having to enter a password
Interactive non-login shells have their own startup file ...
called .bashrc ...
and it must be in your home directory

Non-interactive Shells

A shell scripts is a file containing Unix commands
When you run this file, all the commands in the file are executed
The program that understands these commands and runs them ...
is a shell
So your current shell has to create a sub-shell ...
to run the commands in the shell script
This sub-shell does not give you a prompt ...
so it is not an interactive shell
It is a non-interactive shell
There is no standard startup file for such a shell

Creating Startup Files

A startup file contains Unix commands ...
that are run just before you get a prompt
The system administrator creates a startup file for all users ...
in /etc/profile
After you login shell runs the commands in this file ...
it looks for a startup file in your home directory
The user customizable startup file normally used by Bash
is .bash_profile
This file must be placed in your home directory
We'll discuss this more when we talk about shell variables and aliases

New Material

Running a Startup File after a Change has been Made

Usually, when you change a startup file ...
you want the changes to take place immediately
But if you made a change to .bash_profile ...
the changes won't take effect ...
until the next time you login
Unix gives away to make the changes take effect immediately ...
by running the source command
```
    source .bash_profile
```

source is a built-in

$ help source
source: source filename [arguments]
    Execute commands from a file in the current shell.
    
    Read and execute commands from FILENAME in the current shell.  The
    entries in $PATH are used to find the directory containing FILENAME.
    If any ARGUMENTS are supplied, they become the positional parameters
    when FILENAME is executed.
    
    Exit Status:
    Returns the status of the last command executed in FILENAME; fails if
    FILENAME cannot be read.

Commands that are Symbols

Unix has some commands that are symbols rather than words
I'll just mention them now and go into greater detail in future classes

( )	Runs whatever commands are enclosed in the parentheses in a sub-shell
$( )	Command substitution: runs the commands enclosed in the parentheses in a subshell and returns their value to the command line, replacing the dollar sign, the parentheses and everything in them with this value
(( ))	Evaluates an arithmetic expression: by default, Unix treats everything as text, but this command evaluates whatever it encloses as a numerical, rather than a string, expression
$(( ))	Arithmetic expansion: evaluates an arithmetic expression and returns its value at that place on the command line
[ ]	The test command: used to evaluate a boolean expression in constructs like `if` clauses
[[ ]]	The conditional expression: similar to [ ] but adds string comparisons

File Descriptors

Resources are given to each process when it is created
Every time the shell creates a process ...
it gives that process connections to three "files"
- Standard input
- Standard output
- Standard error
Any program can open other files ...
besides these three standard "files"
So how does Unix keep track these multiple files?
It does so through file descriptors
File descriptors are data structures ...
that Unix creates to handle access to files for processes
File descriptors are the abstract representation ...
of the files that are connected to a process
Each file descriptor is assigned a positive number starting with 0
Think of a file descriptor as an integer ...
that represents a file
Standard input, standard output and standard error ...
each have their own file descriptors

Name File Descriptor

Standard input 0

Standard output 1

Standard error 2
So while we think of standard input, standard output and standard error ...
Unix thinks of the file descriptors 0, 1 and 2
Most of the time, you do not have to worry about file descriptors ...
though they can appear in complex scripts.

Name	File Descriptor
Standard input	0
Standard output	1
Standard error	2

Redirecting Standard Error

Standard error is the "file" into which error messages are usually sent
Redirecting standard error allows a program to separate its output ...
from its error messages
To redirect standard input we use the less than symbol, < ...

followed by a file pathname

    $ ./repeat.sh < test.txt 
    Enter several lines
    Type X on a line by itself when done

    You entered
    -----------
    123456789
    abcdefg
    987654321
    hijklmnop
    foo
    bar
    bletch
    X

< is really a shorthand ...
for a notation using file descriptors
When you type
```
    ./repeat.sh  <  test.txt
```
Unix thinks of this as
```
    ./repeat.sh  0<  test.txt
```
where the 0 in front of the greater than sign ...
is the file descriptor for standard input

Similarly, when we use output redirection

    $ echo Hello there  >  hello.txt

Unix thinks of this as meaning

    $ echo Hello there  1>  hello.txt

Again the file descriptor precedes the redirection symbol, >
So how do we redirect standard error?

We place a 2 in front of the greater than symbol, >

    $ ls xxxx
    ls: cannot access xxxx: No such file or directory

    $ ls xxxx 2> error.txt

    $ cat error.txt 
    ls: cannot access xxxx: No such file or directory

When we redirected standard error using 2> ...
Unix sent the error messages to the file error.txt ...
not to the screen
You can redirect both standard output and standard error ...
to the same file

You do this with ampersand and greater than symbols together, &>

 
    $ cat foo1.txt foo2.txt foo57.txt
    foo to you
    bar to everyone else
    bletch to the universe foo foo foo
    bar bar bar
    bletch
    cat: foo57.txt: No such file or directory
    
    $ cat foo1.txt foo2.txt foo57.txt &> error.txt
    
    $ cat error.txt 
    foo to you
    bar to everyone else
    bletch to the universe foo foo foo
    bar bar bar
    bletch
    cat: foo57.txt: No such file or directory

Shell Scripts

A shell script is a file that contains Unix commands ...
along with their options and arguments
You can think of a shell script as a collection of command line entries
When the shell script is executed ...
each line of the script is run in turn
A shell script can use any shell feature ...
that is available at the command line ...
except those features which are provided by tty
- Command line editing (arrow keys, control key combinations)
- Pathname completion (tab to get more of a filename)
- The history mechanism (up arrow to recall previous command line)
You can use ambiguous file references in a shell script
That is, you have full use of the metacharacters ?, *; and [ ]
You can use redirection in a shell script ...
and pipes
Unix also provides control structures
- If statements
- Loops
Control structures allow you to change the path taken through the script
The shell executes the script one line at a time
The shell behaves exactly as it would ...
if you had typed in the line at the keyboard

Making a Shell Script Executable

You can run a shell script without using bash
if you give the script both read ...
and execute permissions
You need read permission ...
because the shell has to read the contents of the script
You need execute permission so the script can be run ...
without explicitly using bash
If you try to run a script without both permissions ...

you will get an error

    $ ls -l cheer.sh 
    -rw-rw-r-- 1 ghoffman grad 13 Oct 29 14:23 cheer.sh 
    
    $ cat cheer.sh 
    #! /bin/bash

    # this file roots for the home team

    echo "Let's go Red Sox!"
    
    $ ./cheer.sh
    -bash: ./cheer.sh: Permission denied

Of course, you set these permissions using chmod
Normally you would give a shell script file 755 permissions
The owner can read, write and execute

The group and everyone else can read and execute

    $ chmod 755 cheer.sh
    
    $ ls -l cheer.sh 
    -rwxr-xr-x 1 ghoffman grad 13 Oct 29 14:23 cheer.sh
    
    $ ./cheer.sh
    Go Sox!

All scripts for this course must have 755 permissions set
Points will be deducted if you don't

Specifying Which Shell Will Run a Script

The shell is just a program ...
that reads what you enter at the command line ...
and runs programs for you
It stands between you and the operating system
When the shell runs a program for you ...
it normally sleeps until the program is finished ...
unless you tell the shell to run the command in the background
When the shell runs a shell script ...
it creates a new shell inside the process ...
that will run the script
Normally this sub-shell will be the same kind of shell ...
as your login shell
So if your login shell is Bash ...
a Bash sub-shell will run the script
There are significant differences between the various shells ...
that come with Unix and Linux
What if you need to run the script in different shell?
It is always best to run a script ...
in the same shell used by the programmer ...
who wrote the script
Unix provides a way to to specify which shell to run ...
when a script is executed
It is called the hashbang line ...
or sometimes the shebang line
That's because the first two characters on the line ...
must be a hash mark, # ...
followed by an exclamation mark, !
The exclamation mark is sometimes called bang
After these two characters ...
comes the absolute pathname of the shell ...
which will run with the script
The pathname following #! must be an absolute pathname ...
because you don't know where the user will be ...
when the script is run
The hashbang line tells your current shell ...
which shell to use to run your script
The hashbang line must be the first line in the script
Unix looks at the first few characters of a file ...
before running a script
If it sees #! it interprets what follows ...
as the pathname of the program that should run the script
It is good form to always use a hashbang line
You may follow hashbang with a couple of spaces before the pathname
To show you that this really works ...

I'm going to run the script shell_test_1.sh

    $ cat shell_test_1.sh
    #! /bin/sh
    ps -f
    
    $ ./shell_test.sh 
    UID        PID  PPID  C STIME TTY          TIME CMD
    ghoffman    710   709  0 13:25 pts/1    00:00:00 -bash
    ghoffman   2741   710  0 15:35 pts/1    00:00:00 /bin/sh ./shell_test.sh
    ghoffman   2742  2741  0 15:35 pts/1    00:00:00 ps -f

I specified that this script should be run with the sh shell
I did this by specifying it in the hashbang line
```
    #! /bin/sh
```

Now compare this with shell_test_2.sh

    $ cat shell_test_2.sh
    ps -f

    $ ./shell_test_2.sh
    UID        PID  PPID  C STIME TTY          TIME CMD
    ghoffman    710   709  0 13:25 pts/1    00:00:00 -bash
    ghoffman   2893   710  0 15:41 pts/1    00:00:00 -bash
    ghoffman   2894  2893  0 15:41 pts/1    00:00:00 ps -f

The second script has no hashbang line ...
so the script was run in a Bash shell
The shell did this because I did not tell it otherwise
you can leave out the hashbang line ...
and still run a script without calling bash ...
But you must use a hashbang line for scripts ...
written in scripting languages like Perl and Python
Here is a simple Python script that does not have hashbang line
```
    $ cat hello_1.py
    print Hello world!
```

It has read and execute permissions

    $ ls -l hello_1.py 
    -rwxrwxr-x 1 ghoffman grad 21 Jun 19 17:48 hello_1.py

But when I try to run it, there is a problem

    $ ./hello_1.py 
    Warning: unknown mime-type for "Hello world!" -- using "application/octet-stream"
    Error: no such file "Hello world!"

I can only run this script by calling the Python interpreter
```
    $ python hello_1.py 
    Hello world!
```
Here is the same script with a hashbang line ...

that uses the Python interpreter

    $ cat hello_2.py 
    #! /usr/bin/python

    print "Hello world!"

I can run this script directly
```
    $ ./hello_2.py 
    Hello world!
```

Comments in Shell Scripts

Programs are written by people for machines
But programs also have to be read by the people
- Who write the program
- Who maintain the program
- Who use the program
To make clear what is happening inside a program ...
use comments
Comments are text which is ignored ...
by whatever program is running the script
They are only meant for people to read
Anything following a hash mark, #, is a comment ...
except for the hashbang line

Here is an example

    $ cat comment_test.sh 
    #! /bin/sh
    #  demonstrates that comments do not affect the
    #  way the script runs
    echo Hello there
    
    $ ./comment_test.sh 
    Hello there

Comments are a way to document a program ...
within the text of the program itself
This sort of documentation is extremely important
You may create a script today and not use it for a couple of months
When you need to change it, you may have forgotten how it works
A few well placed comments can save you hours of work
It is good practice to place a comment at the top of the shell script ...
after the hashbang line

This comment should say what the script does

    $ cat bother.sh
    #!/bin/bash
    
    # keeps printing something to the screen until it is killed

    while [ 6 -lt 10 ]
        do
        sleep 5
        echo "Excuse me"    
        done

You should also comment any part of a script that ...
does something less than opathious
I'll take off points if you don't