IT 244: Introduction to Linux/Unix
Class 17
Review
New Material
Microphone
Graded Quiz
You can connect to Gradescope to take weekly graded quiz
today during the last 15 minutes of the class.
Once you start the quiz you have 15 minutes to finish it.
You can only take this quiz today.
There is not makeup for the weekly quiz because Gradescope does not permit it.
Readings
If you have the textbook you should read chapter 8,
The Bourne Again Shell.
Homework 8
I have posted Homework 8 here.
As usual, it will be due next Sunday at 11:59 PM.
Answers to Midterm Questions
I have posted the answers to the odd questions
here
and the answers to the even questions
here.
You will find links to them on the class web page.
Midterm Scores
If you believe I have scored one of your Midterm answers incorrectly you
can request a rescore for that question on Gradescope.
If your score on the Final is significantly better than your score on the Midterm,
I will replace your Midterm grade with that of the Final when calculating your
grade for the course.
If you want to discuss how you can improve your grade in this course,
come see me during Office Hours.
You do not have to make an appointment to see me during Office Hours,
but you may have to wait your turn if I am helping another student.
I want students to pass this course, and will do what I reasonably can to
make that happen.
Your Current Standing in This Course
If you want to know your grade as of this moment, send me an email.
In performing this calculation, I will assume that your score on the final
exam will be the same as that of you Midterm score.
I can do this only for your grade as of the Midterm.
I can't do this for your grade later in the semester.
Final Exam
The final for this course will take place
on Tuesday, December 17th, from 11:30 AM - 2:30 PM.
It will NOT take place in this room but in University Hall Y02-2330
You will also find this information at the top of the class web page.
Questions
Are there any questions before I begin?
Review
Pathname Expansion
- Pathname expansion
allows you to specify a file or directory ...
- without typing the full name
- It also allows you to specify more than one file or directory ...
- with a single string of characters
- Pathname expansion uses characters with special meaning to the shell
- These special characters are called
meta-characters
- Meta-characters are also sometimes called
wildcards
- They allow you to specify a pattern
- The shell replaces the pattern with pathnames that match
- The shell then runs this altered command line
- The pattern is called an
ambiguous file reference
- You can use as many meta-characters as you want to form a pattern
- Pathname expansion is different from
pathname completion ...
- which you get by hitting Tab
- The square brackets, [ and ], are also meta-characters
- They work somewhat like ?
- They only match a single character in a pathname
- But the pathname character must match one of the characters ...
- within the brackets
- There can be many characters within the brackets
- But it only matches a single character
- You can use the bracket meta-characters with any program
- You can use a range to avoid listing all characters
- A range is created by listing the first and last characters of a sequence ...
- separated by a dash, -
- The range must be writen in alphabetical order
- The square brackets provide another shortcut
- An exclamation mark, ! ,
or a caret, ^ has special meaning inside
the brackets ...
- but only when it appears as the first character
- This construction matches any character that is NOT included within the brackets
Built-ins
Help for Built-ins
- If you run
man on a built-in command you will get nothing
$ man bg
No manual entry for bg
- For help information on built-ins run
help
- which is itself a built-in ...
$ type help
help is a shell builtin
- If you follow
help with the name of a built-in
it will print some information on that command
$ help bg
bg: bg [job_spec ...]
Move jobs to the background.
Place the jobs identified by each JOB_SPEC in the background, as if they
had been started with `&". If JOB_SPEC is not present, the shell"s notion
of the current job is used.
Exit Status:
Returns success unless job control is not enabled or an error occurs.
- If you run
help with no argument it will show all the built-in commands
$ help
GNU bash, version 4.3.11(1)-release (x86_64-pc-linux-gnu)
These shell commands are defined internally. Type `help" to see this list.
Type `help name" to find out more about the function `name".
Use `info bash" to find out more about the shell in general.
Use `man -k" or `info" to find out more about commands not in this list.
A star (*) next to a name means that the command is disabled.
job_spec [&] history [-c] [-d offset] [n] or history -anrw [filename>
(( expression )) if COMMANDS; then COMMANDS; [ elif COMMANDS; then COMMA>
. filename [arguments] jobs [-lnprs] [jobspec ...] or jobs -x command [args]
: kill [-s sigspec | -n signum | -sigspec] pid | jobspec >
[ arg... ] let arg [arg ...]
[[ expression ]] local [option] name[=value] ...
alias [-p] [name[=value] ... ] logout [n]
bg [job_spec ...] mapfile [-n count] [-O origin] [-s count] [-t] [-u fd] >
...
Ways a Shell Can Be Created
- There are three ways a user can create a shell
- Logging in
- On the command line
- When running a scritp
- There are subtle differences between these three types of shells
The Login Shell
- When you first login to Unix, a shell is created for you
- This shell is your login shell
- The absolute pathname
of this shell is contained in the variable SHELL
- On pe15 the default login shell version is Bash
- When your login shell starts up it runs the commands found in /etc/profile
- This is a file customized by the system administrator for all users
- You can create your own customizations in a
startup file
...
- in you home directory
- The file must have one of these names
- .bash_profile
- .bash_login
- .profile
- We will use .bash_profile
Shells Created at the Command Line
Shells Created to Run a Script
- A shell script is a text file with Unix commands
- You have been creating shell scrips for the Class Exercises ...
- and homework assignments
- A shell script contains Unix commands ...
- which only the shell can understand
- So when a process is created to run a script ...
- two files are loaded into the process RAM
- The text file containing the script
- The Bash binary executable file
- The picture in memory looks like this
- So you always running another shell ...
- when you are running a script
Creating Startup Files
- A startup file contains Unix commands that are run ...
- just before you get a prompt
- The system administrator creates a startup file for all users in
/etc/profile
- After you login shell runs this file ...
- it looks for a startup file in your home directory
- The user customizable startup file normally used by Bash is
.bash_profile
- This file must be placed in your home directory
- We'll discuss this more when we talk about shell variables and aliases
Different Shell Versions
- The shell we have been using is Bash
- Bash stands for Bourne again shell
- The first Unix shell was written by Steve Bourne at AT&T's Bell
Laboratories
- You can still run this shell by typing
sh
- System V Unix introduced the Korn shell,
ksh ...
- which was written by David Korn
- It introduced
aliases and
command line editing
- Machines running Ubuntu have another shell called
dash
dash is a stripped down shell with very few features- It was designed to run shell scripts
- Another popular shell is the C shell
- It is popular with C programmers ...
- because it's scripting language looks like C
- There are two versions of this shell
- But the difference between them is slight
- Both the Korn and C shells cannot run Bash scripts ...
- because of slight differences between them and Bash
- We will only concern ourselves with Bash ...
- and to a lesser extent
sh
Using a Different Shell
- Different version of Unix and Linux have different shells installed
- You can run a different version of the shell as a subshell of Bash
- You do this by simply typing the name of the shell
$ sh
$ ps
PID TTY TIME CMD
13942 pts/5 00:00:00 bash
13953 pts/5 00:00:00 sh
13955 pts/5 00:00:00 ps
- To find out what shells are available on a machine ...
- look at the text file /etc/shells
- Here are the shell available on pe15
$ cat /etc/shells
# /etc/shells: valid login shells
/bin/sh
/bin/bash
/usr/bin/bash
/bin/rbash
/usr/bin/rbash
/bin/dash
/usr/bin/dash
/usr/bin/tmux
/usr/bin/screen
/bin/tcsh
/usr/bin/tcsh
/bin/zsh
/usr/bin/zsh
/usr/bin/fish
POSIX
- A standard exists for how shells should run on Unix
- It was created by Portable Application Standards Committee
- A committee of the IEEE (Institute of Electrical and Electronics Engineers )
- It is called POSIX (Portable Operating System Interface) 1003.2
sh and dash are fully POSIX compliant- But Bash is not
- This is due to certain design decisions made when Bash was created
- If you run Bash with the --posix option ...
- it will will be more compatible with the POSIX standard
Shells For Running Scripts
Attendance
New Material
Running a Startup File after a Change has been Made
- Usually, when you change a startup file
- You want the changes to take effect immediately
- But unless you do something special ...
- the changes won't take effect until you log in again
- You can make the changes take effect immediately ...
- by running the
source command
source .bash_profile
source is a built-in
$ help source
source: source filename [arguments]
Execute commands from a file in the current shell.
Read and execute commands from FILENAME in the current shell. The
entries in $PATH are used to find the directory containing FILENAME.
If any ARGUMENTS are supplied, they become the positional parameters
when FILENAME is executed.
Exit Status:
Returns the status of the last command executed in FILENAME; fails if
FILENAME cannot be read.
source runs a script inside your current shell- When you run a script a new process is created
- The program running in this process is a shell
- If you ran .bash_profile like this
./bash_profile
- Any changes it made would be made in the shell ...
- used to run the script
- But that shell would disappear as soon as the script ended
- So none of the changes would take effect in your current shell
Commands that are Symbols
- Unix has some commands that are symbols rather than words
- I'll just mention them now and go into greater detail in future classes
| ( ) |
Runs whatever commands are enclosed in the parentheses in a sub-shell |
| $( ) |
Command substitution:
runs the commands enclosed in the parentheses in a subshell and returns their value
to the command line, replacing the dollar sign, the parentheses and everything
in them with this value
|
| (( )) |
Evaluates an arithmetic expression:
by default, Unix treats everything as text, but
this command evaluates whatever it encloses as a numerical,
rather than a string, expression
|
| $(( )) |
Arithmetic expansion:
evaluates an arithmetic expression and returns its value at that place on the command line
|
| [ ] |
The test command:
used to evaluate a boolean expression in constructs like if clauses
|
| [[ ]] |
The conditional expression:
similar to [ ] but adds string comparisons
|
File Descriptors
- Resources are given to each process when it is created
- Every time a process is created it is given connections to three data streams
- Any program can open files besides these three standard data streams
- So how does Unix keep track these multiple streans?
- It does so through file descriptors
- File descriptors represent data streams or files that your script can access
- Each file descriptor is assigned a positive number starting with 0
- Think of a file descriptor as an integer that represents a file
- Standard input, standard output and standard error ...
- each have their own file descriptors
| Name | File Descriptor |
|---|
| Standard input | 0 |
| Standard output | 1 |
| Standard error | 2 |
- So while we think of standard input, standard output and standard error
- Unix thinks of the file descriptors 0, 1 and 2
- Most of the time, you do not have to worry about file descriptors
- But they can appear in complex scripts
Redirecting Standard Error
- Standard error is the data stream where error messages are usually sent
- Redirecting standard error allows a program to separate its output ...
- from its error messages
- To redirect standard input we use the less than symbol, <
- Followed by a pathname
$ ./repeat.sh < test.txt
Enter several lines
Type X on a line by itself when done
You entered
-----------
123456789
abcdefg
987654321
hijklmnop
foo
bar
bletch
X
- < is really a shorthand for a notation using file descriptors
- When you type
./repeat.sh < test.txt
Unix thinks of this as
./repeat.sh 0< test.txt
- Where the 0 in front of the greater than sign ...
- is the file descriptor for standard input
- Similarly, when we use output redirection
$ echo Hello there > hello.txt
Unix thinks of this as meaning
$ echo Hello there 1> hello.txt
- Again the file descriptor precedes the redirection symbol, >
- So how do we redirect standard error?
- We place a 2 in front of the greater than symbol, >
$ ls xxxx
ls: cannot access xxxx: No such file or directory
$ ls xxxx 2> error.txt
$ cat error.txt
ls: cannot access xxxx: No such file or directory
- Redirecting standard error using 2> ...
- sends the error messages to the file error.txt
...
- not to the screen
- You can redirect both standard output and standard error to the same file
- You do this with ampersand and greater than symbols together, &>
Shell Scripts
- A shell script is a file that contains Unix commands
- You can think of a shell script as a collection of command line entries
- When the shell script is executed each line of the script is run in turn
- A shell script can use any shell feature that is available at the
command line ...
- except those features which are provided by
tty
- Command line editing(arrow keys, control key combinations)
- Pathname completion (tab to get more of a filename)
- The history mechanism (up arrow to recall previous command line)
- You can use ambiguous file references in a shell script
- That is, you have full use of the metacharacters ?,
*; and [ ]
- You can use redirection in a shell script and pipes
- Unix also provides control structures
- When you run a script
- The shell executes each commmand it comes across
- Normally that means starting at the top ...
- and going straight to the bottom
- Control structures allow you to change the path taken through the script
Making a Shell Script Executable
- You can run a shell script without using Bash ...
- if you give the script both
read
and execute permissions
- You need read permission because the shell has to read the contents of the script
- You need execute permission so the script can be run without calling Bash
- If you try to run a script without both permissions
you will get an error
$ ls -l cheer.sh
-rw-rw-r-- 1 ghoffman grad 13 Oct 29 14:23 cheer.sh
$ cat cheer.sh
#! /bin/bash
# this file roots for the home team
echo "Let's go Red Sox!"
$ ./cheer.sh
-bash: ./cheer.sh: Permission denied
- Of course, you set these permissions using
chmod
- Normally you would give a shell script file 755 permissions
- The owner can read, write and execute
- The group and everyone else can read and execute
$ chmod 755 cheer.sh
$ ls -l cheer.sh
-rwxr-xr-x 1 ghoffman grad 13 Oct 29 14:23 cheer.sh
$ ./cheer.sh
Go Sox!
How Unix Runs a Program
- When you run a program in Unix the kernel creates a process ...
- and gives this process its own chunk of memory
- The most important thing stored in this memory is the code itself
- It consists of binary machine language instructions
- Each of these instructions is executed one at a time
- The CPU needs to know which instruction to execute next
- This is why the computer has a special piece of memory ...
- called the
program counter
- The program counter contains the memory address of the next instruction
- After each instruction finishes ...
- the program counter advances to the next instruction address
Running a Shell Script
- The shell is just a special kind of program
- It reads what you enter at the command line ...
- and runs programs for you ...
- asking the kernel to create a process in which to run the program
- When a program is running ...
- the shell sleeps until the program is finished ...
- unless you tell the shell to run the command in the background
- What is the binary code that is loaded into memory when you run a shell script?
- The script itself is just text which the CPU cannot understand
- To run the script we need to load a binary executable file
- This file must understand the lines of the script ...
- and execute each one in turn
- The only binary executable that can understand a shell script is a shell
Specifying Which Shell Will Run a Script
- The kernel needs to load the binary code for a shell ...
- into the process that runs the script
- The picture in memory looks like this
- But each Unix version has many shells that can be run
- Which shell should be loaded to run a script ?
- Normally the shell chosen will be the one you are currently using
- So if you are running Bash to give you a command line ...
- the kernel will load Bash into the process to run the script
- And if you are running
sh, the binary for sh will be used
- What if you need to run the script in different shell?
- It makes little difference whether we use Bash or
sh
to run the scripts for this course
- But it would make a big difference if we were writing an install script for a package
- Install scripts have to be able to run on any Unix machine
- So install scripts are written to run on POSIX compatible shells
- But Bash is not fully POSIX compatible
- Unix provides us with a solution to this problem
- It allows us to specify which shell to run when a script is executed
- It does this using something called the
hashbang line ...
- sometimes called the shebang line
- This line must be the first line of the script
- And the first two characters on the line must be a hash mark, #, ...
- followed by an exclamation mark, !
- The exclamation mark is sometimes called bang
- After these two characters comes the absolute pathname of the shell ...
- that will run with the script
- The pathname following #! must be an absolute pathname
- Because you don't know where the user will be when the script is run
- The hashbang line tells your current shell which shell to use to run your script
- The hashbang line must be the first line in the script
- Unix looks at the first few characters of a file before running a script
- If it sees #! ...
- it runs the script using the shell whose pathname follows #!
- For this course, the hashbang line should be
#! /bin/bash
- It is good form to always use a hashbang line
- You may follow hashbang with a couple of spaces before the pathname
- To show you that this really works I'm going to run the script shell_test_1.sh
$ cat shell_test_1.sh
#! /bin/sh
ps -f
$ ./shell_test.sh
UID PID PPID C STIME TTY TIME CMD
ghoffman 710 709 0 13:25 pts/1 00:00:00 -bash
ghoffman 2741 710 0 15:35 pts/1 00:00:00 /bin/sh ./shell_test.sh
ghoffman 2742 2741 0 15:35 pts/1 00:00:00 ps -f
- I specified that this script should be run with the sh shell
- I did this by specifying it in the hashbang line
#! /bin/sh
- Now compare this with shell_test_2.sh
$ cat shell_test_2.sh
ps -f
$ ./shell_test_2.sh
UID PID PPID C STIME TTY TIME CMD
ghoffman 710 709 0 13:25 pts/1 00:00:00 -bash
ghoffman 2893 710 0 15:41 pts/1 00:00:00 -bash
ghoffman 2894 2893 0 15:41 pts/1 00:00:00 ps -f
- The second script has no hashbang line ...
- so the script was run in a Bash shell
- The shell did this because I did not tell it otherwise
- You can leave out the hashbang line and still run a script without calling Bash
- But you must use a hashbang line for scripts written in scripting languages ..
- like Perl and Python
- Here is a simple Python script that does not have hashbang line
$ cat hello_1.py
print("Hello world!")
- It has read and execute permissions
$ ls -l hello_1.py
-rwxr-xr-x 1 ghoffman it244-1 22 Mar 29 09:59 hello_1.py
- But when I try to run it, there is a problem
$ ./hello_1.py
./hello_1.py: line 1: syntax error near unexpected token `"Hello world!"'
./hello_1.py: line 1: `print("Hello world!")'
- I can only run this script by calling the Python interpreter
$ python hello_1.py
Hello world!
- Here is the same script with a hashbang line that references the Python interpreter
$ cat hello_2.py
#! /usr/bin/python
print("Hello world!")
- I can run this script directly
$ ./hello_2.py
Hello world!
- You will not have to use the hashbang line for homework assignments ...
- until you get to homework 11
- Homework 11 is the last homework assignment
- The scripts you will write for that assignment must all have the following
hashbang line
#! /bin/bash
- Programs are written by people for machines
- But programs also have to be read by the people
- Who write the program
- Who maintain the program
- Who use the program
- To make clear what is happening inside a program, use comments
- Comments are text which is ignored by whatever program is running the script
- They are only meant for people to read
- Anything following a hash mark, #, is a comment ...
- except for the hashbang line
- Here is an example
$ cat comment_test.sh
#! /bin/sh
# demonstrates that comments do not affect the
# way the script runs
echo Hello there
$ ./comment_test.sh
Hello there
- Comments are a way to document a program ...
- within the text of the program itself
- This sort of documentation is extremely important
- You may create a script today and not use it for a couple of months
- When you need to change it, you may have forgotten how it works
- A few well placed comments can save you hours of work
- It is good practice to place a comment at the top of the shell script ...
- after the hashbang line
- This comment should say what the script does
$ cat bother.sh
#!/bin/bash
# prints something to the terminal 1000 times
count=0
while [ $count -lt 1000 ]
do
sleep 5
echo Excuse me
(( count++ ))
done
- This script uses features of bash that we will discuss in future classes
- You should also comment any part of a script that does something less than obvious
- You will not have to write comments, other than Step comments, until homework 11
- Homework 11 is the last homework assignment
Class Exercise
Class Quiz