IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 19

Microphone

Questions

Are there any questions before I begin?

Homework 9

I have posted Homework 9 here.

As usual, it will be due next Sunday at 11:59 PM.

Graded Quiz Today

After I finish talking I will pass out the papers for today's graded quiz.

Today's Class

Today I will discuss how scripts can use command line arguments, processes and how the shell runs programs.

Tips and Examples

Don't Use Google for Homework Assignments

I often see Unix commands in homework assignments that I have not talked about in class
When this happens it usually means that a student has used Google ...
to find a command required to complete a Step
Doing this is shortsighted ...
and often leads to errors that result in points lost
Everything you need to complete a homework assignment ...
is contained in the Class Notes
It may seem easier to complete your homework by Googling ...
but you learn less when you do this
The purpose of the homework assignments is to give you hands on experience ...
with the material in the Class Notes
The act of trying something and having it work or not work ...
is critical to the learning process
Sometimes using a command properly ...
requires you to think a bit about the material
Doing this thinking is again a fundamental part of the learning process
When you Google for an answer you are avoiding the hard work of learning
You are trading a better score on an assignment ...
for the knowledge that will enable you do to better on tests ...
and in life
And sometimes Google gives you the wrong answer
For example, some students gave the wrong answer to Step 10 of homework 7
That Step read
Using meta-characters, perform a long listing of the files foo21.txt, foo25.txt and foo29.txt.
This must be done using a single command.
Some of you wrote the following
```
ls -l foo{21,25,29}.txt
```
But the { and } characters are NOT metacharacters
They are used in a feature of Unix called Brace Expansion
Think, don't Google ...
or use AI

Using Control Z with an Editor

fg is used to bring a background job into the foreground
But it will also bring a suspended job back to life
This allows you to use Control Z to switch in and out of a text editor ...
while working on a script
You launch an editor to work on a script
When you want to test your script you hit Control Z
This suspends the editor and brings you back to the command line
There you can run your script to test it
When you want to return to the editor simply enter fg
You can also use this trick with man and info

Redirecting Output From Background Job

To run a job in the background, you put an ampersand, & at the end of the line
If you want to redirect output, you must do this before the &
That means you want to do something like this
```
./bother.sh > /dev/null &
```
NOT this
```
./bother.sh & > /dev/null
```
The same is true for redirecting standard input or standard error
The & must be the last thing on the command line

Review

Shell Environment

Your shell needs certain information to do its job properly
It needs to know things like the address of your home directory ...
and what your prompt should look like
Much of this information is stored in variables

Shell Variables

A variable is a name given to a word in memory that holds a single value
Shell variables are variables that are defined in a shell ...
and can be used inside that shell
To get the value of a shell variable ...

put a dollar sign, $, in front of the variable name

$ echo $PATH
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

Some variables are set and maintained by the shell itself
They are called keyword shell variables
Or just keyword variables
A keyword is a word with special meaning to the shell
Many of these keyword variables can be changed by the user
Other variables can be created by the user
They are called user-created variables

Creating a Shell Variable

You create a variable using the following format
```
VARIABLE_NAME=VALUE
```

Like this

$ username=ghoffman

$ echo $username
ghoffman

There must be no spaces on either side of =

If you include spaces you will get an error

$ username = ghoffman
username: command not found

The value you assign to a variable must not contain a space
```
$ name=Glenn Hoffman
Hoffman: command not found
```
There are two ways to deal with this
- Escape the whitespace characters with a \
- Quote the entire value
The backslash character turns off the special meaning ...

of the character that follows it

$ name=Glenn\ Hoffman

$ echo $name
Glenn Hoffman

Of course you must put the backslash in front of every whitespace character ...

or you will get an error

$ team=Boston\ Red Sox
Command 'Sox' not found

For this reason it is usually better to quote the entire value

You can use either a single quote '

$ team='Boston Red Sox'

$ echo $team
Boston Red Sox

Or a double quote "

$ team="Boston Red Sox"

$ echo $team
Boston Red Sox

There is a difference between the two quotes
But we will save that for another class
Be sure to use the same quote at the end of the value ...
as you used at the start

Scope

Where a variable can be used is called the scope of the variable
Shell variables have two scopes
- Local
- Global
Local variables only have meaning in the shell in which they are defined
They have no meaning in any other subshell
Global variables are variables defined in one shell ...
and have meaning in all sub-shells created from that shell
All keyword shell variables are global

Local Variables

By default every shell variable you create is a local variable
You must do something special to make a variable global
If I create the variable name ...

I can use it in that shell

$ name='Glenn Hoffman'

$ echo $name
Glenn Hoffman

But not in a subshell

$ bash

ghoffman@itserver6:~$ ps
  PID TTY          TIME CMD
29566 pts/16   00:00:00 bash
29792 pts/16   00:00:00 bash
29796 pts/16   00:00:00 ps

ghoffman@itserver6:~$ echo $name

$

This means we cannot use a local variable defined in the login shell ...
inside a shell script ...
because a script always runs in a sub-shell

Global Variables

To make a variable global use the export command

You can run export on a variable you have already created

$ export name

$ cat print_name.sh 
#! /bin/bash
# prints the value of the variable called name
echo name: $name

$ ./print_name.sh 
name = Glenn Hoffman

Notice that I did not put a $ in front of the name
That is because I am not changing its value
More often you use export when you are defining the variable
```
$ export name="Glenn Hoffman"
```
If you do not use export when defining a variable ...
it will be a local variable

Where Does a Shell Get It's Environment?

Every shell has it's own envirnoment
Your login shell gets it's environment from the startup files
When you create a subshell, that shell inherits the global variables ...
from the shell that created it
But not the local variables

Choosing the Scope of a Variable

In general, local variables are only used inside shell scripts
For example in all my test scripts I define the variable ex_no
```
ex_no=19
```
I can then use this variable in any part of the script
But this variable has no use outside this script
So there is no need to make it global
But every test script needs to know the absolute pathname of the course directory
Because that is where your Class Exercise scripts are located
I want to define this variable once so I can use it in every script
So I define it inside my .bash_profile
```
export lncd=/courses/it244/f24/ghoffman
```
Since lncd is a global variable ...
it can be used by any script I run

Defining Global Variables

You can define local variables inside a script
And they will be available every time you run the script
What about global variables?
If a global variable is defined on the command line ...
it will be available during that login session
But it will disappear as soon as you log out
To make them permanent you need to define them in a start-up file
like .bash_profile
.bash_profile is a startup file
This file contains Linux commands that are run when you log in
All startup files are contained in your home directory

Changing Keyword Shell Variables

Keyword shell variables are also called keyword variables
They have special meaning to the shell
By convention, the names of keyword variables are always CAPITALIZED
Many keyword variables can be changed by the user
This is normally done in the startup file .bash_profile
When you do this, you usually want to keep the current value of the variable ...
but add something to it
You do this using the following pattern
```
VARIABLE=$VARIABLE:NEW_VALUE
```
For example, to add my shell directory to my PATH I would write
```
PATH=$PATH:/home/ghoffman/bin/shell
```
in my .bash_profile
Notice the : between the list of directories in $PATH ...
and the directory I have added

Important Keyword Shell Variables

There are a number of keyword variables that affect your Unix session

Some of the more important are

Variable	Value
HOME	The absolute pathname of your home directory
PATH	The list of directories the shell must search to find the executable file to run a command
SHELL	The absolute pathname of your default shell
PWD	The absolute pathname off your current directory
PS1	Your command line prompt - what you see after entering each command
PS2	The secondary prompt - what you see if you continue a command to a second line

User-created Variables

User-created variables are any variables you create
By convention, the names of user-created variables are lower case
```
lncd=/courses/it244/f24/ghoffman
```
User-created variables can be either local or global in scope
The variables I create in my scripts are user-created variables ...
but I have many global variables

I define a global variable for the class directories of each class I teach

export lncd=/courses/it244/f24/ghoffman
export s1cd=/courses/it116/f24/ghoffman
export s2cd=/courses/it117/f24/ghoffman

`env` - Show All Global Variables

If you run env without an argument ...

it prints the values of all global variables

$ env
TERM=xterm-color
SHELL=/bin/bash
SSH_CLIENT=66.92.76.9 53785 22
OLDPWD=/home/it244gh
SSH_TTY=/dev/pts/8
USER=it244gh
...

The `read` command

Most scripts need input from the user
One way to get this input is with the read command
read is used to set the value of a local variable using the following format
```
read VARIABLE_NAME
```
When Bash gets to this line in a script ...
it stops and waits for the user to type something
When the user hit Enter or Return ...
Bash assigns whatever the user typed ...
to the variable whose name appears after read
If you run read with the -p option it will print a prompt
```
read -p PROMPT VARIABLE_NAME
```

Here is an example

#! /bin/bash

read -p "Please enter a value: " value
echo $value

When I run this I get

$ ./read_2.sh
Please enter a value: hi there
hi there

The text in blue is user input

Separating and Grouping Commands

You can enter more that one command on a single command line

But you must enter a semicolon, ;, after each command

$ echo Here are the contents of my home directory ; ls ~ ; echo
Here are the contents of my home directory
bin   course_files  it116       it244       mail         submitted    tmp
code  html          it116_test  it244_test  public_html  tests_taken  xrchiv

Each command is executed in the order it was typed at the command line

| (pipe) and & (ampersand) as Command Separators

The pipe, | , and ampersand, & , characters are also command separators
| sends the output of one command into the input of the next
Each command is a separate process
& after a command causes the command to run in the background
When we do this, two things happen
- The command is disconnected from the keyboard
- The command will run at the same time
  as the next command you enter
But the ampersand is also a command separator

So we can use it to run many commands in the background at the same time

$ ./bother.sh > /dev/null & ./bother.sh  > /dev/null & ./bother.sh  > /dev/null & jobs
[1] 1794
[2] 1795
[3] 1796
[1]   Running                 ./bother.sh > /dev/null &
[2]-  Running                 ./bother.sh > /dev/null &
[3]+  Running                 ./bother.sh > /dev/null &

Attendance

New Material

Positional and Special Parameters

Positional and special parameters are variables set by Unix
The values of these variable change each time you enter a command
We have already encountered the special parameter ?

It contains the status code returned by the last command

$ ls bar.txt 
bar.txt

$ echo $?
0

$ ls xxx
ls: cannot access xxx: No such file or directory

$ echo $?
2

Positional parameters are used by shell scripts to get command line arguments
Each word on the command line is assigned to a positional parameter
The first word is the name or pathname of the script
It is assigned to positional parameter 0

Each succeeding word on the command line is assigned to the next integer

$ cat print_positionals.sh
#!/bin/bash
#
# Prints the value of the first four positional arguments

echo 0: $0
echo 1: $1
echo 2: $2
echo 3: $3

$ ./print_positionals.sh foo bar bletch
0: ./print_positionals.sh
1: foo
2: bar
3: bletch

Positional parameters are the usual way input is given to a script
Another special parameter is #

# contains the number of arguments passed to a program from the command line

$ cat print_arg_numbers.sh 
#!/bin/bash
#
# Prints the number of arguments sent to this script

echo This script received $# arguments

$ ./print_arg_numbers.sh foo bar bletch
This script received 3 arguments

Notice that # counts the arguments to the script
It does not count the name or pathname of the script itself

Quoting and the Evaluation of Variables

Whenever the value of a variable contains spaces, you must quote the string ...
or escape the whitespace character
There are two kinds of quotes
- Single quotes, ' '
- Double quotes, " "
Single quotes are the most restrictive
Everything surrounded by single quotes ...
appears in the value exactly as you typed it

This means that special meaning of characters like $ are ignored

$ team="Red Sox"

echo $team
Red Sox

$ cheer='Go $team'

$ echo $cheer
Go $team

Double quotes also preserve spaces in the strings they contain
But you can use a $ in front of a variable name ...
and the $ will get the value of the variable ...

inside the quotes

$ cheer="Go $team"

$ echo $cheer
Go Red Sox

Quotes affect everything they enclose
The backslash, \, only effects the character immediately following it
```
$ foo=bar

$ echo $foo
bar

$ foo3=\$foo

$ echo $foo3
$foo
```

Processes

A process is a running program
Every process has resources that it needs to do its job
Unix is a multitasking operating system
This means many processes can run at the same time
Every time you run a program ...
a new process is created ...
except when you run a builtin
Running a builtin command does not create a process
Because it runs inside the shell's process
When you run a shell script ...
bash creates a subshell for the script
This subshell runs in a new process

System Calls

When you run a command that is not a builtin
The shell asks the kernel to create a process
How does the shell make this request?
It uses a system call
A system call allows a program written in language like C ...
to run a procedure built into the kernel
When a program needs the kernel to do something ...
it makes a system call

The History of Computer Languages

Before discussing how processes are created we need to know some history
Computer programs are a series of binary instructions
Written in the machine language that the CPU understands
Each type of CPU has its own unique machine language
The commands in these languages are binary numbers
Binary numbers are a series of 1's and 0's
In the early days, programmers wrote code in the binary numbers ...
of the CPU's machine language
So a program would look like this
```
1010101010
1010100101
1010101010
...
```
It was very difficult to program in machine language
Something better was needed
That something better was assembly language
Instead of writing a binary number for each instruction ...
you wrote a few letters that stood for that instruction
Programs written in this language were run through a program called an assembler
This program translated the letter codes into binary instructions

A line of assembly code might look like this

MOV AL, 61h       ; Load AL register with 97 decimal (61 hex)

; means what comes after it is a comment ...
not assembly code
Asembly language coding was better than writing binary
But it still forced programmers to think like the machine
There were no loops or if statements ...
or easy ways to break a big program into functions
This made writing programs very tedious
To improve this situation higher level languages were created
These languages featured control structures like loops and conditionals
They also allowed programmers to break up the code into functions
C, Fortran and Cobol are examples of higher level languages
C is sometimes called a middle-level language
While it has all the features of a higher level language ...
it also allows you to work in 1's and 0's of machine language
Most operating systems are written in C

Creating a New Process

So how does Bash ask the kernel to create a process to run a program?
Bash, like most of Unix, is written in C
Bash creates a process by making the system call fork
It does this with the following line of C code
```
pid = fork()
```
fork causes the kernel to create a new process
This new process is an exact duplicate of the your command line Bash processs ...
but with a different process ID
The varialbe fork gets the value of this new process ID
The picture in RAM looks like this

This original process is called the parent process ...
and the new process is called the child process
Notice that the process ID of the login shell ...
is the parent process ID of the new process

How the Shell Runs a Program

After the C code in Bash calls fork we have two identical process
They only differ in their process ID
In both processes the executable binary code is Bash
So how can the new process run a different program?
To understand how this happens we have to look at the Bash C code
A new process is created with following C statement
```
pid = fork()
```
The value of the variable pid is the process ID of the new process
The running code in both processes is Bash
And both processes have the variable pid
The next line of the Bash C code in both processes ...
is an if statement that looks at the value of pid
It is this if statement that lets the new process know ...
that it needs to run different code
If the value of this variable does not match the PID of the running process ...
as it does in the original shell process ...
the process knows it is shell that ran the command
That is, the parent process
This process has nothing to do until the child process finishes
So it makes a system call to sleep ...
and goes into a state of suspended animation
If, however, the PID, matches the value of the variable pid ...
as it does in the new process ...
we have to replace the Bash executable code ...
with the code the process needs to run
To do this it makes system call to exec
exec replaces the Bash code in process memory ...
with the binary code of the program ...
that you entered at the command line
The situation in RAM now looks like this

When this child process code finishes running ...
it sends an exit status to its parent process
The parent process is your login Bash shell
When it gets the status code from the child process ...
the login shell process wakes up again ...
and is ready for you next command
The sequence of events that occur when running ls on the command line ...
looks like this

This will not be on the Final exam

Process Structure

Every process on a computer was created by a parent process ...
with one exception
How is the very first process created?
Whenever a Unix system is turned on ...
a special first process is created automatically
This special process then creates other process
These processes can create other processes ...
and so on
This creates a hierarchical structure much like the file system
In the filesystem the special directory at the top called the root
Every other file or directory is contained in this directory ...
or one of its subdirectories
Until recently the special first process was init
But for various reasons this is no longer the case with Ubuntu
Though it still may be true in other versions of Linux
The first process on an Ubuntu system is now systemd

Booting a Unix Machine

Whenever you turn on a Unix machine
A spontaneous process is automatically created
The code for this process is either init ...
or systemd
Either way, it has a process ID of 1
This process then creates all other processes ...
which are needed to run the services the machine provides
Unix can be run in a mode with only one user
But this is only done under special circumstances
When run in multiuser mode ...
systemd or init will create processes ...
that allow users to log in
There are a number of different programs that can do this
- getty
- mingetty
- agetty
Our systems run agetty
Many of agetty process are always running ...
each of which can handle an ssh cinnection
Each one of these processes is connected to a physical or virtual terminal
When the process code detects an attempt to connect ...
it displays a login prompt
It then waits for the user to respond
When that happens it runs login to verify the password
login is located in the /bin directory
If the password is accepted a system call is made to exec() ...
to run Bash ...
and this process becomes your login shell
Again, this will not be on the Final exam

Process Identification

Each process has a unique Process ID (PID) number
As long as the process runs, it has the same PID
After a process terminates its PID can be assigned to a new process

ps -f displays a full listing for each of the user's running processes

$ ps -f
UID        PID  PPID  C STIME TTY          TIME CMD
it244gh  26374 26373  0 13:41 pts/5    00:00:00 -bash
it244gh  27891 26374  0 13:57 pts/5    00:00:00 ps -f

The UID column shows the Unix username of the account that started the process
The PID column is the process ID of the process
The PPID column is the process ID of the parent process
That is, the process that created this process
The CMD column lists the command that is running in the process
If I were to run sleep in the background ...
and then run ps -f

I would see

$ sleep 10 & ps -f
[1] 27352
UID        PID  PPID  C STIME TTY          TIME CMD
ghoffman  27292 27287  0 15:12 pts/1    00:00:00 -bash
ghoffman  27352 27292  0 15:13 pts/1    00:00:00 sleep 10
ghoffman  27353 27292  0 15:13 pts/1    00:00:00 ps -f

Notice that the parent process of both sleep and ps -f is my login shell
pstree will display a tree of all currently running processes

If I run pstree on pe15 I can see the process structure

systemd─┬─accounts-daemon─┬─{gdbus}
        │                 └─{gmain}
        ├─acpid
        ├─agetty
        ├─atd
        ├─automount───10*[{automount}]
        ├─bash
        ├─bother.sh───sleep
        ├─cron
        ├─cups-browsed─┬─{gdbus}
        │              └─{gmain}
        ├─dbus-daemon
        ├─dhclient
        ├─irqbalance
        ├─2*[iscsid]
        ├─lvmetad
        ├─lxcfs───4*[{lxcfs}]
        ├─master─┬─pickup
        │        └─qmgr
        ├─mdadm
        ├─ntpd
        ├─polkitd─┬─{gdbus}
        │         └─{gmain}
        ├─rpc.statd
        ├─rpcbind
        ├─rsyslogd─┬─{in:imklog}
        │          ├─{in:imuxsock}
        │          └─{rs:main Q:Reg}
        ├─rwhod───rwhod
        ├─screen───bash
        ├─screen───bash───vim
        ├─snapd───7*[{snapd}]
        ├─sshd─┬─3*[sshd───sshd───bash───alpine]
        │      ├─2*[sshd───sshd───bash]
        │      ├─sshd───sshd───bash───pstree
        │      └─sshd───sshd───bash───ssh
        ├─10*[systemd───(sd-pam)]
        ├─systemd-journal
        ├─systemd-logind
        ├─systemd-udevd
        └─ypbind───2*[{ypbind}]

Where you see a number followed by a * ...
it means there multiple copies of that program ...
are running in different processes
You can see this more clearly if you run pstree with the -p option

Now pstree will show each process along with its process ID

systemd(1)─┬─accounts-daemon(2099)─┬─{gdbus}(2136)
           │                       └─{gmain}(2134)
           ├─acpid(2076)
           ├─agetty(2481)
           ├─atd(2103)
           ├─automount(42937)─┬─{automount}(42938)
           │                  ├─{automount}(42939)
           │                  ├─{automount}(42942)
           │                  ├─{automount}(42945)
           │                  ├─{automount}(42946)
           │                  ├─{automount}(42947)
           │                  ├─{automount}(42948)
           │                  ├─{automount}(42949)
           │                  ├─{automount}(42950)
           │                  └─{automount}(42951)
           ├─bash(27573)
           ├─bother.sh(16996)───sleep(47645)
           ├─cron(2087)
           ├─cups-browsed(40751)─┬─{gdbus}(40833)
           │                     └─{gmain}(40832)
           ├─dbus-daemon(2089)
           ├─dhclient(2246)
           ├─irqbalance(2465)
           ├─iscsid(2355)
           ├─iscsid(2356)
           ├─lvmetad(1199)
           ├─lxcfs(2101)─┬─{lxcfs}(2126)
           │             ├─{lxcfs}(2127)
           │             ├─{lxcfs}(3081)
           │             └─{lxcfs}(29814)
           ├─master(21825)─┬─pickup(44495)
           │               └─qmgr(31008)
           ├─mdadm(2148)
           ├─ntpd(2507)
           ├─polkitd(2164)─┬─{gdbus}(2180)
           │               └─{gmain}(2178)
           ├─rpc.statd(2782)
           ├─rpcbind(2053)
           ├─rsyslogd(2071)─┬─{in:imklog}(2132)
           │                ├─{in:imuxsock}(2131)
           │                └─{rs:main Q:Reg}(2133)
           ├─rwhod(2449)───rwhod(2450)
           ├─screen(19504)───bash(19505)
           ├─screen(20288)───bash(20289)───vim(21881)
           ├─snapd(2056)─┬─{snapd}(2118)
           │             ├─{snapd}(2119)
           │             ├─{snapd}(2120)
           │             ├─{snapd}(2121)
           │             ├─{snapd}(2150)
           │             ├─{snapd}(2151)
           │             └─{snapd}(2152)
           ├─sshd(2341)─┬─sshd(3104)───sshd(3160)───bash(3161)───alpine(8255)
           │            ├─sshd(4288)───sshd(4347)───bash(4348)───alpine(4354)
           │            ├─sshd(15840)───sshd(15919)───bash(15926)
           │            ├─sshd(40522)───sshd(40572)───bash(40588)
           │            ├─sshd(41273)───sshd(41339)───bash(41346)───alpine(41362)
           │            ├─sshd(46574)───sshd(46658)───bash(46665)───pstree(47648)
           │            └─sshd(48106)───sshd(48199)───bash(48200)───ssh(48213)
           ├─systemd(2874)───(sd-pam)(2878)
           ├─systemd(10316)───(sd-pam)(10320)
           ├─systemd(15847)───(sd-pam)(15860)
           ├─systemd(48118)───(sd-pam)(48124)
           ├─systemd(46589)───(sd-pam)(46600)
           ├─systemd(41295)───(sd-pam)(41303)
           ├─systemd(27537)───(sd-pam)(27542)
           ├─systemd(17838)───(sd-pam)(17842)
           ├─systemd(14978)───(sd-pam)(14983)
           ├─systemd(20176)───(sd-pam)(20181)
           ├─systemd-journal(1164)
           ├─systemd-logind(2095)
           ├─systemd-udevd(1207)
           └─ypbind(2476)─┬─{ypbind}(2477)
                          └─{ypbind}(2478)

The part in red is my current login shell
Its child process is running pstree

Tips and Examples

Review

New Material

Questions

Homework 9

Graded Quiz Today

Today's Class

Tips and Examples

Don't Use Google for Homework Assignments

Using Control Z with an Editor

Redirecting Output From Background Job

Review

Shell Environment

Shell Variables

Creating a Shell Variable

Scope

Local Variables

Global Variables

Where Does a Shell Get It's Environment?

Choosing the Scope of a Variable

Defining Global Variables

Changing Keyword Shell Variables

Important Keyword Shell Variables

User-created Variables

env - Show All Global Variables

The read command

Separating and Grouping Commands

| (pipe) and & (ampersand) as Command Separators

Attendance

New Material

Positional and Special Parameters

Quoting and the Evaluation of Variables

Processes

System Calls

The History of Computer Languages

Creating a New Process

How the Shell Runs a Program

Process Structure

Booting a Unix Machine

Process Identification

Class Exercise

Class Quiz

`env` - Show All Global Variables

The `read` command