IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 6

Microphone

Questions

Are there any questions before I begin?

Announcement

The UMB IT Club and Boston Linux User Group will hold a Linux InstallFest on Wednesday, September 24th, from 4 to 8 in the University Hall Room 3-3540.

If you have a machine on which you would like to install Linux and would like some help in doing this, bring it the InstallFest.

Volunteers from the Boston Linux User Group with be on hand to help with the installation.

They will also help you install the Window Subsystem for Linux (WSL) on your machine or install Linux as a dual boot.

You can also bring your questions about Linux or Unix to the InstallFest.

The Boston Linux and Unix User Group counts among its members some of the most knowledgeable Linux and Unix people in the Boston area.

Quiz 1

I have posted the answers to Quiz 1 here.

Homework 3

I have posted homework 3.

As usual, it will be due this coming Sunday at 11:59 PM.

Course Requirements

Shell Scripts for Homework Assignments

Starting with homework 3, you will need to write shell scripts for all homework assignments
This makes it easier for me to grade your homework
These script files must meet certain requirements or you will lose points
I've created a document which lists the requirements which you will find here

Scripts for Class Exercises

For now on, all Class Exercises will ask you to also write scripts
You do not have to follow all the requirements for homework scripts for Class Exercise scripts
For example there are no Step comments
But there are a few thing you should not do in these scripts
Do not do any of the following
- Run man or info
- Run more or less
- Run nano, vi, vim, emacs or any text editors
All these commands ask the user for input
I run scripts to check your Class Exercises
If you use any of the commands above the scripts will not run the way they should
You will get no credit for the assignment until you remove these commands

Tips and Examples

Copying and Pasting at the Command Line

You can have many ssh windows open at the same time
Each of these windows runs inside an ssh client
These programs allow you to copy from one window and paste into another
On my Mac I log into pe15 using the MacOS Terminal application
Once there, I can copy text by selecting it ...
and holding down the Apple Command key while pressing C
I can paste something I've copied onto the command line of another window with Command V
In PuTTY simply hold down the left mouse button selecting text will copy it
You do not have to use Control C
To paste into PuTTY simply click the left mouse button
Or use select Paste from the Edit menu

A Space in the Wrong Place

A single wrong character on the command line can cause a command to fail
I discovered a great example of this some semesters ago
I was helping a student with a ex5.sh script

The problem involved two lines of the script

...
cd   /home/ghoffman/course_files/it244_files
...
grep  -r  foo  .
...

The script was supposed to go to my it244_files directory ...
and search for lines that had "foo" in them in all files and subdirectories
The error was caused by putting a single space in the wrong place in the cd command
Instead of typing the cd command given above, the student typed
```
cd  / home/ghoffman/course_files/it244_files
```
Notice the space between the / and "home"
Because of this extra space, cd caused the student to go to the directory /
This directory is the top of the filesystem
The text after the space was ignored because cd takes only one argument
The -r option to grep causes it to search files in all subdirectories
This command will print all lines containing "foo" in the entire filesystem
Needless to say this command keep going and going
The only way to stop it was to use Control C

Finding Errors in a Script

Shell scripts are small programs
If a script contains a command that makes no sense ...
running the script will give you an error

The following script contains an error

$ cat go_home_bad.sh
# this script contains an error
cd /home/ghofman
pwd

When I run it I get

$ bash go_home_bad.sh
go_home_bad.sh: line 2: cd: /home/ghofman: No such file or directory
/home/ghoffman/course_files/it244_file

Even though there was an error on the first line ...
the script continued to run
Here the error was a spelling mistake
The name of my home directory is ghoffman not ghofman
But what if the error message appeared after several lines of output?
You might not see the error as the output scrolled off the screen
The following script produces many lines of output ...

but it has the same error on the first line

$ cat go_home_bad_2.sh
# this script contains an error which is 
# hard to see in the output
cd /home/ghofman
ls -l

When I run this script it prints an error message
But that message is hard to see ...

because it scrolls by quickly off the top of the screen

$ bash go_home_bad_2.sh
go_home_bad_2.sh: line 3: cd: /home/ghofman: No such file or directory
-rw-rw-r-- 1 ghoffman faculty   51 Jun  8 11:54 numbers1.txt
-rw-rw-r-- 1 ghoffman faculty   51 Jun  8 10:55 numbers2.txt
-rw-rw-r-- 1 ghoffman faculty   50 Jun  8 10:55 numbers.txt
-rwxrwxr-x 1 ghoffman faculty   94 Jun  8 10:55 print_arg_numbers.sh
-rwxrwxr-x 1 ghoffman faculty  156 Jun  8 10:55 print_foo_bar.sh
-rwxrwxr-x 1 ghoffman faculty   71 Jun  8 10:55 print_foo.sh
-rwxrwxr-x 1 ghoffman faculty  132 Jun  8 10:55 print_positionals.sh
-rwxrwxr-x 1 ghoffman faculty  122 Jun  8 10:55 read_1.sh
-rwxrwxr-x 1 ghoffman faculty  141 Jun  8 10:55 read_2.sh
-rw-rw-r-- 1 ghoffman faculty 1214 Jun  9 10:35 red_sox.txt
-rwxrwxr-x 1 ghoffman faculty  260 Jun  8 10:55 repeat.sh
....

There is a trick that you can use to see only the error messages

$ bash go_home_bad_2.sh > /dev/null
go_home_bad_2.sh: line 3: cd: /home/ghofman: No such file or directory

When I follow the command with
```
> /dev/null
```
Unix throws away everything except the error messages
I used this trick in the "Error check" section of my Class Exercise test scripts
You'll see why this works when we discuss redirection in a future class

Review

`echo` - Print Text to the Terminal

echo prints text to the screen
```
$ echo Hello
Hello
```
You can use echo to print the value of a system variable ...
if you put $ before the variable name
```
$ echo $SHELL
/bin/bash
```

`hostname` - Print the Name of Your Host Machine

hostname prints the network name of the machine that you have logged on to
```
$ hostname
pe15
```
When used with the -i option hostname will print the IP address of the machine
```
hostname -i
158.121.104.215
```

Pagers - View a File One Screen at a Time

Pagers are programs that display the contents of a file one screenful at a time
The two pagers that Unix supplies are more and less
Hitting the Space bar advances to the next screen
Hitting the Enter or Return key takes you down one line
less, just to be confusing, has more features than more

Pathname Completion

When typing a long file name it is easy to make a mistake
To make life easier, Unix provides a feature called pathname completion
You type a few characters, then hit the Tab key ...
and Unix will try to supply the rest
If there is only one file or directory that matches ...
Unix will fill in the rest of the name
If there is more than one match ...
Unix will supply as much as it can and then beep
If you hit Tab again it will show you the possible matches
If there is no match, it will also beep

`grep` - Finding Strings inside Files

grep is used to find all lines in a file that contain a string
grep takes two arguments
- The string you are searching for
- The file or files in which to search
grep has the following format
```
grep STRING FILE [FILE ...]
```
grep -r will search recursively through a directory
That means it looks through the directory given as a second argument ..,
and all the subdirectories
grep, like Unix, is case sensitive
It thinks of "foo" and "FOO" as two different strings
To have grep ignore case, run it with the -i option
grep -v returns all lines that do not match the search string

`head` - View the Top of a File

head prints the first 10 lines of any file
For a different number of lines, follow head with a dash, -, and a number
```
head -20 foo.txt
```

`tail` - View the Bottom of a File

tail prints the last 10 lines of any file
For a different number of lines, follow tail with a dash, -, and a number
```
tail -5 foo.txt
```

`sort` - Print a File in Sorted Order

sort prints all the lines in a file in sorted order
sort does not change the file
To sort in reverse order use the -r option
sort, by default, sorts in alphabetical order
This is a problem when the first characters on a line are numbers
That's because 11 will sort before 2
To sort in numerical order use sort -n
To sort in reverse numerical order use sort -nr

Attendance

New Material

The Shell Prompt

In the last class I mentioned shell variables
We'll talk a lot more about them in a few weeks
But right now I'd like to say a few words about one particular variable
PS1 is the variable that determines your Unix prompt
This variable can be customized in many ways
By default PS1 tells you three things
- Your Unix username
- The machine to which you are connected
- The location of your current directory
Some of you may have noticed that my prompt looks different from yours
That's because I have changed the value of PS1
But before I did this, my prompt looked like this
```
ghoffman@pe15:~/it244/work$
```
The characters before the @ are my Unix username
The characters between @ and : are the hostname of the machine I am using
The characters from : to $ show my current directory
The ~ indicates my home directory
We'll talk more about this in a future class
So when I got this prompt, I was in the directory
```
/home/ghoffman/it244/work
```
If you look carefully at your prompt
You can always see where you are in the Unix filesystem

Pipes - Stringing Programs Together

Pipes are one of the most powerful features of Unix
A pipe takes the output of one Unix command ...
and feeds it into the input of another
Using pipes you can string programs together
This way many programs can work together to do something ...
that none of them could do by itself
A pipe consists of two or more Unix commands ...
with each command separated from the one that came before by |
Using a pipe, we can build up a sequence of commands ...
to get exactly what we want
Let's say we want to find all Red Sox wins against the Rays
But sorted in reverse order by date

From the file red_sox.txt

$ cat red_sox.txt 
2011-07-02      Red Sox @  Astros       Win 7-5
2011-07-03      Red Sox @  Astros       Win 2-1
2011-07-04      Red Sox vs Blue Jays    Loss 7-9
2011-07-05      Red Sox vs Blue Jays    Win 3-2
2011-07-06      Red Sox vs Blue Jays    Win 6-4
2011-07-07      Red Sox vs Orioles      Win 10-4
2011-07-08      Red Sox vs Orioles      Win 10-3
2011-07-09      Red Sox vs Orioles      Win 4-0
2011-07-10      Red Sox vs Orioles      Win 8-6
2011-07-15      Red Sox @  Rays         Loss 6-9
2011-07-16      Red Sox @  Rays         Win 9-5
2011-07-17      Red Sox @  Rays         Win 1-0
2011-07-18      Red Sox @  Orioles      Win 15-10
2011-07-19      Red Sox @  Orioles      Loss 2-6
2011-07-20      Red Sox @  Orioles      Win 4-0
2011-07-22      Red Sox vs Mariners     Win 7-4
2011-07-23      Red Sox vs Mariners     Win 3-1
2011-07-24      Red Sox vs Mariners     Win 12-8
2011-07-25      Red Sox vs Royals       Loss 1-3
2011-07-26      Red Sox vs Royals       Win 13-9
2011-07-27      Red Sox vs Royals       Win 12-5
2011-07-28      Red Sox vs Royals       Loss 3-4
2011-07-29      Red Sox @  White Sox    Loss 1-3
2011-07-30      Red Sox @  White Sox    Win 10-2
2011-07-31      Red Sox @  White Sox    Win 5-3

First let's find all games against the Rays

$ grep Rays red_sox.txt
2011-07-15      Red Sox @  Rays         Loss 6-9
2011-07-16      Red Sox @  Rays         Win 9-5
2011-07-17      Red Sox @  Rays         Win 1-0

Now let's feed this into another command that selects the games the Sox won

$ grep Rays red_sox.txt | grep Win
2011-07-16      Red Sox @  Rays         Win 9-5
2011-07-17      Red Sox @  Rays         Win 1-0

Now we can use sort to get the results in the order we want

$ grep Rays red_sox.txt | grep Win | sort -r
2011-07-17      Red Sox @  Rays         Win 1-0
2011-07-16      Red Sox @  Rays         Win 9-5

The Unix tool philosophy is simple programs doing one thing well
Pipes are an essential element in making this philosophy work
As we progress through this course ...
you will have many opportunities to use pipes
Pipes are one of the reasons that Unix is so powerful

`uniq` - Eliminate Duplicate Lines

uniq prints the lines in a file, removing adjacent identical lines

$ cat numbers2.txt 
4
5
6
7
8
9
10
11
11
11
12
13
14
15
16
17
18
19
20

$ uniq numbers2.txt 
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

Note, the duplicate lines must be adjacent for uniq to work
They must appear one right after the other

So uniq will not work with this file

$ cat numbers3.txt
4
5
6
7
8
9
10
11
12
13
11
14
15
11
17
18
19
20

$ uniq numbers3.txt
4
5
6
7
8
9
10
11
12
13
11
14
15
11
17
18
19
20

For this reason, you should pipe the output of sort into uniq

$ sort -n numbers3.txt | uniq
4
5
6
7
8
9
10
11
12
13
14
15
17
18
19
20

uniq -i ignores case when looking for duplicate lines

`diff` - Differences between Files

diff compares two files and displays the lines that are different

The default output format of diff is confusing

$ cat numbers1.txt 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

$ cat numbers2.txt 
4
5
6
7
8
9
10
11
11
11
12
13
14
15
16
17
18
19
20

$ diff  numbers1.txt  numbers2.txt 
1,3d0
< 1
< 2
< 3
10a8,9
> 11
> 11

diff was created for use with the Unix patch utility
patch creates a new version of a file from the changes given by diff
This output was never meant to be read by people

The most readable output is obtained by using the -y option

$ diff  -y  numbers1.txt numbers2.txt 
1                                 <
2                                 <
3                                 <
4                               4
5                               5
6                               6
7                               7
8                               8
9                               9
10                              10
                                >  11
                                >  11
11                              11
12                              12
13                              13
14                              14
15                              15
16                              16
17                              17
18                              18
19                              19
20                              20

diff -i ignores case when looking for differences.

`file` - Show the File Type

The file utility takes an argument of one or more files ...

and shows the type of each

$ file *
bin:                 directory
cars.txt:            ASCII text
cmds:                directory
dead.letter:         ASCII news text
downloads:           directory
exercises_it244:     directory
homework_it244:      directory
it244:               symbolic link to `/courses/it244/s19/ghoffman/'
java:                directory
...

You can also run file on individual files

$ file break.sh
break.sh: Bourne-Again shell script, ASCII text executable

$ file make_upper.py 
make_upper.py: Python script, ASCII text executable

$ file index.html 
index.html: HTML document, ASCII text

`date` - Get the Date and Time

date displays the time and date
```
$ date
Tue Aug  7 20:02:48 EDT 2012
```
You can change the way the date is displayed ...

by following date with a + and a format string

$ date +"%Y-%m-%d %r"
2012-08-07 08:19:44 PM

The format string consists mostly of character pairs ...
the first character of which is a %
In the string above, %Y stands for the four digit year
To get more information on the various formatting options use
```
info date
```

Now move the cursor down to the line that reads

* Date conversion specifiers::    %[aAbBcCdDeFgGhjmuUVwWxyY]

Then hit the Enter key
An interesting format option is s
When date is run with this option
```
$ date +%s
1525711270
```
It displays the number of seconds since January 1, 1970
This date is called the Unix epoch
and is used to compare dates in Unix

`wc` - Counting characters

Most of the files we use in Unix are text files
Text files are a collection of lines
A line is a series of characters with the newline at the end

wc counts the lines, words and characters in a file

$ cat lines.txt 
Line 1
Line 2
Line 3
Line 4
Line 5

$ wc lines.txt 
 5 10 35 lines.txt

This output says there are 5 lines, 10 words and 35 characters in lines.txt

You can use options to tell wc what to count

c - bytes (characters)
w - words
l - lines

$ wc -c lines.txt 
35 lines.txt

$ wc -w lines.txt 
10 lines.txt

$ wc -l lines.txt 
5 lines.txt

`which` - Finding a Program File

Unix commands are programs which exist as binary files
These files contain the binary codes that the computer's processor understands
These commands are located in different directories in the Unix file system
You can use the Unix utility which ...
to find the exact location of a binary program file
To find the location of the tar program file we would run
```
$ which tar
/bin/tar
```
which shows that the executable file for tar is located in the /bin directory
which uses the PATH system variable to find the location of the file
We'll discuss PATH in a future class

`whereis` - Finding Files Used by a Program

whereis is another program that can be used to locate program files
whereis takes an approach different from that of which
Every Unix or Linux system has certain standard places ...
where it stores programs and the files they use
whereis searches these locations
which only shows the location of a command
```
$ which tar
/usr/bin/tar
```

When we run whereis on tar we get more information than which returned

$ whereis tar
tar: /usr/bin/tar /usr/include/tar.h /usr/share/man/man1/tar.1.gz

The first entry is the executable file /usr/bin/tar
The second entry is a header file /usr/include/tar.h
The program needs the header file to get certain information
The third entry is the file that man uses to provide information about tar
A word of caution about using which and whereis
Some commands are actually built into the shell itself
These command are called built-ins and we will talk about them in a future class
If you run which or whereis on these programs you will get nothing back
```
$ which cd

$
```

`find` - Search for Files Using Different Criteria

The most powerful Linux/Unix command for finding files is find
Unfortunately, it's power makes it hard to use
find can be used to search for a file based on many different things such as
- The name of the file
- The last time the file was used
- The last time the file was changed
- The access permissions on the file
An in-depth discussion of find is beyond the scope of this course
The simplest way to use find is searching for files by name
You do this using the following format
```
find DIRECTORY -name FILENAME
```
Where DIRECTORY it the directory where the search begins ...
and FILENAME is the name of the file

Here is an example

$ find  /home/ghoffman  -name red_sox.txt
/home/ghoffman/course_files/it244_files/red_sox.txt

find will not accept a partial file name
```
$ find  /home/ghoffman  -name red

$
```
You can get the effect of searching on a part of a filename by using the * character

But since * has special meaning on the command line you have to escape it

$ find /home/ghoffman -name memo.\*
/home/ghoffman/memo.bak
/home/ghoffman/memo.txt
/home/ghoffman/tmp/memo.bak
/home/ghoffman/tmp/memo.txt
/home/ghoffman/tmp/memo.backup

`who` - See Users Logged On

who prints a list of all users currently logged on to the machine

$ who
ghoffman  pts/0        2012-08-12 13:41 (dsl092-066-161.bos1.dsl.speakeasy.net)
rouilj    pts/1        2012-08-12 04:25 (pool-74-104-161-40.bstnma.fios.verizon.net)
eb        pts/2        2012-08-12 08:19 (pool-96-237-251-11.bstnma.fios.verizon.net)

who also provides information about each user's login session
It shows the time they logged in
It also shows the machine from which the user connected

who am i will show the user who is logged into a specific terminal

$ who am i
ghoffman  pts/0      2012-08-12 13:41 (dsl092-066-161.bos1.dsl.speakeasy.net)

This can be useful if you find an unattended terminal
You can run the same command without the spaces but it gives less information
```
$ whoami
ghoffman
```

`finger` - Get information on Users

finger, like mv, has two functions

When used without an argument finger shows every user currently logged in

$ finger
Login    Name                  Tty      Idle  Login Time   Office     Office Phone
admerrib Adil Merribi          pts/23     35  Sep  8 12:19 (75.69.126.23)
apera    Antonio Pera         *pts/2   23:47  Sep  7 04:43 (73.16.254.131)
bnsevak  Birva Sevak           pts/12     14  Sep  8 11:45 (65.96.118.54)
gabrgabr Gabriel Park          pts/13    45d  Jul 17 09:46 (tmux(29686).%0)
gabrgabr Gabriel Park          pts/14    52d  Jul 17 16:09 (tmux(29686).%5)
gabrgabr Gabriel Park          pts/17    52d  Jul 17 16:23 (tmux(29686).%6)
gabrgabr Gabriel Park          pts/20    45d  Jul 18 19:39 (tmux(29686).%8)
gghinita Gabriel Ghinita       pts/9      5d  Sep  2 16:00 (192.168.105.101)
ghoffman Glenn Hoffman         pts/21         Sep  8 12:51 (209.6.202.123)
grullnor Raffi T Sahagian      pts/11  20:42  Sep  5 15:27 (192.168.107.156)
grullnor Raffi T Sahagian      pts/26     29  Sep  8 12:28 (73.126.84.97)
ldavid   Leonard David         pts/1      3d  Sep  3 11:34 (158.121.23.66)
lilmnh   Michael Nguyen        pts/16         Sep  8 12:38 (71.174.229.240)
mkhr     Mikhail Khayretdinov  pts/8      19  Sep  8 10:54 (76.28.116.136)
offner   Carl D. Offner        pts/0      2d  Jul 30 07:38 (50.200.93.68)
offner   Carl D. Offner        pts/22     3d  Sep  4 15:20 (192.168.107.170)
rickm    Rick Martin           pts/5   20:43  Aug 17 09:42 (192.168.105.41)
vyiep    Vinson Yiep           pts/10     5d  Jul 24 18:30 (73.47.67.141:S.0)
xiaohui  Xiaohui Liang         pts/7      2d  Sep  6 10:55 (192.168.107.184)

When used with a username, finger provides information about the account

$ finger ghoffman
Login: ghoffman                         Name: Glenn Hoffman
Directory: /home/ghoffman               Shell: /bin/bash
On since Sun Sep  8 12:51 (EDT) on pts/21 from 209.6.202.123
Mail forwarded to glennhoffman@mac.com
Mail last read Fri Sep  6 12:04 2019 (EDT)
Plan:
Office:         McCormack 3-0201-22 (Pterosaur poster on door)
Office Hours:   Tuesday & Thursday, 11:OO AM - 12:00 PM and 5:30 to 6:30 PM
Classes:        IT 116-1   Introduction to Scripting         TuTh    12:30  - 1:45 PM     Wheatley 1-004
                IT 244-1   Introduction to Linux/Unix        TuTh     2:00  - 3:15 PM     McCormack M1-409
                IT 117-1   Intermediate Scripting            TuTh     4:00  - 5:15 PM     University Hall 2-2310
Home Address:   40 Central Street
                Somerville, MA 02143
                617-821-3936 (9am - 9pm only)

Studying

Attending Class

Course Requirements

Tips and Examples

Review

New Material

Studying

Questions

Announcement

Quiz 1

Homework 3

Course Requirements

Shell Scripts for Homework Assignments

Scripts for Class Exercises

Tips and Examples

Copying and Pasting at the Command Line

A Space in the Wrong Place

Finding Errors in a Script

Review

echo - Print Text to the Terminal

hostname - Print the Name of Your Host Machine

Pagers - View a File One Screen at a Time

Pathname Completion

grep - Finding Strings inside Files

head - View the Top of a File

tail - View the Bottom of a File

sort - Print a File in Sorted Order

Attendance

New Material

The Shell Prompt

Pipes - Stringing Programs Together

uniq - Eliminate Duplicate Lines

diff - Differences between Files

file - Show the File Type

date - Get the Date and Time

wc - Counting characters

which - Finding a Program File

whereis - Finding Files Used by a Program

find - Search for Files Using Different Criteria

who - See Users Logged On

finger - Get information on Users

Studying

Class Exercise

Class Quiz

`echo` - Print Text to the Terminal

`hostname` - Print the Name of Your Host Machine

`grep` - Finding Strings inside Files

`head` - View the Top of a File

`tail` - View the Bottom of a File

`sort` - Print a File in Sorted Order

`uniq` - Eliminate Duplicate Lines

`diff` - Differences between Files

`file` - Show the File Type

`date` - Get the Date and Time

`wc` - Counting characters

`which` - Finding a Program File

`whereis` - Finding Files Used by a Program

`find` - Search for Files Using Different Criteria

`who` - See Users Logged On

`finger` - Get information on Users