IT 244: Introduction to Linux/Unix

IT 244: Introduction to Linux/Unix
Class 7

Microphone

Questions

Are there any questions before I begin?

Readings

If you have the textbook you should read chapter 4 of Sobell, The Filesystem.

Homework 4

I have posted Homework 4 here.

As usual, it will be due next Sunday at 11:59 PM.

Graded Quiz Today

After I finish talking I will pass out the papers for today's graded quiz.

Today's Class

Today I will talk about text files in Unix and Windows as well as the untilities for compressing files for download.

I will also begin talking about the hierachical filesystem.

General Advice

Your Mental Health

You cannot do your best work if you often feel down ...
and have difficulty getting things done
How you feel about yourself and things in general ...
affects what you do ...
and how well you do it
If you are feeling down or stressed there are things you can do ...
to keep yourself on track:
- Get out - don't spend all your time at home
- Keep busy - it will take your mind off what's troubling you
- Talk to people - we are social animals and need contact with others
- Cut down online time - games and social media are no substitute for human interaction
If the way you are feeling prevents you from doing your best ...
do something about it
You can contact the Counseling Center by calling 617-287-5690
or sending a text to 988
This will put you in touch with a licensed clinician
This service is available 24 hours a day, 7 days a week
Or you can go to https://988lifeline.org
If you are thinking of hurting yourself ...
you can get help from the Samaritans
And you can always talk to me

Course Requirements

Collecting and Testing Assignments

All homework assignments and Class Exercises are due on the Sunday after they are assigned at 11:59 PM
I do not have the time to score this work until the following weekend
In the week after an assignment is due, I run scripts to collect homework assignments
If my script cannot find your homework file you will receive an email
I also run a script to test Class Exercises
I do this weekdays on Monday, Wednesday and Friday
If the script reveals errors you will receive an email noting the problem
I run these scripts for the last time the following weekend to give your submissions their final score
After I do this you will receive your score in an email
Any assignments not submitted by then receive a 0

Notifying You About Missing Work

If I cannot collect your homework assignment I will send you an email
If I cannot find your Class Exercise I will send you an email
If there is a problem with your Class Exercise I will send you an email describing the problem
This is only done in the week between the due date and the following weekend
Once I score an assignment I will send no more messages about missing work

Late Work

Once I have scored an assignment the grade book is closed
You cannot submit an "improved" copy of the script
You cannot submit a script that you did not get in on time
I simply do not have the time to rescore scripts

Rules for Homework Assignments

All homework assignments must follow certain rules
You will find a link to them on the class page
Let's review them

Tips and Examples

No Appointment Needed For Office Hours

You do not need an appointment for Office Hours
You just show up in my office ...
or connect to the Zoom session
You can use this time to get help with your scripts ...
or talk about other issues
And don't forget the weekend Zoom sessions ...
which I advertise with an email to all my students

Checking Class Exercises

If running my test scripts on your Class Exercises shows no errors ...
then your script is OK
If you cannot finish the exercise in class ...
you may do so at home
If you did not create a script for the exercise ...
you will receive an email from me
You will also get an email if there is a problem with your script
If you receive no email from me, your script is OK

An Easy Way to Create Homework Scripts

You may create your homework scripts anyway you like
It is only important that they meet the homework specification
But I would suggest the following approach
You can have more than one ssh window open at any one time
putty and the Mac Terminal program allow you to copy and paste ...
between different windows
You can use this feature in creating homework scripts
First, open two ssh windows connected to pe15.cs.umb.edu
In both windows go to the homework directory for the assignment
In one window run nano to create the homework script file
The filename must be the one specified in the assignment
In this script file write the comment
```
# Step 1
```
Now create two empty lines
Repeat this procedure for every step in the assignment
No go to the other window
Enter the Unix commands that will perform a step
When you get the commands working, copy and paste them into the script file
Make sure you put each command under the appropriate step number
And remember that some steps require more than one command

Read the Specification

It's important to read the homework assignments carefully
If you leave out a step, or a part of step, you will lose points
I try to be careful to say exactly what I want in an assignment
If you do not do everything I have asked ...
you will not get full credit
I do not do this to be obnoxious
I do this to prepare you for professional work
In technology, details are important
If you misspell a command in Linux/Unix you will get an error
If you skip a step in an installation procedure ...
the install will not work properly
In a well-run organization every project will have a specification
If you miss a step in this specification you have not completed the project
Treat the homework assignments like a specification and read them carefully
It is a good idea to read a specification more than once
When I was in school, I read each assignment immediately after class
That way I could look for anything that might cause me trouble
I could ask questions if I did not understand some point
Read over your homework assignment after you have done the work
Do this to make sure that you have not skipped something
One of the purposes of a college education ...
is to teach you habits that will help you later in life
Reading the specification carefully is a habit you should learn now

Always Check Your Work

Always run your scripts to see that they work before the assignment is due
If your script does not run, or runs with errors, you will lose points
You should never assume that a script that you have written will work
You may read a script and think it looks fine
But spelling or other errors are easy to overlook
Whenever you make a change to a script you should run it again to look for errors
Always check your work

Common Mistakes with Pipes

There are some mistakes that student seem to make each semester
Two of these mistakes involve pipes
The first mistake is to use cat at the beginning of a pipeline when it is not needed
Students will sometimes use cat to create input sent to the following commands
- grep
- sort
- head
- tail
You don't need cat in a pipe with these commands ...
because these utilities can read a file directly

So instead of using

cat names.txt | grep Glenn	# WRONG no need for cat with grep
cat names.txt | sort		# WRONG no need for cat with sort
cat names.txt | head		# WRONG no need for cat with head
cat names.txt | tail		# WRONG no need for cat with tail

You should simply use

grep Glenn names.txt
sort names.txt
head names.txt
tail names.txt

This mistake is harmless because it does not affect the result
But the second mistake will cause an error
If you call grep more than once in a pipeline ...
only the first call to grep gets two arguments
The search string and the filename
Every other call to grep gets one argument
Just the search string
If I wanted to find all Red Sox games where the Sox won at home I would write
```
grep Win red_sox.txt | grep vs
```

I would not write

grep Win red_sox.txt | grep vs red_sox.txt  # WRONG

If you do this, the output of the first grep will be thrown away ...
and you will only get the output of the second grep

Review

The Shell Prompt

By default, the shell prompt tells you three things
- Your Unix username
- The machine to which you are connected
- The location of your current directory
But before I changed it, my shell prompt looked like this
```
ghoffman@pe15:~/it244/work$
```
The characters before the @ show my Unix username
The characters between @ and : are the hostname of the machine I am using
And the characters from : to $ show my current directory
The ~ indicates your home directory
We'll talk more about this in a future class

Pipes - Stringing Programs Together

A pipe takes the output of one command ..
and feeds it into the input of another command
Pipes allow you to chain together several Unix commands ...
into a single command line
Commands joined in this way are sometimes called pipelines
Pipes are essential to the Unix philosophy of simple tools
Pipes let you string together many commands ...
to achieve exactly what you want

You form a pipe by placing the vertical line character, |, between two commands

$ head -5 red_sox.txt
2011-07-31  Red Sox @  White Sox    Win 5-3
2011-07-02  Red Sox @  Astros       Win 7-5
2011-07-03  Red Sox @  Astros       Win 2-1
2011-07-04  Red Sox vs Blue Jays    Loss 7-9
2011-07-05  Red Sox vs Blue Jays    Win 3-2

$ head -5 red_sox.txt | sort
2011-07-02  Red Sox @  Astros       Win 7-5
2011-07-03  Red Sox @  Astros       Win 2-1
2011-07-04  Red Sox vs Blue Jays    Loss 7-9
2011-07-05  Red Sox vs Blue Jays    Win 3-2
2011-07-31  Red Sox @  White Sox    Win 5-

Notice that sort does not have an argument
Normally sort requires an argument that specifies the file to sort
In a pipe, each command after the first ...
takes its input from the output of the preceding command
You never need to specify the input ...
when using a command in a pipeline ...
except for the first command

`uniq` - Eliminate Duplicate Lines

uniq prints the lines of a file, leaving out any adjacent lines that are identical
Notice that the identical lines must be adjacent
One right after the other
For this reason, you should pipe the output of sort into uniq
uniq -i ignores case when looking for duplicate lines
If the file has duplicate lines but they are not adjacent ...
you must pipe the output from sort into uniq

`diff` - Differences between Files

diff compares two files and notes their differences
diff was created for use with the patch utility
Run diff with the -y option to get output that is easier to read

`file` - See the File Type

The file utility can be used the determine the type of a file

$ file *
class_notes.css:           ASCII text
common_unix_commands.html: HTML document text
cs285L:                    directory
emacs_cheat_sheet.html:    HTML document text
index.html:                HTML document text
it244:                     directory
tips.html:                 HTML document text
unix_cheat_sheet.html:     HTML document text
work.txt:                  ASCII text

`date` - Get the Date and Time

The Unix date command will give the time and the date
```
$ date
Tue Aug 21 10:20:05 EDT 2012
```
If we follow the command with a + and a string ...
we can change the format
```
$ date +"%Y-%m-%d"
2012-08-21
```
Use info or man to see the various formatting options date provides

`wc` - Counting characters

wc counts the lines, words and characters in a file

$ cat lines.txt 
Line 1
Line 2
Line 3
Line 4
Line 5

$ wc lines.txt 
 5 10 35 lines.txt

This output says there are 5 lines, 10 words and 35 characters in lines.txt

You can use options to tell wc what to count

c - bytes (characters)
w - words
l - lines

$ wc -c lines.txt 
35 lines.txt

$ wc -w lines.txt 
10 lines.txt

$ wc -l lines.txt 
5 lines.txt

`which` - Finding a Program File

Unix commands are programs that are located somewhere in the filesystem
The Unix utility which gives the location of an executable file
To find where the executable file for less is located ...
we can run which like this
```
$ which less
/usr/bin/less
```
which uses the PATH system variable to find the executable file
We'll discuss PATH in a future class

`whereis` - Finding Files Used by a Program

whereis is another program that can be used to locate program files
whereis takes a different approach than which
Every Unix or Linux system has certain standard places where it stores programs ...
and the files they use
whereis searches these locations
It returns a list of all files associated with a program
When we run whereis on the tar utility ...

we get more information than which returned

$ whereis tar
tar: /bin/tar /usr/include/tar.h /usr/share/man/man1/tar.1.gz

The first entry is the executable file
The second is a header file
The third is the file that man uses to provide information about tar

`who` - See Users Logged On

who prints a list of all users currently on the machine

$ who
offner   pts/0        2019-07-30 07:38 (50.200.93.68)
rickm    pts/1        2019-07-26 17:05 (192.168.105.41)
grullnor pts/2        2019-08-06 16:04 (192.168.107.156)
ldavid   pts/3        2019-07-31 07:47 (158.121.23.66)
...

who also provides information about each user's login session

who am i will show the user who is logged into a terminal

$ who am i
ghoffman  pts/0     2012-08-12 13:41 (dsl092-066-161.bos1.dsl.speakeasy.net)

`finger` - Get information on Users

finger provides information about a user account

$ finger ghoffman
Login: ghoffman                         Name: Glenn Hoffman
Directory: /home/ghoffman               Shell: /bin/bash
On since Wed Aug  7 07:06 (EDT) on pts/5 from 209.6.202.123
    8 hours 21 minutes idle
On since Wed Aug  7 19:21 (EDT) on pts/19 from 209.6.202.123
   2 seconds idle
Mail forwarded to glennhoffman@mac.com
Mail last read Mon Jul 15 10:58 2019 (EDT)
Plan:
Office:         McCormack 3-0201-22 (Pterosaur poster on door)
Office Hours:   Tuesday & Thursday, 11:OO AM - 12:00 PM and 5:30 to 6:30 PM
Classes:        IT 116-1   Introduction to Scripting     TuTh    12:30  - 1:45 PM     Wheatley 1-004
                IT 244-1   Introduction to Linux/Unix    TuTh     2:00  - 3:15 PM     McCormack M1-409
                IT 117-1   Intermediate Scripting        TuTh     4:00  - 5:15 PM     University Hall 2-2310
Home Address:   40 Central Street
                Somerville, MA 02143

finger, like mv, has two functions

When used without an argument, finger shows every user currently logged in

$ finger
Login     Name                Tty      Idle  Login Time   Office     Office Phone
apera     Antonio Pera        pts/12   7:48  Aug  7 09:53 (73.16.254.131)
apera     Antonio Pera       *pts/15   4:29  Aug  7 15:18 (192.168.105.83)
gabrgabr  Gabriel Park        pts/13    14d  Jul 17 09:46 (tmux(29686).%0)5
...

Attendance

New Material

Text File Conversion Programs

Text files on Unix are slightly different from text files on Winows
A text files is a collection of characters ...
with a non-printing character that marks the end of the line
Unix text files use the newline character, \n to mark the end of the line
Windows text files use two non-printing character do to the same
They use carriage return, \r and linefeed \n
Carriage return moves to the begining of the line
The term comes from old style typewriters
When you got the end of a line ...
you had to hit a vertical bar to move the carriage back into position

If you want to work with a text file from a Windows machine ...
you have to change \r\n to \n
There are two packages that provide software ...
to convert between Unix and Windows text files
- tofrodos
- unix2dos
We won't be using either of them in this course ...
but you should know about them
The tofrodos package provides todos ...
to convert Unix text files to Windows text files
The same package has fromdos to go the other way
The unix2dos package uses unix2dos ...
to convert Unix text files to Windows text files
It also provides dos2unix to go the other way
You will not be expected to know this for a quiz or exam

Compressing Files with `bzip2`

On the Internet there are many files which are free for the taking
Many of these files are huge
and downloading them can take a long time
To speed up the process, big files are usually compressed ...
to reduce their size
Compression utilities are used to do this
bzip2 is one such utility
It achieves the highest compression ratio of all common compression utilities
You run bzip2 like this
```
bzip2 FILENAME
```
bzip2 compresses the file ...
creating a new file with the extension .bz2
It then deletes the original file
If you need to keep the original file run bzip2 with the -k (for keep) option
To decompress a file created by bzip2 ...
use the bunzip2 utility like this
```
bunzip2 FILENAME.bz2
```

For example the html file for todays Class Notes is 41K

$ ls -lh 07_class_notes_it244.html 
-rw-r--r-- 1 ghoffman faculty 41K May 27 09:24 07_class_notes_it244.html

If I run bzip2 on the file ...

it is compressed to 11K

$ bzip2 07_class_notes_it244.html

$ ls -lh 07_class_notes_it244.html.bz2 
-rw-r--r-- 1 ghoffman faculty 11K May 27 09:24 07_class_notes_it244.html.bz2

bunzip2 decompresses the .bz2 file to recreate the file

$ bunzip2 07_class_notes_it244.html.bz2 

[506] (pe15) ghoffman - ~/public_html/it244_html/class_notes_it244
$ ls -lh 07_class_notes_it244.html 
-rw-r--r-- 1 ghoffman faculty 41K May 27 09:24 07_class_notes_it244.html

This file will have the .bz2 extension removed ...
and the compressed file is also deleted
A file that bzip2 has compressed is unreadable
To look at the contents of a .bz2 compressed file ...
without uncompressing it, use bzcat
bzcat will print the uncompressed contents of a file to the screen
bzcat does not alter the original, compressed file
You will not be expected to know this for a quiz or exam

`gzip` - the GNU Compression Utility

The GNU project created gzip to compress files
It is older, and less efficient, than bzip2
But many open source packages are compressed with this program

It is similar in operation to bzip2

$ gzip 07_class_notes_it244.html 

[508] (pe15) ghoffman - ~/public_html/it244_html/class_notes_it244
$ ls -lh 07_class_notes_it244.html.gz 
-rw-r--r-- 1 ghoffman faculty 12K May 27 09:24 07_class_notes_it244.html.gz

The compressed files gzip creates have a .gz extension
To show the contents of a file without converting it use zcat
To decompress a gzipped file, use gunzip
These utilities have nothing to do with the zip and unzip programs ...
that are frequently used on Wintel machines
You will not be expected to know this for a quiz or exam

`tar` - Packaging Files for Transfer or Storage

Most software packages consist of many files
But to distribute the package efficiently ...
you want to turn these into a single file
This also needs to be done when backing up a directory
tar (tape archive) is the Unix utility used for this purpose
tar does not compress files
It stuffs multiple files into a single file
This file is often called a tarball
tar is usually used along with a compression program
First you run tar to create a single file
Then you run another utility to compress the file
You also use tar to unpack the files
You run tar with different options ...
to pack, or unpack, a set of files
To "tar up" a set of files, run
```
tar cvf ARCHIVE_NAME.tar DIRECTORY_NAME
```
The options stand for create, verbose, file
The .tar extension is a convention
Though you do not have to use this extension ...
you would be foolish not to
To see the files contained in a tar file without unpacking them use
```
tar -tvf ARCHIVE_FILE
```
To unpack a tarball use
```
tar -xvf ARCHIVE_FILE
```
The -x option stands for extract
Normally, you create a tarball
and then run a compression program on the archive file
If bzip2 is used for compression ...
the new file will often have a .tar.bz2 or .tbz extension
If gzip is used the extensions usually are .tar.gz, or .tgz ...
or sometimes simply .gz
Again, these are simply conventions
But don't violate these conventions unless you have a very good reason
Life is complicated enough as it is
You will not be expected to know this for a quiz or exam

The Hierarchical Filesystem

A filesystem is an arrangement of data on a storage device ...
and the software needed to store and retrieve that data
All files are contained in directories
On Windows and the Mac these directories are called folders
A directory can contain other directories
A hierarchical filesystem starts with a special directory at the top
This directory is called the root directory

In Unix, we represent this directory with a single slash character, /

$ ls /
bin     initrd.img  proc tmp
boot    lib         root        tools
courses lib32       run         users
data    lib64       sbin        usr
dev     lost+found  selinux     var
etc     media       sources     vmlinuz
groups  mnt         spool
home    nobackup    srv
home.ORIG  opt      sys

All other directories are contained within the root directory ...
or one of its many subdirectories
So the shape of the hierarchical filesystem is like a pyramid

On a machine, the hierarchical filesystem might look like this

This structure is called a tree because it looks like a tree turned upside down
Unlike in nature, this tree grows down from the root
A hierarchical filesystem is endlessly extensible
We can have as many files and directories as we want
As long as there is enough disk space to hold them
Since a hierarchical filesystem resembles a family tree ...
we often use genealogical terms when talking about directories
The directory one level up is called the parent directory
All directories above the current directory are called ancestors
All directories inside the current directories are called child directories
All directories inside or below the current directory can be called descendants
Directories and files within the same directory are called siblings

Unix Files and Directories

Files are linear arrangements of data on some storage device
Linear means it's like a string ...
with a beginning and an end
Some files are programs
Program files contain binary instructions that the CPU understands
Some files contain data in various formats such as text
Some Unix text files contain configuration information
These configuration files modify your Unix environment
They perform the same function as the Windows Registry
These files are text so you can read and edit them easily
You only need a text editor to read or change them ...
not some special utility like the Windows Registry Editor

Filenames

When you ask Unix for a file you must give it two pieces of information
- The name of the file
- The location of the file in the hierarchical file system
Every file has a name - the filename
Different Unix flavors allow filenames of different lengths
Most Unix systems allow filenames to be no more than 255 characters
It's best to keep filenames short
This makes them easier to type
You're less likely run into trouble ...
if you move the file to another system with a different maximum
To learn the maximum length of a filename on any machine use stat ...

piped into grep to find the right line

$ stat -f /home | grep Namelen
    ID: 2b00000000 Namelen: 255     Type: autofs

You can use a space in filenames if you enclose them in quotes
But this is not a good idea
Why?
Because the name has two parts and looks like two files
```
$ touch homework 11

$ ls
11  homework
```
To avoid problems, only use the following characters in file names
- Lowercase letters (a-z)
- Digits (0-9)
- Underscore _
- Dash -
- Period .
You cannot have two files with the same name in the same directory

Case Sensitivity

Unix is case sensitive
This means that Foo, foo and FOO are three different files
```
$ ls

$ touch Foo foo FOO

$ ls
foo  Foo  FOO
```
Unix commands are always lower case
This is just a convention
But it is a helpful convention
Otherwise you would have to remember capitalization ...
as well as the name of the command
Some operating systems are not case sensitive
Like Windows
Stick to the convention of always using lowercase letters in filenames
This will make it easier to move between different operating systems

Filename Extensions

Extensions are strings of characters ...
that appear at the end of the filename after a period, .
Unix ignores extensions
As far as Unix is concerned, extensions are just part of the filename
However, some Unix programs, do require certain extensions
The C compiler, gcc, expects the filenames of source code files to end in .c
The Java compiler, javac, expects Java source files to have .java extension
These programs enforce this rule
Unix couldn't care less
As far a Unix is concerned an extension is just part of the filename
It has no special significance
You may have noticed that all my text file have a ".txt" extention ...
and all my scripts use ".sh"
I do that so I know which are text files ...
and which are bash scripts
But Unix does not force me to do this

Current Directory

The way a command works depends on your Unix environment
One of the most important parts of your environment is your current directory
Your current directory is your current location in the filesystem
pwd (print working directory) command will always tell you your current directory
If you run ls without an argument it will list your current directory

Your Home Directory

Whenever you log in to a Unix machine ...
you will always find yourself inside your home directory
This directory belongs to your Unix account only
You can do anything you want in your home directory
Including blowing it all away
You can create files or directories in your home directory
You can let other users have access to the contents of your home directory
You can always get back to your home directory ...
by running cd (change directory) with no argument
```
$ cd

$ pwd
/home/ghoffman
```
Your home directory contains a number of "hidden" files ...
which are used to customize your Unix environment
The names of these files begin with a period, .
They do not appear in an ls listing ...
unless you use the -a option
On most Unix systems, your home directory will be in the /home directory
On a Mac it will appear in the /Users directory

Any file or directory in the hierarchical filesystem ...
will be one of four positions relative to your current directory
- Inside your current directory
- Below your current directory
- Above your current directory
- Off to the side of your current directory
In this last case, you must go up ...
before you can go down to reach the file or directory

General Advice

Course Requirements

Tips and Examples

Review

New Material

Questions

Readings

Homework 4

Graded Quiz Today

Today's Class

General Advice

Your Mental Health

Course Requirements

Collecting and Testing Assignments

Notifying You About Missing Work

Late Work

Rules for Homework Assignments

Tips and Examples

No Appointment Needed For Office Hours

Checking Class Exercises

An Easy Way to Create Homework Scripts

Read the Specification

Always Check Your Work

Common Mistakes with Pipes

Review

The Shell Prompt

Pipes - Stringing Programs Together

uniq - Eliminate Duplicate Lines

diff - Differences between Files

file - See the File Type

date - Get the Date and Time

wc - Counting characters

which - Finding a Program File

whereis - Finding Files Used by a Program

who - See Users Logged On

finger - Get information on Users

Attendance

New Material

Text File Conversion Programs

Compressing Files with bzip2

gzip - the GNU Compression Utility

tar - Packaging Files for Transfer or Storage

The Hierarchical Filesystem

Unix Files and Directories

Filenames

Case Sensitivity

Filename Extensions

Current Directory

Your Home Directory

Navigating the Hierarchical File Systems

Class Exercise

Class Quiz

`uniq` - Eliminate Duplicate Lines

`diff` - Differences between Files

`file` - See the File Type

`date` - Get the Date and Time

`wc` - Counting characters

`which` - Finding a Program File

`whereis` - Finding Files Used by a Program

`who` - See Users Logged On

`finger` - Get information on Users

Compressing Files with `bzip2`

`gzip` - the GNU Compression Utility

`tar` - Packaging Files for Transfer or Storage