IT 116: Introduction to Scripting

IT 116: Introduction to Scripting
Class 19

Today's Topics

Reading Assignment

You should read Chapter 7, Lists and Tuples, from our textbook, Starting Out with Python, before next week's class.

Homework 9

I have posted homework 9 here.

It is due this coming Sunday at 11:59 PM.

Quiz 6

I have posted the answers to Quiz 6 here.

Tips and Examples

A Simple Function

A function is just a group of Python statements that is given a name
You should create a function whenever you do something more than once
A simple example of a function is one that converts Celsius to Fahrenheit
There is a simple formula for this conversion
```
fahrenheit = 9/5 * celsius + 32
```
If I only needed to calculate this value once I would not write a function for it
But if I needed to do this more than once I would write the following function
```
def celsius_to_fahrenheit(celsius):
    return round(9/5 * celsius + 32)
```
If I used this in more than one script I would put it in a module
I could then imported it into any script where it was needed

Don't Use readline() in a `for` Loop

The readline() method of a file object is only needed in certain unusual situations
Most of the time when you want to read a file, use a for loop
NEVER USE readline() in a for loop
If you do, you will not read every line

Here is an example of what happens when you use a readline() in a for loop

$ cat 1-5.txt 
1
2
3
4
5

>>>  file = open('1-5.txt', 'r')
>>>  for line in file:
...     line = file.readline()
...     print(line.strip())
... 
2
4

Review

Getting Rid of Newline Characters

The lines in a text file all have a newline character, \n, at the end of each line
You must remove this newline character when printing the line
Or there will be a blank line after each printed line
As you will see in a few weeks, strings are objects
Like most objects, they have methods
rstrip is a string method that removes newlines at the end of a string
You would use it like this
```
line = line.rstrip()
```

Writing and Reading Numbers

In this course we will only be dealing with text files
Any numbers you read from a file will be strings
You can't perform arithmetic on strings
You need to convert them into numbers using either int or float
When writing a number to a file, you must convert it into a string
You can do this with the str conversion function
If you don't convert a number to a string when you are writing to a text file you will get an error

Appending Data to an Existing File

If you write something to a file that already exists it will erase the current text
And will overwrite it with new text
To avoid doing use the append access mode, "a"
Writing to a file open for appending adds text to the bottom of the fine

Reading Files with a `for` Loop

The easiest way to read a file in Python is to use a for loop
The header of for loop has this format
```
for LOOP_VARIABLE in LIST_OF_SOME_KIND:
```
The last item is some sort of collection
A text file is a collection of lines
This means we can use a file object in a for loop
If you file object for reading pointed to by the variable file you can print the file like this
```
for line in file:
    print(line.rstrip())
```

New Material

Running a Script Using `python3`

To write a program in the computer language C you create a text file
This file contains C statements and called the source code
You cannot run this file
In order to create a working program this file must be run through a compiler
The compiler creates a binary file
The binary file consists of instructions written in machine language
Machine language is binary code the CPU understands
The CPU does not understand C or Python
Our Python scripts are text files written in Python programming language
The computer does not understand this text so we cannot run Python scripts directly
Only the Python interpreter understands Python scripts
The Python interpreter was written in a language like C and compiled into an executable file
When we type python3 at the command line we are running this binary file
To run the script we give this command the filename of the Python script
python3 then executes all the Python statements in the file

So if I have the Python script hello_1.py

# prints a friendly message

print("Hello world!")

I can run it on Unix command line like this
```
$ python3 hello_1.py 
Hello world!
```

Running a Script Without `python3`

Unix gives us another way to run a Python script

Look at the Bash script hello1.sh

# prints a friendly message

echo Hello world

I can run this script using the bash command
```
$ bash hello_1.sh
Hello world!
```
The program bash can run a bash script just like the program python3 runs a Python script
But if I change the access permission on this Bash script using the Unix command chmod
And give everyone read and execute permission
```
$ chmod 755  hello_1.sh
```
I can now run the script directly without using bash
```
$ ./hello_1.sh 
Hello world!
```
I have to put . / before the script name for security reasons
This only works if your current directory contains the script
If I give the same permissions to a Python script hello_1.py
```
$ chmod 755  hello_1.py
```

And try to run it, I will get an error

$ ./hello_1.py
./hello_1.py: line 3: syntax error near unexpected token `"Hello world!"'
./hello_1.py: line 3: `print("Hello world!")'

It won't work because Unix does not know what program to use when running the script
When this happens it uses bash
bash does not understand Python, so you get an error message
Unix has a feature which allows us to name the program that will run the script
But we have to use the absolute pathname of the program
On our Unix systems, the Python interpreter python3 is located in the bin directory inside the usr directory
So the absolute pathname is /usr/bin/python3
I can tell Unix to run the script with python3 if the very first line of the script is
```
#! /usr/bin/python3
```

So the script now looks like this

#! /usr/bin/python3

# prints a friendly message

print("Hello world!")

I can now run the script without running python3
```
$ ./hello_2.py 
Hello world!
```
This special line is called the hashbang line
This line must be the first line in the script
And the first two characters must be #!
These two characters must be followed by the absolute pathname of the program to run the script
If the script were written in Perl the hashbang line would be
```
#! /usr/bin/perl
```
From now on I want you to follow these procedures you make all scripts executable
All scripts must have the following as the first line
```
#! /usr/bin/python3
```
And you must run chmod on them
```
chmod  755  FILENAME
```
I have changed the script specification document to add this requirement

Why Make a Script Executable?

Why go through the trouble of making a script executable?
Typing python3 before the name of a file is no great hardship
I have a script that sends an email to all students who have not submitted a homework assignment
The script is email_missing.py
What's the big deal about typing the following at the command line?
```
python3  email_missing.py
```
The problem is this won't work
It would work if email_missing.py were in my current directory
and I put . / in front of it
But the script is never in the directory where I need to use it
The script has to be run in the directory I used to collect homework assignments
That directory would be a subdirectory of /home/ghoffman/submitted/it116_submitted/hw_submitted_it116
But email_missing.py is in the directory /home/ghoffman/bin/python

So I used python3 to run the file I would have to write

python3  /home/ghoffman/bin/python/umb/email_missing.py

Not only that, but the pathname could not have any mistakes
If I was off by one letter I would get an error message
I want to be able to run this script by simply typing
```
email_missing.py
```
I can achieve this if I do two things
- Make the script executable
- Put the script in a directory I have added to the PATH shell variable
As you will learn in IT 244, PATH tells Unix were to look for the executable files that correspond to Unix commands
I have included /home/ghoffman/bin/python/umb in my PATH variable
So when I type the name of the script I run the script

Averaging the Numbers in a File

Let's use a for loop to average the numbers in a file
Say we have a text file containing integers one to a line
```
$ cat numbs.txt
1
2
3
4
5
6
7
8
9
10
```
We need to create a file object, so let's ask the user for the filename

We can use this to create a file object for reading

filename = input("Filename: ")
file = open(filename, "r")

We will need to total the numbers, but we also need to count them
So we must initialize two accumulators to 0
```
total = 0
count = 0
```
Now we need a for loop that will read each line
```
count = 0
```
In the body of the loop we do two things
First we increment the count
```
count += 1
```
Then we convert the line to integer and add it to the total
```
total += int(line)
```

Finally we calculate the average and print it

average = round(total/count, 2)
print("Average:", str(average))

This gives us the following script

#! /usr/bin/python3

# reads numbers from a file and averages them

filename = input("Filename: ")
file = open(filename, "r")

total = 0
count = 0
for line in file:
    count += 1
    total += int(line)

average = round(total/count, 2)
print("Average:", str(average))

The script has a hashbang line and I have made it executable

So I can run the script like this

$ ./for_average.py 
Filename: numbs.txt
Average: 5.5

Records

For any sporting event involving teams we have at least 5 pieces of information
- Date
- Home team
- Opposing team
- Home team score
- Opposing team score
The collection of data about a specific event is called a record
The individual pieces of data within a record are called fields
A text file containing data usually has one record per line

For example, we might have a file of dates and the highest temperature

$ cat temps.txt 
2017-06-01 67
2017-06-02 71
2017-06-03 69
2017-06-04 88
2017-06-05 74
...

Here the fields are
- Date
- Highest temperature

Reading Records from a File

How can we store two pieces of information from a single line?
To do this we need to know where one field ends and the next begins
The characters used to separate one field from the next are called delimiters
In temps.txt the delimiter is a space
When you export data from Excel you can choose many formats
Very often people chose CSV, which stands for Comma Separated Values
In these files a comma separates the fields
We know how to read in a text file line by line
But how do we get each field?
The easiest way to do this is with another string method named split

split breaks up the line into individual fields

>>> line = '2017-06-01   67'
>>> line.split()
['2017-06-01', '67']

We can use multiple assignment to assign the value of each field to a variable
```
>>> date, temp = line.split()
```

With this approach we can read each record in temps.txt

$ cat temps_print.py
#! /usr/bin/python3

# prints each field in temps.txt

file = open('temps.txt', 'r')

for line in file:
    date, temp = line.split()
    print(date, temp)
    
$ ./temps_print.py
2017-06-01 67
2017-06-02 71
2017-06-03 69
2017-06-04 88
2017-06-05 74
...

Finding the Average From a File of Records

Usually we read a file of records to do some processing of the data
The simplest processing of numbers is to calculate the average
We can create a script to do this starting with temps_print.py
Then we can make the appropriate changes
First we need add two accumulators
```
count = 0
total = 0
```
The for loop header stays the same
But we need to add a statement to count lines
```
count += 1
```
We need to update total
But before we do this we must turn temp into an integer
```
temp   = int(temp)
total += temp
```
Outside the loop we need to calculate and print the average
```
average = round(total/count, 2)
print('Average:', average)
```

Putting this all together we get

$ cat temps_average.py 
#! /usr/bin/python3

# calculates the average the temperature in temps.txt

file = open('temps.txt', 'r')

count = 0
total = 0
for line in file:
    count += 1)
    date, temp = line.split()
    temp   = int(temp)
    total += temp
    
average = round(total/count, 2)
print('Average:', average)

The added statements are shown in red
Now we run it
```
$ ./temps_average.py 
Average: 77.27
```

Additional Processing of Records

In the code above we only calculated the average
But we can do more than that while processing the file
Why not compute the highest temperature and the lowest?

To compute the highest temperature we use the following algorithm

set a variable to the lowest possible value
for each value in the file
    if the value is greater than the variable
        set the variable to this value

Why did do we have to set the variable to a low value?
To make sure it is replaced by some value we comes across in the loop
If we set the variable to 100 it would never be replaced by any value in temps.txt
To calculate the minimum we need to set the variable before entering the loop
```
max = -100
```
Then we need to add an if statement to the loop
```
    if temp > max:
        max = temp
```

A similar algorithm will calculate the lowest temperature

set a variable to the highest possible value
for each value in the file
    if the value is less than the variable
        set the variable to this value

Once again we need to set the the variable before entering the loop
```
min = 200
```
And update this value inside the loop
```
    if temp < min:
        min = temp
```

Here is the script with the additions in red

$ cat temps_max_min_average.py 
#! /usr/bin/python3

# calculates the average, maximum and minimum in temps.txt

file = open('temps.txt', 'r')

count = 0
total = 0
max   = -100
min   = 200
for line in file:
    count += 1
    date, temp = line.split()
    temp  = int(temp)
    total += temp
    if temp > max:
        max = temp
    if temp < min:
        min = temp

average = round(total/count, 2)
print('Average:', average)
print('Maximum:', max)
print('Minimum:', min)

When we run this we get

$ ./temps_max_min_average.py 
Average: 77.27
Maximum: 89
Minimum: 66

Closing a File Object

You get rid of a file object by using the close method on the variable that points to the object
```
file.close()
```
The interpreter will close the file object for you when the script ends
But it is good practice for your script to close a file object when you are done with it

Looping Through a File More Than Once

What if we want to determine the number of days when the temperature was above average
We need to loop through the file more than once to do this
We loop through it once to get the average
Then we loop through it again to count the days with temperatures above average
But in order to do this, we must close the file after the first loop
We need to do this because when we open a text file for reading we can only go in one direction
Once we have reached the end of a file, there is no way to rewind to the beginning
Instead we have to create a new file object
Before we do that, it is good practice to close the first file object we created

The algorithm we need is similar to that for calculating a maximum

set a variable to 0
for each value in the file
    if the value is greater than the average
        add 1 to the variable

Here is the code we need to add to temps_average.py

file.close()
file = open('temps.txt', 'r')

days_above = 0
for line in file:
    date, temp = line.split()
    temp   = int(temp)
    if temp > average:
        days_above += 1

print("Days above average:", days_above)

When we run this we get

$ ./temps_days_above_average.py
Days above average: 13