IT 116: Introduction to Scripting

IT 116: Introduction to Scripting
Class 18

Today's Topics

Review

Files
Types of Files
Identifying a File
Storing Data on a Hard Disk
Files and the Kernel
Working with a File in Python
Objects
Accessing the Data in Files
Opening a File
Specifying the Location of a File
Writing Data to a File
Reading Data From a File

Homework 8

I have posted homework 8 here.

It is due this coming Sunday at 11:59 PM.

Review

Files

When processing large amount of data we need to use files
A file is simply a linear arrangement of data on some storage medium
That storage medium might be a
- Hard disk
- Flash drive
- CD ROM
- SSD card

Types of Files

All files consist of binary numbers
but those numbers can be interpreted in two ways
- Text
- Binary

Consider the following text file

$ cat fruit.txt 
grapes
pears
oranges
cranberries
apples
melons
blueberries

To see how this file is really stored on disk I can use the Unix od (octal dump) command

$ od -b fruit.txt 
0000000 147 162 141 160 145 163 012 160 145 141 162 163 012 157 162 141
0000020 156 147 145 163 012 143 162 141 156 142 145 162 162 151 145 163
0000040 012 141 160 160 154 145 163 012 155 145 154 157 156 163 012 142
0000060 154 165 145 142 145 162 162 151 145 163 012
0000073

To see the characters, I can run od with the -c option

od -c fruit.txt 
0000000   g   r   a   p   e   s  \n   p   e   a   r   s  \n   o   r   a
0000020   n   g   e   s  \n   c   r   a   n   b   e   r   r   i   e   s
0000040  \n   a   p   p   l   e   s  \n   m   e   l   o   n   s  \n   b
0000060   l   u   e   b   e   r   r   i   e   s  \n
0000073

A binary file, such as a JPEG image also consists of numbers
But the numbers do not represent characters
Text files have a simple structure
Each file is a collection of lines
A line is a series of characters with the newline character, \n, at the end
We will only be dealing with text files in this course

Identifying a File

Every file has two unique attributes that uniquely identify it
- Name
- Location
The location of a file is given by a path
A path is a list of directories
The last directory in a path is the one that holds the file

Storing Data on a Hard Disk

Hard disk consists of rapidly spinning disks
The disk is made up of material that can be magnetized
Small regions on the disk are magnetized point up or down
Each of these fields is a binary digit called a bit
The surface of the disk is broken up into rings called tracks
Each track is broken up into sectors
Most files consist of many sectors scattered throughout the disk

Files and the Kernel

At the core of every operating system is a program called the kernel
The kernel handles all interactions with hardware
To read a file, the interpreter must use the kernel
The kernel will then use a special program called a device driver
The device driver handles communication between the hardware and the kernel
Each piece of hardware attached to the computer must have a device driver
The device driver is created by the manufacturer of the device

Working with a File in Python

A file lives on a some long term storage device
But a program works with the file in RAM
RAM is fast and expensive
External storage is slower and cheap
RAM is limited and files can be very large
So there is usually only a small part of the file in RAM
The interpreter needs to know where to get the rest of the file
The data needed to do this is stored in a file object
This object contains all the bookkeeping data needed to work with the file

Objects

A variable holds only one piece of information
Objects can hold many such pieces
The data is contained in variables inside the object
Objects also hold functions that interact with the data
The functions that work on this data are called methods
Objects have no name, just a location in RAM
We create an object by using a special function
This function returns a pointer to the object in RAM
We store this value of this pointer in a variable
Then we use this variable to work with the object

Accessing the Data in Files

There are two ways to get data from a file
- Sequential access
- Direct access
Sequential access is like a tape recorder
It starts at the beginning and runs to the end
It can only go in one direction
Direct access goes to a particular place in the file
We will only be using sequential access in this class

Opening a File

To work with a file we have to create a file object
We do this using the built-in function open
The open function does two things
- It creates a file object
- It returns a reference to the object
The reference is stored in a variable
We then use the variable to work with the object
Here is the format for using the open function
```
FILE_VARIABLE = open(FILENAME, MODE)
```
FILE_VARIABLE is the name of the variable that hold the reference
FILENAME is a string specifying the file
MODE determines what we can do with the file

The three most commonly used access modes are

Mode	Description
r	Open a file for reading only
w	Open a file for writing
a	Open a file for writing to the end of the file

A file open for reading cannot be changed
If you open a file that does not exist it will be created
If the file exists it will be overwritten with new text
The a in the last mode stands for "append"
Text is added to the end of the file so nothing is lost

Specifying the Location of a File

A file is identified by two attributes
- It's name
- It's location
If the file is in the same directory as the script, you just need the name
Otherwise you need both name and location
This is done by using a pathname
The pathname has two parts
- The path to the file
- The filename
A path is simply a list of directories
The last directory in the list holds the file
In Linux and Unix the name of a directory is followed by a /
In Windows you have to use a backslash, \, instead
There are two types of path
- Absolute
- Relative
The difference between between them is the directory where they start
A relative path starts with your current directory
The current the directory is the one that holds the script which opens the file
An absolute path starts with the special directory at the very top of the filesystem
This directory is called the root directory
On Unix the root directory is written /

Writing Data to a File

To work with a file in Python, we must first create a file object
This object contains methods that allow us to talk to the file
The functions inside objects are called methods
To call a method you must use dot notation
Dot notation for methods has the following format
```
OBJECT_VARIABLE.METHOD_NAME
```
We first open the file for writing
```
teams_file = open("teams.txt", "w")
```

Now we can send text to the file using write method of this file object

$ cat write_teams.py
# writes the names of the teams in the American League East

teams_file = open("teams.txt", "w")

teams_file.write("Boston Red Sox\n")
teams_file.write("Baltimore Orioles\n")
teams_file.write("Toronto Blue Jays\n")
teams_file.write("Tampa Bay Rays\n")
teams_file.write("New York Yankees\n")


$ python3 write_teams.py 

$ cat teams.txt 
Boston Red Sox
Baltimore Orioles
Toronto Blue Jays
Tampa Bay Rays
New York Yankees

Notice that I had to put the newline character, \n, at the end of each line
Otherwise all the text would be on one line

Reading Data From a File

There is more than one method that can read a line

The simplest is read which returns a string containing everything in the file

$ cat read_teams.py
# reads the contents of the file teams.txt

teams_file = open("teams.txt", "r")

teams = teams_file.read()

print(teams)

$ python3 read_teams.py 
Boston Red Sox
Baltimore Orioles
Toronto Blue Jays
Tampa Bay Rays
New York Yankees

It turns out that read is not very useful
Normally we want to read a file line by line

New Material

Reading a File Line By Line

What if you had a file with numbers and you wanted to add them?
The read method could not be used to do this
We really want to be able to read in each line and get the number
Then we could add that number to our running total
One way to do this is to use the readline method of the file object
We use sequential access with text files
The file object keeps track of were we are in the file
It has a variable that points to the next line to be read
Each time we call readline it reads in the next line
If we have a file with three lines, we can call readline three times to read the file
A line is a string of characters with a newline, \n at the end
readline reads the entire line, including the newline

Getting Rid of Newline Characters

Here is a file with four lines

$ cat teams.txt 
Red Sox
Patriots
Celtics
Bruins

Here is a script that will read and print each line

$ cat read_teams.py 
# reads and prints the lines in teams.txt

file = open('teams.txt', 'r')

line_1 = file.readline()
print(line_1)
line_2 = file.readline()
print(line_2)
line_3 = file.readline()
print(line_3)
line_4 = file.readline()
print(line_4)

When we run this script we get

$ python3 read_teams.py 
Red Sox

Patriots

Celtics

Bruins

Why the blank lines?
Remember, print adds a newline to the end of what it prints
But each line already has a newline character
The newline in the line causes the output to go to the next line
The newline supplied by print makes it go down another line
It turns out that strings are also objects
We'll talk more about this in a few weeks
Strings have methods and one of them is rstrip
rstrip removes newlines from the end of the string

So we can rewrite the script to print single lines like this

$ cat read_teams_2.py
# reads and prints the lines in teams.txt

file = open('teams.txt', 'r')

line_1 = file.readline()
print(line_1.rstrip())
line_2 = file.readline()
print(line_2.rstrip())
line_3 = file.readline()
print(line_3.rstrip())
line_4 = file.readline()
print(line_4.rstrip())

When we run this version we get

$ python3 read_teams_2.py
Red Sox
Patriots
Celtics
Bruins

Writing and Reading Numbers

Let's say we wanted to create a file of pseudorandom numbers one number to a line

We might write code like this

# creates a series of random numbers and writes them to a file
# this script will not work

import random

filename = input("Filename: ")
how_many = int(input("How many numbers: "))
highest  = int(input("Highest number: "))

file = open(filename, "w")
for i in range(how_many):
    file.write(random.randint(1, highest))

But when we run it we get an error

$ python3 random_numbers_create.py 
Filename: numbers.txt
How many numbers: 10
Highest number: 5
Traceback (most recent call last):
  File "random_numbers_create.py", line 11, in <module>
    file.write(random.randint(1, highest))
TypeError: write() argument must be str, not int

The problem is that we opened the file using the "w" access mode
The three access modes we use in this class
- r Read
- w Write
- a Append
only work with text files
The write method of our file object will only accept text
Because we opened it with an access mode that is used for text files
The integers created by calling randint are not strings
If we wanted to write integers to a file we need access modes that work on binary files
These access modes exist
But we won't be dealing with them in this class
We will only be using text files in this course
So we must convert any numbers we write into strings

Here is a corrected version of the script

# creates a series of random numbers and writes them to a file

import random

filename = input("Filename: ")
how_many = int(input("How many numbers: "))
highest  = int(input("Highest number: "))

file = open(filename, "w")
for i in range(how_many):
    file.write(str(random.randint(1, highest)) + "\n")

We can run this script without getting an error

$ python3 random_numbers_create_2.py 
Filename: numbers.txt
How many numbers: 10
Highest number: 5

And the results are what we expect
```
$ cat numbers.txt 
2
2
4
2
5
4
3
3
1
5
```
When we read in a text file of numbers they will be strings
We will need to use either int or float to convert them back into numbers

Appending Data to an Existing File

To add lines to a file that already exists you use the append access mode
Let's say that I want to keep my own record of Red Sox games

At the beginning of the baseball season I create the file sox_games.txt

$ cat sox_games.txt 
2017-04-04  Red Sox @  Astros   Win 7-5
2017-04-05  Red Sox @  Astros   Win 2-1
2017-04-07  Red Sox vs Yankees  Loss 7-9

I want to create a script to keep this file up-to-date

This means I have to open the file in append mode

# adds the results of a Red Sox game to the file

date      = input("Date: ")
opponent  = input("Opponent: ")
sox_score = input("Sox score: ")
opp_score = input("Opponent score: ")
home_game = input("Was this a home game (y/n): ")

result = "Win"
if int(sox_score) < int(opp_score):
    result = "Loss"
line = date + "\tRed Sox " 
if home_game == "y":
    line += "vs"
else: 
    line += "@"
line += " " + opponent + "\t" + result + " " + sox_score + "-" + opp_score + "\n"

game_file = open("sox_games.txt", "a")
game_file.write(line)

Notice that I had to convert the text entered at the keyboard into integers
I do this since I want to calculate whether the Sox won or lost the game
Strings would be useless for this calculation, so I need integers

Let's run the script and see what happens

$ python3 sox_game_add.py 
Date: 2017-04-08
Opponent: Yankees
Sox score: 12
Opponent score: 1
Was this a home game (y/n): y

$ cat sox_games.txt 
2017-04-04  Red Sox @  Astros   Win 7-5
2017-04-05  Red Sox @  Astros   Win 2-1
2017-04-07  Red Sox vs Yankees  Loss 7-9
2017-04-08  Red Sox vs Yankees  Win 12-1

Using Loops to Process Files

The code above asks the user for the number of lines in the file
This makes the user count the lines in the file
We should never ask the user to do something the computer can do
The information in the file object can tell when we have reached the end of a file
This means we can run through the file using loops
There are two ways to do this
First I will show you a technique using readline and while loops
But it turns out that this method is seldom used
There is a much better way
So why do I show you while loops first?
Because it uses the information in the file object more directly
I want you to get comfortable with the concept of a file object
Then I can show you a better way to process a file

Reading Files with `while` and readline

If you use a loop to process a file, eventually you will reach the end
You can't read past the end of a file
The file object will not let you
The readline method is part of the file object
So readline knows when it reaches the end of a file
It will return a string with nothing in it, "", when it gets to the end of the file
This is called the empty string
The empty string is a perfectly good string
It serves the same purpose for strings as 0 does for integers

With this in mind we can use the following algorithm to read through a file

open the file for reading
call readline to get the first line
while the line is not the empty string:
    process the last line from the file
    read in the next line

I want a loop that will keep going until something happens
This means I need a while loop
I want a loop that keeps going until it reaches the end of the file
That means a loop that keeps going until readline returns the empty string
Notice that I had to call readline twice
Once before the loop
And once inside the loop
I need the first call because line must have a value that I can test in the while loop

Here is a script to read a file and print each line

$ cat while_loop_print_file.py
# reads and prints the lines in a file
# using readline and a while loop

filename = input("Filename: ")
file = open(filename, "r")

line = file.readline()
while line != "":
    print(line.rstrip())
    line = file.readline()

$ python3 while_loop_print_file.py
Filename: teams.txt
Red Sox
Patriots
Celtics
Bruins

Reading Files with a `for` Loop

The most programming languages read a file using a while loop
But Python has a better and simpler way
We can use a for loop

Here is the format for a Python for loop

for LOOP_VARIABLE in LIST_OF_SOME_KIND:
    STATEMENT
    STATEMENT
    ...

Python has objects called lists that we will discuss in a few weeks
But a for loop can use anything which is a collection of things
Remember when I said a text file was really a list of strings with a newline at the end?
So a text file is a collection of lines
How do we refer to this collection in the header of a for loop?
We use a file object

Here is a for that reads a file and prints each line

$ cat for_loop_print_file.py
# reads and prints the lines in a file

filename = input("Filename: ")
file = open(filename, "r")

for line in file:
    print(line.rstrip())

$ python3 for_loop_print_file.py
Filename: teams.txt
Red Sox
Patriots
Celtics
Bruins

Let's compare the two scripts that read a file and print each line

I can do in 2 lines with a forloop

for line in file:
    print(line.rstrip())

What takes me 4 lines with a while loop

line = file.readline()
while line != "":
    print(line.rstrip())
    line = file.readline()

Short and simple is always better than long and complicated
This is why I usually use a for loop when reading files

Today's Topics

Review

New Material

Homework 8

Review

Files

Types of Files

Identifying a File

Storing Data on a Hard Disk

Files and the Kernel

Working with a File in Python

Objects

Accessing the Data in Files

Opening a File

Specifying the Location of a File

Writing Data to a File

Reading Data From a File

New Material

Reading a File Line By Line

Getting Rid of Newline Characters

Writing and Reading Numbers

Appending Data to an Existing File

Using Loops to Process Files

Reading Files with while and readline

Reading Files with a for Loop

Attendance

Reading Files with `while` and readline

Reading Files with a `for` Loop