IT 116: Introduction to Scripting

IT 116: Introduction to Scripting
Class 18

Review

Files
Types of Files
Identifying a File
Storing Data on a Hard Disk
Files and the Kernel
Working with a File in Python
Objects
Accessing the Data in Files
Opening a File
Writing Data to a File
Reading Data From a File

New Material

Reading a File Line By Line
Getting Rid of Newline Characters
Writing and Reading Numbers
Using Loops to Process Files
Reading Files with while and readline
Reading Files with a for Loop
Professional Responsibility
Codes of Conduct

Microphone

Homework 8

I have posted homework 8 here.

It is due this coming Sunday at 11:59 PM.

Quiz 6

I have posted the answers to Quiz 6 here.

Questions

Are there any questions before I begin?

Review

Files

When processing large amount of data we need to use files
A file is simply a linear arrangement of data on some long term storage device
That storage device might be a
- Hard disk
- Flash drive
- CD ROM
- SSD card

Types of Files

All files consist of binary numbers
but those numbers can be interpreted in two ways
- Text
- Binary

Consider the following text file

$ cat fruit.txt 
grapes
pears
oranges
cranberries
apples
melons
blueberries

To see how this file is really stored on disk I can use the Unix od (octal dump) command

$ od -b fruit.txt 
0000000 147 162 141 160 145 163 012 160 145 141 162 163 012 157 162 141
0000020 156 147 145 163 012 143 162 141 156 142 145 162 162 151 145 163
0000040 012 141 160 160 154 145 163 012 155 145 154 157 156 163 012 142
0000060 154 165 145 142 145 162 162 151 145 163 012
0000073

To see the characters, I can run od with the -c option

od -c fruit.txt 
0000000   g   r   a   p   e   s  \n   p   e   a   r   s  \n   o   r   a
0000020   n   g   e   s  \n   c   r   a   n   b   e   r   r   i   e   s
0000040  \n   a   p   p   l   e   s  \n   m   e   l   o   n   s  \n   b
0000060   l   u   e   b   e   r   r   i   e   s  \n
0000073

A binary file, such as a JPEG image also consists of numbers
But the numbers do not represent characters
Text files have a simple structure
Each file is a collection of lines
A line is a series of characters with the newline character, \n, at the end
We will only be dealing with text files in this course

Identifying a File

Every file has two attributes that uniquely identify it
- Name
- Location
The location of a file is given by a path
A path is a list of directories
The last directory in a path is the one that holds the file
A pathname is a path ...
followed by the file name
A pathname gives the name and location of a file
So it uniquely specifies each file

Storing Data on a Hard Disk

Hard disk consists of rapidly spinning disks
The disk is made up of material that can be magnetized
Small regions on the disk are magnetized to point up or down
Each of these regions stands for a binary digit called a bit
The surface of the disk is broken up into rings called tracks
Each track is broken up into sectors
Most files consist of many sectors scattered throughout the disk

Files and the Kernel

At the core of every operating system is a program called the kernel
The kernel handles all interactions with hardware
To read a file, the interpreter must use the kernel
The kernel will then call a special program called a device driver
The device driver handles communication between the hardware and the kernel
Each hardware device attached to the computer must have a device driver
The device driver is created by the manufacturer of the device

Working with a File in Python

A file lives on a long term storage device
Most files contain data
Some files contain the binary code for programs ...
that can work on data files
The program works with a data file in RAM
RAM is fast and expensive
External storage is slower and cheaper
RAM is limited and files can be very large ...
so usually only a small part of the data file is loaded into RAM
The other parts will be loaded as needed
To keep track of this ...
the kernel needs some bookkeeping information
A file object contains the bookkeeping data ...
needed to work with the data file

Objects

A variable holds only one piece of information
Objects can hold many such pieces
The data is contained in variables inside the object
Objects also hold functions that interact with the data
The functions inside an object are called methods
Objects have no name ...
just a location in RAM
We create an object by using a special function called a constructor
The constructor returns a pointer to the object in RAM
We store this value of this pointer in a variable
Then we use the variable to work with the object

Accessing the Data in Files

There are two ways to get data from a file
- Sequential access
- Direct access
Sequential access is like a tape recorder
It starts at the beginning and runs to the end
It can only go in one direction
Direct access goes directly to a particular place in the file
We will only be using sequential access in this class

Opening a File

To work with a file we have to create a file object
We do this using the built-in function open
The open function does two things
- It creates a file object
- It returns a reference to the object
The reference is stored in a variable
The picture in memory looks like this

We then use the variable to work with the object
Here is the format for using the open function
```
FILE_VARIABLE = open(FILENAME, MODE)
```
FILE_VARIABLE is the name of the variable that holds the reference
FILENAME is a string specifying the file
MODE determines what we can do with the file

The three most commonly used access modes are

Mode	Description
"r"	Open a file for reading only
"w"	Open a file for writing
"a"	Open a file for writing to the end of the file

The a in the last mode stands for "append"
Text is added to the end of the file so nothing is lost
A file open for reading cannot be changed
If you try to open a file for reading ...
and the file does not exist ...
you will get an error
If you try to open a file for writing or appending ...
that does not exists ...
the kernel will create the file for you
If you open a file for writing that already exists ...
its contents will be replaced by whatever you write to it

Writing Data to a File

To work with a file in Python, we must first create a file object
This object contains methods that allow us to talk to the file
The functions inside objects are called methods
To call a method you must use dot notation
Dot notation for methods has the following format
```
OBJECT_VARIABLE.METHOD_NAME
```
We first open the file for writing
```
teams_file = open("teams.txt", "w")
```

Now we can send text to the file using write method of this object

$ cat write_teams.py
# writes the names of the teams in the American League East

teams_file = open("teams.txt", "w")

teams_file.write("Boston Red Sox\n")
teams_file.write("Baltimore Orioles\n")
teams_file.write("Toronto Blue Jays\n")
teams_file.write("Tampa Bay Rays\n")
teams_file.write("New York Yankees\n")


$ python3 write_teams.py 

$ cat teams.txt 
Boston Red Sox
Baltimore Orioles
Toronto Blue Jays
Tampa Bay Rays
New York Yankees

Notice that I had to put the newline character, \n ...
at the end of each line
Otherwise all the added text would be on one line

Reading Data From a File

There is more than one method that can read a line
The simplest is read

Which returns a string containing everything in the file

$ cat read_teams.py
# reads the contents of the file teams.txt

teams_file = open("teams.txt", "r")

teams = teams_file.read()

print(teams)

$ python3 read_teams.py 
Boston Red Sox
Baltimore Orioles
Toronto Blue Jays
Tampa Bay Rays
New York Yankees

It turns out that read is not very useful
Normally we want to read a file line by line

Attendance

New Material

Reading a File Line By Line

What if you had a file with numbers, one to each line ...
that you needed to add?
The read method cannot do this
We need to read in one line at a time ...
to get each number ...
and then add it to the running total
One way to do this is to use the readline method of the file object
Remember, objects have functions built inside them called methods ...
that work with the data inside the object
Each time you call readline it returns the next line in the file
By calling readline many times ...
we can read the entire contents of a file

So if wanted to read and print the text file teams.txt created in the last class

$ cat teams.txt 
Boston Red Sox
Baltimore Orioles
Toronto Blue Jays
Tampa Bay Rays
New York Yankees

We can do this calling readline 4 times

$ cat readline_teams.py 
# reads and prints the lines in teams.txt

file = open('teams.txt', 'r')

line_1 = file.readline()
print(line_1)
line_2 = file.readline()
print(line_2)
line_3 = file.readline()
print(line_3)
line_4 = file.readline()
print(line_4)
line_5 = file.readline()
print(line_5)

$ python3 readline_teams.py 
Boston Red Sox

Baltimore Orioles

Toronto Blue Jays

Tampa Bay Rays

New York Yankees

Getting Rid of Newline Characters

Why are there blank lines between the teams in the output above?
That's because readline reads in the entire line ...
including the newline character, \n, at the end
There is a simple fix to this problem using a string method
String have methods because they are objects
I didn't mention this earlier because I didn't want to confuse you
The string method we need is strip() ...
which produces a copy of the string ...
with spaces removed from both ends

Here is an improved version of the script

$ cat readline_teams_2.py 
# reads and prints the lines in teams.txt

file = open('teams.txt', 'r')

line_1 = file.readline()
print(line_1.strip())
line_2 = file.readline()
print(line_2.strip())
line_3 = file.readline()
print(line_3.strip())
line_4 = file.readline()
print(line_4.strip())
line_5 = file.readline()
print(line_5.strip())


$ python3 readline_teams_2.py 
Boston Red Sox
Baltimore Orioles
Toronto Blue Jays
Tampa Bay Rays
New York Yankees

Writing and Reading Numbers

Let's say we wanted to create a file of pseudorandom numbers ...
one number to a line

We might write code like this

# creates a series of random numbers and writes them to a file
# THIS SCRIPT WILL NOT WORK

import random

filename = input("Filename: ")
how_many = int(input("How many numbers: "))
highest  = int(input("Highest number: "))

file = open(filename, "w")
for i in range(how_many):
    file.write(random.randint(1, highest))

But when we run it we get an error

$ python3 random_numbers_create.py 
Filename: numbers.txt
How many numbers: 10
Highest number: 5
Traceback (most recent call last):
  File "random_numbers_create.py", line 11, in <module>
    file.write(random.randint(1, highest))
TypeError: write() argument must be str, not int

The problem lies with the access mode
The three access modes we use in this class
- "r" Read
- "w" Write
- "a" Append
only work with text files
So the write method only accepts text as an argument
The integers created by calling randint are not strings
If we wanted to write integers to a file ...
we would need an access mode that works on binary files
These access modes exist ...
but we won't be using them in this class
We will only be using text files in this course
So we must convert any numbers we write into strings

Here is a corrected version of the script

# creates a series of random numbers and writes them to a file

import random

filename = input("Filename: ")
how_many = int(input("How many numbers: "))
highest  = int(input("Highest number: "))

file = open(filename, "w")
for i in range(how_many):
    file.write(str(random.randint(1, highest)) + "\n")

We can run this script without getting an error

$ python3 random_numbers_create_2.py 
Filename: numbers.txt
How many numbers: 10
Highest number: 5

And the results are what we expect
```
$ cat numbers.txt 
2
2
4
2
5
4
3
3
1
5
```
When we read in a text file of numbers ...
those numbers will be strings
and we need to use either int or float ..
to convert them back into numbers

Using Loops to Process Files

In the program readline_teams_2.py we knew how many times to call readline ...
because we knew how many lines there were in the file
But that situation does not occur often
So we need a better way to proceed ...
and that involves using loops
File objects know when they have reached the end of a file ...
and will not let us go beyond that point
There are two ways to loop through a file
First I will show you a technique using readline ...
and while loops
But this method is seldom used
There is a much better way
So why do I show you while loops first?
Because it uses the information in the file object more directly
I want you to get comfortable with the concept of a file object
Then I can show you a better way to process a file

Reading Files with `while` and readline

Normally when you call readline ...
it returns the next line in the file ...
including the newline character, \n, at the end
But readline can't do this when it reaches the end of the file
It needs some way to tell us that there are no more lines ...
and it does this is a very simple way
When readline gets to the end of a file ...
the next time it is called it will return the empty string
The empty string is a perfectly good string
It serves the same purpose for strings as 0 does for integers

With this in mind we can use the following algorithm

open the file for reading
call readline to get the first line
while the line is not the empty string:
    process the last line from the file
    read in the next line

Notice that I had to call readline twice
Once before the loop ...
and once inside the loop
The while loop needs a line to test ...
before entering the loop
So I need to call readline before the loop

Here is a script to read a file and print each line

$ cat while_loop_print_file.py
# reads and prints the lines in a file
# using readline and a while loop

filename = input("Filename: ")
file = open(filename, "r")

line = file.readline()
while line != "":
    print(line.strip())
    line = file.readline()

$ python3 while_loop_print_file.py
Filename: teams.txt
Boston Red Sox
Baltimore Orioles
Toronto Blue Jays
Tampa Bay Rays

Reading Files with a `for` Loop

Most programming languages read a file using a while loop
But Python has a better and simpler way
We can use a for loop

Here is the format for a Python for loop

for LOOP_VARIABLE in LIST_OF_SOME_KIND:
    STATEMENT
    STATEMENT
    ...

Python has objects called lists that we will discuss in a few weeks
But a for loop can use anything which is a collection of things
A text file is collection of lines
So we can run through it with a for loop
But what do we put in the header of the loop ...
after the keyword in?
We use a file object

Here is a for loop that reads a file and prints each line

$ cat for_loop_print_file.py
# reads and prints the lines in a file

filename = input("Filename: ")
file = open(filename, "r")

for line in file:
    print(line.strip())

$ python3 for_loop_print_file.py
Filename: teams.txt
Red Sox
Patriots
Celtics
Bruins

Let's compare the two scripts that read a file and print each line

I can do in 2 lines with a for loop

for line in file:
    print(line.strip())

What takes me 4 lines with a while loop

line = file.readline()
while line != ":
    print(line.strip())
    line = file.readline()

Short and simple is always better than long and complicated
This is why I usually use a for loop when reading files

Professional Responsibility

Codes of Conduct