IT 116: Introduction to Scripting

IT 116: Introduction to Scripting
Class 26

Today's Topics

Final Exam

The final exam will be given on Thursday, May 21st. It will be on Blackboard. I will use the same format as the Midterm. The exam will be a two parts exam (Questions and writing short segments of Python code).

The last day of our class will be on Tuesday, May 12th, We will have an optional Zoom meeting for any question you may have. I will send you the Zoom information.

Class 28, on Thursday, May 7th, will be a review session.

You will only be responsible for the material in Class Notes 28 and the review for the Mid-term, which you will find here.

The final will be an online exam.

Tips and Examples

A Practical Example of Reading Files into Lists

At the start of each semester I need to create tuples for each IT student containing the following information
- Student ID
- First name
- Last name
- UMB email
I get an Excel file from the Registrar with this information

I export this file as a CSV text file where the field values are separated by commas

Program Status Code,Term Code,Academic Career Code,Person ID,Person Name,First Name,Last Name,Program Code,Plan Code,Student Email Address,Personal Email Address
AC,2610,UGRD,01459837,"Smith,John",John,Smith,CSM-U,IT-BS,John.Smith001@umb.edu,jsmith@gmail.com
AC,2610,UGRD,01636476,"Jacobs,Jane",Jane,Jacobs,CSM-U,IT-BS,Jane.Jacobs002@umb.edu,jjacobs@gmail.com
AC,2610,UGRD,01523746,"Miller,Alan",Alan,Miller,CSM-U,IT-BS,Alan.Miller001@umb.edu,amiller@gmail.com
...

This is not the real file
I can't show that to you because of privacy rules
My code uses split to break up the line into a list
Whenever you use split you have to tell it where to break the string into list elements
The boundaries are determined by a string of one or more characters called a delimiter
If you run split with no argument it uses spaces as the delimiter
But that won't work with this file
Here the delimiter is a comma, ,
To use another delimiter you have to give split an argument
That argument must be a string
In the script I use ',' as the argument
I can then use indexing to get the values I need

Here is the code

#! /usr/bin/python3
# takes a CSV file from the Registrar and turns it into
# tuples of the form
# (STUDENT_ID, FIRST_NAME, LAST_NAME, UMB_EMAIL)

# takes a stripped string, quotes it and adds a comma
def quote_comma(string):
    return '"' + string.strip() + '", '

in_filename = input("Input filename: ")
in_file = open(in_filename, 'r')

tuple = ''
for line in in_file:
    line = line.strip()
    values = line.split(',')
    if 'UGRD' in values[2]: # skip first line with labels
        tuple += '('
        id = values[3]
        first_name = values[6]
        last_name  = values[7]
        umb_email  = values[10]
        tuple += quote_comma(id) + quote_comma(first_name) + quote_comma(last_name) + umb_email  
        tuple += '),\n'
print(tuple)

When I run it I get

("01459837", "John", "Smith", John.Smith001@umb.edu),
("01636476", "Jane", "Jacobs", Jane.Jacobs002@umb.edu),
("01523746", "Alan", "Miller", Alan.Miller001@umb.edu),
...

I can copy this text into a Python tuple for all IT students

Review

List Elements

The elements of a list can be anything

>>> list_1 = [1 , 2.5, True, "foo"]
>>> for element in list_1:
...     print(element, type(element))))
... 
1 <class 'int'>
2.5 <class 'float'>
True <class 'bool'>
foo <class 'str'>

The elements of a list can even be another list

>>> list_2 = [ 1, 2, 3, 4, [5, 6, 7,]]
>>> for element in list_2:
...     print(element, type(element))
... 
1 <class 'int'>
2 <class 'int'>
3 <class 'int'>
4 <class 'int'>
[5, 6, 7] <class 'list'>

Two-Dimensional Lists

When all the elements of a list are lists, we have a two-dimensional list

>>> two_d_list = [[1, 2, 3], [4, 5, 6], [7, 8, 9]] 
>>> two_d_list
[[1, 2, 3], [4, 5, 6], [7, 8, 9]]

Each element of two_d_list is also a list

>>> for element in two_d_list:
...     print(element)
... 
[1, 2, 3]
[4, 5, 6]
[7, 8, 9]

We can use indexing to get each sublist

>>> two_d_list[0]
[1, 2, 3]
>>> two_d_list[1]
[4, 5, 6]
>>> two_d_list[2]
[7, 8, 9]

How do we get each element within the sublist?

With another [ ]

>>> two_d_list[0][0]
1
>>> two_d_list[0][1]
2
>>> two_d_list[0][2]
3

Tuples

A tuple is a sequence of values that cannot be changed
We can create a tuple literal by enclosing comma separated values inside parentheses
```
>>> tuple_1 = (1, 2, 3, 4, 5)
>>> tuple_1 = (1, 2, 3, 4, 5)
(1, 2, 3, 4, 5)
```

Tuples can contain elements of any type

>>> tuple_2 = (1, 2.5, False, "Sam")
>>> tuple_2
(1, 2.5, False, 'Sam')

You can access the elements of a tuple with an index
```
>>> tuple_1[0]
1
```

You can use a for loop to print all the elements

>>> for number in tuple_1:
...     print(number)
... 
1
2
3
4
5

The `tuple` Conversion Function

Every one of the basic data types in Python has a conversion function
- int
- float
- str
- bool
- list
The conversion function for tuples is tuple

It will work on strings

>>> name = 'Glenn'
>>> tuple(name)
('G', 'l', 'e', 'n', 'n')

and lists

>>> digits = [1, 2, 3, 4, 5, 6, 7, 8, 9, 0]
>>>  tuple(digits)
(1, 2, 3, 4, 5, 6, 7, 8, 9, 0)

Why Use Tuples?

Lists and tuples are similar
But tuples have many limitations
So why should we use tuples?
There are two reasons
- Performance
- Security
Tuples are simpler that lists
So a list takes more memory than a tuple of the same size
The smaller size makes them somewhat faster to work with
The savings can be significant when working with big data sets
Consider the data collected by the European Gaia satellite
It measured the position and movement of 1.7 billion stars
You would not want to use lists for that data set
Tuples cannot be changed
So any data you put in a tuple cannot be modified or corrupted
This can be important in team projects where many people work with the same data

New Material

Strings Are Sequences

There are many different data types in Python
Some hold only one value
- Integer
- Float
- Boolean
Other data types hold more than one value
A sequence is a family of data types that contain many different values stored one right after the other
The last chapter in our textbook dealt with two types of sequences
- Lists
- Tuples
Strings are sequences where the elements are individual characters
Strings are objects
And they have methods that work on the characters they contain

Accessing Characters With Indexing

The characters in a string have a definite order, so we can use indexing to get a particular character
If we have a string
```
>>> team = "Red Sox"
```

we can use the [ ] operator to get a specific character
Inside the brackets we can use a integer literal
```
>>> team[0]
'R'
```
We can also use a variable as an index
```
>>> num = 4
>>> team[num]
'S'
```
Or a calculation using operators
```
>>> team[num +1]
'o'
```
Or a function call that returns an integer
```
>>> team[len(team) - 1]
'x'
```
Notice that I had to subtract 1 from the length of the string

If I did not do this I would get an exception

>>> team[len(team)]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: string index out of range

Negative Numbers as Indexes

You can't use any number as an index
It must be an integer

And it cannot be longer than the length of the list minus 1

>>> letters = 'abcde'
>>> len(letters)
5
>>> letters[4]
'e'
>>> letters[5]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
IndexError: string index out of range

You might think that using a negative number as an index would raise and exception
But you would be wrong
```
>>> letters
'abcde'
>>> letters[-1]
'e'
```
A negative index is added to the length of the string when using [ ]
In practices this means that -1 is the index of the last character
-2 is the index of the second from last

and so on

>>> letters[-1]
'e'
>>> letters[-2]
'd'
>>> letters[-3]
'c'
>>> letters[-4]
'b'
>>> letters[-5]
'a'
>>> letters[-len(letters)]
'a'

This feature can be useful
After all it's easier to write
```
letters[-1]
```
than to write
```
letters[len(team) - 1]
```
They are particularly useful with strings of different lengths
You can get the last character in the string using -1 as the index
Negative index values can be used with any sequence

They can be used with lists

>>> numbers = [1, 2, 3, 4, 5]
>>> numbers[-1]
5

and with tuples

>>> values = (1, 2, 3, 4, 5)
>>> values[-1]
5

Using a `for` Loop with Strings

You can also access the individual characters in a string using a for loop
```
>>> for ch in team:
...   print(ch)
... 
R
e
d

S
o
x
```
This allows us to write some very interesting functions that work with strings

We can write a function to count the number of times a certain character occurs in a string

>>> def character_count(string, char):
...    count = 0
...    for ch in string:
...        if ch == char:
...            count += 1
...    return count
... 
>>> character_count("Mississippi", "i")
4
>>> character_count("Mississippi", "s")
4
>>> character_count("Mississippi", "p")
2

We can also use a for loop to reverse a string

>>> def string_reverse(string):
...     new_string = ""
...     for ch in string:
...         new_string =  ch + new_string
...     return new_string
... 
>>> string_reverse('Mississippi')
'ippississiM'
>>> string_reverse('radar')
'radar'

To reverse a string I need to create a new string starting with the empty string
Then I add each character in the original string to the new string
I do this by placing each character before the previous version of the new string

Strings Are Immutable

Why did I write the function string_reverse
Surely I could have used the reverse method of the string object
The graphic below lists the string methods

But you won't find reverse among them
That's because strings cannot be changed
Strings are immutable

You might think that the following code shows that you can change a string

>>> string_1 = "Red"
>>> string_1 = string_1 + " Sox"
>>> string_1
'Red Sox'

But that is not what is happening here
You cannot change a string
But you can use one string to create another
In the first statement above I created the string "Red"
The string variable string_1 points to this new string
Here is how things look in memory

When I wrote
```
string_1 += " Sox"
```
two things are happening
First, I created a new string consisting of the characters in the first string and " Sox"
Second, the object variable string_1 now points to this new string
Nothing points to the old string "Red"
But that string has not gone away
Here is the new memory situation

Garbage Collection

In the example above we have two strings in memory
- "Red"
- "Red Sox"
The string variable string_1 points to the second string
So we can do things with it
But nothing points to the first string
We can't do anything with it and it takes up space

I could created still another new string from the second string

>>> string_1
'Red Sox'
>>> string_1 = "Go " + string_1
>>> string_1
'Go Red Sox'

Now I have three string in memory

But I can only work with the newest one
If I keep doing this I would run out of memory
If I did this in a endlessly in a script you might think it would eventually run out of memory
And crash
If the program were written in C it would crash
This is a type of bug called a "memory leak"
Of course memory cannot leak away
But if you keep creating object you will eventually run out of memory the script can use
This does not happen in Python and most scripting languages
It also does not happen in Java
In these languages the interpreter does something called garbage collection
I works like this
The interpreter keeps a table of all objects in memory
And the variables that point to them
Every so often the interpreter looks at this table and deletes objects that have no variable pointing to them
So programs written in Python will never have a memory leak

String Slicing

One of the most common string operations is finding a smaller string inside a bigger string
A string that is contained inside another string is called a substring
Many computer languages have functions that can be used to extract a substring
But Python gives us an easier way
We can use slices
Slicing works with all sequence objects including strings
So if we have the following string
```
>>> team = "Boston Red Sox"
```
we can use a slice to extract the city substring from the larger string
```
>>> team[0:6]
'Boston'
```
In the above example I had to count the characters of the string to find where the space was
But the index method of a string object can do that work for us
```
>>> team.index(' ')
6
```
We can use this function call in the slice
```
team[0:team.index(' ')]
'Boston'
```
We are starting the slice at the beginning of the string
So we can omit the 0
```
>>> team[:team.index(' ')]
'Boston'
```
We can use this technique to create functions that will split a full name into first names and last names
To do this we will assume that a full name consists of two names

And that the names have no spaces inside them

>>> def first_name(full_name):
...     return full_name[:full_name.index(' ')]
... 
>>> first_name('Glenn Hoffman')
'Glenn'
>>> def last_name(full_name):
...     return full_name[full_name.index(' ') + 1:]
... 
>>> last_name('Glenn Hoffman')
'Hoffman'

Writing Programs Work with Words

The first computers were created to do numerical calculations
But people soon learned that they could also be used with words
Think of the Big Five in Tech today
- Apple
- Google
- Amazon
- Microsoft
- Facebook
Almost all of their products work with words not numbers
Below I will give some examples of programs that work with numbers

String Programs 1 - Turning Numbers into Words

My first example of a program that works with words is somewhat silly
I want to write a program that turns an integer into words
So 512 would become "five one two"
To do this we need a way to turn a digit into its appropriate word
This can be done by creating a list of the names of digits
We can use each digit in the integer as an index to get the correct digit name
But we have to be smart about doing this
If we defined the list like this
```
digits = ["one", "two", "three", ...
```
The index of the word "one" would be 0

So let's start the list with "zero"

>>> digits = ["zero", "one", "two", "three", "four"]
>>> digits += ["five", "six", "seven", "eight", "nine"]

Let's iterate through this list using a for loop

>>> for i in range(len(digits)):
...    print(str(i) + ":\t" + digits[i])
... 
0:  zero
1:  one
2:  two
3:  three
4:  four
5:  five
6:  six
7:  seven
8:  eight
9:  nine

Here is the code

>>> def digits_to_words(number):
...     new_string = ""
...     number_string = str(number)
...     for digit in number_string:
...         digit = int(digit)
...         new_string += digits[digit] + " "
...     return new_string 
... 
>>> digits_to_words(512)
'five one two '
>>> digits_to_words(10428)
'one zero four two eight '

We start with the empty string
Now we need some sort of sequence object to use in a for loop
A string is such an object and we can create one using the str conversion function on the number
The for loop with give us each digit in turn
To use the digit as an index we need to turn it back into an integer using int
The index gives us the name of each digit
And we build the result string by adding a space after each digit name

String Programs 2 - Checking for Anagrams

An anagram is "a word, phrase, or name formed by rearranging the letters of another"
To make this problem easier we will only deal with words, not phrases
Phrases have spaces in them which makes things more complicated
So we need to look at two strings and see if every letter in the first is also in the second
But this is not enough
What if a character appears twice in the first word but only once in the second?
The best idea would to loop through the characters in the first word removing each character from the second word
But there is one simple test we should perform first
For one word to be an anagram of another they need to have the same length

Here is the algorithm

if two words do not have the same length:
    return False
for each character in word 1:
    if that character is not in word 2:
        return False
    else
        remove the character from word 2
return True

Looks good but there is one problem
How do we remove a character from a string when strings are immutable?
The answer is to use the list conversion function
Turning the second word into a list of characters means we can use the list method pop

Here is the code

def is_anagram(word_1, word_2):
    if len(word_1) != len(word_2):
        return False
    word_2 = list(word_2) # turn into list so we can use pop
    for ch1 in word_1:
        if ch1 not in word_2:
            return False
        ch2_index = word_2.index(ch1)
        word_2.pop(ch2_index)
    return True

String Programs 3 - Creating Phony Student IDs

I have created many scripts to automate as much of my work as possible
I use some of these scripts as examples for this class
A lot of these scripts deal with student data
But I can't show you the real files my scripts work on
That would be a violation of student confidentiality
Creating phony names and email addresses is not hard
But I also need phony student IDs
I could make them up, but it's easier for me to run a script that does it for me
A student ID has 8 digits
And the first digit is always 0

So you might think that I would use the following algorithm

set a string to "0"
for seven times:
    add a randomly selected digit to the end of the string

But that is not what I do

I found a simpler approach

print ('0' + str(random.randrange(1000000,9999999)))

randrange will create a randomly number between two other numbers
Actually not quite
The arguments work like range
But close enough for our purposes

Running the script gives

$ ./phony_id_create.py
05494043

$ ./phony_id_create.py
04385371

String Programs 4 - Creating Passwords

Every time you go to a new URL that sells something, you have to create a password
The Mac has a feature that suggests a password and remembers it afterwards
Unfortunately, for some reason this feature does not work on my Mac
I spent a little time investigating the issue
In the end I decided fixing it would take more time than I had available
Besides, I could write a Python program to create my own passwords
I needed to create a list of characters which could be randomly chosen
I decided to use four groups of characters
- Symbols
- Lowercase letters
- Uppercase letters
- Digits
Each group of characters would be stored in a tuple
And these four tuples would be stored inside another tuple
Characters would be chosen in two steps
- Chose a class of characters
- Chose a character from the class
In each step I would use randrange with the length of each list as its argument
This would randomly select a valid index for the chosen tuple

Here is the script

PWD_LENGTH = 20
CHARACTERS = (
    ('!','@', '#', '$', '%', '^', '&', '*'),
    ('a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'j', 'k', 'm', 'n', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z'),
    ('A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'J', 'K', 'M', 'N', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z'),
    ('1', '2', '3', '4', '5', '6', '7', '8', '9', '0')
    )

def random_char():
    tuple_index = random.randrange(4)
    tuple       = CHARACTERS[tuple_index]
    return  tuple[random.randrange(len(tuple))]

passwd = ''
for i in range(PWD_LENGTH):
    passwd += random_char()

print(passwd)

Here's what I get when I run the script

$ ./password_create.py
d*Y&tn%Zk$75G29k*J!N