#iPython Notebook HowTo's

##Graphs
x = numpy.arange(0,2 * math.pi, 0.1)

f1=map(lambda x: 5 * math.sin(x), x)

f2=map(lambda x: 2 * math.sin(2*x), x)

f3=map(lambda x: 1 * math.sin(3*x), x)

pylab.plot(x,f1)

pylab.plot(x,f2,'ro')

pylab.plot(x,f3,'g--')

pylab.show()

##LaTeX in Markdown:
$e^{i \omega t}$

e^{i \omega t}

##Ideas:
General Ideas:
https://github.com/ipython/ipython/tree/master/examples/notebooks#a-collection-of-notebooks-for-using-ipython-effectively

Cython integration: http://nbviewer.ipython.org/urls/raw.github.com/ipython/ipython/3607712653c66d63e0d7f13f073bde8c0f209ba8/docs/examples/notebooks/cython_extension.ipynb

##iPython Display Modifications:

```
#Enable LaTeX printing
from IPython.core.display import *

global MathJax
MathJax = True
def MDPL(string): display(Math(string)) if MathJax else display(Latex(string))

import IPython.core.display
IPython.core.display.HTML("""

div.input {
width: 105ex; /* about 80 chars + buffer */
}

div.text_cell {
width: 105ex /* instead of 100%, */
}

div.text_cell_render {
/*font-family: "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;*/
font-family: "Charis SIL", serif; /* Make non-code text serif. */
line-height: 145%; /* added for some line spacing of text. */
width: 105ex; /* instead of 'inherit' for shorter lines */
}

/* Set the size of the headers */
div.text_cell_render h1 {
font-size: 18pt;
}

div.text_cell_render h2 {
font-size: 14pt;
}

.CodeMirror {
 font-family: Consolas, monospace;
 }

""")
```

div.input {
width: 105ex; /* about 80 chars + buffer */
}

div.text_cell {
width: 105ex /* instead of 100%, */
}

div.text_cell_render {
/*font-family: "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;*/
font-family: "Charis SIL", serif; /* Make non-code text serif. */
line-height: 145%; /* added for some line spacing of text. */
width: 105ex; /* instead of 'inherit' for shorter lines */
}

/* Set the size of the headers */
div.text_cell_render h1 {
font-size: 18pt;
}

div.text_cell_render h2 {
font-size: 14pt;
}

.CodeMirror {
 font-family: Consolas, monospace;
 }

#Cryptography
Notes and code derived from CSCI 466 Cryptography

School: Liberty University

Professor: Mark Shaneck

```
#Basic Encryption/Decryption Algorithms
#Code Block
from binascii import *        #Hex conversion
from collections import deque #Queue
from collections import defaultdict as dd #Dynamic Dictionary
from string import Template   #Output formatting
from heapq import heappush, heappop #Minheap
from functools import reduce as reduce
import re

def type2(a):
    try:
        b=type(a)
    except:
        b='NULL'
    return b

def remDups(seq):
    seen = set()
    seen_add = seen.add
    return reduce(lambda x,y:x+y,[ x for x in seq if x not in seen and not seen_add(x)])

class Cipher:
    def __init__(self):
        pass
    def enc(self,s):
        print("No Encryption Algorithm Added")
    def dec(self,s):
        print("No Decryption Algorithm Added")

class MAS(Cipher):
    """Monoalphabetic (Affine) Substitution Cipher"""
    def __init__(self,a,b):
        self.a=a
        self.b=b
        self.an=Euclid(a,26,retA=1)[1]
    def enc(self,s):
        return reduce(lambda x,y:x+y,[chr(((ord(p)-97)*self.a+self.b)%26+97) if (ord(p)>=97 and ord(p)<=122) else p for p in s])
    def dec(self,s):
        return reduce(lambda x,y:x+y,[chr((((ord(p)-97)-self.b)*self.an)%26+97) if (ord(p)>=97 and ord(p)<=122) else p for p in s])

class PF(Cipher):
    """Playfair Cipher"""
    def __init__(self,s):
        Temp=remDups(s.replace('q','')+'abcdefghijklmnoprstuvwxyz')
        Temp2=[]
        for i in range(5):
            Temp2=Temp2+[Temp[i*5:(i+1)*5]]
        self.Table=Temp2
    def printTable(self):
        for i in self.Table:
            print ('{0}'.format(str(i)))

```

#Necessary Number Theory
Many concepts in Crypto are built on concept from Number Theory

Many Crypto analyses are rooted in probability and statistics

Let's take a moment and go over the fundamentals that we need for the class

##Groups

Group $G$

binary operator $*$: Mapping from $G \times G \mapsto G$ (closed under $*$)

Group named $(G,*)$

Mapping has properties

>Associative: $\forall a,b,c \in G a*(b*c) = (a*b)*c$

>Identity: $\exists e \in G$ such that $\forall a \in G, a*e = e*a = a$

>Inverse: $\forall a \in G \exists a^{-1} \in G $ s.t. $ a*a^{-1} = a^{-1}*a = e$

Abelian if commutative

>Commutative: $\forall a,b \in G $  $ a*b=b*a$

Finite group if $G$ has finite number of elements

$|G|$ is the order of $G$

Exponentiation: repeated application of operation

>$a^0=e$

>$a^1=a$

>$a^2=a*a$

>$a^n=a*a* \dots *a$ ($n$ times)

>$a^{-n}=(a^{-1})^n$

Cyclic if $\exists a \in G $  s.t. $  \forall b \in G $  $  \exists k\in \mathbb{Z} $  s.t. $  a^k=b$

$a$ is called a generator

##Rings

Set with two operations: $(R,+,*)$ s.t.

>$(R,+)$ is an abelian group (Identity $0$ and Inverse $-a$)

>$*$ is closed under $R$ (i.e. $R \times R \mapsto R$)

>$*$ is associative

>$*$ distributes over $+$

Commutative Ring

>$*$ is commutative

Integral Domain

>$\exists 1 \in R $~ s.t. ~$ \forall a \in R $~~$ a1=1a=a$ (Multiplicative Identity)

>No zero divisors: If $a,r \in R$ and $ab=0$ then either $a=0$ or $b=0$

##Fields

$(F,+,*)$ is a field if

$F$ is an integral domain

$\forall a \in F, $  $  a \neq 0, $~~$ \exists a^{-1} \in F $   s.t.  $ a * a^{-1} = a^{-1}*a=1$ (i.e., Multiplicative Inverse)

##Modular Arithmetic

We can do $+, * $ and modulo reduce the answer ($\mod n$)

Wraps around

Can reduce at any point (as long as reduction also takes place at end)

$\mod n$ defines a mapping from $\mathbb{Z}$ to $\{0,1,\dots,n-1\} = \mathbb{Z}_n$

Why is $\mathbb{Z}_n$ not a field?  Zero divisors exist.

e.g. In $\mathbb{Z}_6$,    $4*3 = 0 \mod 6$

##Divisibility

$a,b \in \mathbb{Z}, $  $ a \neq 0$

$a|b $ if $ \exists k \in \mathbb{Z} $  s.t.  $ b=ak$

Properties

>$\forall a \neq 0, $  $ a|0$ and $a|a$

>$1|b $  $ \forall b$

>If $a|b$ and $b|c$ then $a|c$

>If $a|b$ and $a|c$ then $a|(sb+tc)$   $ \forall s,t \in \mathbb{Z} $

##Prime Numbers

Number divisible only by $1$ and itself

Not prime - composite

How many primes?  Infinite!

>Assume otherwise, the $\exists$ largest prime $p$ (well ordered)

>Consider integer $Q=(2*3*5*\dots*p)+1$

>$Q$ cannot be prime (bigger than largest prime $p$)

>$Q$ must be composite - i.e. has prime factors

>But $2, 3, 5, \dots, p$ cannot divide $Q$ (always remainder of $1$

>Thus, $\exists $ prime $q > p$   s.t.   $q|Q$.  Contradiction!

##Prime Number Theorem

$\pi(x) $ is the number of primes $<x$.  $\pi(x) \approx \frac{x}{\ln x}$

Estimate the number of primes, i.e. every $\ln x$ numbers is a prime

Sometimes closer

>Twin Prime Conjecture: There are an infinite number of twin primes

>i.e. $x$ and $x+2$ are both prime

Arbitrarily large gaps

>Given $k$, find $k$ consecutive compositie numbers

>Consider $(k+1)!+2, (k+1)!+3, \dots, (k+1)!+k+1$

##Greatest Common Divisor

Largest positive integer that divides both $a$ and $b$ is $\gcd(a,b)$

$d = \gcd(a,b) $ if

>$d|a$ and $d|b$

>$\forall c, $  $ c|a $ and $c|b \Rightarrow c|d$

If $\gcd(a,b)=1$ then $a$ and $b$ are relatively prime (coprime).

Examples:

>$\gcd(6,4)$

>$\gcd(5,7)$

>$\gcd(24,60)$

###How To Find gcd?

Brute Force: get prime factorization, take all common factors

e.g. $576=2^63^2$, ~~ and ~~$135=3^35$, ~ $\gcd(576,135)=3^2=9$

Problem: factor $1024$ bit numbers.

Fortunately, Euclid was pretty smart

Observed that $\gcd(a,b) = \gcd(b, a\mod b)$

###Proof: $\gcd(a,b) = \gcd(b, a\mod b)$

Let $d=\gcd(a,b)$

By def, $d|a$ and $d|b$

Also, $a=kb+r = r \mod b$ and $a\mod b = r$ for $k,r \in \mathbb{Z}$

Thus, $a \mod b = a-kb$

Since, $d|b$, then $d|kb$.  Also, $d|a$.

Thus, $d|(a \mod b)$

This shows that $d$ is a common divisor of $b$ and $a\mod b$.

How to show its the largest?

If $e|b$ and $e|(a\mod b)$ then $e|kb$ and thus $e|(kb + a \mod b) = a$ so $e|a$.

>This means that the set of all common divisors of $a$ and $b$ is equal to the set of all

>divisors of $b$ and $a\mod b$.  The greatest of one is the same as the greatest of the other.

###How Does That Help?

Euclidean Algorithm - repeatedly apply until $b=0$ ($a$ will be the gcd)

$\gcd(7,5) = \gcd(5,2) = \gcd(2,1) = \gcd(1,0) = 1$

$\gcd(1970,1066) = \gcd(1066,904) = \gcd(904,162) = \gcd(162,94)$

$  = \gcd(94,68) = \gcd(68,26) = \gcd(26,16) = \gcd(16,10) = \gcd(10,6)$

$  = \gcd(6,4) = \gcd(4,2) = \gcd(2,0) = 2$

Also, $\gcd(a,b)=d$ means that $d$ is the smallest integer that can be written in the form $d=ax+by$

How to find $x$ and $y$?

##Extended Euclidean Algorithm

$(A1,A2,A3) = (1, 0, a)$

$(B1, B2, B3) = (0,1,b)$

>If $B3=0$, $\gcd(a,b)=A3$

>If $B3=1$, $\gcd(a,b)=1$

>$Q=A3/B3$

>$(T1,T2,T3) = (A1-QB1, A2-QB2, A3-QB3)$

>$(A1,A2,A3) = (B1,B2,B3)$

>$(B1,B2,B3)=(T1,T2,T3)$

>GOTO Step $1$

At any time, $aB1+bB2 = B3$

##Extended Euclidean Algorithm Running Example

```
#(Extended) Euclid Algorithm
#Code Block
#Derived from Algorithm by Robert W. Sebesta
def Euclid(a,b,ans=0,pt=0,retA=1):
    """Finds the greatest common divisor of a and b and x,y
    such that a*x+b*y=gcd(a,b)
    Three flags vary the output (Any number may be true):
    ans: If true will print the equation a*x+b*y=gcd(a,b)
    pt: If true will show the table method for finding the solution
    retA: If true will return the list [a,x,b,y,gcd(a,b)]
    """
    Temp=_Euc([1,0,a],[0,1,b],[[['A1','A2','A3'],['B1','B2','B3']],[[1,0,a],[0,1,b]]])
    T=Temp[0]
    if ans:
        print ('\n({0})({1})+({2})({3})={4}'.format(a,T[0],b,T[1],T[2]))
    if pt:
        for x in Temp[1]: print ('{0}\t{1}\t{2}\t{3}\t{4}\t{5}'.format(x[0][0],x[0][1],x[0][2],x[1][0],x[1][1],x[1][2]))
        print()
    if retA:
        return [a,T[0],b,T[1],T[2]]
def _Euc(a,b,s):
    """Main recursive loop for Extended Euclidean Algorithm"""
    if b[2]==0: return [b[0],b[1],a[2]],s
    if b[2]==1: return b,s
    else:
        Q=a[2]//b[2]
        t=[a[n]-Q*b[n] for n in range(3)]
        a=b
        b=t
        return _Euc(a,b,s+[[a,b]])
def MI(a,b):
    """Returns the multiplicative inverse of a mod b"""
    temp = Euclid(a,b,ans=0,pt=0,retA=1)[1]
    if temp<0:temp+=b
    return temp
```

```
#Extended Euclidean Algorithm
#Test Block
Euclid(139,91,ans=1,pt=1,retA=1)
#Euclid(4,9,pt=1)
#Euclid(26,3)
#Euclid(1000,800,pt=1)
#Euclid(5423,76357,pt=1)
```

    (139)(-36)+(91)(55)=1
    A1	A2	A3	B1	B2	B3
    1	0	139	0	1	91
    0	1	91	1	-1	48
    1	-1	48	-1	2	43
    -1	2	43	2	-3	5
    2	-3	5	-17	26	3
    -17	26	3	19	-29	2
    19	-29	2	-36	55	1

    [139, -36, 91, 55, 1]

##Multiplicative Inverses
$ax+by=\gcd(a,b)$

If $\gcd(a,b)=1$ then $ax+by=1$

$ax+by = ax \mod b = 1$

Thus, $x = a^{-1} \mod b$

Only elements $a$ in $\mathbb{Z}_n$ where $\gcd(a,n)=1$ have multiplicative inverses $\mod n$.

##Back To Primes
If $p$ is prime, and $p|ab$ then $p|a$ or $p|b$ (or both).

Proof:

Suppose $p$ prime and $p|ab$.

Either $p|a$ or $p \nmid a$.

If $p \nmid a$, then $\gcd(a,p)=1$

>Why?  Only divisors of $p$ are $p$ and $1$

If $\gcd(a,p)=1$   $\exists x,y$ ~s.t.  $ax+py=1 \Rightarrow bax+bpy=b$

So $p|bpy$ (obviously) and $p|bax$

>Why? Since $p|ab$

So $p|b$

##Fundamental Theorem of Arithmetic

Every positive integer is a unique product of primes

Proof:

Suppose otherwise, that $n$ has two different factorizations

$n = p_1p_2\dots p_r = q_1q_2\dots q_s$

Divide out common factors so prime appears on both sides

But $p_1|n \Rightarrow p_1|q_1q_2\dots q_s \Rightarrow p_1|q_i$ for at least one $i$

Thus $p_1=q_i$.  Contradiction.

##Congruences

$a,b,n \in \mathbb{Z}$ with $n\neq 0$

$a \equiv b \mod n$ if $a-b$ is a multiple of $n$ (or $n|(a-b)$ )

Properties:

>$a \equiv 0 \mod n \Leftrightarrow n|a$

>$a \equiv a \mod n$

>$a \equiv b \mod n \Leftrightarrow b \equiv a \mod n$

>$a \equiv b \mod n$ and $ b\equiv c \mod n \Rightarrow a \equiv c \mod n$

>$a,b,c,d,n \in \mathbb{Z}$, $n\neq 0$, $a\equiv b \mod n$ and $c \equiv d \mod n$, then

>>$a+c \equiv b+d \mod n$

>>$a-c \equiv b-d \mod n$

>>i.e. arithmetic is well defined in $\mathbb{Z}_n$

###Division $\mod n$

$a,b,c,n \in \mathbb{Z}$, $n\neq 0$, and $\gcd(a,n)=1$

$ab \equiv ac \mod n \Rightarrow b \equiv c \mod n$

>Since $\gcd(a,n)=1$, $\exists x,y$ ~s.t.~ $ax+ny=1$

>Thus $(ab-ac)x + n(b-c)y = b-c$

>Since left side is multiple of $n$ then $b-c$ is a multiple of $n$

>Thus $b\equiv c \mod n$

I.e., we can divide by $a$ if $a$ has an inverse.

###What If?

What if $\gcd(a,n) > 1$, i.e. $\gcd(a,n)=d$

Solve $ax\equiv b \mod n$

>If $d\nmid b$, there is no answer.

>>e.g. $2x\equiv 5 \mod 6$

>If $d|b$

>>Consider $\frac{a}{d}x_0 \equiv \frac{b}{d} \mod \frac{n}{d}$

>>Note $\gcd(\frac{a}{d}, \frac{n}{d})=1$

>>Solve for $x_0$

>>Original solutions are $x_0, x_0+\frac{n}{d}, x_0+\frac{2n}{d}, \dots, x+0+\frac{(d-1)n}{d}\mod n$

>>e.g. Solve $12x \equiv 21 \mod 39$ ~~ $\gcd(12,39)=3 \Rightarrow 3$ solutions

###Solving $x^2\equiv a \mod n$

What about $x^2\equiv 1 \mod 7$?

Answers? $x=1,6$, (or $x=\pm 1 \mod 7$)

Exactly $2$ solutions when $n$ is odd prime

What about composite? Can have more

e.g. $x^2\equiv 1 \mod 8$?  Try all.  Hint: $4$ solutions.

Before we can compute, some extra techniques are needed.

##Chinese Remainder Theorem
Figured out by Sun Tzuin 3$^{rd}$ century (Not ``Art of War'' guy).

Main purpose: speed up modular computations

Reverse of following: let $x\equiv 25 \mod 42$

Since $42=6*7$ we get two equations:

>$x\equiv 25 \mod 7 \equiv 4 \mod 7$

>$x \equiv 25 \mod 6 \equiv 1 \mod 6$

So we now have a system of modular equations

CRT lets us solve for $x \mod 6*7$

Let $M = m_1m_2\dots m_k$ where all $m_i$ are pairwise coprime

(i.e. $\forall i,j $,~~$ 1\leq i,j \leq k$, ~~$i\neq j$, ~~ $\gcd(m_i,m_j)=1$)

If $x\equiv a_1 \mod m_1$, $x \equiv a_2 \mod m_2$, $\dots$, $x\equiv a_k \mod m_k$

then there is a unique solution for $x \mod M$.

Proof:

>Consider $\frac{M}{m_j}$.  $\gcd(\frac{M}{m_j},m_j)=1$ (Why?)

>Thus, $\exists (\frac{M}{m_j})^{-1} \mod m_j$.  Call it $c_j$.

>Thus we have $(\frac{M}{m_j})(c_j) \equiv 1 \mod m_j$ and $(\frac{M}{m_j})(c_j) \equiv 0 \mod m_i (i\neq j)$ (Why?)

>Consider $x_0 = \sum^{k}_{j=1}(\frac{M}{m_j})(c_j)(a_j)$.

>It solves all the conguences:

>$x_0 \equiv \frac{M}{m_i}c_ia_i \equiv a_i \mod m_i$

>Thus $x_0$ is a solution - Is it unique?

>Assume another solution exists: $x_1$

>$x_0 \equiv x_1 \mod m_i $~~$ \forall i$ thus $m_i|(x_0-x_1) \forall i$

>Since all $m_i$ are pairwise coprime:

>$\pi(m_i)|(x_0-x_1) \Rightarrow M|(x_0-x_1) \Rightarrow x_0 \equiv x_1 \mod M$

>So yes, unique $\mod M$.

##Chinese Remainder Theorem Running Example:

```
#Chinese Remainder Theorum
#Code Block
pprint=MDPL
def CRT(A,ptable=0):
    """A is a list of lists such that [[a1,a2],[b1,b2],...] is equivalent to
    x = a1 mod a2, x = b1 mod b2,... where a,b,... are integers
    The flag ptable if true will output the work to find the solution"""
    M=reduce(lambda x,y:x*y,[i[1] for i in A])
    if ptable: pprint( 'M={0}'.format(M))
    count=1;Ans=[];outS='';sumS=0
    for i in A:
        Ans=Ans+[[count,i[0],i[1],M//i[1],Euclid(M//i[1],i[1],retA=1)[1]%i[1]]]
        #print 'a({0})={1} m({0})={2} M/M({0})={3} c({0})={4}'.format(Ans[count-1][0],Ans[count-1][1],Ans[count-1][2],Ans[count-1][3],Ans[count-1][4])
        if ptable: pprint('a_{{{0}}}={1},\ m_{{{0}}}={2},\ \\frac{{M}}{{M_{{{0}}}}}={3},\ c_{{{0}}}={4}'.format(Ans[count-1][0],Ans[count-1][1],Ans[count-1][2],Ans[count-1][3],Ans[count-1][4]))
        outS=outS+'({0}*{1}*{2})+'.format(Ans[count-1][3],Ans[count-1][4],Ans[count-1][1])
        sumS=sumS+(Ans[count-1][3]*Ans[count-1][4]*Ans[count-1][1])
        count=count+1
    if ptable:
        pprint( outS[:-1] + ' = {0}'.format(sumS) )
        pprint( '= {0}\ mod\ {1}'.format(sumS%M,M) )
    return sumS%M
#ChRem([[5,7],[7,11],[3,13]]);print
CRT([[5,8],[3,5]],ptable=1);print()
#ChRem([[7,11],[9,7],[2,3],[1,2]])
```

$$M=40$$

$$a_{1}=5,\ m_{1}=8,\ \frac{M}{M_{1}}=5,\ c_{1}=5$$

$$a_{2}=3,\ m_{2}=5,\ \frac{M}{M_{2}}=8,\ c_{2}=2$$

$$(5*5*5)+(8*2*3) = 173$$

$$= 13\ mod\ 40$$

##So What's The Point?
We can do arithmetic on smaller numbers then recombine later using CRT to get full answer.

Consider: $973 \mod 1813$ where $1813 = 37*49$.

$m_1 = 37$    $ a_1=11\mod 37$

$m_2 = 49$    $ a_2=42\mod 49$

Let's write $973$ as $(11,42)$

Similarly, $678 = (12,41)$

To add $973 + 678$, instead add $(12+11 \mod 37, 41+42 \mod 49) = (23,34)$

Use CRT to recombine:

>$m_1=37$~~$a_1=23$~~$\frac{M}{m_1}=49$~~$49^{-1}\mod 37 = 34$

>$m_2=49$~~$a_2=34$~~$\frac{M}{m_2}=37$~~$37^{-1}\mod 49 = 4 $

>$x=(49*34*23)+(37*4*34) = 43350 \equiv 1651 \mod 1813$

Check: $678+973 = 1651 \mod 1813$

Turns out, operations $\mod m_1,m_2$ are much more efficient (more noticable when $M$ is $1024$ bits or more).

#Modular Exponentiation

Point $1$: modular reduction can be distributed

So $x^{1024}\mod 5$ never needs to be fully exponentiated and then reduced $\mod 5$.  All operations need digits $1-4$.

Point $2$: how to compute $x^{16}$?

Option $1$: $\underbrace{x*x*\dots*x}_{16 ~\text{times}}$?

Option $2$: $(((x^2)^2)^2)^2$?  Much fewer computations this way.

This is known as Fast Exponentiation or Square and Multiply.

$k=10 ~ a=5 ~ a^k = 5^{10} = 5^85^2 = 5^{2^3}5^2$

$10$ in binary is $1010$.

##Fast Modular Exponentiation Algorithm ($s=a^k$)

Start with $s=1$.

Step through exponent $k$ bit by bit (MSB to LSB).

Always square.

If exponent bit is $1$, multiply by $a$.

Example: $5^{10} = 5^{1010_2}$

>$s=1$

>square ($1$).  bit is $1$, $s = s*5 = 5$

>square ($5^2$).  bit is $0$.

>square ($5^{2^2}$).  bit is $1$.  $s = (5^{2^2})5$

>square ($(5^{2^3})5^2$).  bit is $0$.

##$\mathbb{Z}_m^*$

$\mathbb{Z}_m^*$ is the set of all integers $k$, $0 < k < m$ s.t. $\gcd(k,m)=1$

>E.g. $\mathbb{Z}_6^* = \{1, 5\}$

>E.g. $\mathbb{Z}_7^* = \{1, 2, 3, 4, 5,6\}$

>E.g. $\mathbb{Z}_{10}^* = \{1, 3, 7, 9\}$

#Fermat's Little Theorem

Now that we have some Number Theory under our belts, let's look at the meat.

>If $p$ is prime and $\gcd(a,p)=1$ then

>>a^{(p-1)} \equiv 1\mod p

>Proof:

>>Consider $\mathbb{Z}_p^* = \{1,2,\dots,p-1\}$

>>Multiply each element in $\mathbb{Z}_p^*$ by $a\mod p$ to get $S=\{a\mod p, 2a\mod p, \dots, a(p-1)\mod p\}$

>>No element in $S$ is $0$. (Why?)

>>Now check if any two elements in $S$ are the same (Is there $x,y \in S$, s.t. $x=y$?)

>>If $x=y$ then $aj\equiv ak \mod p$ for some $j,k$.  Since $\gcd(a,p)=1$, then $a^{-1} \mod p$ exists, thus $j\equiv k \mod p$.

>>Thus, $S$ has all distinct elements with no $0$.

>>Thus, $S$ is a permutation of $\mathbb{Z}_p^*$.

>>Thus,

>>> \prod_{x \in S} x = \prod_{y \in \mathbb{Z}_p^*} y

>>$a * 2a * \dots * a(p-1) \equiv 1*2*\dots*(p-1) \mod p$

>>$a^{(p-1)}(p-1)! \equiv (p-1)! \mod p$

>>Since $p$ is prime, $(p-1)!^{-1}$ exists, so $a^{p-1}\equiv 1\mod p$

#Euler's Totient Function $\phi(n)$
$\phi(n) = |\mathbb{Z}_n^*|$

Properties

>If $p$ is prime, $\phi(p)=p-1$.

>$\phi(n)$ is multiplicative.  if $\gcd(m,n)=1$ then $\phi(mn)=\phi(m)\phi(n)$.

>If $n=p^k$, then $\phi(n) = (1-\frac{1}{p})p^k = p^k-p^{k-1}$ (i.e. remove all multiples of $p$)

>General case: If $n=p_1^{e_1}p_2^{e_2}\dots p_k^{e_k}$ then $\phi(n) = n(1-\frac{1}{p_1})(1-\frac{1}{p_2})\dots (1-\frac{1}{p_k})$

Key point to remember for later - to compute $\phi(n)$ you need the prime factors of $n$.

##Euler's Theorem

If $\gcd(a,n)=1$, then $a^{\phi(n)}\equiv 1 \mod n$

Proof:

>$\mathbb{Z}_n^* = \{x_1,x_2,\dots,x_{\phi(n)}\}$

>Multiply by $a$ to get $S=\{ax_1,ax_2\dots,ax_{\phi(n)}\}$

>$S$ is a permutation of $\mathbb{Z}_n^*$. Why?

>>$\gcd(a,n)=1$ and $\gcd(x_i,n)=1$, so $\gcd(ax_i,n)=1$.  So $ax_i \mod n \in \mathbb{Z}_n^*$

>>If $ax_i\equiv ax_j \mod n$, since $\gcd(a,n)=1$ then $x_i\equiv x_j \mod n$

So

>>>\prod_{x\in S}x = \prod_{y\in \mathbb{Z}_n^*} y

>$ax_1 * ax_2 * \dots * ax_{\phi(n)} \equiv x_1*x_2*\dots *x_{\phi(n)}$

>$a^{\phi(n)}*x_1*x_2*\dots * x_{\phi(n)} \equiv x_1*x_2*\dots * x_{\phi(n)} \mod n$

>$a^{\phi(n)} \equiv 1 \mod n$

##Now Compute The Last $3$ Digits Of $7^{803}$

Last $3$ digits $\Rightarrow \mod 1000$

$\phi(1000) = 1000(1-\frac{1}{2})(1-\frac{1}{5}) = 400$

$7^{803} = 7^{800}7^3 = (7^{400})^27^3 \equiv 1^27^3 \equiv 343 \mod 1000$

In general, if $\gcd(a,n)=1$ then $a^k \mod n \equiv a^{k \mod \phi(n)}\mod n$

Compute: $2^{43210} \mod 101$

#Square Roots $\mod n$

How to find?

e.g. $\sqrt{6} \mod 10 = 4,6 $

Let's start with roots $\mod p$.

In particular, let's look at roots $\mod p$ where $p\equiv 3 \mod 4$ (turns out $p\equiv 1 \mod p$ is a lot harder)

Proposition: Let $p\equiv 3 \mod 4$ be prime, $y\in \mathbb{Z}$.  Let $x\equiv y^{\frac{p+1}{4}}\mod p$.  Then,

>If $y$ has a square root $\mod p$, they are $\pm x$

>If $y$ has no square roots  $\mod p$, then $-y$ does and they are $\pm x$

Proof:

>Assume $y \not\equiv 0 \mod p$.  Fermat says $y^{p-1}\equiv 1 \mod p$.

>Thus, $x^4 \equiv y^{p+1} \equiv y^2y^{p-1} \equiv y^2 \mod p$

>This implies that $(x^2+y)(x^2-y)\equiv 0 \mod p$

>So $x^2 \equiv \pm y\mod p$

>Can $x$ equal both?  Can both $+y$ and $-y$ be squares?

>If yes (and $y=a^2$, $-y=b^2$), then $\frac{y}{-y} \equiv -1 \equiv (\frac{a}{b})^2 \mod p \Rightarrow -1 $ is a square $\mod p$

>This can't be if $p\equiv 3 \mod 4$ (Reason on next slide)

>Only $y$ or $-y$ can be a square.  Then $y=x^2 \Rightarrow \sqrt{y}=\pm x$ or $-y=x \Rightarrow \sqrt{-y}=\pm x$

Why can't $-1$ be a square $\mod p$ when $p \equiv 3 \mod 4$?

Suppose it is: $x^2\equiv -1 \mod p$.

Consider $(x^{\frac{p-1}{2}})^2 \equiv (-1)^{\frac{p-1}{2}} \mod p$

$x^{p-1} \equiv (-1)^{\frac{p-1}{2}} \mod p$

$x^{p-1} \equiv 1 \mod p$ by Fermat

So can $(-1)^{\frac{p-1}{2}}  \equiv 1 \mod p$?

$\frac{p-1}{2}$ is odd, since $p\equiv 3 \mod 4$

$p \equiv 3 \mod 4 \Rightarrow p=4k+3 \Rightarrow \frac{(4k+3)-1}{2} = 2k+1$ which is odd

So how to find square roots?

Raise to the $\frac{p+1}{4}$ power.

e.g. $\sqrt{5} \mod 11$ (since $11 \equiv 3 \mod 4$)

$5^{\frac{11+1}{4}} = 5^3 \equiv 4 \mod 11$

Check: $4^2 \equiv 16 \equiv 5 \mod 11$

Check: $(-4)^2 \equiv 7^2 \equiv 49 \equiv 5 \mod 11$

Find $\sqrt{2} \mod 11$

What about square roots mod composite $n$?

If prime factors are $\equiv 3 \mod 4$, comptue roots mod each prime factor, recombine using CRT.

$x^2 \equiv 71 \mod 77=7*11$

So, $x^2 \equiv 71 \equiv 1 \mod 7$ and $x^2 \equiv 71 \equiv 5 mod 11$

From last slide: $x\equiv \pm 1 \mod 7$ and $x\equiv \pm 4 \mod 11$

CRT lets us recombine in $4$ ways to get $4$ roots.

Doing so gives us: $\pm 15, \pm 29$ as roots of $71 \mod 77$

So, knowing factors lets us take roots.  Does knowing roots let us factor?

##Roots $\Rightarrow$ Factoring

Let $x^2 \equiv y \mod n$ where $n=pq$, $p,q \equiv 3 \mod 4$

$x=\pm a, \pm b$

By last slide's construction,  $a\equiv b \mod p$ and $a \equiv -b\mod q$

Thus, $p|(a-b)$ but $q \not| (a-b)$

Thus, $\gcd(a-b,n) = p$ which is a nontrivial factor of $n$

In other words, factoring is computationally equivalent to taking square roots $\mod n$

#Primality Tests
They become rare as numbers get bigger
##Trial Division
Requires $\sqrt{n}$ test divisions

For $1024$ bit prime, requires $2^{512}$ tests...
##Fermat
Remember: $\gcd(a,p)=1 \Rightarrow a^{p-1}\equiv 1 \mod p$

Test it for a bunch of $a$ values

If any are not equal to $1$, output COMPOSITE

If all are equal to $1$, output (PROBABLY) PRIME
##Carmichael
Carmichael number  pass the Fermat test, but are composite.

They become rare as numbers get bigger
##Miller Rabin
Very efficient, popular, effective

Gives positive proof of composite

Gives (adjustable) probability of primality

Based on fact that in $\mathbb{Z}_p$, $1$ only has $\pm 1$ as
    square roots

Let $n$ be odd - testing for primality

Write $n-1$ as $2^km$ where $m$ is odd

Why? If $n$ is odd, $n=2x+1$, so $n-1=2x$.  Keep dividing until
    $x$ is odd

Important fact:

Let $p$ be a prime $>2$

Let $p-1 = 2^sr$ where $r$ is odd

Let $a$ be any integer s.t. $\gcd(a,p)=1$

Then one of the following is true:

$a^r\equiv 1 \mod p$

$a^{2^jr}\equiv -1 \mod p$ for some $j$, ~~ $0\leq j \leq s-1$
###Proof
Fermat says that $a^{p-1}\equiv 1 \mod p$.  Since $p-1=2^sr$,
  then $a^{2^sr}\equiv 1 \mod p$

Consider sequence $a^r, a^{2r}, a^{2^2r}, \dots, a^{2^{s-1}r},
  a^{2^sr}$

The last item is $\equiv 1 \mod p$ by Fermat.  Also, each item
  in the sequence is the square of the previous

We have two possibilities:

$a^r\equiv 1 \mod p$ (and thus all subsequent are also $1$)

Some number in the sequence is not $1$ but squares to $1$.
  i.e. it is $-1 \mod p \equiv p-1 \mod p$

This suggests an algorithm for primality testing
###Miller Rabin Primality Test
Given $n$

Find: $k,q,k>0, q$ odd such that $n-1=2^kq$

Select random base $a$ between $1$ and $n-1$

If $a^q \equiv 1 \mod n$ reteurn ``Probably Prime''

For $j=0$ to $k-1$

~~~~~~ If $a^{2^jq} \equiv n-1 \mod  n$ return ``Probably
  Prime''

Return ``Composite''

If ``Composite'' returned, $n$ is definitely composite

If $n$ is a prime, it will never return ``Composite''

For composite $n$, MR will return ``Probably Prime'' with
  probability $<\frac{1}{4}$

i.e. about $\frac{1}{4}$ of the possibilities for $a$ are MR
  liars

Solution?  Run MR $t$ times.  Composite $n$ will get verdict of
  ``Composite'' with probability $1-(\frac{1}{4})^t$

For more certainty, run it more iterations

Used by OpenSSL

Other tests include Solovay Strassen (uses number theory we
  didn't cover).
###Interesting Observation
If composite $n$ and there are an intermediate pair of values
  $b_i=a^{2^iq}, b_{i+1}=a^{2^{i+1}q}$
such that $b_{i+1}\equiv 1
  \mod n$ but $b_i \not\equiv \pm 1 \mod n$

Then $\gcd(b_i-1,n) = $ a nontrivial factor of $n$

Remember square roots $\mod n$?

If $x^2\equiv y^2 \mod n$ but $x\not\equiv \pm y\mod n$ then
  $\gcd(x-y,n)$ is a nontrivial factor of $n$

Recall: $x^2\equiv y^2 \mod n \Rightarrow x^2-y^2\equiv 0 \mod n
  \Rightarrow (x+y)(x-y) \equiv 0 \mod n \rightarrow n|(x+y)(x-y)$
###Example
$n=561$, $n-1 = 560 = 16*35 = 2^4*35$

$a=2 , q=35, k=4$

$2^{35} \equiv 263 \mod 561$

$263^2 \equiv 166 \mod 561$

$166^2 \equiv 67  \mod 561$

$67^2  \equiv 1   \mod 561$

$67^2 \equiv 1^2 \mod 561$

$\gcd(67-1, 561) = 33$
###Miller Rabin Primality Test Example
$n=221, n-1 = 220 = 2^2*55$

Randomly select $a=174$

$174^{55} \equiv 47 \mod 221$

$47^2    \equiv 220 \mod 221 \equiv -1 \mod 221$ - Probably
  Prime!

Randomly select $a=137$

$137^{55} \equiv 188 \mod 221$

$188^2   \equiv 205 \not\equiv -1 \mod 221$ - Composite!

```
#Primality Tests
##Code Block
from random import randint as ri
ppt=MDPL
class PTest:
    def __init__(self):
        pass
    def TryDiv(self,a):
        return [(i,a//i) for i in range(2,int(a**.5)) if a%i==0]
    def Fermat(self,a):
        pass
    def Carmic(self,a):
        pass
def MilRab(n,rep=5,showwork=0):
    if n<=3: return 1
    if n%2==0:
        if showwork: ppt("{0}\ is\ even".format(n))
        return 0
    n1=q=(n-1)
    k=0
    while q%2==0:
        q=(q)//2
        k+=1
    c=pow(2,q,n)
    d,s=q,k #Other explanations use this notation
    if showwork: ppt("{2}-1 = 2^{0} {1}".format(k,q,n)) #Show n-1 factored
    if q==1: return 1
    def _MRHelp(a):
        if pow(a,d,n)==1: return 0
        for i in range(s): #Witness Loop
            temp2=pow(a,2**i*d,n)
            if temp2==n-1: return 0
        return 1
    for temp in range(rep):
        a=ri(2,n-2)
        if _MRHelp(a): return 0
    return 1 #Since nothing says otherwise, return probably prime
```

```
#Primality Tests
##Test Block
outS={0:'Composite',1:'Probably Prime'}
for a in range(20):
    rnd=2*ri(1,10000)-1
    print (rnd)
    print (outS[MilRab(rnd,showwork=1)])
    print ("Factor for verification: ",fc(rnd), "\n\n")
#Timer=0
#a=42
#for i in [["Trial Division",TEST.TryDiv],["Fermat",TEST.Fermat],["Carmichael",TEST.Carmic],["Miller-Rabin",TEST.MilRab],["Universal Exponent Factorization",TEST.Univ]]:
#    print i[0]+": ",i[1](a)
#    if Timer:
#        %timeit TEST.TryDiv(a)

```

#Factorization
##Trial Division
Trial Division - try all primes $<\sqrt{n}$ - SLOW!
##Fermat Factorization
Express $n$ as a difference of squares

$n=x^2-y^2 = (x+y)(x-y) \Rightarrow x+y$ is a factor, $x-y$ is a
  factor

Example: factor $n=295927$

Compute $n+1^2, n+2^2, n+3^2, \dots$ until we find a square

In this case, $295927+3^2 = 295936 = 544^2$

Thus  $n = (544-3)(544+3) = 541*547$

Example: Factor $n=56$

$n+1^2 = 57$

$n+2^2=60$

$n+3^3 = 65$

$n+4^2 = 72$

$n+5^2 = 81 = 9^2$

$56=(9+5)(9-5)$

Works very well for $n=pq$ when $p$ and $q$ are very close to
  each other

Security: In RSA pick $p$ and $q$ of slightly different sizes
  ($p$ of 513 bits and $q$ of 511 bits) - whats the complexity to factor?
##Pollard $p-1$
Works when $p-1$ has only small prime factors

Find prime factors $p$ of $n$ when $p-1$ is $B$-smooth (i.e. all
  prime factors are $<B$)

Lots of variances in descriptions/explanantions

Basic Idea: If $p|n$ then $a^{K(p-1)}\equiv 1 \mod p$ (if
  $\gcd(a,n)=1$).

If $x\equiv 1 \mod p$ then $\gcd(x-1,n)=d$ is a factor of $n$

Why? $x-1\equiv 0 \mod p$, so $x-1$ is a multiple of $p$

Intuition: Compute $a^Q$ where $Q$ is a multiple of $p-1$

If $p-1$ is $B$-smooth, make it a product of powers of all small
  primes $< B$
\[ Q = \prod_{\text{primes~} q<B} q^{\log_q n} \]

Select a random $a$, ~ $2\leq a\leq n-1$ where $\gcd(a,n)=1$

What to do if $\gcd(a,n) \neq 1$?

Compute $gcd(a^Q-1,n)=d$.  If $d>1$ yay!

Else, increase $B$, add to $Q$ and retry

Repeat until you factor, or until you get bored and give up

Strong Primes! $n$ should be product of strong primes:

$p$ is large

$p-1$ has large factors: $p=a_1q_1+1$ for some integer $a_1$ and
  a large prime $q_1$

$q_1$ has large prime factors: $q_1=a_2q_2+1$ for some integer
  $a_2$ and large prime $q_2$

$p+1$ has large prime factors: $p=a_3q_3-1$ for some integer
  $a_3$ and large prime $q_3$

Caveat: strong prime do not protect against new, faster
  factorization methods, so often strong prime rules are not
  enforced. (Large size is most important)
##Quadratic Sieve
Basis for modern factorization methods:

If $x^2\equiv y^2 \mod n$ and $x \not\equiv \pm y\mod n$
  then $n$ is composite and $\gcd(x-y,n)=d$ is a non-trivial factor of
  $n$

Motivating example: $n=3837523$

Observe:

$ 9398^2 \equiv 5^5*19 \mod n\\ $

$ 19095^2 \equiv 2^2*5*11*13*19 \mod n\\ $

$ 1964^2 \equiv 3^2*13^3 \mod n\\ $

$ 17078^2 \equiv 2^6*3^2*11 \mod n $

Multiply all together

$ \[9398*19095*1964*17078 \equiv (2^4*3^2*5^3*11*13^2*19)^2 \mod n\] $

$ \[2230387^2 \equiv 2586705^2 \mod n\] $

Can factor by taking $\gcd(2586705-2230387, 3837523) = 1093$
###What Did We Just Do?
Took a bunch of numbers that, when squared, were relatively
  small when reduced $\mod n$ and thus can be written as a product of
  small primes

Need to multiply together to get even powers of our small primes
  (called factor base)

Selecting factor base requires material we skipped...
###How To Do?
 This Is The Quadratic Sieve Part

Generate numbers of the form $\lceil \sqrt{in}+j \rceil$ for
  small $j$ and various $i$

When squared, $in+2j\sqrt{in}+j^2 \equiv 2j\sqrt{in}+j^2 \mod n$

For small $i,j$, this number is small and likely to be the
  product of small primes

Factor this small number using trial division and factor base

Interpret prime powers as rows in a matrix

$2^2 * 3 * 5^3 \Rightarrow (2,0,3) = (0,0,1) \mod 2$

Generate several lines and look for lines that add up to all
  zeros (Linear Algebra stuff)

If $x^2\equiv y^2\ mod\ n$ and $x\equiv ±y\ mod\ n$ then n is composite and $gcd(x-y,n)=d$ is a non-trivial factor of n

```
#Factorization
#Code Block
class Factorizer:
    def __init__(self):
        pass
    def TryDiv(self,a):
        return [(i,a/i) for i in range(2,int(a**.5)) if a%i==0]
    def Fermat(self,a):
        ret=[]
        for i in range(1,100):
            temp=(a+i**2)**.5
            if temp==int(temp):
                temp=int(temp)
                ret+=[[temp+i,temp-i]]
        return ret
    def Pollard(self,a):
        pass
    def QuadSieve(self,a):
        pass
```

```
#Factorization Tests
##Test Block
TEST=Factorizer()
Timer=0
a=56
for i in [["Trial Division",TEST.TryDiv],["Fermat",TEST.Fermat],["Pollard p-1",TEST.Pollard],["Quadratic Sieve",TEST.QuadSieve]]:
    print (i[0]+": ",i[1](a))
    if Timer:
        %timeit TEST.TryDiv(a)

```

#Discrete Logarithms
##Intro
One more major topic related to public key

A Number Theory reference problem - Discrete Logarithms

Given prime $p$, $\alpha, \beta$ non zero integers in
  $\mathbb{Z}_p^*$ where $\beta \equiv \alpha^x \mod p$

Find $x$  (Book notation: $x=L_{\alpha}(\beta)$

e.g. $p=11, \alpha = 2, \beta = 9$, then $x=6$ since $2^6 \equiv
  9 \mod 11$, or $L_2(9)=6$

In crypto protocols, usually $\alpha$ is a primitive root (or
  generator) $g$

This makes $L_g(\beta)$ defined for all $\beta \in
  \mathbb{Z}_p^*$

Believed to be hard (about as hard as factoring)

How to compute?

My notation for DLP: $y=g^x \mod p$
##$x \mod 2$
Its easy to determine $x \mod 2$ (even/odd)

Note: $(g^{\frac{p-1}{2}})^2 \equiv g^{p-1} \equiv 1 \mod p$

So $g^{\frac{p-1}{2}} \equiv \pm 1 \mod p$

Since $p$ prime, no other exponent less than $p-1$ results in $1$, so $g^{\frac{p-1}{2}} \not\equiv 1 \mod p$

So $g^{\frac{p-1}{2}} \equiv -1 \mod p$

So $y^{\frac{p-1}{2}} \equiv (g^x)^{\frac{p-1}{2}} \equiv (g^{\frac{p-1}{2}})^x \equiv -1^x \mod p$

So if $y^{\frac{p-1}{2}} \equiv 1 \mod p$ then $x$ is even ($x \equiv 0 \mod 2$)

If $y^{\frac{p-1}{2}} \equiv -1 \mod p$ then $x$ is odd ($x \equiv 1 \mod 2$)
###Example
$2^x \equiv 9 \mod 11$

$9^{\frac{11-1}{2}} \equiv 9^5 \equiv 1 \mod 11 \Rightarrow x$
  is even

Recall: $x=6$
##Pohlig Hellman
Extend that idea to get a way to compute DL when $p-1$ only has small factors.

Suppose $p-1 = \prod_i q_i^{r_i}$ where each $q_i$ is a prime

Compute $L_g(y) \mod q_i^{r_i}$ for all $i$ and recombine with CRT!

How to solve $L_g(y) \mod q^r$?

Write $x=x_0 + x_1q + x_2q^2 + \dots + x_{r-1}q^{r-1}$ with each $0 \leq x_i \leq q-1$

i.e. write $x$ in base $q$

Determine each $x_i$ successively to get $x\mod q^r$

Note that $x(\frac{p-1}{q}) = x_0(\frac{p-1}{q}) + (p-1)(x_1+x_2q+\dots x_{r-1}q^{r-2}) = x_0(\frac{p-1}{q}) + (p-1)n$ for some $n \in \mathbb{Z}$

Since $y\equiv g^x \mod p$ then

$ \[y^{\frac{p-1}{q}} \equiv g^{(x_0\frac{p-1}{q})}\underbrace{g^{n(p-1)} \equiv g^{x_0(\frac{p-1}{q})}}_{\text{By Fermat}} \mod p  \] $

Find $x_0$ by looking at $g^{k\frac{p-1}{q}} \mod p$ for $k=0$

  to $q-1$ until one matches $y^{\frac{p-1}{q}}$

##Running Example

```
#Pohlig Hellman Discrete Log
#Code Block
#Source: http://www-math.ucdenver.edu/~wcherowi/courses/m5410/phexam.html
from sympy import factorint as fc
ppt=MDPL #LaTeX-style print function
def DLog(g,p,y,ShowWork=0):
    """Prompt for a generator g, prime p, and y=g^x value.
    Essentially finds x
    Compute x using the Pohlig Hellman algorithm."""
    pf=fc(p-1) #Factor p-1
    if ShowWork:
        ppt("Pollig-Hellman\ Discrete\ Log\ Example")
        ppt("Given\ p={0},g={1},\ and\ y={2}\ such\ that\ {1}^x\equiv {2}\mod {0} \equiv {3}\mod {0},\ find\ x:".format(p,g,y,y%p))
        ppt("$p-1={0}-1={1}$".format(p,"".join([str(i)+"^"+str(pf[i])+" * " for i in pf]))[:-3])
        ppt("Now\ we\ need\ to\ find\ a\ congruency\ for\ each\ factor:")
    x=[]
    for i in pf: #For each factor
        q=i #Factor
        r=pf[i] #Power
        if ShowWork:
            print
            ppt("Find\ x_{{{0}}} \mod {1}\ [x_{{{0}}}={2}]:".format(q,q**r,"".join(["c_{{{0}}}*({{{1}}})+".format(ii,q**ii) for ii in range(r)])[:-1]))
        yy=y #Initially set base to y for each factor
        NewCLast=dict((pow(g,ii*(p-1)//q,p),ii%q) for ii in range(1,q+1)) #Dictionary of possible values for c_? to hit
        if ShowWork: ppt("y^{{ k \\frac{{p-1}}{{q^{{c_0}}}} }}\ must\ be\ one\ of\ these:\ {0}".format("".join(["{2}^{{{0}*\\frac{{p-1}}{{{3}}}}} \equiv {1},\ ".format(NewCLast[ii],ii,g,q) for ii in NewCLast.keys()])[:-3] ) )
        xTemp=0 #Using cumulative sum for finding each c_?
        for k in range(r): #For each c_?
            for j in range(1,q+1): #Try values until temp is in the NewCLast dictionary
                j=j%q #Since it would normally start with 0, making 0 last by wrapping with mod
                trypow=j*(p-1)//(q**(k+1)) #Generate power to raise yy by
                temp=pow(yy,trypow,p)
                if temp in NewCLast.keys():
                    j=NewCLast[temp] #Find the power of yy that makes temp
                    xTemp+=(q**k)*j #Add the cumulating sum for x
                    if ShowWork: ppt("$ y^{{ k \\frac{{p-1}}{{q^{{{9}}}}} }} \equiv {0}^{{ {7} \\frac{{{1}}}{{{2}}} }} \equiv {0}^{{{3}}} \equiv {4} \mod {6},\ so\ c_{{{8}}} = {7} $".format(yy,p-1,q**(k+1),trypow,temp,temp-p,p,j,k,k+1))
                    #cLast=[j]
                    if k<r-1: #Skip division if unneccessary
                        if ShowWork: ppt("We found\ c_{{{1}}},\ so\ divide\ {0}\ by\ {3}^{{{2}*c_{{{1}}} }}\ ({3}^{{ -{5}*{4} }} \equiv {6})".format(yy,k,q**k,g,j,q**k, (MI(g,p)**(q**j))%p ) )
                        yy=( yy*MI(g,p)**( q**(k)*(j) ) )%p
                    break
        if ShowWork: ppt("So,\ x_{{{0}}}={1} \mod {2}".format(q,xTemp,q**r))
        x+=[[xTemp,q**r]] #Add equivalence to list for CRT
    if ShowWork:
        print
        ppt("Now\ use\ the\ Chinese\ Remainder\ Theorem:")
    return CRT(x,ptable=ShowWork)

```

```
#Pohlig Hellman
#Test Block
#g is a
#p is p
#y is a^x
#DLog(g,p,y)
DLog(7,41,12,1)#(6,8101,7531,1)#(7,41,12)
```

$$Pollig-Hellman\ Discrete\ Log\ Example$$

$$Given\ p=41,g=7,\ and\ y=12\ such\ that\ 7^x\equiv 12\mod 41 \equiv 12\mod 41,\ find\ x:$$

$$p-1=41-1=2^3 * 5^1 $$

$$Now\ we\ need\ to\ find\ a\ congruency\ for\ each\ factor:$$

$$Find\ x_{2} \mod 8\ [x_{2}=c_{0}*({1})+c_{1}*({2})+c_{2}*({4})]:$$

$$y^{ k \frac{p-1}{q^{c_0}} }\ must\ be\ one\ of\ these:\ 7^{1*\frac{p-1}{2}} \equiv 40,\ 7^{0*\frac{p-1}{2}} \equiv 1$$

$$ y^{ k \frac{p-1}{q^{1}} } \equiv 12^{ 1 \frac{40}{2} } \equiv 12^{20} \equiv 40 \mod 41,\ so\ c_{0} = 1 $$

$$We found\ c_{0},\ so\ divide\ 12\ by\ 7^{1*c_{0} }\ (7^{ -1*1 } \equiv 36)$$

$$ y^{ k \frac{p-1}{q^{2}} } \equiv 31^{ 0 \frac{40}{4} } \equiv 31^{10} \equiv 1 \mod 41,\ so\ c_{1} = 0 $$

$$We found\ c_{1},\ so\ divide\ 31\ by\ 7^{2*c_{1} }\ (7^{ -2*0 } \equiv 6)$$

$$ y^{ k \frac{p-1}{q^{3}} } \equiv 31^{ 1 \frac{40}{8} } \equiv 31^{5} \equiv 40 \mod 41,\ so\ c_{2} = 1 $$

$$So,\ x_{2}=5 \mod 8$$

$$Find\ x_{5} \mod 5\ [x_{5}=c_{0}*({1})]:$$

$$y^{ k \frac{p-1}{q^{c_0}} }\ must\ be\ one\ of\ these:\ 7^{2*\frac{p-1}{5}} \equiv 16,\ 7^{0*\frac{p-1}{5}} \equiv 1,\ 7^{3*\frac{p-1}{5}} \equiv 18,\ 7^{4*\frac{p-1}{5}} \equiv 10,\ 7^{1*\frac{p-1}{5}} \equiv 37$$

$$ y^{ k \frac{p-1}{q^{1}} } \equiv 12^{ 3 \frac{40}{5} } \equiv 12^{8} \equiv 18 \mod 41,\ so\ c_{0} = 3 $$

$$So,\ x_{5}=3 \mod 5$$

$$Now\ use\ the\ Chinese\ Remainder\ Theorem:$$

$$M=40$$

$$a_{1}=5,\ m_{1}=8,\ \frac{M}{M_{1}}=5,\ c_{1}=5$$

$$a_{2}=3,\ m_{2}=5,\ \frac{M}{M_{2}}=8,\ c_{2}=2$$

$$(5*5*5)+(8*2*3) = 173$$

$$= 13\ mod\ 40$$

    13

#Project 3
For this assignment you will develop a number theory/public key encryption toolkit that implements several of the functions that we have discussed. You will need to provide an interface that allows me to choose which function to run and input the relevant parameters. The interface can be as simple or as complex as you like (from text based to GUI). It just needs to let me test your functions. You may use either any language. The programs should support arbitrarily large numbers so you will need to make use of appropriate libraries (e.g. gmp for C++ [hHp://gmplib.org/], or the BigInteger class in Java). Many of the functions that I ask you to write are provided already by the respective libraries, you may not use them, but rather you should implement them yourselves. You are of course free to use the low-level functions (add, mod, etc). The functions to be implemented are listed below. For each the user should be able to input the appropriate parameters.

1. isProbablyPrime – allow the user to input a number and run the miller rabin test. Should ask for a number of repetitions.

2. getProbablePrime – allow the user to input a bit length and number of repetitions and your program should generate a random number of appropriate bit length that is probably prime (using your function 1).

3. Gcd – allow user to input two numbers and your program will implement the Euclidean algorithm to find the gcd.

4. Multiplicative inverse – your program will allow the user to input a number and a modulus and it will implement the extended Euclidean algorithm to find the multiplicative inverse for the modulus.

5. Generate RSA key pair – generate large (probable) primes p and q, and appropriate e and d and print them to the screen. It would be helpful to allow interaction with a file (put the public key information into one file and private key information into another file for input into later functions).

6. RSA encrypt – prompt for a message and public key information (file I/O for faster input of large key) and should encrypt using underlying fast exponention from the respective libraries

7. RSA decrypt – prompt for ciphertext and private key info and should decrypt using underlying exponentiation library functions. Print the decrypted message to the screen.

8. Pohlig Hellman Discrete Logarithm – prompt for a generator g, prime p, and y=$g^x$ value, compute x using the Pohlig Hellman algorithm.

Extra Credit: (10 points) Implement CRT speedup for RSA decryption

```
#P3
#Code Block
from sympy import factorint as fc
from sympy import randprime as rp
from random import randint as ri
from binascii import hexlify
from binascii import unhexlify
from binascii import a2b_qp as a2b

def isProbablyPrime(a,b=5):
    """Uses Miller-Rabin to return if a is prime using b repititions"""
    return MilRab(a,rep=b,showwork=0)
def getProbablePrime(a,b,speedup=0):
    """Return a prime of bit length a using b repititions of Miller-Rabin"""
    r=4
    if speedup: r=rp(2**(a-1),2**(a))
    else:
        while not(isProbablyPrime(r,b)):
            r=ri(2**(a-2),2**(a-1))*2-1#int(reduce(lambda x,y:x+y,[str(ri(0,1)) for i in range(a)]),2)
    return r
def Gcd(a,b):
    """Returns gcd of integers a and b"""
    return Euclid(a,b,ans=0,pt=0,retA=1)[4]
def MultiplicativeInverse(a,b):
    """Returns the inverse of a mod b"""
    temp = Euclid(a,b,ans=0,pt=0,retA=1)[1]
    if temp<0:temp+=b
    return temp
def GenerateRSAKeyPair():
    """generate large (probable) primes p and q,
    and appropriate e and d and print them to the screen.
    It would be helpful to allow interaction with a file
    (put the public key information into one file and
    private key information into another file for input
    into later functions)."""
    p=getProbablePrime(1024,5,1)
    q=getProbablePrime(1024,5,1)
    n=p*q
    e=ri(3,2**10+3)
    phiN=(p-1)*(q-1)
    while Gcd(e,phiN)!=1: e=ri(3,2**10+3)
    d=MultiplicativeInverse(e,phiN)
    return (e,n),(d,n)

def RSAEncrypt(p,e):
    e,n=e
    m=int(hexlify(a2b(p)),16)
    m=hex(pow(m,e,n))[2:-1]
    if len(m)%2==1: m='0'+m
    m=unhexlify(m)
    return m
def RSADecrypt(c,d):
    return RSAEncrypt(c,d)
def PohligHellmanDiscreteLogarithm(g,p,y):
    """Prompt for a generator g, prime p, and y=g^x value.
    Essentially finds x
    Compute x using the Pohlig Hellman algorithm."""
    return DLog(g,p,y)

```

```
#P3
#Test Block
import locale
q=locale.setlocale(locale.LC_ALL,'en_US')
def commafy(a): return locale.format("%d",a,grouping=True)
import re
#def commafy(q): reduce(lambda x,y:x+', '+y,re.findall(r'[0-9]{3}|[0-9]+','0'*(len(str(q))%3-1)+str(q)))

outS={0:'Composite',1:'Probably Prime'}
rnd=ri(1,1000000)*2-1;print ("\nisProbablyPrime ({0})\n".format(rnd),outS[isProbablyPrime(rnd,5)])
print( "\nRandom Prime: ",getProbablePrime(1024,5,1) )
print( "\nGcd (5,89)\n",Gcd(5,89) )
print( "\nMultiplicative Inverse (5,89)\n",MultiplicativeInverse(5,89) )
e,d=GenerateRSAKeyPair()
print( "\nRSA Key Pair" )
print( "e: {0}".format(commafy(e[0])) )
print( "d: {0}".format(commafy(d[0])) )
print( "n: {0}".format(commafy(d[1])) )
c=RSAEncrypt("TestersTesting123",e)
p=RSADecrypt(c,d)
print( "\nRSA Encrypt: \n",c )
print( "\nRSA Decrypt: \n",p )
print( "\nPohlig-Hellman Discrete Logarithm (6,8101,7531)\n",PohligHellmanDiscreteLogarithm(6,8101,7531) )
```

    isProbablyPrime (1785337)
     Probably Prime

    Random Prime:  100882513874432988285239837291483141528119818136348662234663661983608140817170114599018330843314220988839479982472904892152304551549403588555086896049426638987395284282979327732605575979765183993133561764731786032866153175693930678009394219979971806256408796401018644851008249977207458767149159537907167118843

    Gcd (5,89)
     1

    Multiplicative Inverse (5,89)
     18

    RSA Key Pair
    e: 347
    d: 11,345,284,867,406,717,705,520,722,646,314,588,947,458,386,909,417,094,415,856,341,085,515,473,241,643,052,278,003,155,098,800,400,106,769,337,286,756,065,923,124,995,950,675,203,977,579,599,554,286,613,080,923,431,153,798,942,074,013,948,296,556,771,355,392,893,680,708,105,725,160,337,505,112,839,687,678,155,076,721,092,116,142,875,788,097,319,985,266,576,263,853,480,230,625,184,064,360,232,317,654,491,031,875,514,887,171,698,353,547,048,846,323,209,757,669,480,316,004,822,764,143,474,364,907,132,128,316,966,170,753,201,189,593,151,477,796,551,295,855,362,908,901,941,913,815,556,949,901,458,667,661,653,860,928,527,371,667,303,301,344,860,246,489,519,319,801,041,861,823,523,504,938,225,920,254,271,993,426,684,584,256,503,829,586,459,926,665,191,899,446,187,868,429,959,091,178,627,947,711,287,768,240,582,556,371
    n: 27,338,985,062,431,465,582,053,408,043,549,738,644,222,640,677,553,692,793,764,933,032,457,425,103,125,966,253,243,714,022,803,741,923,951,111,378,502,464,411,974,816,631,140,942,918,195,285,037,065,657,910,280,768,127,557,172,914,464,167,075,730,553,196,675,931,300,039,671,434,934,979,960,237,190,080,724,443,136,265,409,474,316,513,183,817,847,464,496,541,413,591,372,500,187,075,488,423,615,376,570,197,139,641,845,153,363,694,787,646,489,114,118,215,398,240,054,434,396,982,082,094,151,960,637,281,321,416,580,385,660,215,530,315,054,794,989,036,863,642,607,137,237,496,827,018,037,107,922,430,138,102,483,515,807,378,953,963,181,008,399,478,017,562,274,830,083,772,782,215,220,554,792,367,011,118,104,718,784,221,493,717,039,963,377,154,965,224,586,491,763,724,607,976,908,446,990,036,892,354,254,042,497,086,813,421

    RSA Encrypt:
     b'\x05u\xad<Qo\xc7\xb4\x1c\xcb\x1c\x81J\x02\xa2\xc5\xb3\xf6\x8d\xaa\xddJ\xec\xc7\x82e\xa3\x0f&\xd4\x05\xbdd\xc9\x1c\x9a\x90o\xec\x17\xaa\xec(b\x16+\xd7\x13\xc0\xd3EF\x92\x93\xab\x01\x15\xdf\xcdL\xd9\x85\x7fz\x1d8A\xd5w\x0b!\x96\xf1@\x11U\xa0\x8f\xaf\x17\xfeB\xb0I\x94\xa42\xbc\xa2 C\xdc\xa8\xddO\xbfxZ\xd7\xfch\xdc\xf1\xb9\x04\xcb\x05Tr\xe21\xbc=m\xd3\xc7tII\x19`b\xade>\x1e\x1c\xdaJ,-\xb5\xc8\x8d!Lx\xd8\x87"\xee\xce\xc4\xff\x1b.Q\xbdhY\xf0~\xfbZ\xda\xe7\xbc;\xb3\xf6\xe0uvLlN\xb5HI@\x00\xd7\x18\x99\x18\xc4\x94\x14\xbec\xbaS\x07v\x9c*\x14w@\x05\xfa%\xdfj{t\x07lY\x81\xe4\xf5/\xbfQ\xe2&/\xe2NS\xc2\x90\xe2zn\xfaS\xd9\x10\x13\x82\x9aX\xec\'H\x19_\x85\xaf\xc8\xde}\xc6\x8d\xafOzc@\xc1Q\xab\x15\x96\xbf\xa3"\x10Y\xf6\xd05\xd2\xb6'

    RSA Decrypt:
     b"\x03\x0fL\xf2\x93\x01\xf16\x89\xd8\x95'\x9aA\x8c!\xae:U\x88\xec\x85%A\x1e!\xa2\xb9\x05\x89w?iP\x168\xce\xe5\x8f\x0ei\x19\x86\x8f\x01\x8cX\xe56{\x97\x00\x1d8\x83a\x7fG\x12`\xf9\xad\xfd\x13GJm\x9f\xbe\xd296}\xca\x0byyY\xa6Lm \xa21\x92\xfe\xfeU.\xff\xdb\x9b_\x9dQ\xc8\xff\xf2\x16\xba\x03\xad\xd7\xb7k\xd3\x03\xdet\x03\xde\x143\x06\xefH\xba\xda\x07\xc9'\xabPW\x14\xce\xe1m\xb5\x80\xa5\xa1>\x80*=7Ok&y'L\xbc\x08\xa8\xf21P\xb2\xa3\x16\xc2\xb3\xdf2\x88\xdb\xda\x95\x03\x93d\xb8\x19\x81\x88\xf5\x98\xb2\xcd\xd3W\x80\xd5\x91V\xbf\xc0\x93\xd1\xd6\x85\xfd`\xd4)7G\x0c rS\xa8\x86\xc8\x03\xc3$U;w\xf27\x181\x0fH\x08x\x16\x85\x06\x97m\x10/S\x95\rW[\xbek\xba\xfe\x10K\xd0/\x93\x17\xcb\x12Wp\x86\xa5\x14\x95C\xf2\xe7\x03T\xf5q\n\rjX\x19\xa8\x82j\xc8"

    Pohlig-Hellman Discrete Logarithm (6,8101,7531)
     6689

https://comeoncodeon.wordpress.com/2010/09/18/pollard-rho-brent-integer-factorization/
http://stackoverflow.com/questions/4643647/fast-prime-factorization-module
http://userpages.umbc.edu/~rcampbel/Computers/Python/numbthy.html

#Testing
##More advanced formatting for Extended Euclid Algorithm
###Class object

```
from IPython.display import HTML
#(Extended) Euclid Algorithm
#Code Block
#Derived from Algorithm by Robert W. Sebesta
class Euclidean:
    def __init__(self, a, b):
        """Finds the greatest common divisor of a and b and x,y
        such that a*x+b*y=gcd(a,b)"""
        self.a=a
        self.b=b
        self.retstr="" #Return String
        self.anstab=[] #Answer Table
        self.Euclid()
    def changeIn(self,a,b):
        """Change Input"""
        self.__init__(a,b)
        return self
    def __str__(self):
        return self.showSolutionStr()
    def showSolutionStr(self):
        """Will return the equation a*x+b*y=gcd(a,b)"""
        return '({0})({1})+({2})({3})={4}'.format(self.a,self.Temp[0][0],self.b,self.Temp[0][1],self.Temp[0][2])
    def showWorkStr(self):
        """Will show the table method for finding the solution"""
        temp=""
        for x in self.Temp[1]: temp+='\n{0}\t{1}\t{2}\t{3}\t{4}\t{5}'.format(x[0][0],x[0][1],x[0][2],x[1][0],x[1][1],x[1][2])
        return temp
    def showWorkHTML(self):
        """Will return HTML version of table method for finding solution"""
        temp="<table>" #<tr><th>A1</th><th>A2</th><th>A3</th><th>B1</th><th>B2</th><th>B3</th></tr>"
        for x in self.Temp[1]:
            temp+="<tr>"
            for i in range(6):
                temp+="<td>{0}</td>".format(x[i//3][i%3])
        temp+="</tr>"
        temp+="</table>"
        return temp#;print(temp);temp2
    def showList(self):
        """Will return the list [a,x,b,y,gcd(a,b)]"""
        return [self.a,self.Temp[0][0],self.b,self.Temp[0][1],self.Temp[0][2]]
    def Euclid(self): #,a,b,ans=0,pt=0,retA=1,htmlpt=0):
        """Finds the greatest common divisor of a and b and x,y
        such that a*x+b*y=gcd(a,b)"""
        self.Temp=self._Euc([1,0,self.a],[0,1,self.b],[[['A1','A2','A3'],['B1','B2','B3']],[[1,0,self.a],[0,1,self.b]]])
    def _Euc(self,a,b,s):
        """Main recursive loop for Extended Euclidean Algorithm"""
        if b[2]==0: return [b[0],b[1],a[2]],s
        if b[2]==1: return b,s
        else:
            Q=a[2]//b[2]
            t=[a[n]-Q*b[n] for n in range(3)]
            a=b
            b=t
            return self._Euc(a,b,s+[[a,b]])
    ##End Euclidean Class##

```

```
#Test Block
Q=Euclidean(389813,642401)
print("Solution:\t",Q)#.showSolutionStr())
print("List:    \t",Q.showList())
print(Q.changeIn(655,32))
print("Work Text:\n",Q.showWorkStr())
HTML(Q.showWorkHTML())
```

    Solution:	 (389813)(-9021)+(642401)(5474)=1
    List:    	 [389813, -9021, 642401, 5474, 1]
    (655)(15)+(32)(-307)=1
    Work Text:

    A1	A2	A3	B1	B2	B3
    1	0	655	0	1	32
    0	1	32	1	-20	15
    1	-20	15	-2	41	2
    -2	41	2	15	-307	1

<table><tr><td>A1</td><td>A2</td><td>A3</td><td>B1</td><td>B2</td><td>B3</td><tr><td>1</td><td>0</td><td>655</td><td>0</td><td>1</td><td>32</td><tr><td>0</td><td>1</td><td>32</td><td>1</td><td>-20</td><td>15</td><tr><td>1</td><td>-20</td><td>15</td><td>-2</td><td>41</td><td>2</td><tr><td>-2</td><td>41</td><td>2</td><td>15</td><td>-307</td><td>1</td></tr></table>

```
def Euclid(a,b,ans=0,pt=0,retA=1,htmlpt=0):
    """Finds the greatest common divisor of a and b and x,y
    such that a*x+b*y=gcd(a,b)
    Three flags vary the output (Any number may be true):
    ans: If true will print the equation a*x+b*y=gcd(a,b)
    pt: If true will show the table method for finding the solution
    retA: If true will return the list [a,x,b,y,gcd(a,b)]
    """
    Temp=_Euc([1,0,a],[0,1,b],[[['A1','A2','A3'],['B1','B2','B3']],[[1,0,a],[0,1,b]]])
    T=Temp[0]
    if ans:
        print ('\n({0})({1})+({2})({3})={4}'.format(a,T[0],b,T[1],T[2]))
    if pt:
        for x in Temp[1]: print ('{0}\t{1}\t{2}\t{3}\t{4}\t{5}'.format(x[0][0],x[0][1],x[0][2],x[1][0],x[1][1],x[1][2]))
        print()
    if htmlpt:
        temp="<table>" #<tr><th>A1</th><th>A2</th><th>A3</th><th>B1</th><th>B2</th><th>B3</th></tr>"
        for x in Temp[1]:
            temp+="<tr>"
            for i in range(6):
                temp+="<td>{0}</td>".format(x[i//3][i%3])
            temp+="</tr>"
        temp+="</table>"
        return HTML(temp)#;print(temp);temp2
    if retA:
        return [a,T[0],b,T[1],T[2]]
def _Euc(a,b,s):
    """Main recursive loop for Extended Euclidean Algorithm"""
    if b[2]==0: return [b[0],b[1],a[2]],s
    if b[2]==1: return b,s
    else:
        Q=a[2]//b[2]
        t=[a[n]-Q*b[n] for n in range(3)]
        a=b
        b=t
        return _Euc(a,b,s+[[a,b]])
def MI(a,b):
    """Returns the multiplicative inverse of a mod b"""
    temp = Euclid(a,b,ans=0,pt=0,retA=1)[1]
    if temp<0:temp+=b
    return temp
```

```
from IPython.display import HTML
import sympy
def GCD2():
    a,x,b,y,d=sympy.symbols('a x b y d')
    A=sympy.Matrix([[x,y,d],[-b,a,0]])
    display_latex(A)
    B=sympy.Matrix([[1,0,a],[0,1,b]])
    display_latex(B)
GCD2()
MI(389813,642401)
Euclid(389813,642401,ans=1,pt=1,retA=1,htmlpt=1)

```

    (389813)(-9021)+(642401)(5474)=1
    A1	A2	A3	B1	B2	B3
    1	0	389813	0	1	642401
    0	1	642401	1	0	389813
    1	0	389813	-1	1	252588
    -1	1	252588	2	-1	137225
    2	-1	137225	-3	2	115363
    -3	2	115363	5	-3	21862
    5	-3	21862	-28	17	6053
    -28	17	6053	89	-54	3703
    89	-54	3703	-117	71	2350
    -117	71	2350	206	-125	1353
    206	-125	1353	-323	196	997
    -323	196	997	529	-321	356
    529	-321	356	-1381	838	285
    -1381	838	285	1910	-1159	71
    1910	-1159	71	-9021	5474	1

<table><tr><td>A1</td><td>A2</td><td>A3</td><td>B1</td><td>B2</td><td>B3</td></tr><tr><td>1</td><td>0</td><td>389813</td><td>0</td><td>1</td><td>642401</td></tr><tr><td>0</td><td>1</td><td>642401</td><td>1</td><td>0</td><td>389813</td></tr><tr><td>1</td><td>0</td><td>389813</td><td>-1</td><td>1</td><td>252588</td></tr><tr><td>-1</td><td>1</td><td>252588</td><td>2</td><td>-1</td><td>137225</td></tr><tr><td>2</td><td>-1</td><td>137225</td><td>-3</td><td>2</td><td>115363</td></tr><tr><td>-3</td><td>2</td><td>115363</td><td>5</td><td>-3</td><td>21862</td></tr><tr><td>5</td><td>-3</td><td>21862</td><td>-28</td><td>17</td><td>6053</td></tr><tr><td>-28</td><td>17</td><td>6053</td><td>89</td><td>-54</td><td>3703</td></tr><tr><td>89</td><td>-54</td><td>3703</td><td>-117</td><td>71</td><td>2350</td></tr><tr><td>-117</td><td>71</td><td>2350</td><td>206</td><td>-125</td><td>1353</td></tr><tr><td>206</td><td>-125</td><td>1353</td><td>-323</td><td>196</td><td>997</td></tr><tr><td>-323</td><td>196</td><td>997</td><td>529</td><td>-321</td><td>356</td></tr><tr><td>529</td><td>-321</td><td>356</td><td>-1381</td><td>838</td><td>285</td></tr><tr><td>-1381</td><td>838</td><td>285</td><td>1910</td><td>-1159</td><td>71</td></tr><tr><td>1910</td><td>-1159</td><td>71</td><td>-9021</td><td>5474</td><td>1</td></tr></table>

```

```


# Regression in Python

***
This is a very quick run-through of some basic statistical concepts, adapted from [Lab 4 in Harvard's CS109](https://github.com/cs109/2015lab4) course. Please feel free to try the original lab if you're feeling ambitious :-) The CS109 git repository also has the solutions if you're stuck.

* Linear Regression Models
* Prediction using linear regression

Linear regression is used to model and predict continuous outcomes with normal random errors. There are nearly an infinite number of different types of regression models and each regression model is typically defined by the distribution of the prediction errors (called "residuals") of the type of data. Logistic regression is used to model binary outcomes whereas Poisson regression is used to predict counts. In this exercise, we'll see some examples of linear regression as well as Train-test splits.

The packages we'll cover are: `statsmodels`, `seaborn`, and `scikit-learn`. While we don't explicitly teach `statsmodels` and `seaborn` in the Springboard workshop, those are great libraries to know.
***

***

```python
# special IPython command to prepare the notebook for matplotlib and other libraries
%matplotlib inline

import numpy as np
import pandas as pd
import scipy.stats as stats
import matplotlib.pyplot as plt
import sklearn

import seaborn as sns

# special matplotlib argument for improved plots
from matplotlib import rcParams
sns.set_style("whitegrid")
sns.set_context("poster")

```

***
# Part 1: Introduction to Linear Regression
### Purpose of linear regression
***

 Given a dataset containing predictor variables $X$ and outcome/response variable $Y$, linear regression can be used to:

 Build a predictive model to predict future values of $\hat{Y}$, using new data $X^*$ where $Y$ is unknown.

 Model the strength of the relationship between each independent variable $X_i$ and $Y$

 Many times, only a subset of independent variables $X_i$ will have a linear relationship with $Y$

 Need to figure out which $X_i$ contributes most information to predict $Y$

It is in many cases, the first pass prediction algorithm for continuous outcomes.

### A Brief Mathematical Recap
***

[Linear Regression](http://en.wikipedia.org/wiki/Linear_regression) is a method to model the relationship between a set of independent variables $X$ (also knowns as explanatory variables, features, predictors) and a dependent variable $Y$.  This method assumes the relationship between each predictor $X$ is **linearly** related to the dependent variable $Y$. The most basic linear regression model contains one independent variable $X$, we'll call this the simple model.

$$ Y = \beta_0 + \beta_1 X + \epsilon$$

where $\epsilon$ is considered as an unobservable random variable that adds noise to the linear relationship. In linear regression, $\epsilon$ is assumed to be normally distributed with a mean of 0. In other words, what this means is that on average, if we know $Y$, a roughly equal number of predictions $\hat{Y}$ will be above $Y$ and others will be below $Y$. That is, on average, the error is zero. The residuals, $\epsilon$ are also assumed to be "i.i.d.": independently and identically distributed. Independence means that the residuals are not correlated -- the residual from one prediction has no effect on the residual from another prediction. Correlated errors are common in time series analysis and spatial analyses.

* $\beta_0$ is the intercept of the linear model and represents the average of $Y$ when all independent variables $X$ are set to 0.

* $\beta_1$ is the slope of the line associated with the regression model and represents the average effect of a one-unit increase in $X$ on $Y$.

* Back to the simple model. The model in linear regression is the *conditional mean* of $Y$ given the values in $X$ is expressed a linear function.

$$ y = f(x) = E(Y | X = x)$$

http://www.learner.org/courses/againstallodds/about/glossary.html

* The goal is to estimate the coefficients (e.g. $\beta_0$ and $\beta_1$). We represent the estimates of the coefficients with a "hat" on top of the letter.

$$ \hat{\beta}_0, \hat{\beta}_1 $$

* Once we estimate the coefficients $\hat{\beta}_0$ and $\hat{\beta}_1$, we can use these to predict new values of $Y$ given new data $X$.

$$\hat{y} = \hat{\beta}_0 + \hat{\beta}_1 x_1$$

* Multiple linear regression is when you have more than one independent variable and the estimation involves matrices
    * $X_1$, $X_2$, $X_3$, $\ldots$

* How do you estimate the coefficients?
    * There are many ways to fit a linear regression model
    * The method called **least squares** is the most common methods
    * We will discuss least squares

$$ Y = \beta_0 + \beta_1 X_1 + \ldots + \beta_p X_p + \epsilon$$

### Estimating $\hat\beta$: Least squares
***
[Least squares](http://en.wikipedia.org/wiki/Least_squares) is a method that can estimate the coefficients of a linear model by minimizing the squared residuals:

$$ \mathscr{L} = \sum_{i=1}^N \epsilon_i = \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2  = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

where $N$ is the number of observations and $\epsilon$ represents a residual or error, ACTUAL - PREDICTED.

#### Estimating the intercept $\hat{\beta_0}$ for the simple linear model

We want to minimize the squared residuals and solve for $\hat{\beta_0}$ so we take the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta_0}$

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_0}} &= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_0}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= -2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \hspace{25mm} \mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \\
&= -2 \left[ \left( \sum_{i=1}^N y_i \right) - n \hat{\beta_0} - \hat{\beta}_1 \left( \sum_{i=1}^N x_i
\right) \right] \\
& 2 \left[ n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum_{i=1}^N y_i \right] = 0 \hspace{20mm} \mbox{(Set equal to 0 and solve for $\hat{\beta}_0$)} \\
& n \hat{\beta}_0 + \hat{\beta}_1 \sum_{i=1}^N x_i - \sum{i=1}^N y_i = 0 \\
& n \hat{\beta}_0 = \sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i - \hat{\beta}_1 \sum_{i=1}^N x_i}{n} \\
& \hat{\beta}_0 = \frac{\sum_{i=1}^N y_i}{n} - \hat{\beta}_1 \frac{\sum_{i=1}^N x_i}{n} \\
& \boxed{\hat{\beta}_0 = \bar{y} - \hat{\beta}_1 \bar{x}}
\end{align}
$

Using this new information, we can compute the estimate for $\hat{\beta}_1$ by taking the partial derivative of $\mathscr{L}$ with respect to $\hat{\beta}_1$.

$
\begin{align}
\frac{\partial \mathscr{L}}{\partial \hat{\beta_1}} &= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \epsilon^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \hat{y}_i \right)^2 \\
&= \frac{\partial}{\partial \hat{\beta_1}} \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right)^2 \\
&= 2 \sum_{i=1}^N \left( y_i - \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) \right) \left( -x_i \right) \hspace{25mm}\mbox{(by chain rule)} \\
&= -2 \sum_{i=1}^N x_i \left( y_i - \hat{\beta}_0 - \hat{\beta}_1 x_i \right) \\
&= -2 \sum_{i=1}^N x_i y_i - \hat{\beta}_0 x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \left( \bar{y} - \hat{\beta}_1 \bar{x} \right) x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \sum_{i=1}^N x_i y_i - \bar{y}x_i + \hat{\beta}_1\bar{x}x_i - \hat{\beta}_1 x_i^2 \\
&= -2 \left[ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i + \hat{\beta}_1\bar{x} - \hat{\beta}_1 x_i^2 \right] \\
&= -2 \left[ \hat{\beta}_1 \left\{ \bar{x} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i^2 \right\} + \left\{ \sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i \right\}\right] \\
& 2 \left[ \hat{\beta}_1 \left\{ \sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i \right\} + \left\{ \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right\} \right] = 0 \\
& \hat{\beta}_1 = \frac{-\left( \bar{y} \sum_{i=1}^N x_i - \sum_{i=1}^N x_i y_i \right)}{\sum_{i=1}^N x_i^2 - \bar{x}\sum_{i=1}^N x_i} \\
&= \frac{\sum_{i=1}^N x_i y_i - \bar{y} \sum_{i=1}^N x_i}{\sum_{i=1}^N x_i^2 - \bar{x} \sum_{i=1}^N x_i} \\
& \boxed{\hat{\beta}_1 = \frac{\sum_{i=1}^N x_i y_i - \bar{x}\bar{y}n}{\sum_{i=1}^N x_i^2 - n \bar{x}^2}}
\end{align}
$

The solution can be written in compact matrix notation as

$$\hat\beta =  (X^T X)^{-1}X^T Y$$

We wanted to show you this in case you remember linear algebra, in order for this solution to exist we need $X^T X$ to be invertible. Of course this requires a few extra assumptions, $X$ must be full rank so that $X^T X$ is invertible, etc. Basically, $X^T X$ is full rank if all rows and columns are linearly independent. This has a loose relationship to variables and observations being independent respective. **This is important for us because this means that having redundant features in our regression models will lead to poorly fitting (and unstable) models.** We'll see an implementation of this in the extra linear regression example.

***
# Part 2: Exploratory Data Analysis for Linear Relationships

The [Boston Housing data set](https://archive.ics.uci.edu/ml/datasets/Housing) contains information about the housing values in suburbs of Boston.  This dataset was originally taken from the StatLib library which is maintained at Carnegie Mellon University and is now available on the UCI Machine Learning Repository.

## Load the Boston Housing data set from `sklearn`
***

This data set is available in the [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.load_boston.html#sklearn.datasets.load_boston) python module which is how we will access it today.

```python
from sklearn.datasets import load_boston
import pandas as pd

boston = load_boston()
```

```python
boston.keys()
```

```python
boston.data.shape
```

```python
# Print column names
print(boston.feature_names)
```

```python
# Print description of Boston housing data set
print(boston.DESCR)
```

Now let's explore the data set itself.

```python
bos = pd.DataFrame(boston.data)
bos.head()
```

There are no column names in the DataFrame. Let's add those.

```python
bos.columns = boston.feature_names
bos.head()
```

Now we have a pandas DataFrame called `bos` containing all the data we want to use to predict Boston Housing prices.  Let's create a variable called `PRICE` which will contain the prices. This information is contained in the `target` data.

```python
print(boston.target.shape)
```

```python
bos['PRICE'] = boston.target
bos.head()
```

## EDA and Summary Statistics
***

Let's explore this data set.  First we use `describe()` to get basic summary statistics for each of the columns.

```python
bos.describe()
```

### Scatterplots
***

Let's look at some scatter plots for three variables: 'CRIM' (per capita crime rate), 'RM' (number of rooms) and 'PTRATIO' (pupil-to-teacher ratio in schools).

```python
plt.scatter(bos.CRIM, bos.PRICE)
plt.xlabel("Per capita crime rate by town (CRIM)")
plt.ylabel("Housing Price")
plt.title("Relationship between CRIM and Price")
```

Part 2 Checkup Exercise Set I

Exercise: What kind of relationship do you see? e.g. positive, negative?  linear? non-linear? Is there anything else strange or interesting about the data? What about outliers?

Exercise: Create scatter plots between *RM* and *PRICE*, and *PTRATIO* and *PRICE*. Label your axes appropriately using human readable labels. Tell a story about what you see.

Exercise: What are some other numeric variables of interest? Why do you think they are interesting? Plot scatterplots with these variables and *PRICE* (house price) and tell a story about what you see.

```python
# your turn: describe relationship

```

```python
# your turn: scatter plot between *RM* and *PRICE*

```

```python
# your turn: scatter plot between *PTRATIO* and *PRICE*

```

```python
# your turn: create some other scatter plots

```

### Scatterplots using Seaborn
***

[Seaborn](https://stanford.edu/~mwaskom/software/seaborn/) is a cool Python plotting library built on top of matplotlib. It provides convenient syntax and shortcuts for many common types of plots, along with better-looking defaults.

We can also use [seaborn regplot](https://stanford.edu/~mwaskom/software/seaborn/tutorial/regression.html#functions-to-draw-linear-regression-models) for the scatterplot above. This provides automatic linear regression fits (useful for data exploration later on). Here's one example below.

```python
sns.regplot(y="PRICE", x="RM", data=bos, fit_reg = True)
```

### Histograms
***

```python
plt.hist(np.log(bos.CRIM))
plt.title("CRIM")
plt.xlabel("Crime rate per capita")
plt.ylabel("Frequencey")
plt.show()
```

Part 2 Checkup Exercise Set II

Exercise: In the above histogram, we took the logarithm of the crime rate per capita. Repeat this histogram without taking the log. What was the purpose of taking the log? What do we gain by making this transformation? What do you now notice about this variable that is not obvious without making the transformation?

Exercise: Plot the histogram for *RM* and *PTRATIO* against each other, along with the two variables you picked in the previous section. We are looking for correlations in predictors here.

```python
#your turn

```

## Part 3: Linear Regression with Boston Housing Data Example
***

Here,

$Y$ = boston housing prices (called "target" data in python, and referred to as the dependent variable or response variable)

and

$X$ = all the other features (or independent variables, predictors or explanatory variables)

which we will use to fit a linear regression model and predict Boston housing prices. We will use the least-squares method to estimate the coefficients.

We'll use two ways of fitting a linear regression. We recommend the first but the second is also powerful in its features.

### Fitting Linear Regression using `statsmodels`
***
[Statsmodels](http://statsmodels.sourceforge.net/) is a great Python library for a lot of basic and inferential statistics. It also provides basic regression functions using an R-like syntax, so it's commonly used by statisticians. While we don't cover statsmodels officially in the Data Science Intensive workshop, it's a good library to have in your toolbox. Here's a quick example of what you could do with it. The version of least-squares we will use in statsmodels is called *ordinary least-squares (OLS)*. There are many other versions of least-squares such as [partial least squares (PLS)](https://en.wikipedia.org/wiki/Partial_least_squares_regression) and [weighted least squares (WLS)](https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares).

```python
# Import regression modules
import statsmodels.api as sm
from statsmodels.formula.api import ols
```

```python
# statsmodels works nicely with pandas dataframes
# The thing inside the "quotes" is called a formula, a bit on that below
m = ols('PRICE ~ RM',bos).fit()
print(m.summary())
```

#### Interpreting coefficients

There is a ton of information in this output. But we'll concentrate on the coefficient table (middle table). We can interpret the `RM` coefficient (9.1021) by first noticing that the p-value (under `P>|t|`) is so small, basically zero. This means that the number of rooms, `RM`, is a statisticall significant predictor of `PRICE`. The regression coefficient for `RM` of 9.1021 means that *on average, each additional room is associated with an increase of $\$9,100$ in house price net of the other variables*. The confidence interval gives us a range of plausible values for this average change, about ($\$8,279, \$9,925$), definitely not chump change.

In general, the $\hat{\beta_i}, i > 0$ can be interpreted as the following: "A one unit increase in $x_i$ is associated with, on average, a $\hat{\beta_i}$ increase/decrease in $y$ net of all other variables."

On the other hand, the interpretation for the intercept, $\hat{\beta}_0$ is the average of $y$ given that all of the independent variables $x_i$ are 0.

####  `statsmodels` formulas
***
This formula notation will seem familiar to `R` users, but will take some getting used to for people coming from other languages or are new to statistics.

The formula gives instruction for a general structure for a regression call. For `statsmodels` (`ols` or `logit`) calls you need to have a Pandas dataframe with column names that you will add to your formula. In the below example you need a pandas data frame that includes the columns named (`Outcome`, `X1`,`X2`, ...), but you don't need to build a new dataframe for every regression. Use the same dataframe with all these things in it. The structure is very simple:

`Outcome ~ X1`

But of course we want to to be able to handle more complex models, for example multiple regression is doone like this:

`Outcome ~ X1 + X2 + X3`

In general, a formula for an OLS multiple linear regression is

`Y ~ X1 + X2 + ... + Xp`

This is the very basic structure but it should be enough to get you through the homework. Things can get much more complex. You can force statsmodels to treat variables as categorical with the `C()` function, call numpy functions to transform data such as `np.log` for extremely-skewed data, or fit a model without an intercept by including `- 1` in the formula. For a quick run-down of further uses see the `statsmodels` [help page](http://statsmodels.sourceforge.net/devel/example_formulas.html).

Let's see how our model actually fit our data. We can see below that there is a ceiling effect, we should probably look into that. Also, for large values of $Y$ we get underpredictions, most predictions are below the 45-degree gridlines.

Part 3 Checkup Exercise Set I

Exercise: Create a scatterplot between the predicted prices, available in `m.fittedvalues` (where `m` is the fitted model) and the original prices. How does the plot look? Do you notice anything interesting or weird in the plot? Comment on what you see.

```python
# your turn
```

### Fitting Linear Regression using `sklearn`

```python
from sklearn.linear_model import LinearRegression
X = bos.drop('PRICE', axis = 1)

# This creates a LinearRegression object
lm = LinearRegression()
lm
```

#### What can you do with a LinearRegression object?
***
Check out the scikit-learn [docs here](http://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html). We have listed the main functions here. Most machine learning models in scikit-learn follow this same API of fitting a model with `fit`, making predictions with `predict` and the appropriate scoring function `score` for each model.

Main functions | Description
--- | ---
`lm.fit()` | Fit a linear model
`lm.predit()` | Predict Y using the linear model with estimated coefficients
`lm.score()` | Returns the coefficient of determination (R^2). *A measure of how well observed outcomes are replicated by the model, as the proportion of total variation of outcomes explained by the model*

#### What output can you get?

```python
# Look inside lm object
# lm.<tab>
```

Output | Description
--- | ---
`lm.coef_` | Estimated coefficients
`lm.intercept_` | Estimated intercept

### Fit a linear model
***

The `lm.fit()` function estimates the coefficients the linear regression using least squares.

```python
# Use all 13 predictors to fit linear regression model
lm.fit(X, bos.PRICE)
```

Part 3 Checkup Exercise Set II

Exercise: How would you change the model to not fit an intercept term? Would you recommend not having an intercept? Why or why not? For more information on why to include or exclude an intercept, look [here](https://stats.idre.ucla.edu/other/mult-pkg/faq/general/faq-what-is-regression-through-the-origin/).

Exercise: One of the assumptions of the linear model is that the residuals must be i.i.d. (independently and identically distributed). To satisfy this, is it enough that the residuals are normally distributed? Explain your answer.

Exercise: True or false. To use linear regression, $Y$ must be normally distributed. Explain your answer.

```python
# your turn

```

### Estimated intercept and coefficients

Let's look at the estimated coefficients from the linear model using `1m.intercept_` and `lm.coef_`.

After we have fit our linear regression model using the least squares method, we want to see what are the estimates of our coefficients $\beta_0$, $\beta_1$, ..., $\beta_{13}$:

$$ \hat{\beta}_0, \hat{\beta}_1, \ldots, \hat{\beta}_{13} $$

```python
print('Estimated intercept coefficient: {}'.format(lm.intercept_))
```

```python
print('Number of coefficients: {}'.format(len(lm.coef_)))
```

```python
# The coefficients
pd.DataFrame({'features': X.columns, 'estimatedCoefficients': lm.coef_})[['features', 'estimatedCoefficients']]
```

### Predict Prices

We can calculate the predicted prices ($\hat{Y}_i$) using `lm.predict`.

$$ \hat{Y}_i = \hat{\beta}_0 + \hat{\beta}_1 X_1 + \ldots \hat{\beta}_{13} X_{13} $$

```python
# first five predicted prices
lm.predict(X)[0:5]
```

Part 3 Checkup Exercise Set III

Exercise: Histogram: Plot a histogram of all the predicted prices. Write a story about what you see. Describe the shape, center and spread of the distribution. Are there any outliers? What might be the reason for them? Should we do anything special with them?

Exercise: Scatterplot: Let's plot the true prices compared to the predicted prices to see they disagree (we did this with `statsmodels` before).

Exercise: We have looked at fitting a linear model in both `statsmodels` and `scikit-learn`. What are the advantages and disadvantages of each based on your exploration? Based on the information provided by both packages, what advantage does `statsmodels` provide?

```python
# your turn

```

### Evaluating the Model: Sum-of-Squares

The partitioning of the sum-of-squares shows the variance in the predictions explained by the model and the variance that is attributed to error.

$$TSS = ESS + RSS$$

#### Residual Sum-of-Squares (aka $RSS$)

The residual sum-of-squares is one of the basic ways of quantifying how much error exists in the fitted model. We will revisit this in a bit.

$$ RSS = \sum_{i=1}^N r_i^2 = \sum_{i=1}^N \left(y_i - \left(\beta_0 + \beta_1 x_i\right)\right)^2 $$

```python
print(np.sum((bos.PRICE - lm.predict(X)) ** 2))
```

#### Explained Sum-of-Squares (aka $ESS$)

The explained sum-of-squares measures the variance explained by the regression model.

$$ESS = \sum_{i=1}^N \left( \hat{y}_i - \bar{y} \right)^2 = \sum_{i=1}^N \left( \left( \hat{\beta}_0 + \hat{\beta}_1 x_i \right) - \bar{y} \right)^2$$

```python
print(np.sum(lm.predict(X) - np.mean(bos.PRICE)) ** 2)
```

### Evaluating the Model: The Coefficient of Determination ($R^2$)

The coefficient of determination, $R^2$, tells us the percentage of the variance in the response variable $Y$ that can be explained by the linear regression model.

$$ R^2 = \frac{ESS}{TSS} $$

The $R^2$ value is one of the most common metrics that people use in describing the quality of a model, but it is important to note that *$R^2$ increases artificially as a side-effect of increasing the number of independent variables.* While $R^2$ is reported in almost all statistical packages, another metric called the *adjusted $R^2$* is also provided as it takes into account the number of variables in the model, and can sometimes even be used for non-linear regression models!

$$R_{adj}^2 = 1 - \left( 1 - R^2 \right) \frac{N - 1}{N - K - 1} = R^2 - \left( 1 - R^2 \right) \frac{K}{N - K - 1} = 1 - \frac{\frac{RSS}{DF_R}}{\frac{TSS}{DF_T}}$$

where $N$ is the number of observations, $K$ is the number of variables, $DF_R = N - K - 1$ is the degrees of freedom associated with the residual error and $DF_T = N - 1$ is the degrees of the freedom of the total error.

### Evaluating the Model: Mean Squared Error and the $F$-Statistic
***
The mean squared errors are just the *averages* of the sum-of-squares errors over their respective degrees of freedom.

$$MSE = \frac{ESS}{K}$$
$$MSR = \frac{RSS}{N-K-1}$$

**Remember: ** Notation may vary across resources particularly the use of *R* and *E* in *RSS/ESS* and *MSR/MSE*. In some resources, E = explained and R = residual. In other resources, E = error and R = regression (explained). **This is a very important distinction that requires looking at the formula to determine which naming scheme is being used.**

Given the MSR and MSE, we can now determine whether or not the entire model we just fit is even statistically significant. We use an $F$-test for this. The null hypothesis is that all of the $\beta$ coefficients are zero, that is, none of them have any effect on $Y$. The alternative is that *at least one* $\beta$ coefficient is nonzero, but it doesn't tell us which one in a multiple regression:

$$H_0: \beta_i = 0, \mbox{for all $i$} \\
H_A: \beta_i > 0, \mbox{for some $i$}$$

$$F = \frac{MSR}{MSE} = \left( \frac{R^2}{1 - R^2} \right) \left( \frac{N - K - 1}{K} \right)$$

Once we compute the $F$-statistic, we can use the $F$-distribution with $N-K$ and $K-1$ degrees of degrees of freedom to get a p-value.

**Warning!** The $F$-statistic mentioned in this section is NOT the same as the F1-measure or F1-value discused in Unit 7.

Part 3 Checkup Exercise Set IV

Let's look at the relationship between `PTRATIO` and housing price.

Exercise: Try fitting a linear regression model using only the 'PTRATIO' (pupil-teacher ratio by town) and interpret the intercept and the coefficients.

Exercise: Calculate (or extract) the $R^2$ value. What does it tell you?

Exercise: Compute the $F$-statistic. What does it tell you?

Exercise: Take a close look at the $F$-statistic and the $t$-statistic for the regression coefficient. What relationship do you notice? Note that this relationship only applies in *simple* linear regression models.

```python
# your turn

```

Part 3 Checkup Exercise Set V

Fit a linear regression model using three independent variables

 'CRIM' (per capita crime rate by town)

 'RM' (average number of rooms per dwelling)

 'PTRATIO' (pupil-teacher ratio by town)

Exercise: Compute or extract the $F$-statistic. What does it tell you about the model?

Exercise: Compute or extract the $R^2$ statistic. What does it tell you about the model?

Exercise: Which variables in the model are significant in predicting house price? Write a story that interprets the coefficients.

```python
# your turn
```

## Part 4: Comparing Models

During modeling, there will be times when we want to compare models to see which one is more predictive or fits the data better. There are many ways to compare models, but we will focus on two.

### The $F$-Statistic Revisited

The $F$-statistic can also be used to compare two *nested* models, that is, two models trained on the same dataset where one of the models contains a *subset* of the variables of the other model. The *full* model contains $K$ variables and the *reduced* model contains a subset of these $K$ variables. This allows us to add additional variables to a base model and then test if adding the variables helped the model fit.

$$F = \frac{\left( \frac{RSS_{reduced} - RSS_{full}}{DF_{reduced} - DF_{full}} \right)}{\left( \frac{RSS_{full}}{DF_{full}} \right)}$$

where $DF_x = N - K_x - 1$ where $K_x$ is the number of variables in model $x$.

### Akaike Information Criterion (AIC)

Another statistic for comparing two models is AIC, which is based on the likelihood function and takes into account the number of variables in the model.

$$AIC = 2 K - 2 \log_e{L}$$

where $L$ is the likelihood of the model. AIC is meaningless in the absolute sense, and is only meaningful when compared to AIC values from other models. Lower values of AIC indicate better fitting models.

`statsmodels` provides the AIC in its output.

Part 4 Checkup Exercises

Exercise: Find another variable (or two) to add to the model we built in Part 3. Compute the $F$-test comparing the two models as well as the AIC. Which model is better?

## Part 5: Evaluating the Model via Model Assumptions and Other Issues
***
Linear regression makes several assumptions. It is always best to check that these assumptions are valid after fitting a linear regression model.

**Linearity**. The dependent variable $Y$ is a linear combination of the regression coefficients and the independent variables $X$. This can be verified with a scatterplot of each $X$ vs. $Y$ and plotting correlations among $X$. Nonlinearity can sometimes be resolved by [transforming](https://onlinecourses.science.psu.edu/stat501/node/318) one or more independent variables, the dependent variable, or both. In other cases, a [generalized linear model](https://en.wikipedia.org/wiki/Generalized_linear_model) or a [nonlinear model](https://en.wikipedia.org/wiki/Nonlinear_regression) may be warranted.

**Constant standard deviation**. The SD of the dependent variable $Y$ should be constant for different values of X. We can check this by plotting each $X$ against $Y$ and verifying that there is no "funnel" shape showing data points fanning out as $X$ increases or decreases. Some techniques for dealing with non-constant variance include weighted least squares (WLS), [robust standard errors](https://en.wikipedia.org/wiki/Heteroscedasticity-consistent_standard_errors), or variance stabilizing transformations.

 **Normal distribution for errors**.  The $\epsilon$ term we discussed at the beginning are assumed to be normally distributed. This can be verified with a fitted values vs. residuals plot and verifying that there is no pattern, and with a quantile plot.
  $$ \epsilon_i \sim N(0, \sigma^2)$$
Sometimes the distributions of responses $Y$ may not be normally distributed at any given value of $X$.  e.g. skewed positively or negatively.

 **Independent errors**.  The observations are assumed to be obtained independently.

e.g. Observations across time may be correlated

There are some other issues that are important investigate with linear regression models.

**Correlated Predictors:** Care should be taken to make sure that the independent variables in a regression model are not too highly correlated. Correlated predictors typically do not majorly affect prediction, but do inflate standard errors of coefficients making interpretation unreliable. Common solutions are dropping the least important variables involved in the correlations, using regularlization, or, when many predictors are highly correlated, considering a dimension reduction technique such as principal component analysis (PCA).

**Influential Points:** Data points that have undue influence on the regression model. These points can be high leverage points or outliers. Such points are typically removed and the regression model rerun.

Part 5 Checkup Exercises

Take the reduced model from Part 3 to answer the following exercises. Take a look at [this blog post](http://mpastell.com/2013/04/19/python_regression/) for more information on using statsmodels to construct these plots.

Exercise: Construct a fitted values versus residuals plot. What does the plot tell you? Are there any violations of the model assumptions?

Exercise: Construct a quantile plot of the residuals. What does the plot tell you?

Exercise: What are some advantages and disadvantages of the fitted vs. residual and quantile plot compared to each other?

Exercise: Identify any outliers (if any) in your model and write a story describing what these outliers might represent.

Exercise: Construct a leverage plot and identify high leverage points in the model. Write a story explaining possible reasons for the high leverage points.

Exercise: Remove the outliers and high leverage points from your model and run the regression again. How do the results change?

```python
# Your turn.

```