# Matrix Factorization via Singular Value Decomposition

Matrix factorization is the breaking down of one matrix in a product of multiple matrices. It's extremely well studied in mathematics, and it's highly useful. There are many different ways to factor matrices, but singular value decomposition is particularly useful for making recommendations.

So what is singular value decomposition (SVD)? At a high level, SVD is an algorithm that decomposes a matrix $R$ into the best lower rank (i.e. smaller/simpler) approximation of the original matrix $R$. Mathematically, it decomposes R into a two unitary matrices and a diagonal matrix:

$$\begin{equation}
R = U\Sigma V^{T}
\end{equation}$$

where R is users's ratings matrix, $U$ is the user "features" matrix, $\Sigma$ is the diagonal matrix of singular values (essentially weights), and $V^{T}$ is the movie "features" matrix. $U$ and $V^{T}$ are orthogonal, and represent different things. $U$ represents how much users "like" each feature and $V^{T}$ represents how relevant each feature is to each movie.

To get the lower rank approximation, we take these matrices and keep only the top $k$ features, which we think of as the underlying tastes and preferences vectors.

```python
import pandas as pd
import numpy as np
```

```python
movies_df = pd.read_csv('movies.csv')
movies_df['movie_id'] = movies_df['movie_id'].apply(pd.to_numeric)
movies_df.head(3)
```

      movie_id
      title
      genres

      0
      1
      Toy Story (1995)
      Adventure|Animation|Children|Comedy|Fantasy

      1
      2
      Jumanji (1995)
      Adventure|Children|Fantasy

      2
      3
      Grumpier Old Men (1995)
      Comedy|Romance

```python
ratings_df=pd.read_csv('ratings.csv')
ratings_df.head(3)
```

      user_id
      movie_id
      rating

      0
      1
      31
      2.5

      1
      1
      1029
      3.0

      2
      1
      1061
      3.0

These look good, but I want the format of my ratings matrix to be one row per user and one column per movie. I'll `pivot` `ratings_df` to get that and call the new variable `R`.

```python
R_df = ratings_df.pivot(index = 'user_id', columns ='movie_id', values = 'rating').fillna(0)
R_df.head()
```

      movie_id
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      ...
      161084
      161155
      161594
      161830
      161918
      161944
      162376
      162542
      162672
      163949

      user_id

      1
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      2
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      3
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      4
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      4.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

      5
      0.0
      0.0
      4.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      ...
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0

5 rows × 9066 columns

The last thing I need to do is de-mean the data (normalize by each users mean) and convert it from a dataframe to a numpy array.

```python
R = R_df.as_matrix()
user_ratings_mean = np.mean(R, axis = 1)
R_demeaned = R - user_ratings_mean.reshape(-1, 1)
```

# Singular Value Decomposition

Scipy and Numpy both have functions to do the singular value decomposition. I'm going to use the Scipy function `svds` because it let's me choose how many latent factors I want to use to approximate the original ratings matrix (instead of having to truncate it after).

```python
from scipy.sparse.linalg import svds
U, sigma, Vt = svds(R_demeaned, k = 50)
```

Done. The function returns exactly what I detailed earlier in this post, except that the $\Sigma$ returned is just the values instead of a diagonal matrix. This is useful, but since I'm going to leverage matrix multiplication to get predictions I'll convert it to the diagonal matrix form.

```python
sigma = np.diag(sigma)
```

# Making Predictions from the Decomposed Matrices

I now have everything I need to make movie ratings predictions for every user. I can do it all at once by following the math and matrix multiply $U$, $\Sigma$, and $V^{T}$ back to get the rank $k=50$ approximation of $R$.

I also need to add the user means back to get the actual star ratings prediction.

```python
all_user_predicted_ratings = np.dot(np.dot(U, sigma), Vt) + user_ratings_mean.reshape(-1, 1)
```

```python
preds_df = pd.DataFrame(all_user_predicted_ratings, columns = R_df.columns)
preds_df.head()
```

      movie_id
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      ...
      161084
      161155
      161594
      161830
      161918
      161944
      162376
      162542
      162672
      163949

      0
      -0.054239
      0.045130
      -0.004835
      -0.019817
      -0.011284
      0.041373
      -0.007822
      -0.017188
      0.012246
      0.037670
      ...
      -0.005258
      -0.005453
      0.012369
      -0.004991
      -0.004639
      -0.019055
      0.021402
      -0.006365
      -0.006098
      -0.004819

      1
      0.419835
      1.406440
      -0.188807
      0.156658
      0.268032
      0.414698
      0.052172
      0.044728
      -0.020198
      2.220256
      ...
      -0.005909
      -0.003974
      -0.012555
      -0.003555
      -0.002711
      -0.071621
      -0.016212
      0.001047
      -0.001468
      -0.006577

      2
      1.345619
      0.266505
      -0.011962
      0.012278
      0.079508
      0.090960
      -0.122094
      0.031327
      -0.018023
      0.141176
      ...
      -0.002647
      -0.002364
      -0.010153
      0.000277
      -0.000116
      -0.018063
      -0.015761
      0.010611
      0.006792
      -0.006357

      3
      1.133455
      1.046982
      0.141275
      0.081841
      -0.339675
      -1.484659
      -0.263096
      -0.169750
      -0.021862
      1.611664
      ...
      0.020805
      0.000410
      0.056040
      -0.002817
      -0.000767
      0.159159
      0.087519
      -0.030854
      -0.021279
      0.048529

      4
      1.389578
      1.466495
      0.605557
      -0.029647
      0.729380
      -0.118539
      -0.026017
      0.065577
      -0.156655
      0.307926
      ...
      -0.007422
      -0.011810
      0.006644
      -0.005159
      -0.001249
      -0.034658
      0.016456
      0.001710
      -0.004166
      -0.001864

5 rows × 9066 columns

```python
def recommend_movies(predictions_df, userID, movies_df, original_ratings_df, num_recommendations):

    # Get and sort the user's predictions
    user_row_number = userID - 1 # UserID starts at 1, not 0
    sorted_user_predictions = preds_df.iloc[user_row_number].sort_values(ascending=False) # UserID starts at 1

    # Get the user's data and merge in the movie information.
    user_data = original_ratings_df[original_ratings_df.user_id == (userID)]
    user_full = (user_data.merge(movies_df, how = 'left', left_on = 'movie_id', right_on = 'movie_id').
                     sort_values(['rating'], ascending=False)
                 )

    print('User {0} has already rated {1} movies.'.format(userID, user_full.shape[0]))
    print('Recommending highest {0} predicted ratings movies not already rated.'.format(num_recommendations))

    # Recommend the highest predicted rating movies that the user hasn't seen yet.
    recommendations = (movies_df[~movies_df['movie_id'].isin(user_full['movie_id'])].
         merge(pd.DataFrame(sorted_user_predictions).reset_index(), how = 'left',
               left_on = 'movie_id',
               right_on = 'movie_id').
         rename(columns = {user_row_number: 'Predictions'}).
         sort_values('Predictions', ascending = False).
                       iloc[:num_recommendations, :-1]
                      )

    return user_full, recommendations
```

```python
already_rated, predictions = recommend_movies(preds_df,11, movies_df, ratings_df, 10)
```

    User 11 has already rated 38 movies.
    Recommending highest 10 predicted ratings movies not already rated.

```python
predictions
```

      movie_id
      title
      genres

      279
      318
      Shawshank Redemption, The (1994)
      Crime|Drama

      6894
      58559
      Dark Knight, The (2008)
      Action|Crime|Drama|IMAX

      2359
      2959
      Fight Club (1999)
      Action|Crime|Drama|Thriller

      530
      608
      Fargo (1996)
      Comedy|Crime|Drama|Thriller

      1356
      1732
      Big Lebowski, The (1998)
      Comedy|Crime

      959
      1213
      Goodfellas (1990)
      Crime|Drama

      7264
      70286
      District 9 (2009)
      Mystery|Sci-Fi|Thriller

      2025
      2542
      Lock, Stock & Two Smoking Barrels (1998)
      Comedy|Crime|Thriller

      520
      593
      Silence of the Lambs, The (1991)
      Crime|Horror|Thriller

      871
      1089
      Reservoir Dogs (1992)
      Crime|Mystery|Thriller

```python
already_rated.head(10)
```

      user_id
      movie_id
      rating
      title
      genres

      0
      11
      50
      5.0
      Usual Suspects, The (1995)
      Crime|Mystery|Thriller

      7
      11
      923
      5.0
      Citizen Kane (1941)
      Drama|Mystery

      36
      11
      104841
      5.0
      Gravity (2013)
      Action|Sci-Fi|IMAX

      18
      11
      26614
      5.0
      Bourne Identity, The (1988)
      Action|Adventure|Drama|Mystery|Thriller

      17
      11
      6598
      5.0
      Step Into Liquid (2002)
      Documentary

      10
      11
      1408
      5.0
      Last of the Mohicans, The (1992)
      Action|Romance|War|Western

      9
      11
      1201
      5.0
      Good, the Bad and the Ugly, The (Buono, il bru...
      Action|Adventure|Western

      19
      11
      48516
      5.0
      Departed, The (2006)
      Crime|Drama|Thriller

      37
      11
      106487
      5.0
      The Hunger Games: Catching Fire (2013)
      Action|Adventure|Sci-Fi|IMAX

      4
      11
      296
      5.0
      Pulp Fiction (1994)
      Comedy|Crime|Drama|Thriller

# Conclusion

We've seen that we can make good recommendations with raw data based collaborative filtering methods (neighborhood models) and latent features from low-rank matrix factorization methods (factorization models).

Low-dimensional matrix recommenders try to capture the underlying features driving the raw data (which we understand as tastes and preferences). From a theoretical perspective, if we want to make recommendations based on people's tastes, this seems like the better approach. This technique also scales **significantly** better to larger datasets.

However, we still likely lose some meaningful signals by using a lower-rank matrix. And though these factorization based techniques work extremely well, there's research being done on new methods. These efforts have resulted in various types probabilistic matrix factorization (which works and scales even better) and many other approaches.


## Defining a vector-space
Let's begin with an example. Consider the following three sentences, that may have each originated from an aviation safety narrative:

A - I lost control of the airplane

B - Air traffic control told us that they lost our mode-c

C - I let go of the airplane control column

If we ignore the "stop words" (those that have purely mechanical use in the language) and the words that are only used in a single sentence, we may focus only on the words "lost," "control," and "airplane." If we count the occurance of each of these words in each sentence, we can construct a vector for each as follows:

```R
A <- c(1,1,1)
B <- c(1,1,0)
C <- c(0,1,1)
rows <- c('lost','control','airplane')

docs <- cbind(A,B,C)
docs
```

A	B	C

1	1	0

1	1	1

1	0	1

Our matrix (docs) or vector-space now corresponds to a sub-set of the information contained in the three sentences. Note that we have lost words and order of the remaining words. We are simple counting the occurance of selected words in each sentence given. Since the sentences are now vectors (and coincidently only containing 3 dimensions), we can plot them on a diagram, where the x, y, and z axes, represent 'lost,' 'control,' 'airplane,' respectively.

```R
#install.packages("plot3D", dependencies=TRUE)
library(plot3D)
par(mfrow = c(1, 2))
arrows3D(x0 = c(0,0,0), y0 = c(0,0,0), z0 = c(0,0,0),
         x1 = docs[1,], y1 = docs[2,], z1 = docs[3,],
         col=c("red","blue","green"), theta=0, phi=30,
         main = "Narrative vector-space (0,30)")
arrows3D(x0 = c(0,0,0), y0 = c(0,0,0), z0 = c(0,0,0),
         x1 = docs[1,], y1 = docs[2,], z1 = docs[3,],
         col=c("red","blue","green"), theta=60, phi=30,
         main = "Narrative vector-space (30,30)")
```

Visually, you can see that the vectors for each document point in the same general direction within the vector-space. Were we to draw a fourth arrow with a head between the heads of the three document vectors, we could define a new vector that would decribe most of the "direction" of the documents. If we guess at a value for the "strongest direction" (or principal direction) collectively of the document vectors, it would be half along the x and z axes and a little more than half way along the y axis.

We can then define another vector that decribes the next most amount of document "direction." We can then repreat the process until we have as many vectors as there are documents in the document vector-space. If we require that each new vector we define is orthonormal (at a right angle) to each of the other new vectors, we have a transformed space in which each dimension uniquely describes "direction" and is in rank order of descriptive ability. This process is called singular-value-decomposition and is the foundation of latent semantic analysis (LSA) and principal component analysis (PCA).

## An alternate interpretation
Suppose that we develop a survey to measure Hofstede's concepts of culture. One aspect of which is power-distance. In the first step of the process, we write a number of questions where respondents who are most comfortable with a steep heirachy score the questions in one way and those less comfortable with heirarchy another. By design, respondents should produce scores for each of these questions that follow the latent (hidden) concept of power-distance. In designing the survey you would similarly design questions for individualism, masculinity, and uncertainty avoidance.

Were we to plot survey question responses on a diagram where each question had its own dimension (we can only visualize this for three dimensions...), if Hofstede's model is consistent we should see common "directions" within the vector-space that describe the most information. These latent concepts should exhibit themselves through the singular-value-decomposition.

## Singular value decompostion
The SVD calculation separates a single matrix into two orthonormal matrices (U and V) and a central diagonal scaling matrix (\sigma). It is commonly written as follows:
$$\begin{align}
X =& U \sigma V^T \\
X_{d\times n} =& U_{d\times m} \sigma_{m\times m} V_{n\times m}^T
\end{align}$$
NOTE: The theory and method for actually decomposing the matrix X is beyond the scope of this notebook. However, be aware that its calculation begins with the assignment of randomized values which through iteration converge towards the solution. Therefor, the calculation is NOT deterministic and will change for overly sparse matrices.

For our document vector-space the calculation results in the following:

```R
lsa <- svd(docs)
d <- diag(lsa$d) #Pad the diagonal into a complete matrix
round(lsa$u,2)
round(d,2)
round(lsa$v,2)
round(t(lsa$v),2) #The transposed V matrix
```

-0.50	-0.71	 0.50

-0.71	 0.00	-0.71

-0.50	 0.71	 0.50

2.41	0   	0.00

0.00	1   	0.00

0.00	0   	0.41

-0.71	 0.00	 0.71

-0.50	-0.71	-0.50

-0.50	 0.71	-0.50

-0.71	-0.50	-0.50

 0.00	-0.71	 0.71

 0.71	-0.50	-0.50

Writing the result in terms of the equation:
$$\begin{align}
X =& U \sigma V^T \\
=& \begin{bmatrix} -0.50 & -0.71 & 0.50 \\ -0.71 & 0 & -0.71 \\ -0.50 & 0.71 & 0.50 \end{bmatrix} \begin{bmatrix} 2.41 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0.41 \end{bmatrix} \begin{bmatrix} -0.71 & 0 & 0.71 \\ -0.50 & -0.71 & -0.50 \\ -0.50 & 0.71 & -0.50 \end{bmatrix} \\
=& \begin{bmatrix} -0.50 & -0.71 & 0.50 \\ -0.71 & 0 & -0.71 \\ -0.50 & 0.71 & 0.50 \end{bmatrix} \begin{bmatrix} 2.41 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0.41 \end{bmatrix} \begin{bmatrix} -0.71 & -0.50 & -0.50 \\ 0 & -0.71 & 0.71 \\ 0.71 & -0.50 & -0.50 \end{bmatrix}^T
\end{align}$$

## Interpretation of the decomposed matrix
Given the components of our document vector-space we can consider how to evaluate each of them. From the U matrix:
$$\begin{align}
U =& \begin{bmatrix} -0.50 & -0.71 & 0.50 \\ -0.71 & 0 & -0.71 \\ -0.50 & 0.71 & 0.50 \end{bmatrix}
\end{align}$$
The U matrix has the same number of rows as there are words in the LSA narratives and the same number of columns as there are documents/sentences. Hence, each column defines the weighting of each word in the principal directions in the space. Under this interpretation, the strongest principal direction is weighted as -0.5, -0.71, and -0.5 on 'lost,' 'control,' 'airplane,' respectively.

In LSA terms, this principal direction found is considered to be a "topic" or "theme." The topic represents words that are commonly used together in a themeatic manner. In PCA terms, this is a component and represents a latent/theoretical concept (such as power-distance).

Since the topics are given as a unit vector in terms of word components, we can plot the topics on the same diagram as the documents themselves. Here the topics are show in shades of gray, with the strongest component in a darker shade. Four different orientations are shown to better get a sense of the three dimensional reality.

```R
orig <- matrix(0L, nrow=3, ncol=6) #Base of the vectors
tip <- cbind(docs[,1], docs[,2], docs[,3],
             lsa$u[,1], lsa$u[,2], lsa$u[,3]) #Tips of the vectors
ucol <- gray.colors(3, start = 0, end = 0.9,
                    gamma = 2.2, alpha = NULL) #Gray color scheme for the topics
par(mfrow = c(2, 2))
arrows3D(x0 = orig[1,], y0 = orig[2,], z0 = orig[3,],
         x1 = tip[1,], y1 = tip[2,], z1 = tip[3,], theta=0, phi=0,
         col=c("red", "blue", "green", ucol[1], ucol[2], ucol[3]),
         main = "U transformation (0,0)")
arrows3D(x0 = orig[1,], y0 = orig[2,], z0 = orig[3,],
         x1 = tip[1,], y1 = tip[2,], z1 = tip[3,], theta=60, phi=00,
         col=c("red", "blue", "green", ucol[1], ucol[2], ucol[3]),
         main = "U transformation (60,0)")
arrows3D(x0 = orig[1,], y0 = orig[2,], z0 = orig[3,],
         x1 = tip[1,], y1 = tip[2,], z1 = tip[3,], theta=0, phi=60,
         col=c("red", "blue", "green", ucol[1], ucol[2], ucol[3]),
         main = "U transformation (0,60)")
arrows3D(x0 = orig[1,], y0 = orig[2,], z0 = orig[3,],
         x1 = tip[1,], y1 = tip[2,], z1 = tip[3,], theta=60, phi=60,
         col=c("red", "blue", "green", ucol[1], ucol[2], ucol[3]),
         main = "U transformation (60,60)")
```

```R

```

## The V matrix
The V matrix has the same number of columns as there are documents in the LSA narratives. We can think of the V matrix as the weight of each document within each topic. Hence, document A has weightings of -0.71, 0, and 0.71 on topic 1, respectively.

$$\begin{align}
V =& \begin{bmatrix} -0.71 & -0.50 & -0.50 \\ 0 & -0.71 & 0.71 \\ 0.71 & -0.50 & -0.50 \end{bmatrix}
\end{align}$$

## The sigma matrix
The central \sigma matrix is diagonal, meaning that it scales the values of a matrix/vector that it operates on without changing the relationship between them. In our case, the sigma matrix tells us about the usage of our newly defined principal directions.
$$\begin{align}
\sigma =& \begin{bmatrix} 2.41 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0.41 \end{bmatrix}
\end{align}$$
Here we can see that the third principal direction has the lowest weighting, meaning that it describes the least amount of information. This is where the key insight lies.

If we assume that there is noise in our system, whether from differring word choices or survey question interpretation, we now have a mechanism to remove it.

## Truncating the sigma matrix
Given that the last entry in the \sigma matrix decribes the least amount of information in our system, if we remove it (just set the value to zero) what happens? Let's first confirm that when our matrices are multiplied we get the expected result and then compare this to the truncated outcome:
$$\begin{align}
X =& U \sigma V^T \\
=& \begin{bmatrix} -0.50 & -0.71 & 0.50 \\ -0.71 & 0 & -0.71 \\ -0.50 & 0.71 & 0.50 \end{bmatrix} \begin{bmatrix} 2.41 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0.41 \end{bmatrix} \begin{bmatrix} -0.71 & 0 & 0.71 \\ -0.50 & -0.71 & -0.50 \\ -0.50 & 0.71 & -0.50 \end{bmatrix}^T \\
X^{\prime} =& U \sigma^{\prime} V^T \\
=& \begin{bmatrix} -0.50 & -0.71 & 0.50 \\ -0.71 & 0 & -0.71 \\ -0.50 & 0.71 & 0.50 \end{bmatrix} \begin{bmatrix} 2.41 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 0 \end{bmatrix} \begin{bmatrix} -0.71 & 0 & 0.71 \\ -0.50 & -0.71 & -0.50 \\ -0.50 & 0.71 & -0.50 \end{bmatrix}^T \\
=& \begin{bmatrix} -0.50 & -0.71 \\ -0.71 & 0 \\ -0.50 & 0.71 \end{bmatrix} \begin{bmatrix} 2.41 & 0 \\ 0 & 1 \end{bmatrix} \begin{bmatrix} -0.71 & 0 \\ -0.50 & -0.71 \\ -0.50 & 0.71 \end{bmatrix}^T
\end{align}$$

Since the last entry of the \sigma matrix has been zeroed, we can drop the last column of the U matrix and last column of the V^T matrix.

```R
X <- lsa$u %*% d %*% t(lsa$v) #The %*% operator completes a matrix multiplocation
round(X, 1) #The original word/sentence matrix is reconstructed
```

1	1	0

1	1	1

1	0	1

This time, let's calculate the outcome with the final value of the \sigma matrix substituted with a zero.

```R
dp <- d #Assign a new d prime matrix
dp[3,3] <- 0 #Zero the final value
Xp <- lsa$u %*% dp %*% t(lsa$v) #Recalculate the original matrix with the "noise" removed
round(Xp, 1)
```

0.9	1.1	0.1

1.2	0.9	0.9

0.9	0.1	1.1

We can see that the truncated \sigma matrix results in a matrix X', that is very close to the original X matrix description. In this manner we can reduce the "noise" or error in the description of a system.

The question then becomes, where does the noise stop and the signal begin? Or in mathematical terms, how many dimensions of the \sigma matrix should be retained?

## How many factors should be retained?
There are two main approaches to deciding where the noise ends and the signal begins when reducing the dimensionality of the data. You may estimate the transition point by ploting the \sigma values, just select by values, or measure the entropy of the model against a randomized form. Here, we will only discuss the first two.

In the Scree test or "hockey stick" approach, you simply plot the \sigma values and estimate where the curve transitions to a shallower profile. For the Kaiser criterion, only values >= 1 are retained. Let's plot our existing sigma diagonal:

```R
plot(lsa$d, type="l")
```

We can see from the curve that the first two values of \sigma provide the most information. The shallowing of the curve occurs between the second and third values, which would imply that if we remove the third value from the calculation, we will improve our signal-to-noise ratio. The third value also falls below 1 and does not meet the Kaiser criterion.

## LSA versus PCA
The demonstration so far has been for the latent semantic analysis approach to finding the best linear subspace description of the data. We began by creating a matrix of word occurance for each document (also known as a term-document matrix). It is in this first step that PCA differs from LSA. Rather than using the term-document matrix, PCA contructs the covariance matrix. The covariance matrix is the scaled mean difference of the originating matrix multipled by its transpose:

```R
mm <- matrix(data=1,nrow=nrow(docs)) %*% cbind(mean(A),mean(B),mean(C)) #mean matrix
dm <- docs - mm #difference matrix
cm <- (nrow(docs)-1)**-1 * t(dm) %*% dm #The scaling term includes Bessel's correction
round(cm, 2)
#cov(docs) #cov is the function in R
```

	A	B	C

A	0    	 0.00	 0.00

B	0    	 0.33	-0.17

C	0    	-0.17	 0.33

### Covariance matrix
Assuming that n is the number of documents and n=j=l, we can write this as:
$$\begin{align}
\bar{X}_j =& \frac{1}{n} \sum_{i=1}^n X_{ij} \\
Y_{ij} =& X_{ij} - \bar{X}_j \\
\Phi_{jl} =& \frac{1}{n-1} Y_{ji} Y_{il}
\end{align}$$

The covariance matrix \Phi is a square symmetric matrix.

In the PCA approach we are first scaling and centering the vector-space, then multiplying the result by its transpose before taking the SVD. Hence, the resultant principal directions can be considered to be a parallel (affine) subspace that maximizes the variance captured by the first \sigma terms.

NOTE: In this particular example, we can see one of the limitations of using the covariance matrix before the SVD. Since the first document vector contains all ones, its centered matrix contains all zeros. Simply, none of its values deviate from the vector's mean.

### Decomposition of the covariance matrix
Given the covariance matrix we may decompose it into three components in the same manner as before:
$$\begin{align}
\Phi =& \Gamma \lambda \Gamma^T
\end{align}$$

Since the \Phi matrix is square symmetric, the U and V matrices seen in the LSA calculation are identical. The \sigma values relate to the \lambda values as follows:
$$\begin{align}
\sigma_i =& \sqrt{n \lambda_i}
\end{align}$$

```R
pca <- svd(cm)
round(pca$d**.5, 3) #Take the squareroot of sigma to demonstrate the PCA output sd
print(pca)
```

0.707

0.408

0

    $d
    [1] 0.5000000 0.1666667 0.0000000

    $u
               [,1]      [,2] [,3]
    [1,]  0.0000000 0.0000000    1
    [2,] -0.7071068 0.7071068    0
    [3,]  0.7071068 0.7071068    0

    $v
               [,1]      [,2] [,3]
    [1,]  0.0000000 0.0000000    1
    [2,] -0.7071068 0.7071068    0
    [3,]  0.7071068 0.7071068    0

## Calculating the PCA directly
Several functions are available to complete the PCA in R. Here we will use the prcomp function. The output of this function provides the \Gamma matrix (equivalent to the U matrix), which it terms as rotation. The values of each vector (document vectors in our case) as transformed into each of the new principal components, or scores, is termed as x.

```R
pca <- prcomp(as.data.frame(docs))
summary(pca)
pca$rotation
pca$x #Scores
```

    Importance of components:
                              PC1    PC2 PC3
    Standard deviation     0.7071 0.4082   0
    Proportion of Variance 0.7500 0.2500   0
    Cumulative Proportion  0.7500 1.0000   1

	PC1	PC2	PC3

A	 0.0000000	 0.0000000	1

B	-0.7071068	-0.7071068	0

C	 0.7071068	-0.7071068	0

PC1	PC2	PC3

-7.071068e-01	 0.2357023   	0

-3.053113e-16	-0.4714045   	0

 7.071068e-01	 0.2357023   	0

When comparing the biplot with the plot that resulted from the LSA calculation it is difficult to visualize how the two relate. However, when we rotate the view, looking directly along the first topic vector, of the LSA U matrix components the similarity between the two transformations becomes clearer.

```R
par(mfrow = c(1, 2))
suppressWarnings(biplot(pca)) #There is a warning generated because of the zeros in the matrix
arrows3D(x0 = orig[1,], y0 = orig[2,], z0 = orig[3,],
         x1 = tip[1,], y1 = tip[2,], z1 = tip[3,], theta=137, phi=37,
         col=c("red", "blue", "green", ucol[1], ucol[2], ucol[3]))
```

## PCA limitations
Size of the variables affects the alignment of the the resulting components. Therefor, if one variable is significantly larger than the others (perhaps units of millibars versus inHg), the first component will be in the same direction as the first variable. The prcomp function allows for scaling of each variable to avoid this issue.

If all values are non-negative, the mean subtraction process will cause zeros to occur in the covariance matrix and negatives in the components that may not correspond to anything interpretable.

Were values are very close to means, some entries of the covariance matrix will have very small values. In some cases, this will cause the SVD not to converge.

```R

```


# ShallowWaters.jl - The equations

The shallow water equations for the prognostic variables velocity $\mathbf{u} = (u,v)$ and sea surface elevation $\eta$ over the 2-dimensional domain $\Omega$ in $x,y$ are

$$
\begin{align}
\partial_t u &+ u\partial_xu + v\partial_yu - fv = -g\partial_x\eta + D_x(u,v,\eta) + F_x \\
\partial_t v &+ u\partial_xv + v\partial_yv + fu = -g\partial_y\eta + D_y(u,v,\eta) + F_y \\
\partial_t \eta &+ \partial_x(uh) + \partial_y(vh) = 0.
\end{align}
$$

where the first two are the momentum equations for $u,v$ and the latter is the continuity equation. The layer thickness is $h = \eta + H$ with $H=H(x,y)$ being the bottom topography. The gravitational acceleration is $g$, the coriolis parameter $f = f(y)$ depends only (i.e. latitude) only and the beta-plane approximation is used. $(D_x,D_y) = \mathbf{D}$ are the dissipative terms

$$
\mathbf{D} = -\frac{c_D}{h}\vert \mathbf{u} \vert \mathbf{u} - \nu \nabla^4\mathbf{u}
$$

which are a sum of a quadratic bottom drag with dimensionless coefficient $c_D$ and a biharmonic diffusion with viscosity coefficient $\nu$. $\mathbf{F} = (F_x,F_y)$ is the wind forcing which can depend on time and space, i.e. $F_x = F_x(x,y,t)$ and $F_y = F_y(x,y,t)$.

## The vector invariant formulation

The Bernoulli potential $p$ is introduced as

$$
p = \frac{1}{2}(u^2 + v^2) + gh
$$

The relative vorticity $\zeta = \partial_xv + \partial_yu$ lets us define the potential vorticity $q$ as

$$
q = \frac{f + \zeta}{h}
$$

such that we can rewrite the shallow water equations as

$$
\begin{align}
\partial_t u &= qhv -g\partial_xp + D_x(u,v,\eta) + F_x \\
\partial_t v &= -qhu -g\partial_yp + D_y(u,v,\eta) + F_y \\
\partial_t \eta &= -\partial_x(uh) -\partial_y(vh)
\end{align}
$$

## Runge-Kutta time discretisation

Let

$$
R(u,v,\eta) = \begin{pmatrix}
                qhv-g\partial_xp+F_x \\
                -qhu-g\partial_yp+F_y \\
                -\partial_x(uh) -\partial_y(vh)
                \end{pmatrix}
$$

be the non-dissipative right-hand side, i.e. excluding the dissipative terms $\mathbf{D}$. Then we dicretise the time derivative with 4th order Runge-Kutta with $\mathbf{k}_n = (u_n,v_n,\eta_n)$ by

$$
\begin{align}
\mathbf{d}_1 &= R(\mathbf{k}_n) \\
\mathbf{d}_2 &= R(\mathbf{k}_n + \frac{1}{2}\Delta t \mathbf{d}_1) \\
\mathbf{d}_3 &= R(\mathbf{k}_n + \frac{1}{2}\Delta t \mathbf{d}_2) \\
\mathbf{d}_4 &= R(\mathbf{k}_n + \Delta t \mathbf{d}_3) \\
u_{n+1}^*,v_{n+1}^*,\eta_{n+1} = \mathbf{k}_{n+1} &= \mathbf{k}_n + \frac{1}{6}\Delta t(\mathbf{d}_1 + 2\mathbf{d}_2 + 2\mathbf{d}_3 + \mathbf{d}_1)
\end{align}
$$

and the dissipative terms are then added semi-implictly.

$$
u_{n+1} = u_{n+1}^* + \Delta t D_x(u_{n+1}^*,v_{n+1}^*,\eta_{n+1}) \\
v_{n+1} = v_{n+1}^* + \Delta t D_y(u_{n+1}^*,v_{n+1}^*,\eta_{n+1})
$$

Consequently, the dissipative terms only have to be evaluated once per time step, which reduces the computational cost of the right=hand side drastically. This is motivated as the Courant number $C = \sqrt{gH_0}$ is restricted by gravity waves, which are caused by $\partial_t\mathbf{u} = -g\nabla\eta$ and $\partial_t\eta = -H_0\nabla \cdot \mathbf{u}$. The other terms have longer time scales, and it is therefore sufficient to solve those with larger time steps. With this scheme, the shallow water model runs stable at $C=1$.

## Strong stability preserving Runge-Kutta with semi-implicit continuity equation

We split the right-hand side into the momentum equations and the continuity equation

$$
R_m(u,v,\eta) = \begin{pmatrix}
                qhv-g\partial_xp+F_x \\
                -qhu-g\partial_yp+F_y \\
                \end{pmatrix}, \quad R_\eta(u,v,\eta) = -\partial_x(uh) -\partial_y(vh)
$$

The 4-stage strong stability preserving Runge-Kutta scheme, with a semi-implicit treatment of the continuity equation then reads as

$$
\begin{align}
\mathbf{u}_1 &= \mathbf{u}_n + \frac{1}{2} \Delta t R_m(u_n,v_n,\eta_n), \quad \text{then}
\quad \eta_1 = \eta_n + \frac{1}{2} \Delta t R_\eta(u_1,v_1,\eta_n) \\
\mathbf{u}_2 &= \mathbf{u}_1 + \frac{1}{2} \Delta t R_m(u_1,v_1,\eta_1), \quad \text{then}
\quad \eta_2 = \eta_1 + \frac{1}{2} \Delta t R_\eta(u_2,v_2,\eta_1) \\
\mathbf{u}_3 &= \frac{2}{3}\mathbf{u}_n + \frac{1}{3}\mathbf{u}_2 + \frac{1}{6} \Delta t R_m(u_2,v_2,\eta_2), \quad \text{then}
\quad \eta_3 = \frac{2}{3}\eta_n + \frac{1}{3}\eta_2 + \frac{1}{6} \Delta t R_\eta(u_3,v_3,\eta_2) \\
\mathbf{u}_{n+1} &= \mathbf{u}_3 + \frac{1}{2} \Delta t R_m(u_3,v_3,\eta_3), \quad \text{then}
\quad \eta_{n+1} = \eta_3 + \frac{1}{2} \Delta t R_\eta(u_{n+1},v_{n+1},\eta_3) \\
\end{align}
$$

## Splitting the continuity equation

```julia
From
$$
\partial_t = -\partial_x(uh) - \partial_y(vh)
$$

```


## Homework 1

In this class, students will likely submit many different correct answers on most assignments.

## Problem 1

Find a formula for the exact gradient of the sum of squares loss function based on a linear regression model with $d+1$ weights.

*Solution.* For linear regression, we have a loss function

$$L(w)=\sum\limits_{j=1}^n\left(y_j-f(x_j)\right)^2=\sum\limits_{j=1}^n\left(y_j-w_0-w_1x_{j1}-\cdots -w_dx_{jd}\right)^2$$

Taking the derivative with respect to $w_0$, we get

$$\frac{\partial L}{\partial w_0}(w) = 2\sum\limits_{j=1}^n\left(y_j-w_0-w_1x_{j1}-\cdots -w_dx_{jd}\right)(-1)$$

For each $i=1,...,d$, we have

$$\frac{\partial L}{\partial w_i}(w) = 2\sum\limits_{j=1}^n\left(y_j-w_0-w_1x_{j1}-\cdots -w_dx_{jd}\right)(-x_{ji})$$

Denote $\hat{f}(x_j)=w_0+\sum\limits_{k=1}^d w_kx_{jk}$, then the gradient is

$$\nabla L(w)=\left(2\sum\limits_{j=1}^n\left(y_j-\hat{f}(x_j)\right)(-1),2\sum\limits_{j=1}^n\left(y_j-\hat{f}(x_j)\right)(-x_{j1}),...,2\sum\limits_{j=1}^n\left(y_j-\hat{f}(x_j)\right)(-x_{jd})\right)$$

### Alternate derivation using matrix notation

Let's define some terms:

$$
X=\begin{pmatrix}
1 & x_{11} & x_{12} & \cdots & x_{1d}\\
1 & x_{21} & x_{22} & \cdots & x_{2d}\\
\vdots & \vdots & \vdots & \ddots & \vdots\\
1 & x_{n1} & x_{n2} & \cdots & x_{nd}
\end{pmatrix}
\hspace{2cm}y=\begin{pmatrix}
y_1 \\ y_2 \\ \vdots \\ y_n
\end{pmatrix}
\hspace{2cm}w=\begin{pmatrix}
w_0 \\ w_1 \\ \vdots \\ w_d
\end{pmatrix}
$$

Then, we have shown the loss function can be written as

$$L(w)=\|Xw-y\|_2^2$$

Let's convert the loss function into a (longer but) simpler form. Transposes can be applied to sums of matrices separately, so

$$L(w)=(Xw-y)^T(Xw-y)=((Xw)^T-y^T)(Xw-y)$$

Using the distributive property of matrix multiplication,

$$L(w)=(Xw)^T Xw-(Xw)^T y-y^T Xw+y^T y$$

If we realize $y$ and $Xw$ are both matrices of shape $n\times 1$, then we should have $(Xw)^T y=y^T Xw$, so the loss function is

$$L(w)=(Xw)^T Xw-2(Xw)^T y+y^T y$$

Since $(AB)^T=B^TA^T$ for matrices, we can simplify the terms as

$$L(w)=w^T X^T Xw-2w^T X^T y+y^T y$$

Now, of course, this is a scalar (because, in the end, the loss is just a number--the sum of squared errors), so we can take the derivatives with respect to $w_0$, ..., $w_d$ and put those into a vector as

$$\nabla L(w)=2X^T Xw-2X^T y$$

This form is a little easier to use because it's easy to implement in code.

## Problem 2

Write an implementation of linear regression by gradient descent using the exact gradient formula **instead of** a function approximating the gradient (e.g. `computeGradient`).

*Solution.* The code below is similar to what we did in class, but the gradient is computed explicitly.

```python
import numpy as np

class LeastSquaresGradient:
    # initialize the hyperparameters
    def __init__(self, alpha = 0.01, tolerance = 0.01):
        self.alpha = alpha
        self.tolerance = tolerance

    # fit the model to the data
    def fit(self, X, y, epochs):
        n = X.shape[0]
        d = X.shape[1]

        # add a column of ones to the left side of X
        X = np.hstack((np.ones([n, 1]), X))

        # initialize weights to 0
        self.w = [0] * (d + 1)

        for epoch in range(epochs):
            gradient = X.T @ (X @ self.w - y)
            gradientNorm = np.linalg.norm(gradient)

            if gradientNorm < self.tolerance:
                print('Gradient descent took', epoch, 'iterations to converge')
                print('The norm of the gradient is', gradientNorm)

                # return the approximate critical value w
                return self.w

            elif epoch == epochs - 1:
                print("Gradient descent failed")
                print('The gradient is', gradient)

                # return the weights (sometimes it is still pretty good)
                return self.w

            self.w -= self.alpha * gradient

    # predict the output from testing data
    def predict(self, X):
        n = X.shape[0]

        # initialize an empty matrix to store the predicted outputs
        yPredicted = np.empty([n, 1])

        # append a column of ones at the beginning of X
        X = np.hstack((np.ones([n, 1]), X))

        # compute predictions
        return np.atleast_2d((X @ self.w)).T
```

Let's test it on the U.S. high school graduation rate dataset just to see that it works properly.

```python
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import scale
import pandas as pd

# import the data from the csv file to an numpy array
data = pd.read_csv('../data/US_State_Data.csv', sep=',').to_numpy()

X = np.array(data[:,1:8], dtype=float)
y = np.array(data[:,8], dtype=float)

# split the data into training and test sets
(trainX, testX, trainY, testY) = train_test_split(X, y, test_size = 0.25, random_state = 1)

trainX = scale(trainX)
testX = scale(testX)

# instantiate a least squares model
model = LeastSquaresGradient(alpha = 0.01, tolerance = 0.001)

# fit the model to the training data (find the w parameters)
print('Fitting the model...\n')
model.fit(trainX, trainY, epochs = 100000)

# return the predicted outputs for the datapoints in the training set
trainPredictions = model.predict(trainX)

# print quality metrics
print('\nThe mean absolute error on the training set is', mean_absolute_error(trainY, trainPredictions))

# return the predicted outputs for the datapoints in the test set
predictions = model.predict(testX)

# print quality metrics
print('The mean absolute error on the testing set is ', mean_absolute_error(testY, predictions))
```

    Fitting the model...

    Gradient descent took 196 iterations to converge
    The norm of the gradient is 0.0009910441012740167

    The mean absolute error on the training set is 3.667144144634513
    The mean absolute error on the testing set is  4.005061979203928

The results are equivalent to what we found in class (just a little better, actually), so our implementation seems to work!

## Problem 3

Add elastic-net regularization to your implementation. Hyperparameters $\lambda_1$ and $\lambda_2$ should be inputs to the class or the fit function.

*Solution.* To add this, we simply need to adjust the gradient to make it the gradient of the following loss function

$$L(w)=\|Xw-y\|_2^2+\lambda_1\|w\|_1+\lambda_2\|w\|^2$$

or, more conveniently,

$$L(w)=(Xw-y)^T(Xw-y) + \lambda_1\sum\limits_{i=0}^d |w_i| + \lambda_2 w^Tw$$

We already found the gradient of $(Xw-y)^T(Xw-y)$ to be $2X^T Xw-2X^T y$. The gradient of $\lambda_2 w^Tw$ is simply $2\lambda_2w$.

For $|w_i|$, the derivative depends on the value of $w_i$, so we have

$$\frac{\partial}{\partial w_i}|w_i|=
\begin{cases}
-1, &\text{if }w_i<0\\
\text{DNE}, &\text{if }w_i=0\\
1, &\text{if }w_i>0
\end{cases}$$

**Problem**: the derivative at $w_i=0$ does not actually exist. Practically, we can simply treat the derivative as 0 when this occurs, which will cause the $L^1$ regularization to not act on the parameter.

If we initialize the weights to 0, the $L^1$ penalty will not do anything in our first iteration, but as gradient descent causes the weights to become nonzero, the $L^1$ penalty will start having an effect. And, it becomes quite unlikely that the weights will return precisely to 0, so this is not much of a practical concern.

Altogether, the gradient of our sum of squared errors loss function with both $L^1$ and $L^2$ regularization will be

$$\nabla L(w)=2X^T Xw-2X^T y+\lambda_1\text{sign}(w)+2\lambda_2w$$

Note that the coefficients of 2 are actually somewhat redundant with the multipliers $\alpha$ in gradient descent weight updates and $\lambda_2$ for the $L^2$ penalty, so these 2's can be left out of the implementation.

Let's implement it!

```python
import numpy as np

class LeastSquaresGradientRegularized:
    # initialize the hyperparameters
    def __init__(self, alpha = 0.01, tolerance = 0.01, lam1 = 0, lam2 = 0):
        self.alpha = alpha
        self.tolerance = tolerance
        self.lam1 = lam1
        self.lam2 = lam2

    # fit the model to the data
    def fit(self, X, y, epochs):
        n = X.shape[0]
        d = X.shape[1]

        # add a column of ones to the left side of X
        X = np.hstack((np.ones([n, 1]), X))

        # initialize weights to 0
        self.w = np.array([0] * (d + 1)).astype('float')

        for epoch in range(epochs):
            gradient = X.T @ (X @ self.w - y) + self.lam1 * np.sign(self.w) + self.lam2 * self.w
            gradientNorm = np.linalg.norm(gradient)

            if gradientNorm < self.tolerance:
                print('Gradient descent took', epoch, 'iterations to converge')

                # return the approximate critical value w
                return self.w

            elif epoch == epochs - 1:
                print("Gradient descent failed to converge")
                print('The norm of the gradient is', gradientNorm)

                # return the weights (sometimes it is still pretty good)
                return self.w

            self.w -= self.alpha * gradient

    # predict the output from testing data
    def predict(self, X):
        n = X.shape[0]

        # initialize an empty matrix to store the predicted outputs
        yPredicted = np.empty([n, 1])

        # append a column of ones at the beginning of X
        X = np.hstack((np.ones([n, 1]), X))

        # compute predictions
        return np.atleast_2d((X @ self.w)).T
```

We will test this regularized implementation in Problem 4 below. Note that, by default, $\lambda_1=\lambda_2=0$, meaning the $L^1$ and $L^2$ penalties are turned off, and it has no effect on the gradient, so the model will run identically to the previous implementation.

If we set these hyperparameters to nonzero values, they will have an impact.

## Problem 4

Load the diabetes dataset from scikit-learn. Tune the hyperparameters of the model to predict disease progression as well as possible. Run the model at least ten different options for the hyperparameters, and document your performance.

*Solution.* Let's read in the data and split into train/dev/test sets.

```python
from sklearn.datasets import load_diabetes

# read in the dataset
diabetes = load_diabetes()

# find the data and labels
X = diabetes.data
Y = diabetes.target

# split the data into train and test sets
trainX, testX, trainY, testY = train_test_split(X, Y, test_size = 0.4)

# split the test set into dev and test sets
devX, testX, devY, testY = train_test_split(testX, testY, test_size = 0.5)

# check the dimensions
print('The dimensions of trainX is', trainX.shape)
print('The dimensions of devX is', devX.shape)
print('The dimensions of testX is', devX.shape)
```

    The dimensions of trainX is (265, 10)
    The dimensions of devX is (88, 10)
    The dimensions of testX is (88, 10)

This confirms we have randomly assigned 60% of the data to the training set, 20% to the dev set, and 20% to the test set.

We need to train the model with different hyperparameter settings. A good approach is to start with minimal use of hyperparameters until we see what sort of behavior we see, so let's start with $\lambda_1=\lambda_2=0$ and tune the learning rate for quick convergence.

```python
for alpha in np.linspace(0.001, 0.007, 7):
    print('================================ Running for alpha =', alpha, '================================')

    # build the linear regression model
    model = LeastSquaresGradientRegularized(alpha = alpha)

    # fit the linear regression model to the training data
    model.fit(trainX, trainY, epochs = 1000000)

    # predict the training Y's
    predictedY = model.predict(trainX)
    print('The mean absolute error on the training set is', mean_absolute_error(predictedY, trainY))

    # predict the labels of the training set
    predictedY = model.predict(devX)
    print('The mean absolute error on the dev set is', mean_absolute_error(predictedY, devY), '\n')
```

    ================================ Running for alpha = 0.001 ================================
    Gradient descent failed to converge
    The norm of the gradient is 0.025587414129761645
    The mean absolute error on the training set is 44.301048099502694
    The mean absolute error on the dev set is 42.657354633785374

    ================================ Running for alpha = 0.002 ================================
    Gradient descent took 616529 iterations to converge
    The mean absolute error on the training set is 44.301511933793556
    The mean absolute error on the dev set is 42.655587893256424

    ================================ Running for alpha = 0.003 ================================
    Gradient descent took 411019 iterations to converge
    The mean absolute error on the training set is 44.30151193557534
    The mean absolute error on the dev set is 42.65558788646962

    ================================ Running for alpha = 0.004 ================================
    Gradient descent took 308264 iterations to converge
    The mean absolute error on the training set is 44.301511937357134
    The mean absolute error on the dev set is 42.65558787968281

    ================================ Running for alpha = 0.005 ================================
    Gradient descent took 246610 iterations to converge
    The mean absolute error on the training set is 44.30151193314113
    The mean absolute error on the dev set is 42.65558789574155

    ================================ Running for alpha = 0.006 ================================
    Gradient descent took 205508 iterations to converge
    The mean absolute error on the training set is 44.30151193372337
    The mean absolute error on the dev set is 42.6555878935238

    ================================ Running for alpha = 0.007 ================================
    Gradient descent took 176150 iterations to converge
    The mean absolute error on the training set is 44.301511939103854
    The mean absolute error on the dev set is 42.65558787302953

Increasing the learning rate results in faster convergence, but but if we increase the learning rate, we eventually reach overflows:

```python
for alpha in np.linspace(0.007, 0.008, 11):
    print('================================ Running for alpha =', alpha, '================================')

    # build the linear regression model
    model = LeastSquaresGradientRegularized(alpha = alpha)

    # fit the linear regression model to the training data
    model.fit(trainX, trainY, epochs = 1000000)

    # predict the training Y's
    predictedY = model.predict(trainX)
    print('The mean absolute error on the training set is', mean_absolute_error(predictedY, trainY))

    # predict the labels of the training set
    predictedY = model.predict(devX)
    print('The mean absolute error on the dev set is', mean_absolute_error(predictedY, devY), '\n')
```

(for homework, run more experiments...)

## Problem 5

Prove $\sigma'(z)=\sigma(z)(1-\sigma(z))$ for the sigmoid function $\sigma$.

*Solution*. By definition,

$$\sigma(z)=\frac{1}{1+e^{-z}}$$

Using the chain rule, the derivaive is

$$
\begin{align}
\sigma'(z)&=(-1)\frac{1}{\left(1+e^{-z}\right)^2}e^{-z}(-1)
\\&=\frac{1}{1+e^{-z}}\frac{e^{-z}}{1+e^{-z}}
\end{align}
$$

Adding and subtracting 1 from the numerator of the second fraction,

$$
\begin{align}
\sigma'(z)&=\frac{1}{1+e^{-z}}\frac{1+e^{-z}-1}{1+e^{-z}}
\\&=\frac{1}{1+e^{-z}}\left(1-\frac{1}{1+e^{-z}}\right)
\\&=\sigma(z)\left(1-\sigma(z)\right)
\end{align}
$$

## Problem 6

As above, let's define:

$$
X=\begin{pmatrix}
1 & x_{11} & x_{12} & \cdots & x_{1d}\\
1 & x_{21} & x_{22} & \cdots & x_{2d}\\
\vdots & \vdots & \vdots & \ddots & \vdots\\
1 & x_{n1} & x_{n2} & \cdots & x_{nd}
\end{pmatrix}
\hspace{2cm}y=\begin{pmatrix}
y_1 \\ y_2 \\ \vdots \\ y_n
\end{pmatrix}
\hspace{2cm}w=\begin{pmatrix}
w_0 \\ w_1 \\ \vdots \\ w_d
\end{pmatrix}
$$

where we assume the $y_i$'s are labels in the set $\{0,1\}$.

Then, we have shown the loss function can be written as

$$L(w)=\|\sigma(Xw)-y\|_2^2$$

with the $\sigma$ being applied elementwise to the vector $Xw$.

Let's convert the loss function into a (longer but) simpler form. Using the argument from Problem 1 with $Xw$ replaced by $\sigma(Xw)$, we get

$$L(w)=(\sigma(Xw))^T \sigma(Xw)-2(\sigma(Xw))^Ty +y^T y$$

The gradient is

$$
\begin{align}
\nabla L(w) &= 2X^T\sigma(Xw)\sigma'(Xw)-2X^T\sigma'(Xw) y
\end{align}
$$

Using the result from Problem 5, we can simplify the sigmoid derivatives to find

$$
\begin{align}
\nabla L(w) &= 2X^T\sigma(Xw)\sigma(Xw)\left(1 - \sigma(Xw)\right)-2X^T\sigma(Xw)\left(1 - \sigma(Xw)\right)y
\\&= 2X^T\sigma(Xw)(1 - \sigma(Xw))\left(\sigma(Xw) - y\right)
\end{align}
$$

## Problem 7

```python
import numpy as np
from scipy.special import expit

class LeastSquaresClassifier:
    # initialize the hyperparameters
    def __init__(self, alpha = 0.01, tolerance = 0.01):
        self.alpha = alpha
        self.tolerance = tolerance

    # fit the model to the data
    def fit(self, X, y, epochs):
        n = X.shape[0]
        d = X.shape[1]

        # add a column of ones to the left side of X
        X = np.hstack((np.ones([n, 1]), X))

        # initialize weights as uniform random variables between -1 and 1
        self.w = 2 * np.random.rand(d + 1) - 1

        # train the model
        for epoch in range(epochs):

            # compute sigmoid(Xw), which is reused many times
            sigmoidXw = expit(X @ self.w)

            # compute the gradient
            gradient = X.T @ ((sigmoidXw * (1 - sigmoidXw)) * (sigmoidXw - y))

            # stopping conditions
            gradientNorm = np.linalg.norm(gradient)

            if gradientNorm < self.tolerance:
                print('Gradient descent took', epoch, 'iterations to converge')
                print('The norm of the gradient is', gradientNorm)

                # return the approximate critical value w
                return self.w

            elif epoch == epochs - 1:
                print("Gradient descent failed")
                print('The gradient is', gradient)

                # return the weights (sometimes it is still pretty good)
                return self.w

            self.w -= self.alpha * gradient

    # predict the output from testing data
    def predict(self, X):
        n = X.shape[0]

        # initialize an empty matrix to store the predicted outputs
        yPredicted = np.empty([n, 1])

        # append a column of ones at the beginning of X
        X = np.hstack((np.ones([n, 1]), X))

        # compute predictions
        return np.atleast_2d(expit(X @ self.w)).T

    def sigmoid(self, z): return np.expit(z)
```

Read in data from the file

```python
import pandas as pd

# read the data
df = pd.read_csv('../data/credit_default.csv', sep = ',', header = 1, index_col = 'ID')

# display the data
df
```

      LIMIT_BAL
      SEX
      EDUCATION
      MARRIAGE
      AGE
      PAY_0
      PAY_2
      PAY_3
      PAY_4
      PAY_5
      ...
      BILL_AMT4
      BILL_AMT5
      BILL_AMT6
      PAY_AMT1
      PAY_AMT2
      PAY_AMT3
      PAY_AMT4
      PAY_AMT5
      PAY_AMT6
      default payment next month

      ID

      1
      20000
      2
      2
      1
      24
      2
      2
      -1
      -1
      -2
      ...
      0
      0
      0
      0
      689
      0
      0
      0
      0
      1

      2
      120000
      2
      2
      2
      26
      -1
      2
      0
      0
      0
      ...
      3272
      3455
      3261
      0
      1000
      1000
      1000
      0
      2000
      1

      3
      90000
      2
      2
      2
      34
      0
      0
      0
      0
      0
      ...
      14331
      14948
      15549
      1518
      1500
      1000
      1000
      1000
      5000
      0

      4
      50000
      2
      2
      1
      37
      0
      0
      0
      0
      0
      ...
      28314
      28959
      29547
      2000
      2019
      1200
      1100
      1069
      1000
      0

      5
      50000
      1
      2
      1
      57
      -1
      0
      -1
      0
      0
      ...
      20940
      19146
      19131
      2000
      36681
      10000
      9000
      689
      679
      0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      29996
      220000
      1
      3
      1
      39
      0
      0
      0
      0
      0
      ...
      88004
      31237
      15980
      8500
      20000
      5003
      3047
      5000
      1000
      0

      29997
      150000
      1
      3
      2
      43
      -1
      -1
      -1
      -1
      0
      ...
      8979
      5190
      0
      1837
      3526
      8998
      129
      0
      0
      0

      29998
      30000
      1
      2
      2
      37
      4
      3
      2
      -1
      0
      ...
      20878
      20582
      19357
      0
      0
      22000
      4200
      2000
      3100
      1

      29999
      80000
      1
      3
      1
      41
      1
      -1
      0
      0
      0
      ...
      52774
      11855
      48944
      85900
      3409
      1178
      1926
      52964
      1804
      1

      30000
      50000
      1
      2
      1
      46
      0
      0
      0
      0
      0
      ...
      36535
      32428
      15313
      2078
      1800
      1430
      1000
      1000
      1000
      1

30000 rows × 24 columns

The data looks okay, but let's:

1. Extract the labels from the right column, convert to a NumPy array, and drop the column
2. Convert `SEX` and `MARRIAGE` to binary (0-1)
3. Convert the categorical input `EDUCATION` into a one-hot representation
4. Extract the data from all remaining columns

```python
df = pd.read_csv('../data/credit_default.csv', sep = ',', header = 1, index_col = 'ID')

# extract the labels
Y = df['default payment next month'].to_numpy()

# drop the labels
df = df.drop(columns = ['default payment next month'])

# convert SEX and MARRIAGE into binary
df['SEX'] -= 1
df['MARRIAGE'] -= 1

# convert EDUCATION to one-hot
df = pd.get_dummies(df, columns = ['EDUCATION'])

# extract the input data
X = df.to_numpy()

# split the data into train and test sets
trainX, testX, trainY, testY = train_test_split(X, Y, test_size = 0.4)

# split the test set into dev and test sets
devX, testX, devY, testY = train_test_split(testX, testY, test_size = 0.5)

# check the dimensions
print('The dimensions of trainX is', trainX.shape)
print('The dimensions of devX is', devX.shape)
print('The dimensions of testX is', devX.shape)
```

    The dimensions of trainX is (18000, 29)
    The dimensions of devX is (6000, 29)
    The dimensions of testX is (6000, 29)

Next, we can build the model and train it.

```python
import seaborn as sns
from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix

# build the linear regression model
model = LeastSquaresClassifier(alpha = 0.00001)

# fit the linear regression model to the training data
model.fit(trainX, trainY, epochs = 1000)

# predict the training Y's of the dev set
predictedY = model.predict(trainX).round()

print('\nTrain Classification Report:\n\n', classification_report(trainY, predictedY))

# predict the dev Y's
predictedY = model.predict(devX).round()
print('\nDev Classification Report:\n\n', classification_report(devY, predictedY))
print('\nDev Confusion Matrix:\n')

sns.heatmap(confusion_matrix(devY, predictedY), annot = True)
```

(run some hyperparameter experiments to improve performance hopefully)


# 14 Partial Differential Equations — 2

## Solving Laplace's or Poisson's equation

Still solving **Poisson's equation** for the electric potential $\Phi(\mathbf{r})$ and the charge density $\rho(\mathbf{r})$:

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

For a region of space without charges ($\rho = 0$) this reduces to **Laplace's equation**

$$
\nabla^2 \Phi(x, y, z) = 0
$$

General solution by iteration:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

#### Jacobi method
Do not change $\Phi_{i,j}$ until a complete sweep has been completed.

```python
def Laplace_Jacobi_slow(Phi):
    # Don't use, very slow AND inefficient
    Phi_new = Phi.copy()
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi_new[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                  + Phi[xi, yj+1] + Phi[xi, yj-1])
    Phi[:, :] = Phi_new
    return Phi
```

Fast implementation using numpy array operations (vectorized, run at C speed, not Python speed) (see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/14_PDEs/14_PDEs-2-LectureNotes.pdf) or [local PDF](14_PDEs-2-LectureNotes.pdf) for an explanation for how to set up the numpy array operations that give you a 100-fold speed up over `Laplace_Jacobi_slow()`):

```python
def Laplace_Jacobi(Phi):
    """One update in the Jacobi algorithm"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2])
    return Phi
```

#### Gauss-Seidel method
Immediately use updated new values for $\Phi_{i-1, j}$ and $\Phi_{i, j-1}$ (if starting from $\Phi_{1, 1}$).

Leads to *accelerated convergence* and therefore *less round-off error* (but distorts symmetry of boundary conditions... hopefully irrelevant when converged but check!)

```python
def Laplace_Gauss_Seidel(Phi):
    """One update in the Gauss-Seidel algorithm"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1])
    return Phi
```

#### Fast Gauss-Seidel-like

Divide the lattice into a checkerboard of black and white cells. Update the odd cells first (like Jacobi) but then use the odd cells to update the even ones (Gauss-Seidel-like; see the [board notes (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/15_PDEs/15_PDEs-2-LectureNotes.pdf) or [local PDF](14_PDEs-2-LectureNotes.pdf) ). This leads to faster convergence *and* can be done with numpy array operations.

```python
def Laplace_Gauss_Seidel_odd_even(Phi):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2] + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2])
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2] + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2])

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2] + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2])
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2] + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2])
    return Phi
```

### *Converged* solution of the wire-in-a-box problem

Solve the box-wire problem and **make sure that the solution is converged to  `tol = 1e-3`.**

Check convergence with the [_Frobenius Norm_](http://mathworld.wolfram.com/FrobeniusNorm.html):

$$
||\mathsf{\Phi}|| := \sqrt{\sum_{i,j}|\Phi_{ij}|^2}
$$

which is implemented as [numpy.linalg.norm()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.norm.html) when the argument is a matrix.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
%matplotlib notebook
```

Only execute the next line if you *don't* want interactive plotting (e.g., when exporting to LaTeX/PDF or html):

```python
%matplotlib inline
```

#### Convience plotting functions.
If you provide a filename then output is only written to a file and figures are close to conserve memory. This allows you to plot files in loops and later assemble them into movies using other programs such as ffmpeg, ImageMagick, mencoder, QuickTime 7, ...

```python
def plot_contour(Phi, filename=None, zlabel=r"potential $\Phi$ (V)",
                 cmap=plt.cm.coolwarm):
    """Plot Phi as a contour plot.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    cmap : colormap
          pick one from matplotlib.cm
    """
    fig = plt.figure(figsize=(5,4))
    ax = fig.add_subplot(111)

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]
    cset = ax.contourf(X, Y, Z, 20, cmap=cmap)
    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_aspect(1)

    cb = fig.colorbar(cset, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax

def plot_surf(Phi, filename=None, offset=-20, zlabel=r'potential $\Phi$ (V)',
             elevation=40, azimuth=-65, cmap=plt.cm.coolwarm):
    """Plot Phi as a 3D plot with contour plot underneath.

    Arguments
    ---------
    Phi : 2D array
          potential on lattice
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    offset : float, optional (default: 20)
          position the 2D contour plot by offset along the Z direction
          under the minimum Z value
    zlabel : string, optional
          label for the Z axis and color scale bar
    elevation : float, optional
          choose elevation for initial viewpoint
    azimuth : float, optional
          chooze azimuth angle for initial viewpoint
    cmap : colormap
          pick one from matplotlib.cm
    """

    x = np.arange(Phi.shape[0])
    y = np.arange(Phi.shape[1])
    X, Y = np.meshgrid(x, y)
    Z = Phi[X, Y]

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    ax.plot_wireframe(X, Y, Z, rstride=2, cstride=2, linewidth=0.5, color="gray")
    surf = ax.plot_surface(X, Y, Z, cmap=cmap, alpha=0.6)
    cset = ax.contourf(X, Y, Z, 20, zdir='z', offset=offset+Z.min(), cmap=cmap)

    ax.set_xlabel('X')
    ax.set_ylabel('Y')
    ax.set_zlabel(zlabel)
    ax.set_zlim(offset + Z.min(), Z.max())

    ax.view_init(elev=elevation, azim=azimuth)

    cb = fig.colorbar(surf, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax
```

#### Wire problem

Use the fastest Poisson solver that we have at the moment: `Laplace_Gauss_Seidel_odd_even()`

```python
Max_iter=30000
tol = 1e-3
Nmax = 100
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("Laplace_Gauss_Seidel_odd_even converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
else:
    print("Laplace_Gauss_Seidel_odd_even did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    Laplace_Gauss_Seidel_odd_even converged in 7558 iterations to 0.0009995267090286367

Plot the result and visualy compare to what we had before:

```python
plot_contour(Phi)
```

```python
plot_surf(Phi)
```

#### Make a movie showing convergence

The following solution can also write out intermediate solution steps as PNG graphics (commented out for speed). Filenames and directory names are hard-coded.(Important: `plt.ioff()` to disable interactive plotting in the notebook frontend, otherwise it is very slow.)

Put all images into a directory `wire`:

```python
import os, shutil

def mk_new_dir(dirname):
    """Create new directory (remove existing one)"""
    if os.path.exists(dirname):
        shutil.rmtree(dirname)
        print("Removed directory {}".format(dirname))
    os.mkdir(dirname)
    print("Created empty directory {}".format(dirname))

mk_new_dir("./wire")
```

    Removed directory ./wire
    Created empty directory ./wire

Make a movie by saving an image of the potential every 100 steps.

```python
plt.ioff()  # suppress interactive plotting

nsave = 100

Max_iter=30000
tol = 1e-3
Nmax = 100
Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("Laplace_Gauss_Seidel_odd_even converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, filename="wire/phi_{0:08d}.png".format(n_iter))
        break
    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, filename="wire/phi_{0:08d}.png".format(n_iter))
else:
    print("Laplace_Gauss_Seidel_odd_even did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

plt.ion()
```

    Laplace_Gauss_Seidel_odd_even converged in 7558 iterations to 0.0009995267090286367

```python
ax = plot_surf(Phi, elevation=40, azimuth=-65)
```

## Successive Over Relaxation (SOR)
Accelerate convergence with the scheme

\begin{align}
r_{i, j} &= \Phi_{i,j}^\text{new} - \Phi_{i, j}^\text{old}\\
\Phi_{i,j}^\text{new} &= \Phi_{i,j}^\text{old} + \omega r_{i,j}
\end{align}

where the new solution is computed with the Gauss-Seidel scheme.

Values of $1 \leq \omega \leq 2$ may work well, $\omega > 2$ can lead to numerical instabilities. Experiment!

### Solve Wire-in-a-box with SOR

Run to convergence (tol = 1e-03). Record the number of steps required (and visually check the solution).
* Start with $\omega = 1$. What should you get?
* Change $\omega$ and try to get faster convergence? Who can get it to converge in the fewest number of steps?

```python
Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
residual = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual  # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    SOR converged in 3797 iterations to 0.0009987117944452034

```python
plot_surf(Phi)
```

### Movie of SOR

Create images in directory `./wire_SOR`

```python
mk_new_dir("./wire_SOR")
mk_new_dir("./wire_SOR_surf")
```

    Removed directory ./wire_SOR
    Created empty directory ./wire_SOR
    Removed directory ./wire_SOR_surf
    Created empty directory ./wire_SOR_surf

```python
# %%timeit
plt.ioff()

nsave = 10

Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
residual = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, "wire_SOR/phi_{0:08d}.png".format(n_iter))
        plot_surf(Phi, "./wire_SOR_surf/phi_{0:08d}.png".format(n_iter),
                  elevation=40, azimuth=-65)
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old

    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, "./wire_SOR/phi_{0:08d}.png".format(n_iter))
        plot_surf(Phi, "./wire_SOR_surf/phi_{0:08d}.png".format(n_iter),
                  elevation=40, azimuth=-65)

else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

plt.ion()
```

    SOR converged in 3797 iterations to 0.0009987117944452034

### Tuning the code
Alternative implementation that uses more in-place array operations. (However, only marginally faster (~5%) than the previous solution.)

```python
%%timeit

Max_iter=10000
tol = 1e-3
Nmax = 100
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[0, :] = 100     # wire at x=0 at 100 V

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Laplace_Gauss_Seidel_odd_even(Phi)
    DeltaPhi = np.linalg.norm(Phi - Phi_old)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    # Phi[:, :] = Phi_old + omega*residual  = omega*Phi + (1-omega)*Phi_old
    Phi *= omega
    Phi += (1-omega)*Phi_old

    if n_iter % 100 == 0:
        print("Iteration {0}".format(n_iter), end="\r")
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    SOR converged in 3797 iterations to 0.0009987117944247595
    SOR converged in 3797 iterations to 0.0009987117944247595
    SOR converged in 3797 iterations to 0.0009987117944247595
    SOR converged in 3797 iterations to 0.0009987117944247595
    SOR converged in 3797 iterations to 0.0009987117944247595
    SOR converged in 3797 iterations to 0.0009987117944247595
    SOR converged in 3797 iterations to 0.0009987117944247595
    SOR converged in 3797 iterations to 0.0009987117944247595
    583 ms ± 24.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

## Simple example of Poisson's equation: Wire and charge
Add a positive charge and a negative charge in the box.

Now we need to solve **Poisson's equation**.

$$
\nabla^2 \Phi(x, y, z) = -4\pi\rho(x, y, z)\\
$$

Finite difference solution:
$$
\Phi_{i,j} = \frac{1}{4}\Big(\Phi_{i+1,j} + \Phi_{i-1,j} + \Phi_{i,j+1} + \Phi_{i,j-1}\Big)
     + \pi\rho_{i,j} \Delta^2
$$

### Poisson Solver
Modify the Laplace solvers to now solve Poisson's equation:

```python
import numpy as np

def Poisson_Jacobi(Phi, rho, Delta=1.):
    """One update in the Jacobi algorithm for Poisson's equation"""
    Phi[1:-1, 1:-1] = 0.25*(Phi[2:, 1:-1] + Phi[0:-2, 1:-1] + Phi[1:-1, 2:] + Phi[1:-1, 0:-2]) \
                      + np.pi * Delta**2 * rho[1:-1, 1:-1]
    return Phi

def Poisson_Gauss_Seidel(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm for Poisson's equation"""
    Nx, Ny = Phi.shape
    for xi in range(1, Nx-1):
        for yj in range(1, Ny-1):
            Phi[xi, yj] = 0.25*(Phi[xi+1, yj] + Phi[xi-1, yj]
                                + Phi[xi, yj+1] + Phi[xi, yj-1]) \
                            + np.pi * Delta**2 * rho[xi, yj]
    return Phi

def Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=1.):
    """One update in the Gauss-Seidel algorithm on odd or even fields"""
    a = np.pi * Delta**2
    # odd update (uses old even)
    Phi[1:-2:2, 1:-2:2] = 0.25*(Phi[2::2, 1:-2:2] + Phi[0:-2:2, 1:-2:2]
                                + Phi[1:-2:2, 2::2] + Phi[1:-2:2, 0:-2:2]) + a * rho[1:-2:2, 1:-2:2]
    Phi[2:-1:2, 2:-1:2] = 0.25*(Phi[3::2, 2:-1:2] + Phi[1:-2:2, 2:-1:2]
                                + Phi[2:-1:2, 3::2] + Phi[2:-1:2, 1:-2:2]) + a * rho[2:-1:2, 2:-1:2]

    # even update (uses new odd)
    Phi[1:-2:2, 2:-1:2] = 0.25*(Phi[2::2, 2:-1:2] + Phi[0:-2:2, 2:-1:2]
                                + Phi[1:-2:2, 3::2] + Phi[1:-2:2, 1:-1:2]) + a * rho[1:-2:2, 2:-1:2]
    Phi[2:-1:2, 1:-2:2] = 0.25*(Phi[3::2, 1:-2:2] + Phi[1:-2:2, 1:-2:2]
                                + Phi[2:-1:2, 2::2] + Phi[2:-1:2, 0:-2:2]) + a * rho[2:-1:2, 1:-2:2]
    return Phi
```

### Solve wire with dipole

Solve the Poisson problem with SOR (as above), but now employing the Poisson solvers (we are using the fastest one that we have):

```python
Nmax = 100
Max_iter = 10000
omega = 1.99

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
rho = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V
rho[25:27, 39:41] = 5.0
rho[75:77, 39:41] = -5.0

Delta = 1.0

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=Delta)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

```

    SOR converged in 3823 iterations to 0.0009991605868815044

```python
plot_surf(Phi, elevation=40, azimuth=20)
```

#### Make a movie

The following code can write partial solutions as images. To make it a bit more comfortable, we generate a directory for the images from Python:

```python
import os

def makedir(dirname):
    try:
        os.mkdir(dirname)
    except OSError:
        pass
    return dirname
```

```python
plt.ioff()

nsave = 100

Nmax = 100
Max_iter = 10000
omega = 1.99

dirname = makedir("dipole_wire")
filename = os.path.join(dirname, "phi_{0:08d}.png")

dirname3 = makedir("dipole_wire_3d")
filename3 = os.path.join(dirname3, "phi_{0:08d}.png")

Phi = np.zeros((Nmax, Nmax), dtype=np.float64)
Phi_old = np.zeros_like(Phi)
rho = np.zeros_like(Phi)

# initialize boundaries
# everything starts out zero so nothing special for the grounded wires
Phi[:, 0] = 100     # wire at y=0 at 100 V
rho[25:27, 39:41] = 5.0
rho[75:77, 39:41] = -5.0

Delta = 1.0

for n_iter in range(Max_iter):
    Phi_old[:, :] = Phi
    Phi = Poisson_Gauss_Seidel_odd_even(Phi, rho, Delta=Delta)
    residual[:, :] = Phi - Phi_old
    DeltaPhi = np.linalg.norm(residual)
    if DeltaPhi < tol:
        print("SOR converged in {0} iterations to {1}".format(n_iter+1, DeltaPhi))
        plot_contour(Phi, filename=filename.format(n_iter))
        break
    # SOR
    Phi[:, :] = Phi_old + omega*residual # = omega*Phi + (1-omega)*Phi_old

    if n_iter % nsave == 0:
        print("Iteration {0}".format(n_iter), end="\r")
        plot_contour(Phi, filename=filename.format(n_iter))
        plot_surf(Phi, filename=filename3.format(n_iter),
                  elevation=40, azimuth=20)
else:
    print("SOR did NOT converge in {0} iterations, DeltaPhi={1}".format(n_iter+1, DeltaPhi))

print("Images in {0} and {1}".format(dirname, dirname3))
plt.ion()

```

    SOR converged in 3823 iterations to 0.0009991605868815044
    Images in dipole_wire and dipole_wire_3d

## Charge density
Given a potential $\Phi$, Poisson's equation gives the charge density:
$$
\rho(\mathbf{x}) = -\frac{1}{4\pi} \nabla^2\Phi(\mathbf{x})
$$

Discretized Laplacian
$$
\nabla^2 \Phi \approx \frac{\Phi(x+\Delta x,y) + \Phi(x-\Delta x,y) +\Phi(x,y+\Delta y) +\, \Phi(x,y-\Delta y) - 4\Phi(x, y)}{\Delta^2}
$$

```python
def laplacian2d(f, Delta=1):
    """Finite difference approximation of Del^2 f.

    Arguments
    ---------
    f : M x N matrix
    Delta : float

    Returns
    -------
    M x N matrix, boundaries set to 0
    """

    L = np.zeros_like(f, dtype=np.float64)
    L[1:-1, 1:-1] = f[1:-1, 2:] + f[1:-1, 0:-2] + f[2:, 1:-1] + f[0:-2, 1:-1] - 4*f[1:-1, 1:-1]
    return L/Delta**2

def laplacian2dsimple(f, Delta=1):
    L = np.zeros_like(f, dtype=np.float64)
    for i in range(1, L.shape[0]-1):
        for j in range(1, L.shape[1]-1):
            L[i, j] = f[i, j+1] + f[i, j-1] + f[i+1, j] + f[i-1, j] - 4*f[i, j]
    return L/Delta**2
```

Compute the charge density from the converged potential
$$
\rho(x, y) = -\frac{1}{4\pi} \nabla^2 \Phi(x, y)
$$

```python
rhox = - laplacian2d(Phi)/(4*np.pi)
```

Does `rhox` show the charges of $+5$ and $-5$ that we introduced with the charge density $\rho$?

```python
print(rhox.min())
print(rhox.max())
```

    -5.00000418672
    5.0

Indeed, the point charges are recovered from the potential.

Plot the charge density:

```python
plot_contour(rhox, zlabel=r"charge density $\rho$")
```

The position of the point charges are found correctly but the charge on the wire appear to be absent. This is expected because the Laplacian is not defined on the boundary. If you wanted to see the charge on the wire you would need to include it inside the integration domain and also assign a thickness (e.g., one or two grid cells).

The 3D plot shows the "point" charges as spikes of the size of one grid cell:

```python
plot_surf(rhox, zlabel=r"charge density $\rho$", elevation=20, azimuth=20, offset=-3)
```

The position of the point charges are found correctly but the charge on the wire appear to be absent. The problem is that the Laplacian is not defined on the boundary.

Also, if the charges are only defined on a single grid cell then the 3D visualization is sometimes not able to show it correctly.

#### Testing the implementation of Laplacian

Making sure that our fancy numpy-version of the Laplacian is producing the same results as the naive implementation with Python loops:

```python
import numpy as np

def test_laplacian2d():
    ftest = np.random.random((200, 200))
    assert np.allclose(laplacian2d(ftest), laplacian2dsimple(ftest))

test_laplacian2d()
```

```python

```


# Nonlinear Equations

We want to find a root of the nonlinear function $f$ using different methods.

1. Bisection method
2. Newton method
3. Chord method
4. Secant method
5. Fixed point iterations

```python
%matplotlib inline
from numpy import *
from matplotlib.pyplot import *
import sympy as sym

```

```python
t = sym.symbols('t')

f_sym = t/8. * (63.*t**4 - 70.*t**2. +15.) # Legendre polynomial of order 5

f_prime_sym = sym.diff(f_sym,t)

f = sym.lambdify(t, f_sym, 'numpy')
f_prime = sym.lambdify(t,f_prime_sym, 'numpy')

phi = lambda x : 63./70.*x**3 + 15./(70.*x)
#phi = lambda x : 70.0/15.0*x**3 - 63.0/15.0*x**5
#phi = lambda x : sqrt((63.*x**4 + 15.0)/70.)

# Let's plot
n = 1025

x = linspace(-1,1,n)
c = zeros_like(x)

_ = plot(x,f(x))
_ = plot(x,c)
_ = grid()

```

```python
# Initial data for the variuos algorithms

# interval in which we seek the solution
a = 0.7
b = 1.

# initial points
x0 = (a+b)/2.0
x00 = b

```

```python
# stopping criteria
eps = 1e-10
n_max = 1000
```

## Bisection method

$$
x^k = \frac{a^k+b^k}{2}
$$
```
                   if (f(a_k) * f(x_k)) < 0:
                      b_k1 = x_k
                      a_k1 = a_k
                   else:
                      a_k1 = x_k
                      b_k1 = b_k
```

```python
def bisect(f,a,b,eps,n_max):
    assert(f(a) * f(b) < 0)
    a_new = a
    b_new = b
    x = mean([a,b])
    err = eps + 1.
    errors = [err]
    it = 0
    while (err > eps and it < n_max):
        if ( f(a_new) * f(x) < 0 ):
            # root in (a_new,x)
            b_new = x
        else:
            # root in (x,b_new)
            a_new = x

        x_new = mean([a_new,b_new])

        #err = 0.5 *(b_new -a_new)
        err = abs(f(x_new))
        #err = abs(x-x_new)

        errors.append(err)
        x = x_new
        it += 1

    semilogy(errors)
    print(it)
    print(x)
    print(err)
    return errors

errors_bisect = bisect(f,a,b,eps,n_max)
```

```python
# is the number of iterations coherent with the theoretical estimation?
```

In order to find out other methods for solving non-linear equations, let's compute the Taylor's series of $f(x^k)$ up to the first order

$$
f(x^k) \simeq f(x^k) + (x-x^k)f^{\prime}(x^k)
$$
which suggests the following iterative scheme
$$
x^{k+1} = x^k - \frac{f(x^k)}{f^{\prime}(x^k)}
$$

The following methods are obtained applying the above scheme where

$$
f^{\prime}(x^k) \approx q^k
$$

## Newton's method
$$
q^k = f^{\prime}(x^k)
$$

$$
x^{k+1} = x^k - \frac{f(x^k)}{q^k}
$$

```python
def newton(f,f_prime,x0,eps,n_max):
    x_new = x0
    err = eps + 1.
    errors = [err]
    it = 0
    while (err > eps and it < n_max):
        x_new = x_new - (f(x_new)/f_prime(x_new))
        err = abs(f(x_new))
        errors.append(err)
        it += 1
    semilogy(errors)
    print(it)
    print(x_new)
    print(err)
    return errors

%time errors_newton = newton(f,f_prime,1.0,eps,n_max)
```

## Chord method

$$
q^k \equiv q = \frac{f(b)-f(a)}{b-a}
$$

$$
x^{k+1} = x^k - \frac{f(x^k)}{q}
$$

```python
def chord(f,a,b,x0,eps,n_max):
    x_new = x0
    err = eps + 1.
    errors = [err]
    it = 0
    while (err > eps and it < n_max):
        x_new = x_new - (f(x_new)/((f(b) - f(a)) / (b - a)))
        err = abs(f(x_new))
        errors.append(err)
        it += 1
    semilogy(errors)
    print(it)
    print(x_new)
    print(err)
    return errors

errors_chord = chord (f,a,b,x0,eps,n_max)
```

## Secant method

$$
q^k = \frac{f(x^k)-f(x^{k-1})}{x^k - x^{k-1}}
$$

$$
x^{k+1} = x^k - \frac{f(x^k)}{q^k}
$$

Note that this algorithm requirs **two** initial points

```python
def secant(f,x0,x00,eps,n_max):
    xk = x00
    x_new = x0
    err = eps + 1.
    errors = [err]
    it = 0
    while (err > eps and it < n_max):
        temp = x_new
        x_new = x_new - (f(x_new)/((f(x_new)-f(xk))/(x_new - xk)))
        xk = temp
        err = abs(f(x_new))
        errors.append(err)
        it += 1
    semilogy(errors)
    print(it)
    print(x_new)
    print(err)
    return errors

errors_secant = secant(f,x0,x00,eps,n_max)
```

## Fixed point iterations

$$
f(x)=0 \to x-\phi(x)=0
$$

$$
x^{k+1} = \phi(x^k)
$$

```python
def fixed_point(phi,x0,eps,n_max):
    x_new = x0
    err = eps + 1.
    errors = [err]
    it = 0
    while (err > eps and it < n_max):
        x_new = phi(x_new)
        err = abs(f(x_new))
        errors.append(err)
        it += 1
    semilogy(errors)
    print(it)
    print(x_new)
    print(err)
    return errors

errors_fixed = fixed_point(phi,0.3,eps,n_max)

```

## Comparison

```python
# plot the error convergence for the methods
loglog(errors_bisect, label='bisect')
loglog(errors_chord, label='chord')
loglog(errors_secant, label='secant')
loglog(errors_newton, label ='newton')
loglog(errors_fixed, label ='fixed')
_ = legend()
```

```python
# Let's compare the scipy implmentation of Newton's method with our..
```

```python
import scipy.optimize as opt
%time opt.newton(f, 1.0, f_prime, tol = eps)
```

    CPU times: user 12 µs, sys: 11 µs, total: 23 µs
    Wall time: 26.5 µs

    0.906179845938664

We see that the scipy method is 1000 times slower than the `scipy` one

```python

```


# INF-495, v0.01, Claudio Torres, ctorres@inf.utfsm.cl. DI-UTFSM
## Textbook: Computational Mathematical Modeling, An Integrated Approach Across Scales by Daniela Calvetti and Erkki Somersalo.

# Chapter 8

## 8.1 Markov process and random walk

```python
import numpy as np
import scipy.sparse.linalg as sp
import sympy as sym
from scipy.linalg import toeplitz

import ipywidgets as widgets
from ipywidgets import IntSlider

import matplotlib.pyplot as plt
%matplotlib inline
from matplotlib import cm
from matplotlib.ticker import LinearLocator, FormatStrFormatter

plt.style.use('ggplot')

import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
sym.init_printing()

from scipy.integrate import odeint
from ipywidgets import interact

def plot_matrices_with_values2(ax,M,flag_values=False):
    N=M.shape[0]
    cmap = plt.get_cmap('GnBu')
    ax.matshow(M, cmap=cmap)
    if flag_values:
        for i in np.arange(0, N):
            for j in np.arange(0, N):
                # I had to change the order of indices to make it right
                ax.text(i, j, '{:.2f}'.format(M[j,i]), va='center', ha='center', color='r')
```

```python
ids = np.arange(64) # Making it Python compatible
I = np.reshape(ids,(8,8),order='F')
print(I)
# Defining occupied squares ([row,columns])
occupied = np.array([[2, 3], [5, 5], [6, 2], [6, 3]])-1 # Subtracting one to make it Python compatible!
print(occupied)
for oc in occupied:
    print(I[oc[0],oc[1]])
```

    [[ 0  8 16 24 32 40 48 56]
     [ 1  9 17 25 33 41 49 57]
     [ 2 10 18 26 34 42 50 58]
     [ 3 11 19 27 35 43 51 59]
     [ 4 12 20 28 36 44 52 60]
     [ 5 13 21 29 37 45 53 61]
     [ 6 14 22 30 38 46 54 62]
     [ 7 15 23 31 39 47 55 63]]
    [[1 2]
     [4 4]
     [5 1]
     [5 2]]
    17
    36
    13
    21

```python
# Defining the peice
piece = 'Knight' # 'King' or 'Knight'
if piece == 'King':
    # Squares relative to the current one reached by the King
    Mask = np.zeros((8,2))
    k=0
    piece_range=[-1,0,1]
    for i in piece_range:
        for j in piece_range:
            if not ((i==0) and (j==0)):
                Mask[k,0]=i
                Mask[k,1]=j
                k=k+1
    print(Mask)
elif piece == 'Knight':
    # Squares relative to the current one reached by the Knight
    Mask = np.zeros((8,2))
    k=0
    piece_range=[-2,-1,0,1,2]
    for i in [-2,-1,0,1,2]:
        for j in [-2,-1,0,1,2]:
            if np.linalg.norm([i,j],ord=1)==3:
                Mask[k,0]=i
                Mask[k,1]=j
                k=k+1
    print(Mask)
else:
    print('Wrong piece!')
```

    [[-2. -1.]
     [-2.  1.]
     [-1. -2.]
     [-1.  2.]
     [ 1. -2.]
     [ 1.  2.]
     [ 2. -1.]
     [ 2.  1.]]

```python
M = np.zeros((64,64))
# Scan through all starting squares
# find all possible destinations
for n in np.arange(8):
    for m in np.arange(8):
        k=I[n,m] # Index to current square
        # Find possible destination square using the mask
        P = np.array([n, m])+Mask
        # Discard occupied squares
        P_tmp=[]
        for p in P:
            flag=0
            for oc_sq in occupied:
                if abs(oc_sq[0]-p[0])+abs(oc_sq[1]-p[1])<1e-10:
                    flag=1
            if flag==0:
                P_tmp.append(p)
        P=np.array(P_tmp,dtype=int)
        # Discarding squares outside the board
        P_tmp=[]
        for p in P:
            if p[0]>=0 and p[0]<=7 and p[1]>=0 and p[1]<=7:
                P_tmp.append(p)
        P=np.array(P_tmp,dtype=int)
        nP = P.shape[0] # number of possible destinations
        # Fill in the probabilities
        for p in P:
            Ip = I[p[0],p[1]] # Destination index
            M[Ip,k] = 1/nP # Moving probability
```

```python
fig,  (ax1)   = plt.subplots(1, 1, figsize=(10,10))
plot_matrices_with_values2(ax1,M,False)
```

## First algorithm: Random walk. Now I added the plot of the resulting probabilities, it is interesting to observe the evolution from the beginning. Compare with the other two outcomes.

```python
def plot_random_walk_output(nsteps=1):
    np.random.seed(0)
    # nsteps = 100000
    prev   = I[1,2]
    ncounts = np.zeros(64)
    squares_ids = [prev]
    for n in np.arange(nsteps):
        # Random draw using the column M[:,prev]
        next_step = np.random.choice(np.arange(64),p=M[:,prev])
        ncounts[next_step] = ncounts[next_step] + 1
        prev = next_step
        squares_ids.append(next_step)
    fig,  (ax1)   = plt.subplots(1, 1, figsize=(10,10))
    plot_matrices_with_values2(ax1,np.log10(np.reshape(ncounts/np.sum(ncounts),(8,8),order='F')),True)
    # print(ncounts)
    # print(squares_ids)
    # print(I)
interact(plot_random_walk_output,nsteps=(0,500000,1))
```

    interactive(children=(IntSlider(value=1, description='nsteps', max=500000), Output()), _dom_classes=('widget-i…

    <function __main__.plot_random_walk_output(nsteps=1)>

## Second algorithm: Power iteration. Warning: For the case of the Knight, this algorithm shows an interesting behavior and that seems to be related there are two eigenvalues with the magnitud equal to 1, one of the is actually equal to 1 but the other is equally to -1. This does not happen for the case of the King.

```python
def plot_P_n(nj, I, M):
    # Defining initial distribution
    (i0,j0)=(0,1)
    pcurr = np.zeros(64)
    pcurr[I[i0,j0]] = 1
    for j in np.arange(nj):
        pcurr = np.dot(M,pcurr)
    pcurrM = np.reshape(pcurr,(8,8),order='F')
    fig,  (ax1)   = plt.subplots(1, 1, figsize=(10,10))
    plot_matrices_with_values2(ax1,np.log10(pcurrM),True)
    # print(pcurr)
interact(lambda nj=0: plot_P_n(nj, I=I, M=M), nj = (0,2000,1))
```

    interactive(children=(IntSlider(value=0, description='nj', max=2000), Output()), _dom_classes=('widget-interac…

    <function __main__.<lambda>(nj=0)>

## Third algorithm: Eigenvalues and eigenvectors. We need to look for the eigenvector associated to the eigenvalue equal to 1.

```python
w, v = np.linalg.eig(M)
```

```python
w
```

    array([-1.00000000e+00+0.00000000e+00j,  1.00000000e+00+0.00000000e+00j,
           -7.50632749e-01+0.00000000e+00j, -7.27823791e-01+0.00000000e+00j,
           -7.01486703e-01+0.00000000e+00j,  7.50632749e-01+0.00000000e+00j,
            7.27823791e-01+0.00000000e+00j,  7.01486703e-01+0.00000000e+00j,
           -6.48633862e-01+0.00000000e+00j,  6.48633862e-01+0.00000000e+00j,
           -6.09881975e-01+0.00000000e+00j, -5.81559313e-01+0.00000000e+00j,
           -5.60397014e-01+0.00000000e+00j, -5.50942218e-01+0.00000000e+00j,
           -5.25497291e-01+0.00000000e+00j, -4.93732549e-01+0.00000000e+00j,
           -4.82696191e-01+0.00000000e+00j, -4.34951590e-01+0.00000000e+00j,
           -3.99042432e-01+0.00000000e+00j, -3.75078686e-01+0.00000000e+00j,
           -3.65281851e-01+0.00000000e+00j, -3.37548113e-01+0.00000000e+00j,
           -3.14489772e-01+0.00000000e+00j, -2.90154101e-01+0.00000000e+00j,
           -2.79989305e-01+0.00000000e+00j, -2.53118528e-01+0.00000000e+00j,
           -2.57620679e-01+0.00000000e+00j, -2.20992118e-01+0.00000000e+00j,
           -1.58521297e-01+0.00000000e+00j, -1.35425433e-01+0.00000000e+00j,
           -1.17147883e-01+0.00000000e+00j, -5.97277053e-02+0.00000000e+00j,
            6.09881975e-01+0.00000000e+00j,  5.81559313e-01+0.00000000e+00j,
            5.60397014e-01+0.00000000e+00j,  5.50942218e-01+0.00000000e+00j,
            5.25497291e-01+0.00000000e+00j,  5.97277053e-02+0.00000000e+00j,
            4.93732549e-01+0.00000000e+00j,  4.82696191e-01+0.00000000e+00j,
            4.34951590e-01+0.00000000e+00j,  1.17147883e-01+0.00000000e+00j,
            1.35425433e-01+0.00000000e+00j,  1.58521297e-01+0.00000000e+00j,
            3.99042432e-01+0.00000000e+00j,  2.20992118e-01+0.00000000e+00j,
            3.75078686e-01+0.00000000e+00j,  3.65281851e-01+0.00000000e+00j,
            3.37548113e-01+0.00000000e+00j,  2.53118528e-01+0.00000000e+00j,
            2.57620679e-01+0.00000000e+00j,  2.79989305e-01+0.00000000e+00j,
            2.90154101e-01+0.00000000e+00j,  3.14489772e-01+0.00000000e+00j,
            4.21403376e-17+6.95691790e-17j,  4.21403376e-17-6.95691790e-17j,
            7.59141459e-17+0.00000000e+00j, -4.25731310e-17+0.00000000e+00j,
            3.36509268e-18+8.68465189e-18j,  3.36509268e-18-8.68465189e-18j,
            0.00000000e+00+0.00000000e+00j,  0.00000000e+00+0.00000000e+00j,
            0.00000000e+00+0.00000000e+00j,  0.00000000e+00+0.00000000e+00j])

```python
v0=np.real(v[:,0])/np.sum(np.real(v[:,0]))
print(v0)
```

    [-3.21099073e+13  9.63297220e+13 -9.63297220e+13  9.63297220e+13
     -9.63297220e+13  1.28439629e+14 -6.42198147e+13  3.21099073e+13
      9.63297220e+13 -1.28439629e+14  1.92659444e+14 -1.28439629e+14
      1.92659444e+14 -0.00000000e+00  1.28439629e+14 -6.42198147e+13
     -1.28439629e+14 -0.00000000e+00 -2.56879259e+14  1.92659444e+14
     -2.56879259e+14 -0.00000000e+00 -1.92659444e+14  9.63297220e+13
      1.28439629e+14 -1.92659444e+14  2.24769351e+14 -1.92659444e+14
      2.24769351e+14 -2.56879259e+14  1.28439629e+14 -9.63297220e+13
     -9.63297220e+13  1.92659444e+14 -2.24769351e+14  2.56879259e+14
     -0.00000000e+00  2.56879259e+14 -1.60549537e+14  1.28439629e+14
      1.28439629e+14 -1.92659444e+14  2.24769351e+14 -2.56879259e+14
      2.56879259e+14 -2.56879259e+14  1.60549537e+14 -1.28439629e+14
     -9.63297220e+13  1.28439629e+14 -1.92659444e+14  1.60549537e+14
     -1.92659444e+14  1.60549537e+14 -1.28439629e+14  9.63297220e+13
      6.42198147e+13 -9.63297220e+13  1.28439629e+14 -1.28439629e+14
      1.28439629e+14 -1.28439629e+14  9.63297220e+13 -6.42198147e+13]

```python
v1=np.real(v[:,1])/np.sum(np.real(v[:,1]))
print(v1)
```

    [ 0.0035461   0.0106383   0.0106383   0.0106383   0.0106383   0.0141844
      0.0070922   0.0035461   0.0106383   0.0141844   0.0212766   0.0141844
      0.0212766  -0.          0.0141844   0.0070922   0.0141844  -0.
      0.02836879  0.0212766   0.02836879 -0.          0.0212766   0.0106383
      0.0141844   0.0212766   0.0248227   0.0212766   0.0248227   0.02836879
      0.0141844   0.0106383   0.0106383   0.0212766   0.0248227   0.02836879
     -0.          0.02836879  0.0177305   0.0141844   0.0141844   0.0212766
      0.0248227   0.02836879  0.02836879  0.02836879  0.0177305   0.0141844
      0.0106383   0.0141844   0.0212766   0.0177305   0.0212766   0.0177305
      0.0141844   0.0106383   0.0070922   0.0106383   0.0141844   0.0141844
      0.0141844   0.0141844   0.0106383   0.0070922 ]

```python
if piece == 'King':
    v_selected = v0
elif piece == 'Knight':
    v_selected = v1
else:
    print('Wrong piece!')
    v_selected=np.full((64),np.nan)
fig,  (ax1)   = plt.subplots(1, 1, figsize=(10,10))
plot_matrices_with_values2(ax1,np.log10(np.reshape(np.real(v_selected),(8,8),order='F')),True)
```

# 8.2 Stochastic predator-prey model

```python
def plot_S_Pred_Prey(seed=0):
    Nj       = 200 # number of X adult pair at t = t(j)
    Mj       = 50  # number of Y adult pair at t = t(j)
    p_mean   = 10  # mean number of X offsprings/adult pair
    p_std    = 3   # std of the number of the X offspring/adult pair
    beta     = 0.3 # Survival probability of X in the absence of Y
    Mstar    = 50  # number of Y that halves the survival prob. of X
    q_mean   = 4   # mean number of Y offspring/adult pair
    q_std    = 2   # std of the number of the Y offsrping/adult pair
    gamma    = 0.6 # Survival prob. of Y in adundance of X
    Nstar    = 400 # Number of X pairs at which the survival prob. of Y is halved

    nrounds  = 40  # Number of generations allowed

    N = np.full(nrounds,np.nan)
    N[0] = Nj
    M = np.full(nrounds,np.nan)
    M[0] = Mj
    Surv = np.full((nrounds-1,2),np.nan)

    np.random.seed(seed)

    for j in np.arange(1,nrounds):
        # Draw the number of offsprings of species X
        Pj = Nj*p_mean+np.sqrt(Nj)*p_std*np.random.randn()
        # Survival probability of the offspring of species X
        s = beta/(1+(Mj/Mstar)**2)
        Nj = s*Pj+np.sqrt(s*Pj*(1-s))*np.random.randn()
        N[j] = 0.5*Nj
        # Draw the number of offsprings of species Y
        Qj = Mj*q_mean+np.sqrt(Mj)*q_std*np.random.randn()
        # Survival probability of the offspring of species Y
        r = gamma*(1-1/(1+(Nj/Nstar)**2))
        Mj = r*Qj+np.sqrt(r*Qj*(1-r))*np.random.randn()
        M[j] = 0.5*Mj
        # Saving the survival probabilities for later analysis
        Surv[j-1,:]=[s,r]

    s=Surv[:,0]
    r=Surv[:,1]

    smean = np.sum(s)/(nrounds-1)
    rmean = np.sum(r)/(nrounds-1)
    Surv_c = Surv-np.array([smean,rmean])
    covSurv = np.dot(Surv_c.T,Surv_c)/(nrounds-1)
    corrcoeff = covSurv[0,1]/np.sqrt(covSurv[0,0]*covSurv[1,1])

    fig,  ((ax1,ax2),(ax3,ax4))   = plt.subplots(2, 2, figsize=(10,10))

    ax4.plot(N,'r.-')
    ax4.plot(M,'b.-')
    ax4.grid(True)

    s=Surv[:,0]
    r=Surv[:,1]
    ax1.plot(s,r,'.')
    ax1.set_xlim(0,1)
    ax1.set_ylim(0,1)
    ax1.set_xlabel(r'$s_i$')
    ax1.set_ylabel(r'$r_i$')
    ax1.text(0.5, 0.2, 'c={:.5f}'.format(corrcoeff), va='center', ha='center', color='k')

    ax2.hist(r,orientation='horizontal')
    ax2.set_ylim(0,1)

    ax3.hist(s)
    ax3.set_xlim(0,1)

    plt.show()
interact(plot_S_Pred_Prey,seed=(0,100,1))
```

```python

```