# First a little bit of statistics review:

# Variance

Variance is a measure of the spread of numbers in a dataset. Variance is the average of the squared differences from the mean. So naturally, you can't find the variance of something unless you calculate it's mean first. Lets get some data and find its variance.

```
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import random

# Lets generate two variables with 50 random integers each.
variance_one = []
variance_two = []
for x in range(50):
  variance_one.append(random.randint(25,75))
  variance_two.append(random.randint(0,100))

variance_data = {'v1': variance_one, 'v2': variance_two}

variance_df = pd.DataFrame(variance_data)
variance_df['zeros'] = pd.Series(list(np.zeros(50)))

variance_df.head()
```

      v1
      v2
      zeros

      0
      56
      11
      0.0

      1
      47
      29
      0.0

      2
      41
      8
      0.0

      3
      51
      93
      0.0

      4
      38
      31
      0.0

```
# Now some scatter plots

plt.scatter(variance_df.v1, variance_df.zeros)
plt.xlim(0,100)
plt.title("Plot One")
plt.show()

plt.scatter(variance_df.v2, variance_df.zeros)
plt.xlim(0,100)
plt.title("Plot Two")
plt.show()
```

Now I know this isn't complicated, but each of the above plots has the same number of points, but we can tell visually that "Plot Two" has the greater variance because its points are more spread out. What if we didn't trust our eyes though? Lets calculate the variance of each of these variables to prove it to ourselves

$\overline{X}$ is the symbol for the mean of the dataset.

$N$ is the total number of observations.

$v$ or variance is sometimes denoted by a lowercase v. But you'll also see it referred to as $\sigma^{2}$.

\begin{align}
v = \frac{\sum{(X_{i} - \overline{X})^{2}} }{N}
\end{align}

How do we calculate a simple average? We add up all of the values and then divide by the total number of values. this is why there is a sum in the numerator and N in the denomenator.

However in this calculation, we're not just summing the values like we would if we were calculateing the mean, we are summing the squared difference between each point and the mean. (The squared distance between each point in the mean.)

```
# Since we generated these random values in a range centered around 50, that's
# about where their means should be.

# Find the means for each variable
v1_mean = variance_df.v1.mean()
print("v1 mean: ", v1_mean)
v2_mean = variance_df.v2.mean()
print("v2 mean: ", v2_mean)

# Find the distance between each point and its corresponding mean

variance_df['v1_distance'] = variance_df.v1-v1_mean
variance_df['v2_distance'] = variance_df.v2-v2_mean

variance_df.head()
```

    v1 mean:  49.08
    v2 mean:  51.66

      v1
      v2
      zeros
      v1_distance
      v2_distance

      0
      56
      11
      0.0
      6.92
      -40.66

      1
      47
      29
      0.0
      -2.08
      -22.66

      2
      41
      8
      0.0
      -8.08
      -43.66

      3
      51
      93
      0.0
      1.92
      41.34

      4
      38
      31
      0.0
      -11.08
      -20.66

```
# Now we'll square the distances from the means
variance_df['v1_squared_distance'] = variance_df.v1_distance**2
variance_df['v2_squared_distance'] = variance_df.v2_distance**2

# Notice that squaring the distances turns all of our negative values into positive ones?

variance_df.head()
```

      v1
      v2
      zeros
      v1_distance
      v2_distance
      v1_squared_distance
      v2_squared_distance

      0
      56
      11
      0.0
      6.92
      -40.66
      47.8864
      1653.2356

      1
      47
      29
      0.0
      -2.08
      -22.66
      4.3264
      513.4756

      2
      41
      8
      0.0
      -8.08
      -43.66
      65.2864
      1906.1956

      3
      51
      93
      0.0
      1.92
      41.34
      3.6864
      1708.9956

      4
      38
      31
      0.0
      -11.08
      -20.66
      122.7664
      426.8356

```
# Now we'll sum the squared distances and divide by the number of observations.
observations = len(variance_df)
print("Number of Observations: ", observations)

Variance_One = variance_df.v1_squared_distance.sum()/observations
Variance_Two = variance_df.v2_squared_distance.sum()/observations

print("Variance One: ", Variance_One)
print("Variance Two: ", Variance_Two)
```

    Number of Observations:  50
    Variance One:  176.31360000000004
    Variance Two:  1032.2644

Woah, so what is the domain of V1 and V2?

Well, V1 goes from 25 to 75 so its range is ~50 and V2 goes from 0 to 100 so its range is about 100

So even though V2 is roughly twice as spread out, how much bigger is its variance than V1?

```
print("How many times bigger is Variance_One than Variance_Two? ", Variance_Two/Variance_One)

# About 3.86 times bigger! Why is that?
```

    How many times bigger is Variance_One than Variance_Two?  5.854706613670187

## A note about my code quality

Why did I go to the trouble of calculating all of that by hand, and add a bunch of extra useless rows to my dataframe? That is some bad code!

Because I wanted to make sure that you understood all of the parts of the equation. I didn't want the function to be some magic thing that you  put numbers in and out popped a variance. Taking time to understand the equation will reinforce your intuition about the spread of the data. After all, I could have just done this:

```
print(variance_df.v1.var(ddof=1))
print(variance_df.v2.var(ddof=1))
```

    179.91183673469385
    1053.3310204081633

But wait! Those variance values are different than the ones we calculated above, oh no! This is because variance is calculated slightly differently for a population vs a sample. Lets clarify this a little bit.

The **POPULATION VARIANCE** $\sigma^{2}$ is a **PARAMETER** (aspect, property, attribute, etc) of the population.

The **SAMPLE VARIANCE** $s^{2}$ is a **STATISTIC** (estimated attribute) of the sample.

We use the sample statistic to **estimate** the population parameter.

The sample variance $s^{2}$ is an estimate of the population variance $\sigma^{2}$.

Basically, if you're calculating a **sample** variance, you need to divide by $N-1$ or else your estimate will be a little biased. The equation that we were originally working from is for a **population variance**.

If we use the ddof=0 parameter (default is ddof=1) in our equation, we should get the same result. "ddof" stands for Denominator Degrees of Freedom.

```
print(variance_df.v1.var(ddof=0))
print(variance_df.v2.var(ddof=0))
```

    176.31359999999998
    1032.2644

# Standard Deviation

If you understand how variance is calculated, then standard deviation is a cinch. The standard deviation is the square root $\sqrt()$ of the variance.

## So why would we use one over the other?

Remember how we squared all of the distances from the mean before we added them all up? Well then taking the square root of the variance will put our measures back in the same units as the mean. So the Standard Deviation is a measure of spread of the data that is expressed in the same units as the mean of the data. Variance is the average squared distance from the mean, and the Standard Deviation is the average distance from the mean. You'll remember that when we did hypothesis testing and explored the normal distribution we talked in terms of standard deviations, and not in terms of variance for this reason.

```
print(variance_df.v1.std(ddof=0))
print(variance_df.v2.std(ddof=0))
```

    13.27831314587813
    32.128871751121295

# Covariance

Covariance is a measure of how changes in one variable are associated with changes in a second variable. It's a measure of how they Co (together) Vary (move) or how they move in relation to each other. For this topic we're not really going to dive into the formula, I just want you to be able to understand the topic intuitively. Since this measure is about two variables, graphs that will help us visualize things in two dimensions will help us demonstrate this idea. (scatterplots)

Lets look at the first scatterplot. the y variable has high values where the x variable has low values. This is a negative covariance because as one variable increases (moves), the other decreases (moves in the opposite direction).

In the second scatterplot we see no relation between high and low values of either variable, therefore this cloud of points has a near 0 covariance

In the third graph, we see that the y variable takes on low values in the same range where the x value takes on low values, and simiarly with high values. Because the areas of their high and low values match, we would expect this cloud of points to have a positive covariance.

Check out how popular this site is:

<https://tylervigen.com>

<https://www.similarweb.com/website/tylervigen.com#overview>

## Interpeting Covariance

A large positive or negative covariance indicates a strong relationship between two variables. However, you can't necessarily compare covariances between sets of variables that have a different scale, since the covariance of variables that take on high values will always be higher than  since covariance values are unbounded, they could take on arbitrarily high or low values. This means that you can't compare the covariances between variables that have a different scale. Two variablespositive covariance variable that has a large scale will always have a higher covariance than a variable with an equally strong relationship, yet smaller scale. This means that we need a way to regularlize

One of the challenges with covariance is that its value is unbounded and variables that take on larger values will have a larger covariance irrespective of

Let me show you what I mean:

```
a = [1,2,3,4,5,6,7,8,9]
b = [1,2,3,4,5,6,7,8,9]
c = [10,20,30,40,50,60,70,80,90]
d = [10,20,30,40,50,60,70,80,90]

fake_data = {"a": a, "b": b, "c": c, "d": d,}

df = pd.DataFrame(fake_data)

plt.scatter(df.a, df.b)
plt.xlim(0,100)
plt.ylim(0,100)
plt.show()

plt.scatter(df.c, df.d)
plt.xlim(0,100)
plt.ylim(0,100)
plt.show()
```

Which of the above sets of variables has a stronger relationship?

Which has the stronger covariance?

# The Variance-Covariance Matrix

In order to answer this problem we're going to use a tool called a variance-covariance matrix.

This is  matrix that compares each variable with every other variable in a dataset and returns to us variance values along the main diagonal, and covariance values everywhere else.

```
df.cov()
```

      a
      b
      c
      d

      a
      7.5
      7.5
      75.0
      75.0

      b
      7.5
      7.5
      75.0
      75.0

      c
      75.0
      75.0
      750.0
      750.0

      d
      75.0
      75.0
      750.0
      750.0

What type of special square matrix is the variance-covariance matrix?

The two sets of variables above show relationships that are equal in their strength, yet their covariance values are wildly different.

How can we counteract this problem?

What if there was some statistic of a distribution that represented how spread out the data was that we could use to standardize the units/scale of the variables?

# Correlation Coefficient

Well, it just so happens that we do have such a measure of spread of a variable. It's called the Standard Deviation! And we already learned about it. If we divide our covariance values by the product of the standard deviations of the two variables, we'll end up with what's called the Correlation Coefficient. (Sometimes just referred to as the correlation).

Correlation Coefficients have a fixed range from -1 to +1 with 0 representing no linear relationship between the data.

In most use cases the correlation coefficient is an improvement over measures of covariance because:

- Covariance can take on practically any number while a correlation is limited: -1 to +1.
- Because of it’s numerical limitations, correlation is more useful for determining how strong the relationship is between the two variables.
- Correlation does not have units. Covariance always has units
- Correlation isn’t affected by changes in the center (i.e. mean) or scale of the variables

[Statistics How To - Covariance](https://www.statisticshowto.datasciencecentral.com/covariance/)

The correlation coefficient is usually represented by a lower case $r$.

\begin{align}
r = \frac{cov(X,Y)}{\sigma_{X}\sigma_{Y}}
\end{align}

```
df.corr()
```

      a
      b
      c
      d

      a
      1.0
      1.0
      1.0
      1.0

      b
      1.0
      1.0
      1.0
      1.0

      c
      1.0
      1.0
      1.0
      1.0

      d
      1.0
      1.0
      1.0
      1.0

Correlation coefficients of 1 tell us that all of these varaibles have a perfectly linear positive correlation with one another.

Correlation and other sample statistics are somewhat limited in their ability to tell us about the shape/patterns in the data.

[Anscombe's Quartet](https://en.wikipedia.org/wiki/Anscombe%27s_quartet)

Or take it to the next level with the [Datasaurus Dozen](https://www.autodeskresearch.com/publications/samestats)

# Orthogonality

Orthogonality is another word for "perpendicularity" or things (vectors or matrices) existing at right angles to one another. Two vectors that are perpendicular to one another are orthogonal.

## How to tell if two vectors are orthogonal

Two vectors are orthogonal to each other if their dot product will be zero.

Lets look at a couple of examples to see this in action:

```
vector_1 = [0, 2]
vector_2 = [2, 0]

# Plot the Scaled Vectors
plt.arrow(0,0, vector_1[0], vector_1[1],head_width=.05, head_length=0.05, color ='red')
plt.arrow(0,0, vector_2[0], vector_2[1],head_width=.05, head_length=0.05, color ='green')
plt.xlim(-1,3)
plt.ylim(-1,3)
plt.title("Orthogonal Vectors")
plt.show()
```

Clearly we can see that the above vectors are perpendicular to each other, what does the formula say?

\begin{align}
a = \begin{bmatrix} 0 & 2\end{bmatrix}
\qquad
b = \begin{bmatrix} 2 & 0\end{bmatrix}
\\
a \cdot b = (0)(2) + (2)(0) = 0
\end{align}

```
vector_1 = [-2, 2]
vector_2 = [2, 2]

# Plot the Scaled Vectors
plt.arrow(0,0, vector_1[0], vector_1[1],head_width=.05, head_length=0.05, color ='red')
plt.arrow(0,0, vector_2[0], vector_2[1],head_width=.05, head_length=0.05, color ='green')
plt.xlim(-3,3)
plt.ylim(-1,3)
plt.title("Orthogonal Vectors")
plt.show()
```

Again the dot product is zero.

\begin{align}
a = \begin{bmatrix} -2 & 2\end{bmatrix}
\qquad
b = \begin{bmatrix} 2 & 2\end{bmatrix}
\\
a \cdot b = (-2)(2) + (2)(2) = 0
\end{align}

# Unit Vectors

In Linear Algebra a unit vector is any vector of "unit length" (1). You can turn any non-zero vector into a unit vector by dividing it by its norm (length/magnitude).

for example if I have the vector

\begin{align}
 b = \begin{bmatrix} 1 \\ 2 \\  2 \end{bmatrix}
\end{align}

 and I want to turn it into a unit vector, first I will calculate its norm

 \begin{align}
 ||b|| = \sqrt{1^2 + 2^2 + 2^2} = \sqrt{1 + 4 + 4} = \sqrt{9} = 3
\end{align}

I can turn b into a unit vector by dividing it by its norm. Once something has been turned into a unit vector we'll put a ^ "hat" symbol over it to denote that it is now a unit vector.

 \begin{align}
 \hat{b} = \frac{1}{||b||}b = \frac{1}{3}\begin{bmatrix} 1 \\ 2 \\  2 \end{bmatrix} = \begin{bmatrix} \frac{1}{3} \\ \frac{2}{3} \\  \frac{2}{3} \end{bmatrix}
\end{align}

You might frequently see mentioned the unit vectors used to denote a certain dimensional space.

$\mathbb{R}$ unit vector: $\hat{i} = \begin{bmatrix} 1 \end{bmatrix}$

$\mathbb{R}^2$ unit vectors: $\hat{i} = \begin{bmatrix} 1 \\ 0 \end{bmatrix}$, $\hat{j} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}$

$\mathbb{R}^3$ unit vectors: $\hat{i} = \begin{bmatrix} 1 \\ 0 \\ 0 \end{bmatrix}$, $\hat{j} = \begin{bmatrix} 0 \\ 1 \\ 0 \end{bmatrix}$,  $\hat{k} = \begin{bmatrix} 0 \\ 0 \\ 1 \end{bmatrix}$

You'll notice that in the corresponding space, these basis vectors are the rows/columns of the identity matrix.

```
# Axis Bounds
plt.xlim(-1,2)
plt.ylim(-1,2)

# Unit Vectors
i_hat = [1,0]
j_hat = [0,1]

# Fix Axes
plt.axes().set_aspect('equal')

# PLot Vectors
plt.arrow(0, 0, i_hat[0], i_hat[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')
plt.arrow(0, 0, j_hat[0], j_hat[1], linewidth=3, head_width=.05, head_length=0.05, color ='blue')
plt.title("basis vectors in R^2")
plt.show()
```

## Vectors as linear combinations of scalars and unit vectors

Any vector (or matrix) can be be described in terms of a linear combination of scaled unit vectors. Lets look at an example.

\begin{align}
c = \begin{bmatrix} 2 \\ 3 \end{bmatrix}
\end{align}

We think about a vector that starts at the origin and extends to point $(2,3)$

Lets rewrite this in terms of a linear combination of scaled unit vectors:

\begin{align}
c = \begin{bmatrix} 2 \\ 3 \end{bmatrix} = 2\begin{bmatrix} 1 \\ 0 \end{bmatrix} + 3\begin{bmatrix} 0 \\ 1 \end{bmatrix} = 2\hat{i} + 3\hat{j}
\end{align}

This says that matrix $\begin{bmatrix} 2 \\ 3 \end{bmatrix}$ will result from scaling the $\hat{i}$ unit vector by 2, the $\hat{j}$ vector by 3 and then adding the two together.

We can describe any vector in $\mathbb{R}^2$ in this way. Well, we can describe any vector in any dimensionality this way provided we use all of the unit vectors for that space and scale them all appropriately. In this examply we just happen to be using a vector whose dimension is 2.

# Span

The span is the set of all possible vectors that can be created with a linear combination of two vectors (just as we described above).

A linear combination of two vectors just means that we're composing to vectors (via addition or subtraction) to create a new vector.

## Linearly Dependent Vectors

Two vectors that live on the same line are what's called linearly dependent. This means that there is no linear combination (no way to add, or subtract scaled version of these vectors from each other) that will ever allow us to create a vector that lies outside of that line.

In this case, the span of these vectors (lets say the green one and the red one for example - could be just those two or a whole set) is the line that they lie on, since that's what can be produced by scaling and composing them together.

The span is the graphical area that we're able to cover via a linear combination of a set of vectors.

## Linearly Independent Vectors

Linearly independent vectors are vectors that don't lie on the same line as each other. If two vectors are linearly independent, then there ought to be some linear combination of them that could represent any vector in the space ($\mathbb{R}^2$ in this case).

```
# Plot Linearly Dependent Vectors

# Axis Bounds
plt.xlim(-1.1,4)
plt.ylim(-1.1,4)

# Original Vector
v = [1,0]

# Scaled Vectors
v2 = np.multiply(3, v)
v3 = np.multiply(-1,v)

# Get Vals for L
axes = plt.gca()
x_vals = np.array(axes.get_xlim())
y_vals = 0*x_vals

# Plot Vectors and L
plt.plot(x_vals, y_vals, '--', color='b', linewidth=1)
plt.arrow(0,0, v2[0], v2[1], linewidth=3, head_width=.05, head_length=0.05, color ='yellow')
plt.arrow(0,0, v[0], v[1], linewidth=3, head_width=.05, head_length=0.05, color ='green')
plt.arrow(0,0, v3[0], v3[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("Linearly Dependent Vectors")
plt.show()
```

```
# Plot Linearly Dependent Vectors

# Axis Bounds
plt.xlim(-2,3.5)
plt.ylim(-1,3)

# Original Vector
a = [-1.5,.5]
b = [3, 1]

# Plot Vectors
plt.arrow(0,0, a[0], a[1], linewidth=3, head_width=.05, head_length=0.05, color ='blue')
plt.arrow(0,0, b[0], b[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("Linearly Independent Vectors")
plt.show()
```

# Basis

The basis of a vector space $V$ is a set of vectors that are linearly independent and that span the vector space $V$.

A set of vectors spans a space if their linear combinations fill the space.

For example, the unit vectors in the "Linearly Independent Vectors" plot above form a basis for the vector space $\mathbb{R}^2$ becayse they are linearly independent and span that space.

## Orthogonal Basis

An orthogonal basis is a set of vectors that are linearly independent, span the vector space, and are orthogonal to each other. Remember that vectors are orthogonal if their dot product equals zero.

## Orthonormal Basis

An orthonormal basis is a set of vectors that are linearly independent, span the vector space, are orthogonal to eachother and each have unit length.

For more on this topic (it's thrilling, I know) you might research the Gram-Schmidt process -which is a method for orthonormalizing a set of vectors in an inner product space.

The unit vectors form an orthonormal basis for whatever vector space that they are spanning.

# Rank

The rank of a matrix is the dimension of the vector space spanned by its columns. Just because a matrix has a certain number of rows or columns (dimensionality) doesn't neccessarily mean that it will span that dimensional space. Sometimes there exists a sort of redundancy within the rows/columns of a matrix (linear dependence) that becomes apparent when we reduce a matrix to row-echelon form via Gaussian Elimination.

## Gaussian Elimination

Gaussian Elimination is a process that seeks to take any given matrix and reduce it down to what is called "Row-Echelon form." A matrix is in Row-Echelon form when it has a 1 as its leading entry (furthest left) in each row, and zeroes at every position below that main entry. These matrices will usually wind up as a sort of upper-triangular matrix (not necessarly square) with ones on the main diagonal.

Gaussian Elimination takes a matrix and converts it to row-echelon form by doing combinations of three different row operations:

1) You can swap any two rows

2) You can multiply entire rows by scalars

3) You can add/subtract rows from each other

This takes some practice to do by hand but once mastered becomes the fastest way to find the rank of a matrix.

For example lets look at the following matrix:

\begin{align}
 P = \begin{bmatrix}
  1 & 0 & 1 \\
  -2 & -3 & 1 \\
  3 & 3 & 0
 \end{bmatrix}
\end{align}

Now, lets use gaussian elimination to get this matrix in row-echelon form

Step 1: Add 2 times the 1st row to the 2nd row

\begin{align}
 P = \begin{bmatrix}
  1 & 0 & 1 \\
  0 & -3 & -3 \\
  3 & 3 & 0
 \end{bmatrix}
\end{align}

Step 2: Add -3 times the 1st row to the 3rd row

\begin{align}
 P = \begin{bmatrix}
  1 & 0 & 1 \\
  0 & -3 & 3 \\
  0 & 3 & -3
 \end{bmatrix}
\end{align}

Step 3: Multiply the 2nd row by -1/3

\begin{align}
 P = \begin{bmatrix}
  1 & 0 & 1 \\
  0 & 1 & -1 \\
  0 & 3 & -3
 \end{bmatrix}
\end{align}

Step 4: Add -3 times the 2nd row to the 3rd row

\begin{align}
 P = \begin{bmatrix}
  1 & 0 & 1 \\
  0 & 1 & -1 \\
  0 & 0 & 0
 \end{bmatrix}
\end{align}

Now that we have this in row-echelon form we can see that we had one row that was linearly dependent (could be composed as a linear combination of other rows). That's why we were left with a row of zeros in place of it. If we look closely we will see that the first row equals the second row plus the third row.

Because we had two rows with leading 1s (these are called pivot values) left after the matrix was in row-echelon form, we know that its Rank is 2.

What does this mean? This means that even though the original matrix is a 3x3 matrix, it can't span $\mathbb{R}^3$, only $\mathbb{R}^2$

# Linear Projections in $\mathbb{R}^{2}$

Assume that we have some line $L$ in $\mathbb{R}^{2}$.

```
# Plot a line
plt.xlim(-1,4)
plt.ylim(-1,4)
axes = plt.gca()
x_vals = np.array(axes.get_xlim())
y_vals = 0*x_vals
plt.plot(x_vals, y_vals, '--', color='b')
plt.title("A Line")
plt.show()
```

We know that if we have a vector $v$ that lies on that line, if we scale that vector in any direction, the resulting vectors can only exist on that line.

```
# Plot a line

# Axis Bounds
plt.xlim(-1.1,4)
plt.ylim(-1.1,4)

# Original Vector
v = [1,0]

# Scaled Vectors
v2 = np.multiply(3, v)
v3 = np.multiply(-1,v)

# Get Vals for L
axes = plt.gca()
x_vals = np.array(axes.get_xlim())
y_vals = 0*x_vals

# Plot Vectors and L
plt.plot(x_vals, y_vals, '--', color='b', linewidth=1)
plt.arrow(0,0, v2[0], v2[1], linewidth=3, head_width=.05, head_length=0.05, color ='yellow')
plt.arrow(0,0, v[0], v[1], linewidth=3, head_width=.05, head_length=0.05, color ='green')
plt.arrow(0,0, v3[0], v3[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("v scaled two different ways")
plt.show()
```

Lets call the green vector $v$

This means that line $L$ is equal to vector $v$ scaled by all of the potential scalars in $\mathbb{R}$. We can represent this scaling factor by a constant $c$. Therefore, line $L$ is vector $v$ scaled by any scalar $c$.

\begin{align}
L = cv
\end{align}

Now, say that we have a second vector $w$ that we want to "project" onto line L

```
# Plot a line

# Axis Bounds
plt.xlim(-1.1,4)
plt.ylim(-1.1,4)

# Original Vector
v = [1,0]
w = [2,2]

# Get Vals for L
axes = plt.gca()
x_vals = np.array(axes.get_xlim())
y_vals = 0*x_vals

# Plot Vectors and L
plt.plot(x_vals, y_vals, '--', color='b', linewidth=1)
plt.arrow(0, 0, v[0], v[1], linewidth=3, head_width=.05, head_length=0.05, color ='green')
plt.arrow(0, 0, w[0], w[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("vector w")
plt.show()
```

## Projection as a shadow cast onto the target vector at a right angle

This is the intuition that I want you to develop. Imagine that we are shining a light down onto lin $L$ from a direction that is exactly orthogonal to it. In this case shining a light onto $L$ from a direction that is orthogonal to it is as if we were shining a light down from directly above. How long will the shadow be?

Imagine that you're **projecting** light from above to cast a shadow onto the x-axis.

Well since $L$ is literally the x-axis you can probably tell that the length of the projection of $w$ onto $L$ is 2.

A projection onto an axis is the same as just setting the variable that doesn't match the axis to 0. in our case the coordinates of vector $w$ is $(2,2)$ so it projects onto the x-axis at (2,0) -> just setting the y value to 0.

### Notation

In linear algebra we write the projection of w onto L like this:

\begin{align}proj_{L}(\vec{w})\end{align}

```
# Axis Bounds
plt.xlim(-1.1,4)
plt.ylim(-1.1,4)

# Original Vector
v = [1,0]
w = [2,2]
proj = [2,0]

# Get Vals for L
axes = plt.gca()
x_vals = np.array(axes.get_xlim())
y_vals = 0*x_vals

# Plot Vectors and L
plt.plot(x_vals, y_vals, '--', color='b', linewidth=1)
plt.arrow(0, 0, proj[0], proj[1], linewidth=3, head_width=.05, head_length=0.05, color ='gray')
plt.arrow(0, 0, v[0], v[1], linewidth=3, head_width=.05, head_length=0.05, color ='green')
plt.arrow(0, 0, w[0], w[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("Shadow of w")
plt.show()
```

The problem here is that we can't just draw a vector and call it a day, we can only define that vector in terms of our $v$ (green) vector.

Our gray vector is defined as:

\begin{align}
cv = proj_{L}(w)
\end{align}

But what if $L$ wasn't on the x-axis? How would calculate the projection?

```
# Axis Bounds
plt.xlim(-1.1,4)
plt.ylim(-1.1,4)

# Original Vector
v = [1,1/2]
w = [2,2]
proj = np.multiply(2.4,v)

# Set axes
axes = plt.gca()
plt.axes().set_aspect('equal')

# Get Vals for L
x_vals = np.array(axes.get_xlim())
y_vals = 1/2*x_vals

# Plot Vectors and L
plt.plot(x_vals, y_vals, '--', color='b', linewidth=1)
plt.arrow(0, 0, proj[0], proj[1], linewidth=3, head_width=.05, head_length=0.05, color ='gray')
plt.arrow(0, 0, v[0], v[1], linewidth=3, head_width=.05, head_length=0.05, color ='green')
plt.arrow(0, 0, w[0], w[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("non x-axis projection")
plt.show()
```

Remember, that it doesn't matter how long our $v$ (green) vectors is, we're just looking for the c value that can scale that vector to give us the gray vector $proj_{L}(w)$.

```
# Axis Bounds
plt.xlim(-1.1,4)
plt.ylim(-1.1,4)

# Original Vector
v = [1,1/2]
w = [2,2]
proj = np.multiply(2.4,v)
x_minus_proj = w-proj

# Set axes
axes = plt.gca()
plt.axes().set_aspect('equal')

# Get Vals for L
x_vals = np.array(axes.get_xlim())
y_vals = 1/2*x_vals

# Plot Vectors and L
plt.plot(x_vals, y_vals, '--', color='b', linewidth=1)
plt.arrow(0, 0, proj[0], proj[1], linewidth=3, head_width=.05, head_length=0.05, color ='gray')
plt.arrow(0, 0, v[0], v[1], linewidth=3, head_width=.05, head_length=0.05, color ='green')
plt.arrow(0, 0, w[0], w[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')
plt.arrow(proj[0], proj[1], x_minus_proj[0], x_minus_proj[1], linewidth=3, head_width=.05, head_length=0.05, color = 'yellow')

plt.title("non x-axis projection")
plt.show()
```

Lets use a trick. We're going to imagine that there is yellow vector that is orthogonal to $L$, that starts at the tip of our projection (gray) and ends at the tip of $w$ (red).

### Here's the hard part

This may not be intuitive, but we can define that yellow vector as $w-proj_{L}(w)$. Remember how two vectors added together act like we had placed one at the end of the other? Well this is the opposite, if we take some vector and subtract another vector, the tip moves to the end of the subtracted vector.

Since we defined $proj_{L}(w)$ as $cv$ (above). We then rewrite the yellow vector as:

\begin{align}
yellow = w-cv
\end{align}

Since we know that our yellow vector is orthogonal to $v$ we can then set up the following equation:

\begin{align}
v \cdot (w-cv) = 0
\end{align}

(remember that the dot product of two orthogonal vectors is 0)

Now solving for $c$ we get

1) Distribute the dot product

\begin{align}
v \cdot w - c(v \cdot v) = 0
\end{align}

2) add $c(v \cdot v)$ to both sides

\begin{align}
v \cdot w = c(v \cdot v)
\end{align}

3) divide by $v \cdot v$

\begin{align}
c = \frac{w \cdot v}{v \cdot v}
\end{align}

Since $cv = proj_{L}(w)$ we know that:

\begin{align}
proj_{L}(w) =  \frac{w \cdot v}{v \cdot v}v
\end{align}

This is the equation for the projection of any vector $w$ onto any line $L$!

Think about if we were trying to project an already orthogonal vector onto a line:

```
# Axis Bounds
plt.xlim(-1.1,4)
plt.ylim(-1.1,4)

# Original Vector
# v = [1,0]
w = [0,2]
proj = [2,0]

# Get Vals for L
axes = plt.gca()
x_vals = np.array(axes.get_xlim())
y_vals = 0*x_vals

# Plot Vectors and L
plt.plot(x_vals, y_vals, '--', color='b', linewidth=1)
plt.arrow(0, 0, w[0], w[1], linewidth=3, head_width=.05, head_length=0.05, color ='red')

plt.title("Shadow of w")
plt.show()
```

Now that you have a feel for linear projections, you can see that the $proj_{L}(w)$ is 0 mainly because $w \cdot v$ is 0.

Why have I gone to all of this trouble to explain linear projections? Because I think the intuition behind it is one of the most important things to grasp in linear algebra. We can find the shortest distance between some data point (vector) and a line best via an orthogonal projection onto that line. We can now move data points onto any given line and be certain that they move as little as possible from their original position.

The square of the norm of a vector is equivalent to the dot product of a vector with itself.

The dot product of a vector and itself can be rewritten as that vector times the transpose of itself.


# Linear Algebra and Python Basics

by Rob Hicks http://rlhick.people.wm.edu/stories/linear-algebra-python-basics.html

In this chapter, I will be discussing some linear algebra basics that will provide sufficient linear algebra background for effective programming in Python for our purposes.  We will be doing very basic linear algebra that by no means covers the full breadth of this topic.  Why linear algebra?  Linear algebra allows us to express relatively complex linear expressions in a very compact way.

Being comfortable with the rules for scalar and matrix addition, subtraction, multiplication, and division (known as inversion) is important for our class.

Before we can implement any of these ideas in code, we need to talk a bit about python and how data is stored.

## Python Primer

There are numerous ways to run python code.  I will show you two and both are easily accessible after installing Anaconda:

1. The Spyder integrated development environment.  The major advantages of Spyder is that it provides a graphical way for viewing matrices, vectors, and other objects you want to check as you work on a problem.  It also has the most intuitive way of debugging code.

    Spyder looks like this:

    Code can be run by clicking the green arrow (runs the entire file) or by blocking a subset and running it.
    In Windows or Mac, you can launch the Spyder by looking for the icon in the newly installed Program Folder Anaconda.

2. The Ipython Notebook (now called Jupyter).  The major advantages of this approach is that you use your web browser for all of your python work and you can mix code, videos, notes, graphics from the web, and mathematical notation to tell the whole story of your python project. In fact, I am using the ipython notebook for writing these notes.
    The Ipython Notebook looks like this:

    In Windows or Mac, you can launch the Ipython Notebook by looking in the newly installed Program Folder Anaconda.

In my work flow, I usually only use the Ipython Notebook, but for some coding problems where I need access to the easy debugging capabilities of Spyder, I use it.  We will be using the Ipython Notebook interface (web browser) mostly in this class.

### Loading libraries

The python universe has a huge number of libraries that extend the capabilities of python. Nearly all of these are open source, unlike packages like stata or matlab where some key libraries are proprietary (and can cost lots of money).  In lots of my code, you will see this at the top:

```python
%matplotlib inline
##import sympy as sympy
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sbn
from scipy import *
```

This code sets up Ipython Notebook environments (lines beginning with `%`), and loads several libraries and functions.  The core scientific stack in python consists of a number of free libraries.  The ones I have loaded above include:

1. sympy: provides for symbolic computation (solving algebra problems)
2. numpy: provides for linear algebra computations
3. matplotlib.pyplot: provides for the ability to graph functions and draw figures
4. scipy: scientific python provides a plethora of capabilities
5. seaborn: makes matplotlib figures even pretties (another library like this is called bokeh).  This is entirely optional and is purely for eye candy.

### Creating arrays, scalars, and matrices in Python

Scalars can be created easily like this:

```python
x = .5
print x
```

    0.5

#### Vectors and Lists

The numpy library (we will reference it by np) is the workhorse library for linear algebra in python.  To creat a vector simply surround a python list ($[1,2,3]$) with the np.array function:

```python
x_vector = np.array([1,2,3])
print x_vector
```

    [1 2 3]

We could have done this by defining a python list and converting it to an array:

```python
c_list = [1,2]
print "The list:",c_list
print "Has length:", len(c_list)

c_vector = np.array(c_list)
print "The vector:", c_vector
print "Has shape:",c_vector.shape
```

    The list: [1, 2]
    Has length: 2
    The vector: [1 2]
    Has shape: (2,)

```python
z = [5,6]
print "This is a list, not an array:",z
print type(z)
```

    This is a list, not an array: [5, 6]
    <type 'list'>

```python
zarray = np.array(z)
print "This is an array, not a list",zarray
print type(zarray)
```

    This is an array, not a list [5 6]
    <type 'numpy.ndarray'>

#### Matrices

```python
b = zip(z,c_vector)
print b
print "Note that the length of our zipped list is 2 not (2 by 2):",len(b)
```

    [(5, 1), (6, 2)]
    Note that the length of our zipped list is 2 not (2 by 2): 2

```python
print "But we can convert the list to a matrix like this:"
A = np.array(b)
print A
print type(A)
print "A has shape:",A.shape
```

    But we can convert the list to a matrix like this:
    [[5 1]
     [6 2]]
    <type 'numpy.ndarray'>
    A has shape: (2, 2)

## Matrix Addition and Subtraction

### Adding or subtracting a scalar value to a matrix

To learn the basics, consider a small matrix of dimension $2 \times 2$, where $2 \times 2$ denotes the number of rows $\times$ the number of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Consider adding a scalar value (e.g. 3) to the A.
$$
\begin{equation}
	A+3=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}+3
	=\begin{bmatrix}
	  a_{11}+3 & a_{12}+3 \\
	  a_{21}+3 & a_{22}+3
	\end{bmatrix}
\end{equation}
$$
The same basic principle holds true for A-3:
$$
\begin{equation}
	A-3=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}-3
	=\begin{bmatrix}
	  a_{11}-3 & a_{12}-3 \\
	  a_{21}-3 & a_{22}-3
	\end{bmatrix}
\end{equation}
$$
Notice that we add (or subtract) the scalar value to each element in the matrix A.  A can be of any dimension.

This is trivial to implement, now that we have defined our matrix A:

```python
result = A + 3
#or
result = 3 + A
print result
```

    [[8 4]
     [9 5]]

### Adding or subtracting two matrices
Consider two small $2 \times 2$ matrices, where $2 \times 2$ denotes the \# of rows $\times$ the \# of columns.  Let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr)$ and $B$=$\bigl( \begin{smallmatrix} b_{11} & b_{12} \\ b_{21} & b_{22} \end{smallmatrix} \bigr)$.  To find the result of $A-B$, simply subtract each element of A with the corresponding element of B:

$$
\begin{equation}
	A -B =
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix} -
	\begin{bmatrix} b_{11} & b_{12} \\
	  b_{21} & b_{22}
	\end{bmatrix}
	=
	\begin{bmatrix}
	  a_{11}-b_{11} & a_{12}-b_{12} \\
	  a_{21}-b_{21} & a_{22}-b_{22}
	\end{bmatrix}
\end{equation}
$$

Addition works exactly the same way:

$$
\begin{equation}
	A + B =
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix} +
	\begin{bmatrix} b_{11} & b_{12} \\
	  b_{21} & b_{22}
	\end{bmatrix}
	=
	\begin{bmatrix}
	  a_{11}+b_{11} & a_{12}+b_{12} \\
	  a_{21}+b_{21} & a_{22}+b_{22}
	\end{bmatrix}
\end{equation}
$$

An important point to know about matrix addition and subtraction is that it is only defined when $A$ and $B$ are of the same size.  Here, both are $2 \times 2$.  Since operations are performed element by element, these two matrices must be conformable- and for addition and subtraction that means they must have the same numbers of rows and columns.  I like to be explicit about the dimensions of matrices for checking conformability as I write the equations, so write

$$
A_{2 \times 2} + B_{2 \times 2}= \begin{bmatrix}
  a_{11}+b_{11} & a_{12}+b_{12} \\
  a_{21}+b_{21} & a_{22}+b_{22}
\end{bmatrix}_{2 \times 2}
$$

Notice that the result of a matrix addition or subtraction operation is always of the same dimension as the two operands.

Let's define another matrix, B, that is also $2 \times 2$ and add it to A:

```python
B = np.random.randn(2,2)
print B
```

    [[ 2.5056974   0.37029763]
     [ 0.94461604 -0.23399752]]

```python
result = A + B
result
```

    array([[ 7.5056974 ,  1.37029763],
           [ 6.94461604,  1.76600248]])

##Matrix Multiplication

###Multiplying a scalar value times a matrix

As before, let $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{smallmatrix} \bigr)$.  Suppose we want to multiply A times a scalar value (e.g. $3 \times A$)

$$
\begin{equation}
	3 \times A = 3 \times \begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}
	=
	\begin{bmatrix}
	  3a_{11} & 3a_{12} \\
	  3a_{21} & 3a_{22}
	\end{bmatrix}
\end{equation}
$$

is of dimension (2,2).  Scalar multiplication is commutative, so that $3 \times A$=$A \times 3$.  Notice that the product is defined for a matrix A of any dimension.

Similar to scalar addition and subtration, the code is simple:

```python
A * 3
```

    array([[15,  3],
           [18,  6]])

### Multiplying two matricies

Now, consider the $2 \times 1$ vector $C=\bigl( \begin{smallmatrix} c_{11} \\
  c_{21}
\end{smallmatrix} \bigr)$

Consider multiplying matrix $A_{2 \times 2}$ and the vector $C_{2 \times 1}$.  Unlike the addition and subtraction case, this product is defined.  Here, conformability depends not on the row **and** column dimensions, but rather on the column dimensions of the first operand and the row dimensions of the second operand.  We can write this operation as follows

$$
\begin{equation}
	A_{2 \times 2} \times C_{2 \times 1} =
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22}
	\end{bmatrix}_{2 \times 2}
    \times
    \begin{bmatrix}
	c_{11} \\
	c_{21}
	\end{bmatrix}_{2 \times 1}
	=
	\begin{bmatrix}
	  a_{11}c_{11} + a_{12}c_{21} \\
	  a_{21}c_{11} + a_{22}c_{21}
	\end{bmatrix}_{2 \times 1}
\end{equation}
$$

Alternatively, consider a matrix C of dimension $2 \times 3$ and a matrix A of dimension $3 \times 2$

$$
\begin{equation}
	A_{3 \times 2}=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22} \\
	  a_{31} & a_{32}
	\end{bmatrix}_{3 \times 2}
	,
	C_{2 \times 3} =
	\begin{bmatrix}
		  c_{11} & c_{12} & c_{13} \\
		  c_{21} & c_{22} & c_{23} \\
	\end{bmatrix}_{2 \times 3}
	\end{equation}
$$

Here, A $\times$ C is

$$
\begin{align}
	A_{3 \times 2} \times C_{2 \times 3}=&
	\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22} \\
	  a_{31} & a_{32}
	\end{bmatrix}_{3 \times 2}
	\times
	\begin{bmatrix}
	  c_{11} & c_{12} & c_{13} \\
	  c_{21} & c_{22} & c_{23}
	\end{bmatrix}_{2 \times 3} \\
	=&
	\begin{bmatrix}
	  a_{11} c_{11}+a_{12} c_{21} & a_{11} c_{12}+a_{12} c_{22} & a_{11} c_{13}+a_{12} c_{23} \\
	  a_{21} c_{11}+a_{22} c_{21} & a_{21} c_{12}+a_{22} c_{22} & a_{21} c_{13}+a_{22} c_{23} \\
	  a_{31} c_{11}+a_{32} c_{21} & a_{31} c_{12}+a_{32} c_{22} & a_{31} c_{13}+a_{32} c_{23}
	\end{bmatrix}_{3 \times 3}
\end{align}
$$

So in general, $X_{r_x \times c_x} \times Y_{r_y \times c_y}$ we have two important things to remember:

* For conformability in matrix multiplication, $c_x=r_y$, or the columns in the first operand must be equal to the rows of the second operand.
* The result will be of dimension $r_x \times c_y$, or of dimensions equal to the rows of the first operand and columns equal to columns of the second operand.

Given these facts, you should convince yourself that matrix multiplication is not generally commutative, that the relationship $X \times Y = Y \times X$ does **not** hold in all cases.
For this reason, we will always be very explicit about whether we are pre multiplying ($X \times Y$) or post multiplying ($Y \times X$) the vectors/matrices $X$ and $Y$.

For more information on this topic, see this
http://en.wikipedia.org/wiki/Matrix_multiplication.

```python
# Let's redefine A and C to demonstrate matrix multiplication:
A = np.arange(6).reshape((3,2))
C = np.random.randn(2,2)

print A.shape
print C.shape
```

    (3, 2)
    (2, 2)

We will use the numpy dot operator to perform the these multiplications.  You can use it two ways to yield the same result:

```python
print A.dot(C)
print np.dot(A,C)
```

    [[ 0.48080757  0.43511698]
     [ 1.47915018  0.72999774]
     [ 2.47749278  1.0248785 ]]
    [[ 0.48080757  0.43511698]
     [ 1.47915018  0.72999774]
     [ 2.47749278  1.0248785 ]]

Suppose instead of pre-multiplying C by A, we post-multiply.  The product doesn't exist because we don't have conformability as described above:

```python
C.dot(A)
```

## Matrix Division
The term matrix division is actually a misnomer.  To divide in a matrix algebra world we first need to invert the matrix.  It is useful to consider the analog case in a scalar work.  Suppose we want to divide the $f$ by $g$.  We could do this in two different ways:
$$
\begin{equation}
	\frac{f}{g}=f \times g^{-1}.
\end{equation}
$$
In a scalar seeting, these are equivalent ways of solving the division problem.  The second one requires two steps: first we invert g and then we multiply f times g.  In a matrix world, we need to think about this second approach.  First we have to invert the matrix g and then we will need to pre or post multiply depending on the exact situation we encounter (this is intended to be vague for now).

###Inverting a Matrix

As before, consider the square $2 \times 2$ matrix $A$=$\bigl( \begin{smallmatrix} a_{11} & a_{12} \\ a_{21} & a_{22}\end{smallmatrix} \bigr)$.  Let the inverse of matrix A (denoted as $A^{-1}$) be

$$
\begin{equation}
	A^{-1}=\begin{bmatrix}
             a_{11} & a_{12} \\
		     a_{21} & a_{22}
           \end{bmatrix}^{-1}=\frac{1}{a_{11}a_{22}-a_{12}a_{21}}	\begin{bmatrix}
		             a_{22} & -a_{12} \\
				     -a_{21} & a_{11}
		           \end{bmatrix}
\end{equation}
$$

The inverted matrix $A^{-1}$ has a useful property:
$$
\begin{equation}
	A \times A^{-1}=A^{-1} \times A=I
\end{equation}
$$
where I, the identity matrix (the matrix equivalent of the scalar value 1), is
$$
\begin{equation}
	I_{2 \times 2}=\begin{bmatrix}
             1 & 0 \\
		     0 & 1
           \end{bmatrix}
\end{equation}
$$
furthermore, $A \times I = A$ and $I \times A = A$.

An important feature about matrix inversion is that it is undefined if (in the $2 \times 2$ case), $a_{11}a_{22}-a_{12}a_{21}=0$.  If this relationship is equal to zero the inverse of A does not exist.  If this term is very close to zero, an inverse may exist but $A^{-1}$ may be poorly conditioned meaning it is prone to rounding error and is likely not well identified computationally.  The term $a_{11}a_{22}-a_{12}a_{21}$ is the determinant of matrix A, and for square matrices of size greater than $2 \times 2$, if equal to zero indicates that you have a problem with your data matrix (columns are linearly dependent on other columns).  The inverse of matrix A exists if A is square and is of full rank (ie. the columns of A are not linear combinations of other columns of A).

For more information on this topic, see this
http://en.wikipedia.org/wiki/Matrix_inversion, for example, on inverting matrices.

```python
# note, we need a square matrix (# rows = # cols), use C:
C_inverse = np.linalg.inv(C)
print C_inverse
```

    [[ 2.97399042  1.966247  ]
     [-3.28628201  0.12551463]]

Check that $C\times C^{-1} = I$:

```python
print C.dot(C_inverse)
print "Is identical to:"
print C_inverse.dot(C)
```

    [[  1.00000000e+00  -4.61031414e-18]
     [ -6.43302442e-18   1.00000000e+00]]
    Is identical to:
    [[  1.00000000e+00   6.11198607e-17]
     [  6.54738800e-18   1.00000000e+00]]

## Transposing a Matrix

At times it is useful to pivot a matrix for conformability- that is in order to matrix divide or multiply, we need to switch the rows and column dimensions of matrices.  Consider the matrix
$$
\begin{equation}
	A_{3 \times 2}=\begin{bmatrix}
	  a_{11} & a_{12} \\
	  a_{21} & a_{22} \\
	  a_{31} & a_{32}
	\end{bmatrix}_{3 \times 2}
\end{equation}
$$
The transpose of A (denoted as $A^{\prime}$) is
$$
\begin{equation}
   A^{\prime}=\begin{bmatrix}
	  a_{11} & a_{21} & a_{31} \\
	  a_{12} & a_{22} & a_{32} \\
	\end{bmatrix}_{2 \times 3}
\end{equation}
$$

```python
A = np.arange(6).reshape((3,2))
B = np.arange(8).reshape((2,4))
print "A is"
print A

print "The Transpose of A is"
print A.T
```

    A is
    [[0 1]
     [2 3]
     [4 5]]
    The Transpose of A is
    [[0 2 4]
     [1 3 5]]

One important property of transposing a matrix is the transpose of a product of two matrices.  Let matrix A be of dimension $N \times M$ and let B of of dimension $M \times P$.  Then
$$
\begin{equation}
	(AB)^{\prime}=B^{\prime}A^{\prime}
\end{equation}
$$
For more information, see this http://en.wikipedia.org/wiki/Matrix_transposition on matrix transposition.  This is also easy to implement:

```python
print B.T.dot(A.T)
print "Is identical to:"
print (A.dot(B)).T
```

    [[ 4 12 20]
     [ 5 17 29]
     [ 6 22 38]
     [ 7 27 47]]
    Is identical to:
    [[ 4 12 20]
     [ 5 17 29]
     [ 6 22 38]
     [ 7 27 47]]

## More python tools

### Indexing

Python begins indexing at 0 (not 1), therefore the first row and first column is referenced by 0,0 **not** 1,1.

### Slicing

Accessing elements of numpy matrices and arrays. This code grabs the first column of A:

```python
print A
A[:,0]
```

    [[0 1]
     [2 3]
     [4 5]]

    array([0, 2, 4])

or, we could grab a particular element (in this case, the second column, last row):

```python
A[2,1]
```

    5

### Logical Checks to extract values from matrices/arrays:

```python
print A
```

    [[0 1]
     [2 3]
     [4 5]]

```python
print A[:,1]>4

A[A[:,1]>4]
```

    [False False  True]

    array([[4, 5]])

### For loops

Create a $12 \times 2$ matrix and print it out:

```python
A = np.arange(24).reshape((12,2))
print A
print A.shape
```

    [[ 0  1]
     [ 2  3]
     [ 4  5]
     [ 6  7]
     [ 8  9]
     [10 11]
     [12 13]
     [14 15]
     [16 17]
     [18 19]
     [20 21]
     [22 23]]
    (12, 2)

The code below loops over the rows (12 of them) of our matrix A.  For each row, it slices A and prints the row values across all columns.  Notice the form of the for loop.  The colon defines the statement we are looping over.  For each iteration of the loop **idented** lines will be executed:

```python
for rows in A:
    print rows
```

    [0 1]
    [2 3]
    [4 5]
    [6 7]
    [8 9]
    [10 11]
    [12 13]
    [14 15]
    [16 17]
    [18 19]
    [20 21]
    [22 23]

```python
for rows in A:
    print rows
```

    [0 1]
    [2 3]
    [4 5]
    [6 7]
    [8 9]
    [10 11]
    [12 13]
    [14 15]
    [16 17]
    [18 19]
    [20 21]
    [22 23]

```python
for cols in A.T:
    print cols
```

    [ 0  2  4  6  8 10 12 14 16 18 20 22]
    [ 1  3  5  7  9 11 13 15 17 19 21 23]

### If/then/else

The code below checks the value of x and categorizes it into one of three values.  Like the for loop, each logical if check is ended with a colon, and any commands to be applied to that particular if check (if true) must be indented.

```python
x=.4

if x<.5:
    print "Heads"
    print 100
elif x>.5:
    print "Tails"
    print 0
else:
    print "Tie"
    print 50
```

    Heads
    100

### While loops

Again, we have the same basic form for the statement (note the colons and indents).  Here we use the shorthand notation `x+=1` for performing the calculation `x = x + 1`:

```python
x=0
while x<10:
    x+=1
    print x<10

print x
```

    True
    True
    True
    True
    True
    True
    True
    True
    True
    False
    10

# Some more

```python
v = np.random.beta(56, 23, 100)
```

```python
plt.hist(v)
```

```python
np.savetxt("myvextor.txt", v.reshape(20,5))
```

```sh
%%sh
head -10 myvextor.txt
```

    7.248008818950451015e-01 6.532244850357984411e-01 6.536199234633465194e-01 7.334928646986760281e-01 5.398165955580682684e-01
    7.139216071922678264e-01 6.728443837682190898e-01 6.494566164553140508e-01 6.683509000127559885e-01 7.644556371656708871e-01
    7.215153654452582943e-01 6.911006139808743010e-01 6.734786066554359074e-01 7.319844378166059373e-01 6.986996622331494988e-01
    6.978137082576147954e-01 6.825797326701168455e-01 7.569021032715560482e-01 6.154717554965444259e-01 7.482810964973004575e-01
    6.908519676137282461e-01 7.373778586062508245e-01 6.414907513325304178e-01 7.671835956420999247e-01 6.534701870485098985e-01
    6.679163640763395859e-01 7.410336013997532723e-01 7.564381573540180925e-01 7.741644043396176400e-01 7.560739681550309177e-01
    6.611087863698059675e-01 6.944074001813811403e-01 6.107849058765042471e-01 7.698996963306875552e-01 7.171169314025702679e-01
    7.722202225663886699e-01 7.263728335966201932e-01 7.350093378174409331e-01 7.552882740562755215e-01 6.027107542833861631e-01
    6.953737452564848764e-01 7.343557137075048535e-01 7.700578612228643482e-01 7.458161986630932327e-01 7.763693151512383039e-01
    6.891582596380744219e-01 7.441501789777352771e-01 7.231777325956401103e-01 6.755128569529142979e-01 5.962737198081802248e-01

```python
w = np.loadtxt("myvextor.txt")
```

```python
(w*v).sum()/(w.)
```

    49.145946056843869

```python
a = v.reshape(10,10)
```

```python
type(a)
```

    numpy.ndarray

```python
np.matrix(a)
```

    matrix([[ 0.72480088,  0.65322449,  0.65361992,  0.73349286,  0.5398166 ,
              0.71392161,  0.67284438,  0.64945662,  0.6683509 ,  0.76445564],
            [ 0.72151537,  0.69110061,  0.67347861,  0.73198444,  0.69869966,
              0.69781371,  0.68257973,  0.7569021 ,  0.61547176,  0.7482811 ],
            [ 0.69085197,  0.73737786,  0.64149075,  0.7671836 ,  0.65347019,
              0.66791636,  0.7410336 ,  0.75643816,  0.7741644 ,  0.75607397],
            [ 0.66110879,  0.6944074 ,  0.61078491,  0.7698997 ,  0.71711693,
              0.77222022,  0.72637283,  0.73500934,  0.75528827,  0.60271075],
            [ 0.69537375,  0.73435571,  0.77005786,  0.7458162 ,  0.77636932,
              0.68915826,  0.74415018,  0.72317773,  0.67551286,  0.59627372],
            [ 0.67069764,  0.6087569 ,  0.71437864,  0.78013279,  0.62574326,
              0.64370709,  0.71247713,  0.67404966,  0.67993396,  0.66871352],
            [ 0.66840764,  0.69701147,  0.68111859,  0.72275825,  0.75494115,
              0.66595713,  0.71336278,  0.6192563 ,  0.70614289,  0.71749587],
            [ 0.72002589,  0.72817938,  0.70982128,  0.75395697,  0.72369636,
              0.67164351,  0.68503003,  0.67223346,  0.72685012,  0.68745557],
            [ 0.682259  ,  0.72198239,  0.77744477,  0.71248248,  0.73918039,
              0.75032694,  0.62653126,  0.67802253,  0.71520116,  0.62238738],
            [ 0.65828396,  0.61692964,  0.72643121,  0.79209095,  0.74067574,
              0.66903045,  0.69260704,  0.71129395,  0.6608246 ,  0.66332163]])

```python
v =
type(v)
```

    numpy.ndarray

```python
v = np.matrix(np.random.rand(10,1))
w = np.matrix(np.random.rand(10,1))

```

```python
a = v.dot(w.transpose())
```

```python
a.shape
```

    (10, 10)

```python
type(a)
```

    numpy.matrixlib.defmatrix.matrix

```python
lag = np.linalg
```

```python

```

    matrix([[  1.29562817e+17,  -6.77337552e+16,  -1.06075205e+17,
               2.38896852e+16,  -2.35331525e+17,   8.74792833e+16,
              -1.06547937e+17,  -1.95293482e+16,  -4.33592894e+16,
               6.57006219e+16],
            [ -0.00000000e+00,  -0.00000000e+00,   8.00000000e+00,
              -1.47117588e+17,   1.44115188e+17,   3.12249574e+17,
              -1.15292150e+18,   9.00719925e+16,  -9.24714206e+16,
               7.20575940e+16],
            [  0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
               1.80143985e+16,   0.00000000e+00,  -4.50359963e+16,
               1.44115188e+17,  -9.00719925e+15,   8.09096783e+15,
               0.00000000e+00],
            [  6.00479950e+15,  -1.80143985e+16,  -1.20095990e+16,
               6.75539944e+15,  -1.20095990e+16,  -6.00479950e+15,
               9.60767921e+16,   1.50119988e+15,   6.94501266e+14,
               0.00000000e+00],
            [ -1.20095990e+16,  -0.00000000e+00,   6.00479950e+15,
               1.50119988e+15,   2.40191980e+16,   9.00719925e+15,
              -4.80383960e+16,   6.00479950e+15,  -6.31845975e+15,
              -0.00000000e+00],
            [ -6.00479950e+15,  -0.00000000e+00,   1.20095990e+16,
              -1.05083991e+16,  -2.40191980e+16,   1.80143985e+16,
              -9.60767921e+16,   3.00239975e+15,   2.66842553e+15,
              -0.00000000e+00],
            [  0.00000000e+00,   0.00000000e+00,  -1.80143985e+16,
               1.80143985e+16,   0.00000000e+00,  -4.50359963e+16,
               1.44115188e+17,  -9.00719925e+15,   1.61044132e+16,
               0.00000000e+00],
            [  0.00000000e+00,   0.00000000e+00,   0.00000000e+00,
              -9.00719925e+15,   0.00000000e+00,   1.80143985e+16,
               0.00000000e+00,   0.00000000e+00,  -1.77888496e+15,
               0.00000000e+00],
            [ -6.00479950e+15,   3.60287970e+16,   1.20095990e+16,
              -1.65131986e+16,  -2.40191980e+16,   1.20095990e+16,
              -9.60767921e+16,   3.00239975e+15,   3.85207751e+14,
              -0.00000000e+00],
            [  6.00479950e+15,   0.00000000e+00,   2.40191980e+16,
               2.55203979e+16,   2.40191980e+16,  -4.80383960e+16,
               9.60767921e+16,  -2.10167983e+16,   8.13151685e+15,
              -3.60287970e+16]])

```python

```


# Computing gradients and derivatives in PyTorch
>"Making good use of the `gradient` argument in PyTorch's `backward` function"

- toc: true
- badges: true
- comments: true
- categories: [mathematics]

---
tags: mathematics pytorch gradients backward automatic differentiation vector-Jacobian product backpropagation

---

# tl;dr
The `backward` function in `PyTorch` can be used to compute the derivatives or gradients of functions.  The `backward` function computes vector-Jacobian products so that the appropriate vector must be determined.  In other words, the correct `gradient` argument must be passed to `backward`, although not passing `gradient` explicitly will cause `backward` to choose the appropriate value but only in the simplest cases.

This notebook explains vector-Jacobian products and how to choose the `gradient` argument in the `backward` function in the general case.

# A brief overview
In the case of a function taking a scalar and returning a scalar, the use of the `backward` function is quite straight-forward:

```python
# collapse-hide
import torch
x = torch.tensor(1., requires_grad=True)
y = x**2
y.backward()
print(f"Derivative at a single point:")
print(x.grad.data)
```

    Derivative at a single point:
    tensor(2.)

However, when
- the function is **multi-valued** (e.g. vector- or matrix-valued); or
- one wishes to compute the derivative of a function at **mulitple** points,

then the `gradient` argument in `backward` must be suitably chosen.  For example:

```python
# collapse-hide
import torch
x = torch.linspace(-2, 2, 5, requires_grad=True)
y = x**2
gradient = torch.ones_like(y)
y.backward(gradient)
print("Derivative at multiple points:")
print(x.grad.data)
```

    Derivative at multiple points:
    tensor([-4., -2.,  0.,  2.,  4.])

Indeed, more precisely, the `backward` function computes vector-Jacobian products, which is not explicit in the function's doc string:

```python
# collapse-hide
print("First line of `torch.Tensor.backward` doc string:")
print("\""+ torch.Tensor.backward.__doc__.split("\n")[0] + "\"")
```

    First line of `torch.Tensor.backward` doc string:
    "Computes the gradient of current tensor w.r.t. graph leaves."

although some explanations are given [in this official tutorial](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html#gradients).  The crucial point is therefore to choose the appropriate vector, which is passed to the `backward` function in its `gradient` argument:

```python
# collapse-hide
import inspect
import torch
print(f"torch.Tensor.backward{inspect.signature(torch.Tensor.backward)}")
print("...")
print("\n".join(torch.Tensor.backward.__doc__.split("\n")[11:18]))
print("...")
```

    torch.Tensor.backward(self, gradient=None, retain_graph=None, create_graph=False)
    ...
            Arguments:
                gradient (Tensor or None): Gradient w.r.t. the
                    tensor. If it is a tensor, it will be automatically converted
                    to a Tensor that does not require grad unless ``create_graph`` is True.
                    None values can be specified for scalar Tensors or ones that
                    don't require grad. If a None value would be acceptable then
                    this argument is optional.
    ...

There is a way around specifying the `gradient` argument.  Revisiting the example above, the derivative at multiple points can be equivalently calculated by adding a `sum()`:

```python
# collapse-hide
import torch
x = torch.linspace(-2, 2, 5, requires_grad=True)
y = (x**2).sum()

y.backward()
print("Derivative at multiple points:")
print(x.grad.data)
```

    Derivative at multiple points:
    tensor([-4., -2.,  0.,  2.,  4.])

Here, the `backward` method is invoked on a different `tensor`:
```
(x**2).backward()
```
if `x` contains a single input,
vs
```
(x**2).sum().backward()
```
if `x` contains multiple inputs.

On the other hand, passing the `gradient` argument, whether `x` contains one or multiple inputs, the same command is used to compute the derivatives:
```
y = (x**2)
y.backward(torch.ones_like(y))
```

Roughly speaking, the difference between the two methods, namely setting `gradient=torch.ones_like(y)` or adding `sum()`, is in the order of the summation and differentiation.

# Usage examples of the `backward` function
The derivative of the **scalar**, **univariate** function $f(x)=x^2$ at a **single** point $x=1$:

```python
import torch
x = torch.tensor(1., requires_grad=True)
y = x**2
y.backward()
x.grad
```

    tensor(2.)

The derivative of the **scalar**, **univariate** function $f(x)=x^2$ at **multiple** points $x= -2, -1, \dots, 2$:

```python
import torch
x = torch.linspace(-2, 2, 5, requires_grad=True)
y = x**2
v = torch.ones_like(y)
y.backward(v)
x.grad
```

    tensor([-4., -2.,  0.,  2.,  4.])

The gradient of the **scalar**, **multivariate** function $f(x_1, x_2)=3x_1^2 + 5x_2^2$ at a **single** point $(x_1, x_2)=(-1, 2)$:

```python
import torch
x = torch.tensor([-1., 2.], requires_grad=True)
w = torch.tensor([3., 5.])
y = (x*x*w).sum()
y.backward()
x.grad
```

    tensor([-6., 20.])

The gradient of the **scalar**, **multivariate** function $f(x_1, x_2) = -x_1^2 + x_2^2$ at **multiple** points $(x_1, x_2)$:

```python
import torch
x = torch.arange(6, dtype=float).view(3, 2).requires_grad_(True)
w = torch.tensor([-1, 1])
y = (x*x*w).sum(1)
v = torch.ones_like(y)
y.backward(v)
x.grad
```

    tensor([[-0.,  2.],
            [-4.,  6.],
            [-8., 10.]], dtype=torch.float64)

The _derivatives_ of the **vector-valued**, **univariate** function $f(x)= (-x^3, 5x)$ at a **single** point $x=1$, i.e. the derivative of
- its first component function $f_1(x)=-x^3$; and
- its second component function $f_2(x)=5x$.

```python
# collapse-hide
import torch
x = torch.tensor(1., requires_grad=True)
y = torch.stack([-x**3, 5*x])

v1 = torch.tensor([1., 0.])
y.backward(v1, retain_graph=True)

print(f"f_1'({x.data.item()}) = {x.grad.data.item():>4}")

x.grad.zero_()

v2 = torch.tensor([0., 1.])
y.backward(v2)
print(f"f_2'({x.data.item()}) = {x.grad.data.item():>4}")
```

    f_1'(1.0) = -3.0
    f_2'(1.0) =  5.0

The _derivatives_ of the **vector-valued**, **univariate** function $f(x)= (-x^3, 5x)$ at **multiple** points, i.e. the derivative of
- its first component function $f_1(x)=-x^3$; and
- its second component function $f_2(x)=5x$.

```python
# collapse-hide
import torch
import itertools
x = torch.arange(3, dtype=float, requires_grad=True)
y = torch.stack([-x**3, 5*x])

ranges = [range(_) for _ in y.shape]

v1 = torch.tensor([1. if i == 0 else 0. for i, j in itertools.product(*ranges)]).view(*y.shape)
y.backward(v1, retain_graph=True)
print(f"Derivative of f_1(x)=-3x^2 at the points {tuple(x.data.view(-1).tolist())}:")
print(x.grad)

x.grad.zero_()

v2 = torch.tensor([1. if i == 1 else 0. for i, j in itertools.product(*ranges)]).view(*y.shape)
y.backward(v2)
print(f"\nDerivative of f_2(x)=5x at the points {tuple(x.data.view(-1).tolist())}:")
print(x.grad)
```

    Derivative of f_1(x)=-3x^2 at the points (0.0, 1.0, 2.0):
    tensor([  0.,  -3., -12.], dtype=torch.float64)

    Derivative of f_2(x)=5x at the points (0.0, 1.0, 2.0):
    tensor([5., 5., 5.], dtype=torch.float64)

The _gradients_ of the **vector-valued**, **multivariate** function
$$
f(x_1, \dots, x_n) = (x_1 + \dots + x_n\,, x_1^2 + \dots + x_n^2)
$$
at a **single** point $(x_1, \dots, x_n)$, i.e. the gradient of
- its first component function $f_1(x_1, \dots, x_n) = x_1 + \dots + x_n$; and
- its second component function $f_2(x_1, \dots, x_n) = x_1^2 + \dots + x_n^2$.

```python
# collapse-show
import torch
x = torch.arange(4, dtype=float, requires_grad=True)
y = torch.stack([x.sum(), (x**2).sum()])

print(f"x                 : {tuple(x.data.tolist())}")
print(f"y = (y_1, y_2)    : {tuple(y.data.tolist())}")

v1 = torch.tensor([1., 0.])
y.backward(v1, retain_graph=True)
print(f"gradient of y_1   : {tuple(x.grad.data.tolist())}")

x.grad.zero_()

v2 = torch.tensor([0., 1.])
y.backward(v2)
print(f"gradient of y_2   : {tuple(x.grad.data.tolist())}")
```

    x                 : (0.0, 1.0, 2.0, 3.0)
    y = (y_1, y_2)    : (6.0, 14.0)
    gradient of y_1   : (1.0, 1.0, 1.0, 1.0)
    gradient of y_2   : (0.0, 2.0, 4.0, 6.0)

The _gradients_ of the **vector-valued**, **multivariate** function
$$
f(x_1, \dots, x_n) = (x_1 + \dots + x_n\,, x_1^2 + \dots + x_n^2)
$$
at **multiple** points, i.e. the gradient of
- its first component function $f_1(x_1, \dots, x_n) = x_1 + \dots + x_n$; and
- its second component function $f_2(x_1, \dots, x_n) = x_1^2 + \dots + x_n^2$.

```python
# collapse-show
import torch
import itertools
x = torch.arange(4*3, dtype=float).view(-1,4).requires_grad_(True)
y = torch.stack([x.sum(1), (x**2).sum(1)])
print("x:")
print(x.data)
print("y:")
print(y.data)

print()

ranges = [range(_) for _ in y.shape]

v1 = torch.tensor([1. if i == 0 else 0. for i, j in itertools.product(*ranges)]).view(*y.shape)
y.backward(v1, retain_graph=True)
print("Gradients of the f1 at multiple points:")
print(x.grad)

x.grad.zero_()

print()
v2 = torch.tensor([1. if i == 1 else 0. for i, j in itertools.product(*ranges)]).view(*y.shape)
y.backward(v2)
print("Gradients of the f2 at multiple points:")
print(x.grad)

```

    x:
    tensor([[ 0.,  1.,  2.,  3.],
            [ 4.,  5.,  6.,  7.],
            [ 8.,  9., 10., 11.]], dtype=torch.float64)
    y:
    tensor([[  6.,  22.,  38.],
            [ 14., 126., 366.]], dtype=torch.float64)

    Gradients of the f1 at multiple points:
    tensor([[1., 1., 1., 1.],
            [1., 1., 1., 1.],
            [1., 1., 1., 1.]], dtype=torch.float64)

    Gradients of the f2 at multiple points:
    tensor([[ 0.,  2.,  4.,  6.],
            [ 8., 10., 12., 14.],
            [16., 18., 20., 22.]], dtype=torch.float64)

# Mathematical preliminaries
## Scalars, vectors, matrices, and tensors

- A **scalar** is a real number.  It is usually denoted with $x$.
- An **$n$-dimensional vector** is a list $(x_1, \dots, x_n)$ of scalars.
- An **$m$-by-$n$ matrix** is an array with $m$ rows and $n$ columns of scalars:
$$
\begin{bmatrix}w_{1,1}&\dots&w_{1,n}\\\vdots&\ddots&\vdots\\w_{m,1}&\dots&w_{m,n}\end{bmatrix}
$$
- A **column vector** of length $n$ is a $n$-by-$1$ matrix:
$$\begin{bmatrix}x_1\\\vdots\\x_n\end{bmatrix}$$
Note that it is distinct from its vector counterpart $(x_1, \dots, x_n)$.
- A **row vector** of length $n$ is a $1$-by-$n$ matrix:
$$\begin{bmatrix}x_1&\dots&x_n\end{bmatrix}$$
Note that it is distinct from its vector and column vector counterparts.

>Note:
For convenience, we may denote a vector, a column vector, or a row vector with a single symbol, typically $x$.

In another post we establish the following correspondence between these mathematical entities and their `tensor` counterparts in `PyTorch`:

|mathematical name|mathematical notation|`tensor` shape|`tensor` dimension|
|---|---|---|---|
|scalar|$x$|`()`|`0`|
|vector|$(x_1, \dots, x_n)$|`(n,)`|`1`|
|matrix|$\begin{bmatrix}w_{1,1}&\dots&w_{1,n}\\\vdots&\ddots&\vdots\\w_{m,1}&\dots&w_{n,m}\end{bmatrix}$|`(m,n)`| `2`|
|column vector|$\begin{bmatrix}x_1\\\vdots\\x_n\end{bmatrix}$|`(n,1)`|`2`|
|row vector|$\begin{bmatrix}x_1&\dots&x_n\end{bmatrix}$|`(1,n)`|`2`|

## Mathematical functions
- We consider functions which are mappings from scalars, vectors, or matrices to scalars, vectors, or matrices.  It is generically denoted $y=f(x)$.
- A **scalar** function $y=f(x)$ is a function returning a scalar, i.e. $y$ is a scalar.
- A **vector-valued** function $y=f(x)$ is a function returning a vector, i.e. $y$ is a vector.  We often write
$$f(x) = (f_1(x), \dots, f_m(x))$$
if the output is $m$-dimensional, where each of $f_1(x), \dots, f_m(x)$ is a scalar function.
- A **univariate** function $y=f(x)$ is a function depending on a scalar $x$.
- A **multivariate** function $y=f(x)$ is a function depending on a vector $x=(x_1, \dots, x_n)$.

In summary

|$y=f(x)$|scalar-valued|vector-valued|
|---|---|---|
|**univariate**|$x$ is a scalar
$y$ is a scalar|$x$ is a scalar
$y$ is a vector|
|**multivariate**|$x$ is a vector
$y$ is a scalar|$x$is a vector
$y$ is a vector|

## Differentiation
### Basic definitions
We do not recall the definitions for:
- the **derivative** $f'(x)$ of a scalar, uni-variate function $y=f(x)$ evaluated at a scalar $x$;
- the **partial derivatives** $\frac{\partial f}{\partial x_i}(x)$, $i=1, \dots, n$, of a scalar, multivariate function $y=f(x)$ with respect to the variables $x_1, \dots, x_n$, and evaluated at $x=(x_1, \dots, x_n)$.

### Derivatives of vector-valued, univariate functions
The **derivative** of a vector-valued, uni-variate function $y=f(x)$ evaluated at a scalar $x$ is the vertical concatenation of the derivatives of its component functions:
$$f'(x) = \begin{bmatrix}f_1'(x)\\\vdots\\f_m'(x)\end{bmatrix}$$

### Gradients
The **gradient** of a scalar-valued function $y=f(x)$, is the *row* vector of its partial derivatives:
$$\nabla f(x) = \begin{bmatrix}\frac{\partial f}{\partial x_1}(x)&\dots&\frac{\partial f}{\partial x_n}(x)\end{bmatrix}$$
with length $n$ if $x$ is $n$-dimensional: $x=(x_1, \dots, x_n)$.

### Jacobians
The **Jacobian** of a vector-valued, multivariate function $y=f(x)$ is the vertical concatenation of the gradients of the component functions $f_1, \dots, f_m$:
$$J_f(x)
\,=\,
\begin{bmatrix}
\nabla f_1(x)\\\vdots\\\nabla f_m(x)
\end{bmatrix}
\,=\,
\begin{bmatrix}
    \frac{\partial f_1}{\partial x_1}(x)&\dots&\frac{\partial f_1}{\partial x_n}(x)\\
    \vdots&\ddots&\vdots\\
    \frac{\partial f_m}{\partial x_1}(x)&\dots&\frac{\partial f_m}{\partial x_n}(x)
\end{bmatrix}
$$
It is thus an $m$-by-$n$ matrix, i.e. with $m$ rows and $n$ columns.

#### Special case: $m=1$
In case $m=1$, the Jacobian agrees with the gradient of a scalar, multivariate function:
$$J_f(x) = \nabla f(x)$$

#### Special case: $n=1$
In case $n=1$, the Jacobian agrees with the derivative of a vector-valued, univariate function.
$$J_f(x) = \begin{bmatrix}f_1'(x)\\\vdots\\f_m'(x)\end{bmatrix}$$

## Vector-Jacobian products
Given a vector-valued, multivariate function $y=f(x)$ and a _column_ vector
$v=\begin{bmatrix}v_1\\\vdots\\v_m\end{bmatrix}$,
the **vector-Jacobian product** is the matrix multiplication
$$v^\top J_f(x) \,=\,
\begin{bmatrix}
v_1&\dots&v_m
\end{bmatrix}
\begin{bmatrix}
    \frac{\partial f_1}{\partial x_1}(x)&\dots&\frac{\partial f_1}{\partial x_n}(x)\\
    \vdots&\ddots&\vdots\\
    \frac{\partial f_m}{\partial x_1}(x)&\dots&\frac{\partial f_m}{\partial x_n}(x)
\end{bmatrix}
$$
which is then a _row_ vector of length $n$.

### Special case
If $v^\top$ happens to be the gradient of a scalar-valued function $z=\ell(y)$ evaluated at $f(x)$, i.e. $v = \nabla \ell(y)$ where $y=f(x)$, then
\begin{equation}
v^\top J_f(x)
\,=\,\nabla (\ell\circ f)(x)
\end{equation}
In other words, $v^\top J_f(x)$ is the gradient of the composition of the function $\ell$ with the function $f$.

>Note:
The vector-Jacobian product can be generalized to cases where $x$ and $y$ are (mathematical) tensors of higher dimensions.  This generalization is in fact used in some of the examples of this post.

### Application: Gradients of vector-valued functions
If $y=f(x)=(f_1(x), \dots, f_m(x))$ is a vector-valued, multivariate function, one computes the gradients $\nabla f_1(x), \dots, \nabla f_m(x)$ one at a time, each time with a suitable vector $v$.  Indeed, fix $i$ between $1$ and $m$, and define $\ell_i(y)=y_i$ the function selecting the $i$-th coordinate of $y=(y_1, \dots, y_m)$, so that
$$f_i(x) = \ell_i(f(x))\,.$$
Noting that
$$\nabla \ell_i(y) = \begin{bmatrix}0&\cdots&0&1&0&\cdots&0\end{bmatrix}$$
where the only non-zero coordinate is in the $i$-th position, then
$$
\begin{align}
\nabla \ell_i(f(x))J_f(x)
& =
\begin{bmatrix}0&\cdots&0&1&0&\cdots&0\end{bmatrix}
\begin{bmatrix}
    \frac{\partial f_1}{\partial x_1}(x)&\dots&\frac{\partial f_1}{\partial x_n}(x)\\
    \vdots&\ddots&\vdots\\
    \frac{\partial f_m}{\partial x_1}(x)&\dots&\frac{\partial f_m}{\partial x_n}(x)
\end{bmatrix}\\
&=
\begin{bmatrix}\frac{\partial f_i}{\partial x_1}(x)&\dots&\frac{\partial f_i}{\partial x_n}(x)\end{bmatrix}
\end{align}
$$

### Application: Derivatives at multiple points
To evaluate the derivative of a scalar, univariate function $f(x)$ at multiple sample points $x^{(1)}, \dots, x^{(N)}$, we create a *new*, vector-valued and multivariate function
$$F(x)=\begin{bmatrix}f\left(x^{(1)}\right)\\ \vdots \\ f\left(x^{(N)}\right)\end{bmatrix}
\qquad\textrm{where}\qquad
x\,=\,(x^{(1)}, \dots, x^{(N)})\,.$$
Thus, its Jacobian is
$$J_F(x)=\begin{bmatrix}
f'(x^{(1)})&&&&\\
&\ddots&&&\\
&&f'(x^{(j)})&&\\
&&&\ddots&\\
&&&&f'(x^{(N)})\end{bmatrix}
$$
where all off-diagonal terms are $0$.
Thus, setting $v=\begin{bmatrix}1\\\vdots\\1\end{bmatrix}$, we obtain the gradient of $f$ evaluated at the $N$ sample points $x^{(1)}\,, \dots\,, x^{(N)}$:
$$\begin{bmatrix}f'(x^{(1)})&\dots& f'(x^{(j)})&\cdots& f'(x^{(N)})\end{bmatrix}
=\left[1\,,\dots\,,1\right]
J_f(x)\,.$$
The interpretation here is that the resulting row vector contains the derivative of $f$ at the samples $x^{(1)}$ to $x^{(N)}$.

### The trick with `sum()`
The trick of adding `sum()` before calling `backward` differs with the previous application only in the order of operations performed: the summation is performed before differentiation.

From a scalar, univariate function $y=f(x)$, construct a new scalar, multivariate function
$$G(x_1, \dots, x_N) = f(x_1) + \dots + f(x_N)$$
Using the rules of vector calculus, the gradient of $G$ at an $n$-dimensional point $(x_1, \dots, x_N)$ is
$$
\begin{align}
\nabla G(x) & = \begin{bmatrix}\frac{\partial G}{\partial x_1}(x)&\cdots&\frac{\partial G}{\partial x_N}\end{bmatrix}\\
& = \begin{bmatrix}f'(x_1)&\cdots&f'(x_N)\end{bmatrix}
\end{align}
$$
The interpretation here is that the resulting row vector contains the gradient of $G$ at the $N$-dimensional point $(x_1, \dots, x_N)$.

# Computing gradients with `PyTorch`
A mathematical function is a mapping, which strictly speaking one should denote $f$.  The denotation $y=f(x)$ is simply to suggest that the typical input will be denoted $x$ and the corresponding output will be denoted $y$.  Otherwise, $y=f(x)$ actually asserts the identity between a value $y$ and the evaluation of the function $f$ at the value $x$.

In `PyTorch`, the primary objects are `tensor`s, which can represent (mathematical) scalars, vectors, and matrices (as well as mathematical tensors).   The way a `PyTorch` function calculates a `tensor`, generically denoted `y` and called the output, from another `tensor`, generically denoted `x` and called the input, reflects the action of a mathematical function $f$ (or $y=f(x)$).

Conversely, a mathematical function $f$ can be evaluated at $x$ using `PyTorch`, and furthermore `PyTorch` allows to evaluate the derivative or gradient of $f$ at $x$ via the method `backward`.  More specifically, the `backward` function performs vector-Jacobian products, where the vector correspond to the `gradient` argument.  The key point in using the `backward` is thus to understand how to choose the `gradient` argument.

The mathematical preliminaries above show how `gradient` should be chosen.  There are two key points:
1. `gradient` has the same shape as `y`;
1. `gradient` is populated with `0.`'s and `1.`'s, and the location of the `1.`'s corresponding to the inputs and outputs of interest.

# Examples revisited

>Note:
The variable `v` is passed to the `gradient` argument in all our examples.

For the derivative of a scalar, univariate function evaluated a single point, we choose `gradient=torch.tensor(1.)`, which is the default value:

```python
import torch
x = torch.tensor(1., requires_grad=True)
y = x**2
v = torch.ones_like(y)
y.backward()
print(f"Shape of x         : {tuple(x.shape)}")
print(f"Shape of y         : {tuple(y.shape)}")
print(f"gradient argument  : {v}")
```

    Shape of x         : ()
    Shape of y         : ()
    gradient argument  : 1.0

Note that if `x` is cast as a `1`-dimensional `tensor`, then (in this particular example) `y` is also a `1`-dimensional `tensor`:

```python
import torch
x = torch.tensor([1.], requires_grad=True)
y = x**2
v = torch.ones_like(y)
y.backward()
print(f"Shape of x         : {tuple(x.shape)}")
print(f"Shape of y         : {tuple(y.shape)}")
print(f"gradient argument  : {v}")
```

    Shape of x         : (1,)
    Shape of y         : (1,)
    gradient argument  : tensor([1.])

Similarly if `x` is cast as `2`-dimensional `tensor`:

```python
import torch
x = torch.tensor([[1.]], requires_grad=True)
y = x**2
v = torch.ones_like(y)
y.backward()
print(f"Shape of x         : {tuple(x.shape)}")
print(f"Shape of y         : {tuple(y.shape)}")
print(f"gradient argument  : {v}")
```

    Shape of x         : (1, 1)
    Shape of y         : (1, 1)
    gradient argument  : tensor([[1.]])

For the derivative of a scalar, univariate function evaluated at multiple points, `gradient` contains all `1.`'s and is of same shape as `y`:

```python
import torch
x = torch.linspace(-1, 1, 5, requires_grad=True)
y = x**2
v = torch.ones_like(y)
y.backward(v)
print(f"Shape of x         : {tuple(x.shape)}")
print(f"Shape of y         : {tuple(y.shape)}")
print(f"gradient argument  : {v}")
```

    Shape of x         : (5,)
    Shape of y         : (5,)
    gradient argument  : tensor([1., 1., 1., 1., 1.])

Casting `x` in a different shape changes the shape of `y`, and thus of `gradient`:

```python
import torch
x = torch.linspace(-2, 2, 5).view(-1,1).requires_grad_(True)
y = x**2
v = torch.ones_like(y)
y.backward(v)
print(f"Shape of x         : {tuple(x.shape)}")
print(f"Shape of y         : {tuple(y.shape)}")
print(f"gradient argument  : ")
print(v)
```

    Shape of x         : (5, 1)
    Shape of y         : (5, 1)
    gradient argument  :
    tensor([[1.],
            [1.],
            [1.],
            [1.],
            [1.]])

For the derivative of a vector-valued, univariate function evaluated at a single point, the derivative of each component function is calculated one at a time, and `gradient` consists of all `0.`'s except for one `1.`, which is located at a position corresponding to the component function.  In the example below, the function is in fact *matrix-valued*, namely we calculate the derivative of
$$f(x) = \begin{bmatrix}1&x\\x^2&x^3\\x^4&x^5\end{bmatrix}\qquad \textrm{at}\quad x\,=\,1\,.$$

```python
# collapse-show
import torch
import itertools
x = torch.tensor(1., requires_grad=True)
y = torch.stack([x**i for i in range(6)]).view(3,2)
ranges = [range(_) for _ in y.shape]

print("x:")
print(x.data)
print("\ny:")
print(y.data)

derivatives = torch.zeros_like(y)

for i, j in itertools.product(*ranges):
    v = torch.zeros_like(y)
    v[i,j] = 1.
    if x.grad is not None: x.grad.zero_()

    y.backward(v, retain_graph=True)
    derivatives[i,j] = x.grad.item()
print("\nDerivatives:")
print(derivatives)
```

    x:
    tensor(1.)

    y:
    tensor([[1., 1.],
            [1., 1.],
            [1., 1.]])

    Derivatives:
    tensor([[0., 1.],
            [2., 3.],
            [4., 5.]])

>Note:
The use of `for` loops can be avoided.

For the gradient of a scalar, multivariate function evaluated at a single point, `gradient=torch.tensor(1.)`:

```python
import torch
x = torch.tensor([-1., 2.], requires_grad=True)
w = torch.tensor([3., 5.])
y = (x*x*w).sum()
v = torch.ones_like(y)
y.backward()
print(f"Shape of x         : {tuple(x.shape)}")
print(f"Shape of y         : {tuple(y.shape)}")
print(f"gradient argument  : {v}")
```

    Shape of x         : (2,)
    Shape of y         : ()
    gradient argument  : 1.0

In the following example, the input `x` is a `(3,2)`-tensor:

```python
x = torch.arange(6, dtype=float).view(3,2).requires_grad_(True)
y = (x**2).sum()
v = torch.ones_like(y)
y.backward(v)

print(f"Shape of x: {tuple(x.shape)}")
print(f"Shape of y: {tuple(y.shape)}")
print(f"gradient argument: {v}")
print("x:")
print(x.data)
print("x.grad:")
print(x.grad.data)
```

    Shape of x: (3, 2)
    Shape of y: ()
    gradient argument: 1.0
    x:
    tensor([[0., 1.],
            [2., 3.],
            [4., 5.]], dtype=torch.float64)
    x.grad:
    tensor([[ 0.,  2.],
            [ 4.,  6.],
            [ 8., 10.]], dtype=torch.float64)