# Announcements
- __Please familiarize yourself with the term projects, and sign up for your (preliminary) choice__ using [this form](https://forms.gle/ByLLpsthrpjCcxG89). _You may revise your choice, but I'd recommend settling on a choice well before Thanksgiving._
- Problem Set 5 posted on D2L, due Oct 20.
- __Outlook__: algorithms for solving high-dimensional linear and non-linear equations; then Boundary Value Problems and Partial Differential Equations.
- Conference for Undergraduate Women in Physics: online event in 2021, [applications accepted until 10/25](https://www.aps.org/programs/women/cuwip/)

This notebook presents as selection of topics from the book "Numerical Linear Algebra" by Trefethen and Bau (SIAM, 1997), and uses notebooks by Kyle Mandli.

# Conditioning and Stability

Once an approximation to a linear system is constructed the next question is how much trust can we put in the approximation? Since the true solution is not known, one of the few tools we have is to ask how well the approximation matches the original equation. In other words, we seek a solution to a system,
$$
    \vec{f}(\vec{x}) = \vec{b}.
$$

We do not have $\vec{x}$ but instead have an approximation, $\hat{x}$, and we hope that
$$
    \vec{f}(\hat{x}) \approx \vec{b}.
$$
In this section the question we explore is to try to determine a bound on the relative error, $\frac{||\vec{x}-\hat{x}||}{||\vec{x}||}$ given the matrix, $A$.

This leads to the notion of conditioning. Conditioning is the behavior of a problem when the solution is a changed a small bit (perturbed), and it is a  mathematical (analytic) property of the original system of equations.  Stability, on the other hand, is concerned with how the algorithm used to obtain an approximation behaves when the approximation is perturbed.

## Conditioning and Condition Numbers

A **well-conditioned** problem is one where a small perturbation to the original problem leads to only small changes in the solution.

Formally we can think of a function $f$ which maps $x$ to $y$

$$
    f(x) = y \quad \text{or} \quad f: X \rightarrow Y.
$$

Let $x \in X$ where we perturb $x$ with $\delta x$ and we ask how the result $y$ changes:

$$
    ||f(x) - f(x + \delta x)|| \leq C ||x - (x+\delta x)||
$$

for some constant $C$ possible dependent on $\delta x$ depending on the type of conditioning we are considering.

### Absolute Condition Number

If we let $\delta x$ be the small perturbation to the input and $\delta f = f(x + \delta x) - f(x)$ be the result the **absolute condition number** $\hat{~\kappa}$ can be defined as

$$
    \hat{\!\kappa} = \sup_{\delta x} \frac{||\delta f||}{||\delta x||}
$$

for most problems (assuming $\delta f$ and $\delta x$ are both infinitesimal).

When $f$ is differentiable we can evaluate the condition number via the Jacobian.  Recall that the derivative of a multi-valued function can be termed in the form of a Jacobian $J(x)$ where
$$
    [J(x)]_{ij} = \frac{\partial f_i}{\partial x_j}(x).
$$

This allows us to write the infinitesimal $\delta f$ as
$$
    \delta f \approx J(x) \delta x
$$
with equality when $||\delta x|| \rightarrow 0$.  Then we can write the condition number as
$$
    \hat{\!\kappa} = ||J(x)||
$$
where the norm is the one induced by the spaces $X$ and $Y$.

### Relative Condition Number

The **relative condition number** is defined similarly and is related to the difference before between the absolute error and relative error as defined previously.  With the same caveats as before it can be defined as
$$
    \kappa = \sup_{\delta x} \left( \frac{\frac{||\delta f||}{||f(x)||}}{\frac{||\delta x||}{||x||}} \right).
$$

Again if $f$ is differentiable we can use the Jacobian $J(x)$ to evaluate the relative condition number as
$$
    \kappa = \frac{||J(x)||}{||f(x)|| ~/ ~||x||}.
$$

#### Example
Calculate the relative condition number of $\sqrt{x}$ for $x > 0$.

$$
    f(x) = \sqrt{x}, \quad J(x) = f'(x) = \frac{1}{2 \sqrt{x}} \\
    \kappa = \frac{||J(x)||}{||f(x)|| / ||x||} = \frac{1}{2 \sqrt{x}} \frac{x}{\sqrt{x}} = \frac{1}{2}
$$

The condition number of a function was discussed in general terms above. Now, we examine the more specific case of a linear function, a matrix-vector multiplication. Here we let $\vec{f}(\vec{x})=Ax$ and determine the condition number by perturbing $x$.

We begin with the definition above,
$$\begin{aligned}
    \kappa &= \sup_{\delta x} \left ( \frac{||A (\vec{x}+\delta x) - A \vec{x}||}{||A\vec{x}||} \frac{||\vec{x}||}{||\delta x||}\right ), \\
    &= \sup_{\delta x} \frac{ ||A \delta x||}{||\delta x||} \frac{||\vec{x}||}{||A\vec{x}||}, \\
    &= ||A|| \frac{||\vec{x}||}{||A \vec{x}||},
\end{aligned}$$
where $\delta x$ is a vector.

If $A$ has an inverse, then we note that
$$
\begin{align}
 \vec{x} &= A^{-1}A \vec{x}, \\
 \Rightarrow ||\vec{x}|| &= || A^{-1}A \vec{x} ||, \\
                         &\leq ||A^{-1}|| || A \vec{x} ||,
\end{align}
$$
which implies that
$$
    \frac{||x||}{||A x||} \leq ||A^{-1}||.
$$
_We can now bound the condition number for a matrix by_
$$
    \kappa \leq ||A|| ||A^{-1}||.
$$

### Condition Number of a Matrix

The condition number of a matrix is defined by the product
$$
    \kappa(A) = ||A||~||A^{-1}||.
$$
where here we are thinking about the matrix rather than a problem.  If $\kappa$ is small than $A$ is said to be **well-conditioned**.  If $A$ is singular we assign $\kappa(A) = \infty$ as the matrix's condition number.

When we are considering the $\ell_2$ norm then we can write the condition number as

$$
    \kappa(A) = \frac{\sqrt{\rho(A^\ast A)}}{\sqrt{\rho((A^\ast A)^{-1})}} = \frac{\sqrt{\max |\lambda|}}{\sqrt{\min |\lambda|}}.
$$

### Condition Number of a System of Equations

Another way to think about the conditioning of a problem we have looked at before is that the matrix $A$ itself is an input to the problem.  Consider than the system of equations $A\vec{x} = \vec{b}$ where we will perturb both $A$ and $\vec{x}$ resulting in
$$
    (A + \delta A)(\vec{x} + \delta x) = \vec{b}.
$$

Assuming we solve the problem exactly we know that $A\vec{x} = \vec{b}$ and that the infinitesimals multiplied $\delta A \delta x$ are smaller than the other term, and the above expression can be approximation by
$$
\begin{aligned}
    (A + \delta A)(\vec{x} + \delta x) &= \vec{b}, \\
    A\vec{x} + \delta Ax + A \delta x + \delta A \delta \vec{x} &= \vec{b} \\
    \delta A\vec{x} + A \delta x & = 0
\end{aligned}
$$

Solving for $\delta x$ leads to
$$
    \delta x = -A^{-1} \delta A \vec{x}
$$
implying
$$
    ||\delta x|| \leq ||A^{-1}|| ~ ||\delta A|| ~ ||\vec{x}||
$$
and therefore
$$
    \frac{\frac{||\delta x||}{||\vec{x}||}}{\frac{||\delta A||}{||A||}} \leq ||A^{-1}||~||A|| = \kappa(A).
$$

We can also say the following regarding the condition number of a system of equations then

**Theorem:**  Let $\vec{b}$ be fixed and consider the problem of computing $\vec{x}$ in $A\vec{x} = \vec{b}$ where $A$ is square and non-singular.  The condition number of this problem with respect to perturbations in $A$ is the condition number of the matrix $\kappa(A)$.

## Stability

We now return to the consideration of the fact that we are interested not only in the well-conditioning of a mathematical problem but in how we might solve it on a finite precision machine.  In some sense conditioning describes how well we can solve a problem in exact arithmetic and stability how well we can solve the problem in finite arithmetic.

### Accuracy and Stability

As we have defined before we will consider **absolute error** as
$$
    ||F(x) - f(x)||
$$
where $F(x)$ is the approximation to the true solution $f(x)$.  Similarly we can define **relative error** as
$$
    \frac{||F(x) - f(x)||}{||f(x)||}.
$$
In the ideal case we would like the relative error to be $\mathcal{O}(\epsilon_{\text{machine}})$.

#### Forwards Stability

A **forward stable** algorithm for $x \in X$ has

$$
    \frac{||F(x) - f(x)||}{||f(x)||} = \mathcal{O}(\epsilon_{\text{machine}})
$$

In other words
> A forward stable algorithm gives almost the right answer to exactly the right question.

#### Backwards Stability

A stronger notion of stability can also be defined which is satisfied by many approaches in numerical linear algebra.  We say that an algorithm $F$ is **backward stable** if for $x \in X$ we have

$$
    F(x) = f(\hat{\!x})
$$

for some $\hat{\!x}$ with

$$
    \frac{||\hat{\!x} - x||}{||x||} = \mathcal{O}(\epsilon_{\text{machine}}).
$$

In other words
> A backward stable algorithm gives exactly the right answer to nearly the right question.

Combining these ideas along with the idea that we should not expect to be able to accurately compute the solution to a poorly conditioned problem we can form the mixed forward-backward sense of stability as for $x \in X$ if

$$
    \frac{||F(x) - f(\hat{\!x})||}{||f(\hat{\!x})||} = \mathcal{O}(\epsilon_{\text{machine}})
$$

for some $\hat{\!x}$ with

$$
    \frac{||\hat{\!x} - x||}{||x||} = \mathcal{O}(\epsilon_{\text{machine}}).
$$

In other words
> A stable algorithm gives nearly the right answer to nearly the right question.

_An important aspect of the above statement is that we can not necessarily guarantee an accurate result.  If the condition number $\kappa(x)$ is small we would expect that a stable algorithm would give us an accurate result (by definition).  This is reflected in the following theorem._

**Theorem:**  Suppose a backward stable algorithm is applied to solve a problem $f: X \rightarrow Y$ with condition number $\kappa$ on a finite precision machine, then the relative errors satisfy
$$
    \frac{||F(x) - f(\hat{\!x})||}{||f(\hat{\!x})||} = \mathcal{O}(\kappa(x) ~ \epsilon_{\text{machine}}).
$$

**Proof:**  By the definition of the condition number of a problem we can write
$$
    \frac{||F(x) - f(\hat{\!x})||}{||f(\hat{\!x})||} \leq (\kappa(x) + \mathcal{O}(\epsilon_{\text{machine}}))\frac{||\hat{\!x} - x||}{||x||}.
$$
Combining this with the definition of backwards stability we can arrive at the statement of the theorem.

To summarize:
> **Backward Error Analysis** - Process of using the condition number of the problem and stability of the algorithm to determine the error.

> **Forward Error Analysis** - Considers the accrual of error at each step of an algorithm given slightly perturbed input.

# Eigenproblems

## Overview

We will now consider eigenproblems of the form
$$
    A x = \lambda x
$$
where $A \in \mathbb C^{m \times m}$, $x \in \mathbb C^m$ and $\lambda \in \mathbb C$.  The vector $x$ is known as the **eigenvector** and $\lambda$ the **eigenvalue**.  The set of all eigenvalues is called the **spectrum** of $A$.

### Eigenvalue Decomposition
Similar to QR factorization, an eigendecomposition is possible such that $A$ can be written as

$$
    A = X \Lambda X^{-1}
$$

where $X$ is the matrix formed by the eigenvectors $x$ as its columns and $\Lambda$ is a diagonal matrix with the eigenvalues along its diagonal.

This equation comes from the similar equation $A X = X \Lambda$ which is of course related to the original problem statement.  This latter equation can be written out as

$$
    \begin{bmatrix}
          &   &   &   &   \\
          &   &   &   &   \\
          &   & A &   &   \\
          &   &   &   &   \\
          &   &   &   &
    \end{bmatrix}
    \begin{bmatrix}
          &   &   &   &   \\
          &   &   &   &   \\
        x_1 & x_2 & \cdots & x_{m-1} & x_m \\
          &   &   &   &   \\
          &   &   &   &
    \end{bmatrix} =
    \begin{bmatrix}
          &   &   &   &   \\
          &   &   &   &   \\
        x_1 & x_2 & \cdots & x_{m-1} & x_m \\
          &   &   &   &   \\
          &   &   &   &
    \end{bmatrix}
    \begin{bmatrix}
        \lambda_1 &   &   &   &   \\
          & \lambda_2 &   &   &   \\
          &   & \ddots &   &   \\
          &   &   & \lambda_{m-1} &   \\
          &   &   &   & \lambda_m
    \end{bmatrix}
$$

Here we note that the eigenpair $(x_j, \lambda_j)$ are matched as the $j$th column of $X$ and the $j$th element of $\Lambda$ on the diagonal.

**Algebraic multiplicity** is the number of times overall an eigenvalue repeats itself.

**Geometric multiplicity** is defined as the number of linearly independent eigenvectors that belong to each eigenvalue.

If the algebraic multiplicity is equal to the geometric multiplicity for all $\lambda$ then we can say that there is a full eigenspace.

#### Example:  Computing Multiplicities

Compute the geometric and algebraic multiplicities for the following matrices.  What is the relationship between the algebraic and geometric multiplicities?

$$A = \begin{bmatrix}
    2 &   &  \\
      & 2 &  \\
      &   & 2
\end{bmatrix}$$

$$B = \begin{bmatrix} 2
      & 1 &   \\
      & 2 & 1 \\
      &   & 2
\end{bmatrix}$$

1. The characteristic polynomial of $A$ is

  $$
      \mathcal{P}_A(z) = (2 - z)(2 - z)(2 - z) = (2 - z)^3
  $$

  so the eigenvalues are all $\lambda = 2$ so we know the algebraic multiplicity is 3 of this eigenvalue.  The geometric multiplicity is determined by the number of linearly independent eigenvectors.  For this matrix we have three eigenvectors that are all linearly independent which happen to be the unit vectors in each direction (check!).  This means that the geometric multiplicity is also 3.

1. The characteristic polynomial of $B$ is the same as $A$ so again we know $\lambda = 2$ but now we need to be a bit careful about the eigenvectors.  In this case the only eigenvector is a scalar multiple of $e_1$ so the geometric multiplicity is 1.

### Interpretations of the Eigenspace

One way to interpret the eigenproblem is that of one that tries to find the subspaces of $\mathbb C^m$ which act like scalar multiplication by $\lambda$.  The eigenvectors associated with one eigenvalue then form a subspace of $S \subseteq \mathbb C^m$.

When an eigenvalue has algebraic multiplicity that equals its geometric then it is called non-defective and otherwise defective.  This property is also inherited to the matrix so in the above example $A$ and $B$ are non-defective and defective matrices respectively.

### Determinant and Trace

Two important properties of matrices have important relationships with their eigenvalues, namely the determinant and trace.  The determinant we have seen, the **trace** is defined as the sum of the elements on the diagonal of a matrix, in other words
$$
    \text{tr}(A) = \sum^m_{i=1} A_{ii}.
$$

The relationship between the determinant and the eigenvalues is not difficult to guess due to the nature of the characteristic polynomial.  The trace of a diagonal matrix is clear and provides another suggestion to the relationship.

**Theorem:** The determinant $\det(A)$ and trace $\text{trace}(A)$ are equal to the product and sum of the eigenvalues of $A$ respectively counting algebraic multiplicity.

### Similarity Transformations

The relationship between a matrix's eigenvalues and its determinant and trace are due to the special relationship between the eigenvalue decomposition and what is called similarity transformations.  A **similarity transformation** is defined as a transformation that takes A and maps it to $X^{-1} A X$ (assuming $X$ is non-singular).  Two matrices are said to be **similar** if there is a similarity transformation between them.

The most important property of similar matrices is that they have the same characteristic polynomial, eigenvalues, and multiplicities.

This allows us to relate geometric and algebraic multiplicity as

**Theorem:** The algebraic multiplicity of an eigenvalue $\lambda$ is at least as great as its geometric multiplicity.

### Schur Factorization

A **Schur factorization** of a matrix $A$ is defined as

$$
    A = Q T Q^\ast
$$

where $Q$ is unitary and $T$ is upper-triangular.  In particular note that due do the structure of the resulting characteristic polynomial that $A$ and $T$ have identical eigenvalues.

**Theorem:** Every matrix $A \in \mathbb C^{m \times m}$ has a Schur factorization.

Note that the above results imply the following
 - An eigen-decomposition $A = X \Lambda X^{-1}$ exists if and only if $A$ is non-defective (it has a complete set of eigenvectors)
 - A unitary transformation $A = Q \Lambda Q^\ast$ exists if and only if $A$ is normal ($A^\ast A = A A^\ast$)
 - A Schur factorization always exists

Note that each of these lead to a means for isolating the eigenvalues of a matrix and will be useful when considering algorithms for finding them.

## Condition Number of a Simple Eigenvalue

Before we discuss a number of approaches to computing eigenvalues it good to consider what the condition number of a given eigenproblem is.

Let
$$
    Ax = \lambda x
$$
define the eigenvalue problem in question.  Here we will introduce a related problem
$$
    y^\ast A = \lambda y^\ast
$$
where $y$ is the **left eigenvector** and from before $x$ is the **right eigenvector**.  These vectors also can be shown to have the relationship $y^\ast x \neq 0$ for a simple eigenvalue.

Now consider the perturbed problem
$$
    (A + \delta A) (x + \delta x) = (\lambda + \delta \lambda) (x + \delta x).
$$
Expanding this and throwing out quadratic terms and removing the eigenproblem we have
$$
    \delta A x + A \delta x = \delta \lambda x + \lambda \delta x.
$$

Multiple both sides of the above by the left eigenvector and use $y^\ast x \neq 0$ to find
$$\begin{aligned}
    y^\ast \delta A x + y^\ast A \delta x &= y^\ast \delta \lambda x + y^\ast \lambda \delta x \\
    y^\ast \delta A x &= y^\ast \delta \lambda x
\end{aligned}$$
where we again use the slightly different definition of the eigenproblem.  We can then solve for $\delta \lambda$ to find
$$
    \delta \lambda = \frac{y^\ast \delta A x}{y^\ast x}
$$
meaning that the ratio between the dot-product of the left and right eigenvectors and the conjugate dot-product of the matrix $\delta A$ then form a form of bound on the expected error in the simple eigenvalue.

## Computing Eigenvalues

The most obvious approach to computing eigenvalues is a direct computation of the roots of the characteristic polynomial.  Unfortunately the following theorem suggests this is not a good way to compute eigenvalues:

**Theorem:** For an $m \geq 5$ there is a polynomial $\mathcal{P}(z)$ of degree $m$ with rational coefficients that has a real root $\mathcal{P}(z_0) = 0$ with the property that $z_0$ cannot be written using any expression involving rational numbers, addition, subtraction, multiplication, division, and $k$th roots.

Not all is lost however, we just cannot use any direct methods to solve for the eigenvalues.  Instead we must use an iterative approach, in other words we want to construct a sequence that converges to the eigenvalues.  How does this relate to how we found roots previously?

Almost all approaches to computing eigenvalues do so through the computation of the Schur factorization.  The Schur factorization, as we have seen, will preserve the eigenvalues.  The steps to compute the Schur factorization are usually broken down into two steps
1. Directly transform $A$ into a **Hessenberg** matrix, a matrix that contains zeros below its first sub-diagonal, directly using Householder reflections.
1. Use an iterative method to change the sub-diagonal into all zeros

### Hessenberg and Tridiagonal form

What we want to do is construct a sequence of unitary matrices that turns $A$ into a Hessenberg matrix to start.  We can use Householder reflections to do this with the important distinction that we only want to remove zeros below the first sub-diagonal.  The sequence would look something like

$$
    \begin{bmatrix}
        \text{x} & \text{x} & \text{x} & \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x} & \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x} & \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x} & \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x} & \text{x} & \text{x}
    \end{bmatrix} \overset{Q_1}{\rightarrow}
    \begin{bmatrix}
        \text{x} & \text{x} & \text{x}& \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x}& \text{x} & \text{x} \\
        0 & \text{x} & \text{x}& \text{x} & \text{x} \\
        0 & \text{x} & \text{x}& \text{x} & \text{x} \\
        0 & \text{x} & \text{x}& \text{x} & \text{x}
    \end{bmatrix} \overset{Q_2}{\rightarrow}
    \begin{bmatrix}
        \text{x} & \text{x} & \text{x}& \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x}& \text{x} & \text{x} \\
        0 & \text{x} & \text{x}& \text{x} & \text{x} \\
        0 & 0 & \text{x}& \text{x} & \text{x} \\
        0 & 0 & \text{x}& \text{x} & \text{x}
    \end{bmatrix} \overset{Q_3}{\rightarrow}
    \begin{bmatrix}
        \text{x} & \text{x} & \text{x}& \text{x} & \text{x} \\
        \text{x} & \text{x} & \text{x}& \text{x} & \text{x} \\
        0 & \text{x} & \text{x}& \text{x} & \text{x} \\
        0 & 0 & \text{x}& \text{x} & \text{x} \\
        0 & 0 & 0 & \text{x} & \text{x}
    \end{bmatrix}
$$

so we have the sequence $Q^\ast_1 A Q_1$.  Note we need both to preserve the entries of the first column that are not being transformed to zeros.

One important special case of this sequence of transformations is that if the matrix $A$ is hermitian (the matrix is its own conjugate transpose, $A = A^\ast$, or symmetric in the real case) then the Hessenberg matrix is tridiagonal.

We now will focus on how to formulate the iteration step of the eigenproblem.  We will also restrict our attention to symmetric, real matrices.  This implies that all eigenvalues will be real and have a complete set of orthogonal eigenvectors.  Generalizations can be made of many of the following algorithms but is beyond the scope of this class.

## Rayleigh Quotient and Inverse Iteration

There are a number of classical approaches to computing the iterative step above which we will review here.  Inverse power iteration in particular is today still the dominant means of finding the eigenvectors once the eigenvalues are known.

### Rayleigh Quotient

The **Rayleigh quotient** of a vector $x \in \mathbb R^m$ is the scalar
$$
    r(x) = \frac{x^T A x}{x^T x}.
$$
The importance of the Rayleigh quotient is made clear when we evaluate $r(x)$ at an eigenvector.  When this is the case the quotient evaluates to the corresponding eigenvalue.

The Rayleigh quotient can be motivated by asking the question, given an eigenvector $x$, what value $\alpha$ acts most like an eigenvalue in an $\ell_2$ sense:
$$
    \min_\alpha ||A x - \alpha x||_2.
$$

This can be reformulated as a least-squares problem noting that $x$ is the "matrix", $\alpha$ is the unknown vector (scalar) and $Ax$ is the right-hand side so we have
$$
    (x^T x) \alpha = x^T (A x)
$$
which can be solved so that
$$
    \alpha = r(x) = \frac{x^T A x}{x^T x}.
$$

### Power Iteration

Power iteration is a straight forward approach to finding the eigenvector of the largest eigenvalue of $A$.  The basic idea is that the sequence
$$
    \frac{x}{||x||}, \frac{Ax}{||Ax||}, \frac{A^2x}{||A^2x||}, \frac{A^3x}{||A^3x||}, \ldots
$$
will converge (although very slowly) to the desired eigenvector.

We implement this method by initializing the algorithm with some vector $v$ with $||v|| = 1$.  We then apply the sequence of multiplications.

The reason why this works can be seen by considering the initial vector $v$ as a linear combination of the orthonormal eigenvectors (which we have assumed exist) such that

$$
    v^{(0)} = a_1 q_1 + a_2 q_2 + \cdots + a_m q_m.
$$

Multiplying $v^{(0)}$ by $A$ then leads to

$$\begin{aligned}
    Av^{(0)} = v^{(1)} &= a_1 A q_1 + a_2 A q_2 + \cdots + a_m A q_m \\
    &= c_1 (a_1 \lambda_1 q_1 + a_2 \lambda_2 q_2 + \cdots + a_m \lambda_m q_m) \\
\end{aligned}$$

where $c_1$ is some constant due to the fact the eigenvectors are not uniquely specified.  Repeating this $k$ times we have

$$\begin{aligned}
    Av^{(k-1)} = v^{(k)} &= a_1 A^k q_1 + a_2 A^k q_2 + \cdots + a_m A^k q_m \\
    &= c_k (a_1 \lambda_1^k q_1 + a_2 \lambda_2^k q_2 + \cdots + a_m \lambda_m^k q_m) \\
    &= c_k \lambda_1^k \left(a_1 q_1 + a_2 \frac{\lambda_2^k}{\lambda_1^k} q_2 + \cdots + a_m \frac{\lambda_m^k}{\lambda_1^k} q_m \right)
\end{aligned}$$

Since $\lambda_1 > \lambda_i$ for

### Inverse Iteration

Inverse iteration uses a similar approach with the difference being that we can use it to find any of the eigenvectors for the matrix $A$.

Consider the matrix

$$
    (A - \mu I)^{-1},
$$

the eigenvectors of this matrix are the same as $A$ with the eigenvalues

$$
    (\lambda_j - \mu)^{-1}
$$

where $\lambda_j$ are the eigenvalues of $A$.

If $\mu$ is close to a particular $\lambda_j$, say $\lambda_J$, then

$$
    (\lambda_J - \mu)^{-1}
$$

will be larger than any of the other $(\lambda_j - \mu)^{-1}$.  In this way we effectively have picked out the eigenvalue we want to consider in the power iteration!

### Rayleigh Quotient Iteration

By themselves the above approaches are not particularly useful but combining them we can iterate back and forth to find the eigenvalue, eigenvector pair:
1. Compute the Rayleigh quotient and find an estimate for $\lambda_j$
1. Compute one step of inverse iteration to approximate $x_j$
1. Repeat...

```python
import numpy
import matplotlib.pyplot as plt

m = 3
A = numpy.array([[2, 1, 1], [1, 3, 1], [1, 1, 4]])

num_steps = 10
v = numpy.empty((num_steps, m))
lam = numpy.empty(num_steps)

v[0, :] = numpy.array([1, 1, 1])
v[0, :] = v[0, :] / numpy.linalg.norm(v[0, :], ord=2)
lam[0] = numpy.dot(v[0,:], numpy.dot(A, v[0, :]))
for k in range(1, num_steps):
    w = numpy.linalg.solve(A - lam[k-1] * numpy.identity(m), v[k-1, :])
    v[k, :] = w / numpy.linalg.norm(w, ord=2)
    lam[k] = numpy.dot(v[k,:], numpy.dot(A, v[k, :]))

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
axes.semilogy(range(10), numpy.abs(lam - numpy.linalg.eigvals(A)[0]), 'o')

axes.set_title("Convergence of eigenvalue")
axes.set_xlabel("Step")
axes.set_ylabel("Error")

plt.show()
```

## QR Algorithm

The most basic use of a $QR$ factorization to find eigenvalues is to iteratively compute the factorization and multiply the resulting $Q$ and $R$ in the reverse order.  This sequence will eventually converge to the Schur decomposition of the matrix $A$.

```python
%precision 6
m = 3
A = numpy.array([[2, 1, 1], [1, 3, 1], [1, 1, 4]])
MAX_STEPS = 10

for i in range(MAX_STEPS):
    Q, R = numpy.linalg.qr(A)
    A = numpy.dot(R, Q)
    print()
    print("A(%s) =" % (i))
    print(A)

print()
print("True eigenvalues: ")
print(numpy.linalg.eigvals(A))
print()
print("Computed eigenvalues: ")
for i in range(m):
    print(A[i, i])
```

    A(0) =
    [[ 4.166667e+00  1.095445e+00 -1.267105e+00]
     [ 1.095445e+00  2.000000e+00  3.187966e-17]
     [-1.267105e+00  2.153667e-16  2.833333e+00]]

    A(1) =
    [[ 5.090909  0.15743   0.623249]
     [ 0.15743   1.86182  -0.54704 ]
     [ 0.623249 -0.54704   2.047271]]

    A(2) =
    [[ 5.198682 -0.075903 -0.207275]
     [-0.075903  2.181835  0.496554]
     [-0.207275  0.496554  1.619484]]

    A(3) =
    [[ 5.211648 -0.065892  0.058223]
     [-0.065892  2.363696 -0.321368]
     [ 0.058223 -0.321368  1.424655]]

    A(4) =
    [[ 5.213778 -0.035826 -0.01528 ]
     [-0.035826  2.430805  0.183737]
     [-0.01528   0.183737  1.355417]]

    A(5) =
    [[ 5.214202e+00 -1.758371e-02  3.920120e-03]
     [-1.758371e-02  2.451920e+00 -1.007513e-01]
     [ 3.920120e-03 -1.007513e-01  1.333877e+00]]

    A(6) =
    [[ 5.214294e+00 -8.392060e-03 -9.988671e-04]
     [-8.392060e-03  2.458212e+00  5.453885e-02]
     [-9.988671e-04  5.453885e-02  1.327494e+00]]

    A(7) =
    [[ 5.214314e+00 -3.973357e-03  2.540044e-04]
     [-3.973357e-03  2.460055e+00 -2.940989e-02]
     [ 2.540044e-04 -2.940989e-02  1.325631e+00]]

    A(8) =
    [[ 5.214318e+00 -1.876920e-03 -6.455358e-05]
     [-1.876920e-03  2.460591e+00  1.584128e-02]
     [-6.455358e-05  1.584128e-02  1.325090e+00]]

    A(9) =
    [[ 5.214319e+00 -8.860220e-04  1.640309e-05]
     [-8.860220e-04  2.460747e+00 -8.529903e-03]
     [ 1.640309e-05 -8.529903e-03  1.324933e+00]]

    True eigenvalues:
    [5.21432  2.460811 1.324869]

    Computed eigenvalues:
    5.214319458186374
    2.4607473568633944
    1.324933184950233

So why does this work?  The first step is to find the $QR$ factorization of $A^{(k-1)}$ which is equivalent to finding

$$
    (Q^{(k)})^T A^{(k-1)} = R^{(k)}
$$

and multiplying on the right leads to

$$
    (Q^{(k)})^T A^{(k-1)} Q^{(k)} = R^{(k)} Q^{(k)}.
$$

In this way we can see that this is a similarity transformation of the matrix $A^{(k-1)}$ since the $Q^{(k)}$ is an orthogonal matrix ($Q^{-1} = Q^T$). This of course is not a great idea to do directly but works great in this case as we iterate to find the upper triangular matrix $R^{(k)}$ which is exactly where the eigenvalues appear.

In practice this basic algorithm is modified to include a few additions:

1. Before starting the iteration $A$ is reduced to tridiagonal form.
1. Motivated by the inverse power iteration we observed we instead consider a shifted matrix $A^{(k)} - \mu^{(k)} I$ for factoring.  The $\mu$ picked is related to the estimate given by the Rayleigh quotient.  Here we have

$$
    \mu^{(k)} = \frac{(q_m^{(k)})^T A q_m^{(k)}}{(q_m^{(k)})^T q_m^{(k)}} = (q_m^{(k)})^T A q_m^{(k)}.
$$

1. Deflation is used to reduce the matrix $A^{(k)}$ into smaller matrices once (or when we are close to) finding an eigenvalue to simplify the problem.

This has been the standard approach until recently for finding eigenvalues of a matrix.

## Alternatives

### Jacobi

Jacobi iteration employs the idea that we know the eigenvalues of a matrix of size equal to or less than 4 (we know the roots of the characteristic polynomial directly).  Jacobi iteration therefore attempts to break the matrix down into at most 4 by 4 matrices along the diagonal via a series of similarity transformations based on only diagonalizing sub-matrices 4 by 4 or smaller.

### Bisection

It turns out if you do not want all of the eigenvalues of a matrix that using a bisection method to find some subset of the eigenvalues is often the most efficient way to get these.  This avoids the pitfall of trying to find the eigenvalues via other root-finding approaches by only needing evaluations of the function and if a suitable initial guess is provided can find the eigenvalue quickly that is closest to the initial bracket provided.

### Divide-and-conquer

This algorithm is actually the one used most often used if both eigenvalues and eigenvectors are needed and performs up to twice as fast as the $QR$ approach.  The basic idea is to split the matrix into two pieces at every iteration by introducing zeros on the appropriate off-diagonals which neatly divides the problem into two pieces.

```python

```


# RNN

RNN专门用于解决序列式输入的输入数据，如天气预报的天气特征需要之前多天的数据，而这些数据之间会有一定的依赖关系。

**RNN的本质是一个线性层**，

其中RNN Cell为各数据共享（一条序列），所以权重数量实际较少，另外，输入数据 $\mathbf{X_t} \in \mathbf{R}^{n \times d}$，隐藏数据 $\mathbf{H_t} \in \mathbf{R}^{n \times h}$，其中 n 为数据数量，或者理解为前后有关联的数据个数，而 d 和 h 分别为输入数据的特征维度以及隐藏数据的“层数”。\begin{equation}
\boldsymbol{H}_{t}=\phi\left(\boldsymbol{X}_{t} \boldsymbol{W}_{x h}+\boldsymbol{H}_{t-1} \boldsymbol{W}_{h h}+\boldsymbol{b}_{h}\right)
\tag{a}\label{eq:1}
\end{equation}
式$(a)$即包含了之前的状态变量对当前状态的影响。为了方便计算，我们可以将$\boldsymbol{X_t}$和$\boldsymbol{H_{t-1}}$连结之后与$\boldsymbol{W_{xh}}$和$\boldsymbol{W_{hh}}$连结之后的矩阵相乘，得到与式 (\ref{eq:1})相同的结果。

换言之，在 RNN 的单元内，有多个权重矩阵，分别管理隐藏层、输入层和输出层，对于文字模型来说，最方便的构造输入向量的方式为独热编码，但是有一些坏处如会占用大量内存。

```python
import torch

#initialize the parameters of dimensions
batch_size = 16
seq_len = 3
input_size = 4
hidden_size = 2

cell = torch.nn.RNNCell(input_szie=input_size, hidden_size=hidden_size)

#(seq_len, batch_size, input_size)
dataset = torch.randn(seq_len, batch_size, input_size)
hidden = torch.randn(batch_size, input_size)

for idx, data in enumerate(dataset):
    hidden = cell(input, hidden)
```

此Cell仅实现一个单元计算，要获得各序列的计算结果需要循环调用。

```python
batch_size = 16
seq_len = 3
input_size = 4
hidden_size = 2
num_layers = 1

cell = torch.nn.RNN(input_size=input_size, hidden_size=hidden_size,
                    num_layers=num_layers)

inputs = torch.randn(seq_len, batch_size, input_size)
out, hidden = cell(inputs, hidden)
```

RNN则直接完成了循环操作

```python

```


# Partial Differential Equations and Gaussian Process with GPy

#### presented at the EBI BioPreDyn Course 'The Systems Biology Modelling Cycle'

#### Mu Niu, Neil Lawrence, 12th May 20014, University of Sheffield

## The Spatio-Temporal Model

In this notebook we consider the latent force model paradigm (Alvarez et al, 2013, Alvarez et al, 2011, Alvarez et al, 2009) to drive spatio-temporal differential equation with a Gaussian process. Latent force models are differential equations whose initial conditions or driving forces are given by a stochastic process. Linear latent force models are linear (partial or ordinary) differential equations. If a linear latent force model is driven by a Gaussian process latent force, this describes a joint Gaussian process distribution across all the variables of interest. The Gaussian process covariance function encodes the relationships between the variables, as proscribed by the differential equation, through the covariance function.

This direction of research is part of our ideas as to how to merge mechanistic and statistical models of data. It also maps onto our ideas about 'Gaussian processes over everything'. The covariance function interelates the protein and mRNA concentrations in the same model.

### Differential Equation Model

A model of post-transcriptional processing is formulated to describe the spatio-temporal Drosophila
protein expression data (Becker et al, 2013, Alvarez et al, 2011). Protein production is considered to be linearly dependent on the concentration of mRNA at an earlier time point. The model also allows for diffusion of protein between nuclei and linear protein decay. These processes are dependent on the diffusion parameter and the degradation rate of protein respectively.

\begin{equation}
 a \frac{\partial ^2 y_{x,t}}{\partial x^2} + b \frac{\partial y_{x,t}}{\partial t} + c y_{x,t}=  f_{x,t}
\end{equation}

The coefficients $a$, $b$ and $c$ are unknown. In this study, we use Gaussian process with an exponentiated quadratic kernel as a prior over $y_{x,t}$ (protein). The kernel of $f_{x,t}$ (mRNA) is derived by applying the partial differential operator on the spatial-temporal kernel of protein. The multi-output Gaussian process are developed by combining the covariance matrix of mRNA and protein and their cross covariance.

```
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import numpy as np
import pylab as pb
import GPy
import pandas
from pandas import read_csv
```

The spatio-temporal multi-output partial differential equation covariance functions have been developed with the kernel name `GPy.kern.ODE_st` in GPy. The inputs are one dimension spatial data, one dimension temporal data and one dimension index which is used to indicate $f$ and $y$.

###

```
data = GPy.util.datasets.drosophila_knirps()
```

The next thing to do is to compose the data ready for presentation to GPy. Here we need to use the time field as the input and a concatanation of the expression levels as the output. We need to provide a corresponding index for to each input to describe what output is being represented.

Now we set up the covariance function with the relevant parameters.

```
kern = GPy.kern.ODE_st(input_dim=3,
                       a=1., b=1., c=1.,
                       variance_Yx=1., variance_Yt=1.,
                       lengthscale_Yx=15.,
                       lengthscale_Yt=15.)
```

With the data correctly presented and the covariance function defined, we are ready to proceed with the Gaussian process regression.

```
m = GPy.models.GPRegression(data['X'],data['Y'],kern)
```

# Initial Fit

The initial value of $a$, $b$ and $c$ are 1. For these choices of covariance function parameters, we can plot the random field of $f$ and $y$ separately.

```
leng = data['X'].shape[0]
m.plot(fixed_inputs=[(2,0)], which_data_rows = slice(0,leng*2/2))
m.plot(fixed_inputs=[(2,1)], which_data_rows = slice(leng*2/2,leng*2))
```

    {'contour': <matplotlib.contour.QuadContourSet instance at 0x10f9a32d8>,
     'dataplot': <matplotlib.collections.PathCollection at 0x10f9b97d0>}

Now we optimize the model, this will take a few minutes.

```
m.optimize(messages=True)
```

     I      F              Scale          |g|
    0003   7.614997e+04   2.500000e-01   2.504914e+09
    0008   1.133084e+04   7.812500e-03   8.476310e+07
    0016   3.600478e+03   3.051758e-05   6.694209e+04
    0022   2.830839e+03   4.768372e-07   5.181429e+02
    0031   2.768527e+03   9.313226e-10   7.879161e+01
    0053   2.747934e+03   1.000000e-15   7.722679e+00
    0069   2.746164e+03   1.000000e-15   5.889059e-02
    0082   2.746153e+03   1.000000e-15   7.165007e-03
    0089   2.746153e+03   1.000000e-15   1.479437e-04
    0089   2.746153e+03   1.000000e-15   1.479437e-04
    converged - relative reduction in objective

After optimization, the estimated value of $a$, $b$ and $c$ can be printed.

```
print m
```

      GP_regression.           |      Value       |  Constraint  |  Prior  |  Tied to
      ode_st.a                 |  0.438768542606  |     +ve      |         |
      ode_st.b                 |   10.0747613738  |     +ve      |         |
      ode_st.c                 |  0.620385966622  |     +ve      |         |
      ode_st.variance_Yt       |   32.8716774332  |     +ve      |         |
      ode_st.variance_Yx       |   32.8716774332  |     +ve      |         |
      ode_st.lengthscale_Yt    |   18.3997301624  |     +ve      |         |
      ode_st.lengthscale_Yx    |   14.3560437898  |     +ve      |         |
      Gaussian_noise.variance  |   3.26998881271  |     +ve      |         |

The plot of the random fields are plotted below.

```
m.plot(fixed_inputs=[(2,0)], which_data_rows = slice(0,leng*2/2))
pb.savefig("gene.pdf")
m.plot(fixed_inputs=[(2,1)], which_data_rows = slice(leng*2/2,leng*2))
pb.savefig("protein.pdf")
```

In Kolja Becker's thesis he showed the value of $\alpha$, D and $\lambda$. Should the Gaussian process estimation of $a$, $b$ and $c$ be the same as the value in the thesis? The value of $\lambda$ is equal to 1/$b$.

#### work funded by the BioPreDyn project, it is a collaboration with Nicolas Durrande, Johannes Jaeger.

```

```


# Agricultural Pricing

## Objective and Prerequisites

This problem is an example of an agricultural pricing problem, where we want to determine the prices and  demand for the dairy products of a country in order to maximize total revenue derived from the sales of those products. This problem is formulated as a quadratic optimization problem using the Gurobi Python API and solved with the Gurobi Optimizer.

This model is example 21 from the fifth edition of Model Building in Mathematical Programming, by H. Paul Williams on pages 276-278 and 333-335.

This modeling example is at the intermediate level, where we assume that you know Python and are familiar with the Gurobi Python API. In addition, you should have some knowledge about building mathematical optimization models.

**Download the Repository**

You can download the repository containing this and other examples by clicking [here](https://github.com/Gurobi/modeling-examples/archive/master.zip).

**Gurobi License**

In order to run this Jupyter Notebook properly, you must have a Gurobi license. If you do not have one, you can request an [evaluation license](https://www.gurobi.com/downloads/request-an-evaluation-license/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-EDU-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_AGRICULTURAL_PRICING_COM_EVAL_GitHub&utm_term=Agricultural%20Pricing&utm_content=C_JPM) as a *commercial user*, or download a [free license](https://www.gurobi.com/academia/academic-program-and-licenses/?utm_source=3PW&utm_medium=OT&utm_campaign=WW-MU-EDU-OR-O_LEA-PR_NO-Q3_FY20_WW_JPME_AGRICULTURAL_PRICING_ACADEMIC_EVAL_GitHub&utm_term=Agricultural%20Pricing&utm_content=C_JPM) as an *academic user*.

## Problem Description

The government of a country wants to decide what prices should be charged for its dairy products: milk, butter, and cheese. All these products
are produced
(directly or indirectly) from the country’s raw milk production operations. This raw milk is divided into two main components: fat and dry matter. After subtracting the quantities of fat and dry matter, which are used for making products for export or consumption on the farms, there is a total yearly availability of 600,000 tons of fat and 750,000 tons of dry matter. This is all available for producing milk, butter and two kinds of cheese for domestic consumption. The percentage composition of the products are given in the following table:

| Composition | Fat (%) | Dry matter (%) |
| --- | --- | --- |
| Milk | 4 | 9 |
| Butter | 80 | 2 |
| Cheese 1 | 35 | 30 |
| Cheese 2 | 25 | 40 |

The table below shows last year's domestic consumption (demand) and prices for the dairy products:

| Dairy
 products | Milk | Butter | Cheese 1 | Cheese 2 |
| --- | --- | --- | --- | --- |
| Demand (1000 tons) | 4.82 | 0.32 |  0.21 | 0.07 |
| Price (dollars/ton) | 297 | 720 | 1050 | 815 |

The elasticities and cross-elasticities are given in the table below:

| Milk | Butter | Cheese 1 | Cheese 2 | Cheese 1 to
 Cheese 2 |  Cheese 2 to
 Cheese 1 |
| --- | --- | --- | --- | --- |  --- |
| 0.4 | 2.7 | 1.1 |  0.4 | 0.1 |  0.4 |

The price index cannot be raised higher than last year.This constraint establishes that the new prices must be such that the total cost of last year’s consumption would not be increased.
Last year's price index is 1.939 (measured in thousand dollars).

The goal is to determine what prices and demand  maximizes the total revenue.

## Model Formulation

### Sets and Indices

$d \in \text{Dairy}=\{\text{milk}, \text{butter}, \text{cheese1}, \text{cheese2} \}$

$c \in \text{Components}=\{\text{fat}, \text{dry_matter} \}$

### Parameters

$\text{capacity}_{c} \in \mathbb{R}^+$: Yearly availability of component $c$ (1000 tons).

$\text{qtyper}_{c,d} \in [0,1]$: Percentage of component $c$ in dairy product $d$.

$\text{consumption}_{d} \in \mathbb{R}^+$: Last year domestic consumption of dairy product  $d$ (1000 tons).

$\text{price}_{d} \in \mathbb{R}^+$: Last year price of dairy product $d$ (dollars/1000 tons).

$\text{elasticity}_{d} \in \mathbb{R}^+$: Last year price elasticity of domestic consumption of dairy product $d$.

$\text{elasticity12} \in \mathbb{R}^+$: Last year price cross-elasticity of domestic consumption of cheese 1 and cheese 2.

$\text{elasticity21} \in \mathbb{R}^+$: Last year price cross-elasticity of domestic consumption of cheese 2 and cheese 1.

$\text{prcIndex} \in \mathbb{R}^+$: Price index reflecting last year total consumption cost.

### Decision Variables

$\text{p}_{d} \in \mathbb{R}^+$: Price of dairy product $d$ (dollars/1000 tons).

$\text{q}_{d} \in \mathbb{R}^+$: Demand of dairy product $d$ (1000 tons).

### Constraints

**Capacity**: The limited availabilities of fat and dry matter are enforced by the following constraints.

\begin{equation}
\sum_{d \in \text{Dairy}}{\text{qtyper}_{c,d}*\text{q}_{d} } \leq \text{capacity}_{c} \quad \forall c \in \text{Components}
\end{equation}

**Price index**: This constraint establishes that the new prices must be such that the total cost of last year’s consumption would not be increased.

\begin{equation}
\sum_{d \in \text{Dairy}}{\text{consumption}_{d}*\text{p}_{d} } \leq \text{prcIndex}
\end{equation}

**Elasticities**: The demand variables $q_{d}$ are related to the price variables $p_{d}$  through the price elasticities relationships. We approximate the elasticities with linear relationships.

Milk elasticity.
$$
(\text{q}_{milk} - \text{consumption}_{milk})/\text{consumption}_{milk}) = -\text{elasticity}_{milk}*(\text{p}_{milk} - \text{price}_{milk})/\text{price}_{milk})
$$

Butter elasticity.
$$
(\text{q}_{butter} - \text{consumption}_{butter})/\text{consumption}_{butter}) = -\text{elasticity}_{butter}*(\text{p}_{butter} - \text{price}_{butter})/\text{price}_{butter})
$$

Cheese 1 elasticity.
$$
(\text{q}_{cheese1} - \text{consumption}_{cheese1})/\text{consumption}_{cheese1}) = -\text{elasticity}_{cheese1}*(\text{p}_{cheese1} - \text{price}_{cheese1})/\text{price}_{cheese1})
$$

$$
+ elasticity12*(\text{p}_{cheese2} - \text{price}_{cheese2})/\text{price}_{cheese2})
$$

Cheese 2 elasticity.
$$
(\text{q}_{cheese2} - \text{consumption}_{cheese2})/\text{consumption}_{cheese2}) = -\text{elasticity}_{cheese2}*(\text{p}_{cheese2} - \text{price}_{cheese2})/\text{price}_{cheese2})
$$

$$
+ elasticity21*(\text{p}_{cheese1} - \text{price}_{cheese1})/\text{price}_{cheese1})
$$

### Objective Function

**Revenue**: The objective is to maximize total revenue.

\begin{equation}
\text{Maximize} \quad \sum_{d \in \text{Dairy}}{\text{q}_{d}*\text{p}_{d} }
\end{equation}

---
## Python Implementation

We import the Gurobi Python Module and other Python libraries.

```python
import numpy as np
import pandas as pd

import gurobipy as gp
from gurobipy import GRB

# tested with Python 3.7.0 & Gurobi 9.0
```

## Input data

We define all the input data for the model.

```python
# List of dairy products.

dairy = ['milk', 'butter', 'cheese1', 'cheese2']

components = ['fat', 'dryMatter']

# Create a dictionary to capture the percentage composition of the products.

cd, qtyper = gp.multidict({
    ('fat','milk'): 0.04,
    ('fat','butter'): 0.8,
    ('fat','cheese1'): 0.35,
    ('fat','cheese2'): 0.25,
    ('dryMatter','milk'): 0.09,
    ('dryMatter','butter'): 0.02,
    ('dryMatter','cheese1'): 0.3,
    ('dryMatter','cheese2'): 0.4
})

# Create a dictionary to capture the yearly availability of components (1000 tons).

components, capacity = gp.multidict({
    ('fat'): 600,
    ('dryMatter'): 750
})

# Create a dictionary to capture last year's domestic consumption and prices

dairy, consumption, price, elasticity = gp.multidict({
    ('milk'): [4.82, 0.297, 0.4],
    ('butter'): [0.32, 0.72, 2.7],
    ('cheese1'): [0.21, 1.05, 1.1],
    ('cheese2'): [0.07, 0.815, 0.4]
})

elasticity12 = 0.1
elasticity21 = 0.4

priceIndex = 1.939
```

## Model Deployment

We create a model and the variables. The decision variables of this model are the prices and demand for the dairy products.

Solving bilinear problems with Gurobi is as easy as configuring the global parameter `nonConvex`, and setting this parameter to the value of 2.

```python
model = gp.Model('AgriculturalPricing')

# Set global parameters.
model.params.nonConvex = 2

# Quantity of dairy products.
qvar = model.addVars(dairy, name="qvar")

# Price of dairy products.
pvar = model.addVars(dairy, name="pvar")

```

    Using license file c:\gurobi\gurobi.lic
    Set parameter TokenServer to value SANTOS-SURFACE-
    Changed value of parameter nonConvex to 2
       Prev: -1  Min: -1  Max: 2  Default: -1

The limited availabilities of fat and dry matter are enforced by the following constraints.

```python
# Capacity constraint.

fatCap = model.addConstrs( (gp.quicksum(qtyper[c,d]*qvar[d] for d in dairy) <= capacity[c] for c in components  ),
                          name='fatCap')
```

This constraint ensures that the new prices must be such that the total cost of last year’s consumption would not be increased.

```python
# Price index constraint.

priceIndex = model.addConstr( (gp.quicksum(consumption[d]*pvar[d] for d in dairy) <= priceIndex ), name='priceIndex')
```

The demand variables are related to the price variables  through the price elasticities relationships. We approximate the elasticities with linear relationships.

```python
# Elasticity constraints

elasMilk = model.addConstr( (qvar['milk']-consumption['milk'])/consumption['milk']
                           == -elasticity['milk']*(pvar['milk']-price['milk'])/price['milk'], name='elasMilk')

elasButter = model.addConstr( (qvar['butter']-consumption['butter'])/consumption['butter']
                           == -elasticity['butter']*(pvar['butter']-price['butter'])/price['butter'], name='elasButter')

elasCheese1 = model.addConstr( (qvar['cheese1']-consumption['cheese1'])/consumption['cheese1']
                           == -elasticity['cheese1']*(pvar['cheese1']-price['cheese1'])/price['cheese1']
                              +elasticity12*(pvar['cheese2']-price['cheese2'])/price['cheese2'] , name='elasCheese1')

elasCheese2 = model.addConstr( (qvar['cheese2']-consumption['cheese2'])/consumption['cheese2']
                           == -elasticity['cheese2']*(pvar['cheese2']-price['cheese2'])/price['cheese2']
                              +elasticity21*(pvar['cheese1']-price['cheese1'])/price['cheese1'] , name='elasCheese2')
```

The objective function is to maximize revenue.

```python
# Quadratic objective function.

obj = gp.quicksum(qvar[d]*pvar[d] for d in dairy)

model.setObjective(obj, GRB.MAXIMIZE)
```

```python
# Verify model formulation

model.write('AgriculturalPricing.lp')

# Run optimization engine

model.optimize()
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 7 rows, 8 columns and 22 nonzeros
    Model fingerprint: 0x4fffdf2e
    Model has 4 quadratic objective terms
    Coefficient statistics:
      Matrix range     [2e-02, 1e+01]
      Objective range  [0e+00, 0e+00]
      QObjective range [2e+00, 2e+00]
      Bounds range     [0e+00, 0e+00]
      RHS range        [1e+00, 8e+02]
    Presolve removed 2 rows and 0 columns

    Continuous model is non-convex -- solving as a MIP.

    Presolve removed 2 rows and 0 columns
    Presolve time: 0.00s
    Presolved: 14 rows, 13 columns, 36 nonzeros
    Presolved model has 4 bilinear constraint(s)
    Variable types: 13 continuous, 0 integer (0 binary)

    Root relaxation: objective 2.791207e+00, 11 iterations, 0.00 seconds

        Nodes    |    Current Node    |     Objective Bounds      |     Work
     Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

         0     0    2.79121    0    4          -    2.79121      -     -    0s
    H    0     0                       2.0422943    2.79121  36.7%     -    0s
         0     0    2.23683    0    4    2.04229    2.23683  9.53%     -    0s
         0     0    2.18216    0    4    2.04229    2.18216  6.85%     -    0s
    H    0     0                       2.0596850    2.18216  5.95%     -    0s
         0     0    2.17036    0    4    2.05968    2.17036  5.37%     -    0s
         0     0    2.16917    0    4    2.05968    2.16917  5.32%     -    0s
         0     0    2.16441    0    4    2.05968    2.16441  5.08%     -    0s
         0     0    2.16252    0    4    2.05968    2.16252  4.99%     -    0s
         0     0    2.15497    0    4    2.05968    2.15497  4.63%     -    0s
    H    0     0                       2.0598297    2.15497  4.62%     -    0s
         0     0    2.15391    0    4    2.05983    2.15391  4.57%     -    0s
    H    0     0                       2.0626177    2.15391  4.43%     -    0s
    H    0     0                       2.0626717    2.15391  4.42%     -    0s
         0     2    2.15391    0    4    2.06267    2.15391  4.42%     -    0s
    *   24    22               5       2.0630153    2.09607  1.60%   1.8    0s
    *   39    32               7       2.0631383    2.08439  1.03%   1.5    0s
    *   48    38               8       2.0631594    2.08439  1.03%   1.4    0s
    *   49    38               9       2.0631614    2.08439  1.03%   1.4    0s
    *   72    56              10       2.0656071    2.08439  0.91%   1.5    0s
    *   74    56              11       2.0657765    2.08439  0.90%   1.4    0s
    *  112    64              12       2.0661200    2.07903  0.62%   1.3    0s
    *  133    70              14       2.0661344    2.07903  0.62%   1.3    0s
    *  134    70              14       2.0661346    2.07903  0.62%   1.3    0s
    *  137    70              15       2.0661465    2.07903  0.62%   1.3    0s
    *  141    70              20       2.0663670    2.07903  0.61%   1.3    0s
    *  142    70              21       2.0663670    2.07903  0.61%   1.3    0s
    *  251    72              22       2.0663877    2.07053  0.20%   1.3    0s
    *  296    50              23       2.0663878    2.06861  0.11%   1.3    0s
    *  331    50              20       2.0663882    2.06861  0.11%   1.2    0s
    *  340    41              21       2.0663884    2.06784  0.07%   1.2    0s
    *  343    41              22       2.0663890    2.06784  0.07%   1.2    0s
    *  344    41              22       2.0663896    2.06784  0.07%   1.2    0s
    *  346    41              22       2.0663898    2.06784  0.07%   1.2    0s
    *  347    41              22       2.0663901    2.06784  0.07%   1.2    0s
    *  439    70              26       2.0664033    2.06683  0.02%   1.2    0s
    *  502    33              25       2.0664036    2.06652  0.01%   1.1    0s
    *  523    33              25       2.0664040    2.06648  0.00%   1.1    0s
    *  537    33              24       2.0664051    2.06645  0.00%   1.1    0s
    *  539    33              25       2.0664079    2.06645  0.00%   1.1    0s
    *  540    33              25       2.0664084    2.06645  0.00%   1.1    0s
    *  554    33              26       2.0664084    2.06645  0.00%   1.1    0s

    Cutting planes:
      RLT: 9

    Explored 581 nodes (655 simplex iterations) in 0.14 seconds
    Thread count was 8 (of 8 available processors)

    Solution count 10: 2.06641 2.06641 2.06641 ... 2.06639

    Optimal solution found (tolerance 1.00e-04)
    Best objective 2.066407339147e+00, best bound 2.066454234042e+00, gap 0.0023%

## Analysis

The table below shows the price (dollars/ton) and demand (tons) at equilibrium, for each dairy product. The total revenue generated is $\$ 2,067,170,173$.

```python
# Output Report
price_demand = pd.DataFrame(columns=["Products", "Price", "Demand"])
for d in dairy:
    price_demand = price_demand.append({"Products": d, "Price": '${:,.2f}'.format(round(1000*pvar[d].x)), "Demand": '{:,.2f}'.format(round(1e6*qvar[d].x))}, ignore_index=True)
price_demand.index=[''] * len(price_demand)
price_demand
```

      Products
      Price
      Demand

      milk
      $322.00
      4,658,539.00

      butter
      $422.00
      677,741.00

      cheese1
      $832.00
      265,745.00

      cheese2
      $1,113.00
      53,921.00

---
## References

H. Paul Williams, Model Building in Mathematical Programming, fifth edition.

Copyright © 2020 Gurobi Optimization, LLC

```python

```


```python
%matplotlib inline
import sympy.physics.mechanics as mech
from sympy import S,Rational,pi
import sympy as sp
```

```python
l1,l2,l,t,m1,m2,m,g= sp.symbols(r'l_1 l_2 l t m_1 m_2 m g')
q1,q2 = mech.dynamicsymbols(r'\theta_1 \theta_2')
q1d,q2d = mech.dynamicsymbols(r'\theta_1 \theta_2', 1)

# Create and initialize the reference frame
N = mech.ReferenceFrame('N')
pointN = mech.Point('N*')
pointN.set_vel(N, 0)

# Create the points
point1 = pointN.locatenew('p_1', l1*(sp.sin(q1)*N.x-sp.cos(q1)*N.y))
point2 = point1.locatenew('p_2', l2*(sp.sin(q2)*N.x-sp.cos(q2)*N.y))

# Set the points' velocities
point1.set_vel(N, point1.pos_from(pointN).dt(N))
point2.set_vel(N, point2.pos_from(pointN).dt(N))

# Create the particles
particle1 = mech.Particle('P_1',point1,m1)
particle2 = mech.Particle('P_2',point2,m2)

# Set the particles' potential energy

# Define forces
forces=[(point1,-particle1.mass*g*N.y),(point2,-particle2.mass*g*N.y)]

# Construct the Lagrangian
L = mech.Lagrangian(N, particle1,particle2)

# Create the LagrangesMethod object
LM = mech.LagrangesMethod(L, [q1,q2], hol_coneqs=None, forcelist=forces, frame=N)

# Form Lagranges Equations
ELeqns = LM.form_lagranges_equations()
sp.simplify(ELeqns)
```

$\displaystyle \left[\begin{matrix}l_{1} \left(g m_{1} \sin{\left(\theta_{1}{\left(t \right)} \right)} + g m_{2} \sin{\left(\theta_{1}{\left(t \right)} \right)} + l_{1} m_{1} \frac{d^{2}}{d t^{2}} \theta_{1}{\left(t \right)} + l_{1} m_{2} \frac{d^{2}}{d t^{2}} \theta_{1}{\left(t \right)} + l_{2} m_{2} \sin{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{2}{\left(t \right)}\right)^{2} + l_{2} m_{2} \cos{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \frac{d^{2}}{d t^{2}} \theta_{2}{\left(t \right)}\right)\\l_{2} m_{2} \left(g \sin{\left(\theta_{2}{\left(t \right)} \right)} - l_{1} \sin{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{1}{\left(t \right)}\right)^{2} + l_{1} \cos{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \frac{d^{2}}{d t^{2}} \theta_{1}{\left(t \right)} + l_{2} \frac{d^{2}}{d t^{2}} \theta_{2}{\left(t \right)}\right)\end{matrix}\right]$

```python
sp.simplify(LM.rhs())
```

$\displaystyle \left[\begin{matrix}\frac{d}{d t} \theta_{1}{\left(t \right)}\\\frac{d}{d t} \theta_{2}{\left(t \right)}\\\frac{g \sin{\left(\theta_{1}{\left(t \right)} - 2 \theta_{2}{\left(t \right)} \right)} + 3 g \sin{\left(\theta_{1}{\left(t \right)} \right)} + 2 l \sin{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{2}{\left(t \right)}\right)^{2} + l \sin{\left(2 \theta_{1}{\left(t \right)} - 2 \theta_{2}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{1}{\left(t \right)}\right)^{2}}{l \left(\cos{\left(2 \theta_{1}{\left(t \right)} - 2 \theta_{2}{\left(t \right)} \right)} - 3\right)}\\\frac{g \sin{\left(2 \theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} - g \sin{\left(\theta_{2}{\left(t \right)} \right)} + 2 l \sin{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{1}{\left(t \right)}\right)^{2} + \frac{l \sin{\left(2 \theta_{1}{\left(t \right)} - 2 \theta_{2}{\left(t \right)} \right)} \left(\frac{d}{d t} \theta_{2}{\left(t \right)}\right)^{2}}{2}}{l \left(2 - \cos^{2}{\left(\theta_{1}{\left(t \right)} - \theta_{2}{\left(t \right)} \right)}\right)}\end{matrix}\right]$

```python

```


# Método dos Mínimos Quadrados (MMQ)

## License

All content can be freely used and adapted under the terms of the
[Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).

## Imports

Coloque **todos** os `import` na célula abaixo. Não se esqueça do `%matplotlib inline` para que os gráficos apareçam no notebook.

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
```

## IMPORTANTE

Agora que vocês sabem técnicas de programação defensiva, eu espero que todo o código que vocês fizerem abaixo utilizem essas técnicas. Crie docstrings para suas funções, cheque as entradas (quando for possível) e cheque as saídas. **Não esqueçam dos comentários**.

## Fabricando dados para teste

Para saber se nosso código está funcionando, precisamos fabricar alguns dados com parâmetros conhecidos. Vamos gerar dados que seguem a equação da reta:

$$
d_i = a x_i + b
$$

**IMPORTANTE**: Eu vou usar a biblioteca numpy as gerar os dados abaixo.
Vocês **não podem** utilizar o numpy para calcular a sua solução.
Uso do numpy deve ser conquistado com ~~sangue~~esforço.
O código abaixo serve como exemplo do que vocês poderão fazer ao utilizar o Python no seu trabalho (fora da aula).

```python
a = 10
b = 50
N = 50
# Vou utilizar a função linspace do numpy para facilitar a vida
# Essa função cria N valores igualmente espaçados entre dois números (5 e 50)
x = np.linspace(5, 50, N)
# Agora podemos usar os valores de x, a e b acima para simular dados observados
dados_obs = a*x + b
# Vamos adicionar erro aleatório aos dados para ficar mais interessante
# O erro seguirá uma distribuição normal com os seguintes parâmetros
media_erro = 0
std_erro = 20
# A linha abaixo faz com que os valores aleatórios não sejam verdadeiramente aleatórios
# veja https://en.wikipedia.org/wiki/Pseudorandom_number_generator
np.random.seed(42)
# Gera a lista de numéros aleatórios
erro = np.random.normal(loc=media_erro, scale=std_erro, size=len(dados_obs))
# Agora podemos adicionar o erro aos dados observados
dados_obs += erro
```

Utilize a célula abaixo para gerar um gráfico de círculos pretos (`ok`) de seus dados.

```python
# Cria um gráfico.
plt.figure()
# Plota os valores das coordenadas x, y em bolas pretas.
plt.plot(x, dados_obs, "ok")
# Coloca título no gráfico.
plt.title("Gráfico 1")
# Dá título ao eixo x.
plt.xlabel("Valores")
# Dá título ao eixo y.
plt.ylabel("Dados Observados")
```

## Forma matricial da equação da reta e a matriz Jacobiana

Temos uma equação da reta para cada valor de $x_i$:

$$
\begin{align}
d_1 &= ax_1 + b \\
d_2 &= ax_2 + b \\
\vdots \\
d_N &= ax_N + b \\
\end{align}
$$

Esse sistema pode ser escrito de forma matricial com os parâmetros sendo $a$ e $b$:

$$
\begin{bmatrix}
d_1 \\ d_2 \\ \vdots \\ d_N
\end{bmatrix} =
\begin{bmatrix}
x_1 & 1 \\
x_2 & 1 \\
\vdots & \vdots \\
x_N & 1
\end{bmatrix}
\begin{bmatrix}
a \\ b
\end{bmatrix}
$$

$$
\bar{d} = \bar{\bar{A}}\bar{p}
$$

## Tarefa

Faça uma função chamada `jacobiana` que calcule e retorne a matrix Jacobiana ($\bar{\bar{A}}$).

**Para pensar**: o que essa função deve receber como argumento? (**Dica**: ela só precisa de 1)

```python
# Criamos uma função que retorna a matriz Jacobiana
def jacobiana(x):
    """
    Cria uma matriz Jacobiana:
    Ex: [[x1, 1], [x2, 1]]
    """
# Criamos esta lista vazia para fazermos a matriz Jacobiana
    jacobiana = []
# Este loop adiciona linhas no formato x,1 de x1 até xN.
    for i in range(N):
        jacobiana.append([x[i], 1])
    return jacobiana
```

```python
# Atribui a uma variavél a matriz jacobiana
jac = jacobiana(x)
```

### Resultado esperado

A célula abaixo testa a sua Jacobiana contra uma produzida pelo numpy.

```python
assert np.allclose(jacobiana(x), np.transpose([x, np.ones_like(x)]))
```

## Tarefa

Calcule dados preditos para o vetor de parâmetros definido abaixo **utilizando a forma matricial da equação**. Guarde o resultado em uma variável chamada `preditos`.

Faça um gráfico dos dados observados (gerados acima) como pontos pretos e os dados preditos que você calculou como uma linha vermelha.

**Dica**: utilize as funções que você criou na aula passada.

```python
p = [5, 15]
```

```python
# Cria uma funçao que multiplica matriz por vetor.
def vmult(m, v):
    # Docstring
    """
    Multiplica uma matriz por um vetor
    """
    assert len(m[0]) == len(v), 'Número de colunas da matriz diferente do número de linhas do vetor.'
    # Cria uma lista vazia.
    U = []
    # Faz um loop que percorre as linhas da matriz.
    for i in range(len(m)):
        # Cria uma variavel que recebe o valor zero.
        soma = 0
        # Faz um loop que percorre as colunas da matriz.
        for k in range(len(m[0])):
            # Realiza a multiplicaçao de matriz por vetor.
            soma = soma + (m[i][k] * v[k])
        # Adiciona os resultados a lista nova gerada.
        U.append(soma)
    return U
```

```python
# Atribui a uma variável a multiplicação da matriz transposta pelo vetor.
preditos = vmult(jac, p)
```

```python
# Cria um gráfico.
plt.figure()
# Plota os valores das coordenadas x, y em bolas pretas.
plt.plot(x, dados_obs, "ok")
# Plota os valores das coordenadas x, y em uma linha vermelha.
plt.plot(x, preditos, "-r")
# Coloca título no gráfico.
plt.title("Gráfico 2")
```

### Resultado esperado

A célula abaixo testa seus resultados contra um calculado com o numpy.

```python
assert np.allclose(preditos, np.dot(jacobiana(x), p))
```

O gráfico deve ser parecido com o abaixo:

## Sistema de equações normais

A solução de mínimos quadrados é o vetor $\bar{p}$ que resolve o sistema linear abaixo (chamado de sistema de equações normais):

$$
\bar{\bar{A}}^T\bar{\bar{A}}\bar{p} = \bar{\bar{A}}^T\bar{d}^o
$$

Para resolver esse sistema, precisamos primeiramente calcular a matriz do sistema $\bar{\bar{A}}^T\bar{\bar{A}}$ e o vetor do lado direito $\bar{\bar{A}}^T\bar{d}^o$.

## Tarefa

Faça uma função chamada `eqnormais_sistema` que calcule e retorne a matriz $\bar{\bar{A}}^T\bar{\bar{A}}$ dada a matriz Jacobiana.

Utilize as funções criadas na aula anterior.

**Dica**: É possível saber quantas linhas e colunas o sistema deve conter. Cheque se o seu resultado possui esse número.

```python
def transpm(m):#matriz m
    #docstring
    """
    Pega uma matriz qualquer e retorna sua transposta.
    Exemplos:
    M = [[1, 2], [3, 4]]
    transpm(M) = [[1, 3], [2, 4]]
    """
    for i in range(1, len(m), 1):
        #garante que todas as linhas da matriz estão completas com o mesmo número de elementos
        assert len(m[i]) == len(m[i - 1]), "Alguma linha da matriz não apresenta o mesmo número de elementos das outras"
    Mt = [] #lista vazia representando a matriz Mt transposta de m
    for i in range(len(m[0])):#i variando no número de colunas de m
        U = []
        for j in range(len(m)):# j variando no número de linhas de m
            L = m[j][i] #pega o elemento da posição i da linha j
            U.append(L) #adiciona a U o elemento na posição i de cada coluna
        Mt.append(U) #adiciona a Mt cada lista U
    return(Mt)

# Define a função multiplicaçao de matrizes.
def mmult(m1, m2):
    # Docstring
    """
    Multiplica duas matrizes
    """
    # Se certifica de que o número de colunas de uma matriz é igual ao numero de linhas de outra.
    assert len(m1[0]) == len(m2), "Número de Colunas de A != Número de Linhas de B"
    # Cria uma lista vazia.
    C = []
    # Faz um loop que percorre todas as linhas da matriz.
    for i in range(len(m1)):
        # Adiciona valores a lista C.
        C.append([])
        # Faz um loop que percorre todas as colunas da matriz 2.
        for j in range(len(m2[0])):
            # Cria uma variavel que recebe o valor zero.
            soma = 0
            # Faz um loop nas colunas da matriz 1.
            for k in range(len(m1[0])):
                # Realiza o somatorio da muitiplicacao das matrizes.
                soma = soma + (m1[i][k]*m2[k][j])
            # Adiciona o resultado a matriz criada.
            C[i].append(soma)
    # Retorna o valor de C.
    return C
```

```python
# Define uma função eqnormais_sistema.
def eqnormais_sistema(x):
    # Docstring
    """
    Realiza a multiplicação da matriz jacobiana transposta pela matriz jacobiana
    """
    # Retorna o valor da multiplicação da nova função.
    return mmult(transpm(x), x)
```

```python
# Função que multiplica a jacobiana transposta pela jacobiana.
resultado = eqnormais_sistema(jac)
```

### Resultado esperado

A célula abaixo testa seus resultados contra um calculado com o numpy.

```python
assert np.allclose(eqnormais_sistema(jacobiana(x)), np.transpose(jacobiana(x)).dot(jacobiana(x)))
```

## Tarefa

Faça uma função chamada `eqnormais_lado_direito` que calcule e retorne o vetor do lado direito do sistema de equações normais.

**Dicas**:

* Essa função deve receber 2 argumentos.
* O que essa função deve retornar é um vetor ou matriz?
* É possível saber o número de elementos que o resultado deve conter. Cheque esse número.

```python
# Cria uma função eqnormais_lado_direito.
def eqnormais_lado_direito(x,y):
    #docstring
    """
    Realiza a multiplicação da matriz jacobiana transposta por um vetor
    """
    # Retorna a multiplicação da jacobiana transposta pelos dados observados.
    return vmult(transpm(x), y)

```

```python
jact = transpm(jac)
```

```python
resultado = eqnormais_lado_direito(jac, dados_obs)
```

```python
assert len(resultado) == len(jact), "Número de linhas do resultado é diferente do número de linhas da transposta."
```

### Resultado esperado

A célula abaixo testa seus resultados contra um calculado com o numpy.

```python
assert np.allclose(eqnormais_lado_direito(jacobiana(x), dados_obs), np.transpose(jacobiana(x)).dot(dados_obs))
```

## Solução de mínimos quadrados

Agora que temos o sistema de equações normais, podemos resolvê-lo numericamente para encontrar os valores de $a$ e $b$ que produzem a reta que melhor ajusta nossos dados.

## Tarefa

Faça uma função chamada `elim_gauss` que resolve um sistema de equações utilizando a eliminação de Gauss. Essa função deve receber como argumento a matriz do sistema e o vetor lado direito e retornar o vetor de solução.

**Dicas**:

* Cheque o número de elementos na matriz e no vetor.
* A matriz deve ser quadrada.

```python
def elim_gauss(A, x):
    """
    Realiza o escalonamento e resolve o sistema linear
    """
    # Feito um loop para rodar a cada linha, sendo k o indice delas. A ultima linha não é utilizada pois não há nada além
    for k in range (0, len(A)-1, 1):
        # Feito um loop que começa a partir da segunda linha, o indice i indica as colunas.
        for i in range (k+1, len(A[0]), 1):
            # Criada uma variável para guardar os valores que escalonam a matriz.
            temp = -A[i][k]/A[k][k]
            # Feito um loop para pegar todas as linhas da matriz. 'k' não é usada pois ignora a última linha. Aqui acontece o escalonamento.
            for j in range (k, len(A), 1):
                # Operação para escalonar a matriz sistema
                A[i][j] = A[i][j] + A[k][j] * temp
            # Operação para gerar o novo resultado do vetor lado direito
            x[i] = x[i] + x[k] * temp
    vetor_y = [0]*len(A)
    # Loop para linhas
    for k in range(len(A)-1, -1, -1):
    # Loop para colunas que o código usará. Elas vão aumentando a cada vez que roda o programa
        for i in range(len(A[0])-1, k, -1):
    # Y é somatório do produto de cada solução por cada elemento correspondente da matriz sistema
            vetor_y[k] = vetor_y[k] + (vetor_y[i]*A[k][i])
    # Segunda parte da operação. lado_direito[k] está aqui pq só é usado 1 vez para cada linha.
        vetor_y[k] = (x[k] - vetor_y[k]) / A[k][k]
    return vetor_y

```

### Resultado esperado

A célula abaixo testa seus resultados contra um calculado com o numpy.

```python
np.random.seed(42)
A_teste = np.random.uniform(10, 50, size=(21, 21))
x_teste = np.random.uniform(5, 20, size=21)
y_teste = A_teste.dot(x_teste)
assert np.allclose(elim_gauss(A_teste, y_teste), x_teste)
```

## Tarefa

Faça uma função `ajuste_reta` que recebe um vetor de valores de x e um de dados observados e retorna a solução de minimos quadrados $\bar{p}$ (vetor com os valores de $a$ e $b$ estimados).

Aplique essa função nos dados observados simulados acima. Cheque se a solução bate com o valor esperado (você pode fazer isso usando um `assert`).

Faça um gráfico dos dados observados (pontos pretos) pelos dados preditos pela solução que você obteve agora (linha vermelha). O gráfico deve conter uma legenda. A legenda para os dados preditos deve ser da forma "y = 234x + 244" (trocando os números pelos valores que você estimou).

**Dica**:r

* Quantos elementos deve ter o vetor retornado?
* Para inserir números em uma string (texto): `"y = {}".format(123.1)"` $\to$ `"y = 123.1"`
* Para formatar os números que você quer inserir numa string: `"y = {:.4f}".format(123.242524536362446353436335)` $\to$  `"y = 123.2425"`

```python
# Define a função ajuste_reta
def ajuste_reta(x, y):
    """
    Retorna a solução dos mínimos quadrados, dando o vetor de parâmetros
    """
    vetor = eqnormais_lado_direito(jacobiana(x), y)
    matriz = eqnormais_sistema(jacobiana(x))
    p = elim_gauss(matriz, vetor)
    return p
# O Arthur sugeriu a seguinte linha de código para a função:
# return elim_gauss(eqnormais_sistema(jacobiana(x)),eqnormais_lado_direito(jacobiana(x),y)). Mas ele foi minoria no grupo.
```

```python
d = ajuste_reta(x, dados_obs)
print(d)

```

    [9.7422960022585752, 52.577381832766335]

```python
"a = {:.4f}".format(9.7422960022585752)
"b = {:.4f}".format(52.577381832766335)
y = a*x + b
```

```python
# Cria um gráfico.
plt.figure()
# Plota os valores das coordenadas x, y em bolas pretas.
plt.plot(x, dados_obs, "ok", label='Dados observados')
plt.plot(x, y, "-r", linewidth=3, label='y = 9.742x + 52.577')
#legenda
legend = plt.legend(loc='upper left', shadow=True, fontsize='large')
# Dá título ao eixo x.
plt.xlabel("x")
# Dá título ao eixo y.
plt.ylabel("y")
# Colocamos legenda bege porque somos diferenciados
legend.get_frame().set_facecolor('#F5F5DC')
```

### Resultado esperado

Os valores estimados para $\bar{p}$ devem ser aproximadamente:

    [9.742296, 52.57738183]

O gráfico deve ser parecido com o abaixo:

## Tarefa Bônus

Podemos utilizar o  método dos mínimos quadrados para ajustar qualquer equação que seja linear com relação as parâmetros ($a$ e $b$ no caso da reta). Isso quer dizer que podemos ajustar uma parábola:

$$
d_i = ax_i^2 + bx + c
$$

Dessa vez, os parâmetros que queremos estimar são $a$, $b$ e $c$. Note que agora temos 3 parâmetros, não 2. Por isso, a Jacobiana terá 3 colunas ao invés de 2.

Faça ao menos as seguintes funções:

* `jacobiana_parabola`: calcula e retorna a matriz Jacobiana para o caso da parábola. Deve receber como argumento somente o vetor de coordenadas x.
* `ajuste_parabola`: calcula a solução de mínimos quadrados para o caso de uma parábola. Deve receber como argumento o vetor de coordenadas x e o vetor de dados. Deve retornar o vetor de parâmetros $\bar{p}$ estimado (contem os valores de $a$, $b$ e $c$)

Teste suas funções com os dados gerados abaixo. Note que estamos usando o mesmo vetor x. Gere gráficos dos dados fabricados e também dos dados preditos pela estimativa (como os que foram feitos acima).

O que acontece se você tentar ajustar uma reta aos dados da parábola? E se tentar ajustar uma parábola aos dados da reta?

**Dicas**:

* Você precisa criar outras funções para montar o sistema de equações normais e calcular a solução do sistema?

```python
#A forma de se resolver o sistema de equações e calcular a solução do sistema para o caso de reta e de parabóla não muda.
#O pensamento é o mesmo, mas com mais variavéis e linhas e colunas
#Podemos utilizar as funções previamente criadas.
#O que muda é a matriz jacobiana e o parâmetro que agora é a,b,c.
```

```python
a_par, b_par, c_par = 2, 20, 200
dados_parabola = a_par*x**2 + b_par*x + c_par + erro
```

```python
# Cria um gráfico.
plt.figure()
# Plota os valores das coordenadas x, y em bolas pretas.
plt.plot(x, dados_parabola, "ok")
# Coloca título no gráfico.
plt.title("Gráfico de Dados")
# Dá título ao eixo x.
plt.xlabel("Valores")
# Dá título ao eixo y.
plt.ylabel("Dados Observados")
```

```python
# Criamos uma função que retorna a matriz Jacobiana para caso de parábola
def jacobiana_parabola(x):
    """
    Cria uma matriz Jacobiana para parábola:
    Ex: [[x1², x1, 1], [x2², x2, 1]] ... [[xn², xn, 1]]
    """
# Criamos esta lista vazia para fazermos a matriz Jacobiana
    jacobiana = []
# Este loop adiciona linhas no formato x,1 de x1 até xN.
    for i in range(N):
        jacobiana.append([x[i]*x[i], x[i], 1])
    return jacobiana
```

```python
#varíavel com a matriz gerada pela função
jac_par = jacobiana_parabola(x)
```

```python
# Define a função ajuste_parabola
def ajuste_parabola(x,y):
    """
    Retorna a solução dos mínimos quadrados, dando o vetor de parâmetros
    """
    vetor = eqnormais_lado_direito(jacobiana_parabola(x), y)
    matriz = eqnormais_sistema(jacobiana_parabola(x))
    p = elim_gauss(matriz, vetor)
    return p
```

```python
l = ajuste_parabola(x, dados_parabola)
print(l)
```

    [2.0211512867558628, 18.578975230686243, 214.85807791856433]

```python
"a_par = {:.4f}".format(2.0211512867558628)
"b_par = {:.4f}".format(18.578975230686243)
"c_par = {:.4f}".format(214.85807791856433)
y = a_par*x*x + b_par*x + c_par
```

```python
# Cria um gráfico.
plt.figure()
# Plota os valores das coordenadas x, y em bolas pretas.
plt.plot(x, dados_parabola, "ok", label='Dados observados')
plt.plot(x, y, "-r", linewidth=3, label='y = 2.021x² + 18.57x + 214.85')
#legenda
legend = plt.legend(loc='upper left', shadow=True, fontsize='large')
# Dá título ao eixo x.
plt.xlabel("x")
# Dá título ao eixo y.
plt.ylabel("y")

```

### Resultado esperado

Os gráficos que você deve gerar deverão ser parecidos com os abaixo:


## setup and notebook configuration

```python
# scientific python stack
import numpy as np
import scipy as sp
import sympy as sym
import orthopy
```

```python
# matplotlib, plotting setup
import matplotlib as mpl
import matplotlib.pyplot as plt
import matplotlib.tri as mtri               # delaunay triangulation
from mpl_toolkits.mplot3d import Axes3D     # surface plotting
import seaborn as sns                       # nice plotting defaults
import cmocean as cmo                       # ocean colormaps
sym.init_printing(use_latex='mathjax')
sns.set()
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
```

    /Users/corbin/virtual_envs/FEMexperiment_env/lib/python3.6/site-packages/cmocean/tools.py:76: MatplotlibDeprecationWarning: The is_string_like function was deprecated in version 2.1.
      if not mpl.cbook.is_string_like(rgbin[0]):

```python
%load_ext autoreload
%autoreload 2
```

```python
# local imports
import src.fem_base.master.mk_basis_nodal as mbn
import src.fem_base.master.mk_master as mkm
```

# creating nodal bases

## 1D basis

make a nodal basis on the master 1D element

```python
b = mbn.Basis_nodal(order=1, dim=1, element=0)
b.plot_elm()
```

each Basis_nodal object has an `eval_at_pts` method which can evaluate the basis at a given set of points. We can use this function to plot the basis functions over the element. Clunky, but this was originally written to accept input like `[[x1, y1, z1], [x2, y2, z2], ]`

```python
fig, ax = plt.subplots(1, 5, figsize = (10, 1))
xx = np.linspace(-1, 1, 100)
pts = [[x, 0, 0] for x in xx]

for order in range(5):
    b = mbn.Basis_nodal(order=order, dim=1, element=0)
    yy = b.eval_at_pts(pts=pts)
    for basis_fn in range(b.nb):
        ax[order].plot(xx, yy[:, basis_fn])
    ax[order].set_title(r'$p = {}$'.format(order))
```

## 2D

# creating master elements

## master triangle

```python
m = mkm.Master_nodal(order=1, dim=2, element=0)
print(dir(m))
```

    ['K', 'L', 'M', '__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_mk_ed_orient', '_mkelmFEM', '_mkelmSHAP', 'basis', 'cube_pts', 'cube_pts2xy', 'cube_wghts', 'dim', 'ed_orient', 'element', 'n', 'n_cube_pts', 'n_ed_type', 'n_quad', 'nb', 'ne', 'nodal_pts', 'nodal_shap_der', 'nr', 'nv', 'shap', 'shap_der']

By default there is a nodal shift that happens to prevent the Vandermonde matrices from being ill-conditioned. This is described in Hesthaven Warburton.

```python
fig, ax = plt.subplots(1, 4, figsize=(9, 2))
for p in range(4):
    m = mkm.Master_nodal(order=p+1, dim=2, element=0)
    xp, yp = m.nodal_pts[:,0], m.nodal_pts[:,1]
    ax[p].scatter(xp, yp, s=20, c='k', label='nodal')
    ax[p].set_title(r'$p = {}$'.format(p+1))
    ax[p].axis('equal')
    xq, yq, wq = m.cube_pts[:,0], m.cube_pts[:,1], m.cube_wghts
    ax[p].scatter(xq, yq, s=20, c='r', alpha=0.5, label='quad')

```

## master quad

## nodal placements

```python
m = mkm.Master_nodal(order=3, dim=2, element=0)
b = m.basis
print(m.M)
print(m.shap.shape)
```

    [[ 2.30158730e-02 -8.40489678e-03  4.68388844e-04  1.19047619e-03
      -8.40489678e-03 -1.29733633e-16  1.19047619e-02  4.68388844e-04
       1.19047619e-02  1.19047619e-03]
     [-8.40489678e-03  1.38888889e-01 -9.92063492e-03  4.68388844e-04
       6.42076980e-02  5.35714286e-02 -2.97619048e-02 -2.97619048e-02
      -2.45251583e-02  1.19047619e-02]
     [ 4.68388844e-04 -9.92063492e-03  1.38888889e-01 -8.40489678e-03
      -2.97619048e-02  5.35714286e-02  6.42076980e-02 -2.45251583e-02
      -2.97619048e-02  1.19047619e-02]
     [ 1.19047619e-03  4.68388844e-04 -8.40489678e-03  2.30158730e-02
       1.19047619e-02 -7.88124783e-17 -8.40489678e-03  1.19047619e-02
       4.68388844e-04  1.19047619e-03]
     [-8.40489678e-03  6.42076980e-02 -2.97619048e-02  1.19047619e-02
       1.38888889e-01  5.35714286e-02 -2.45251583e-02 -9.92063492e-03
      -2.97619048e-02  4.68388844e-04]
     [-1.28002974e-16  5.35714286e-02  5.35714286e-02 -7.79322275e-17
       5.35714286e-02  5.78571429e-01  5.35714286e-02  5.35714286e-02
       5.35714286e-02 -1.09037642e-17]
     [ 1.19047619e-02 -2.97619048e-02  6.42076980e-02 -8.40489678e-03
      -2.45251583e-02  5.35714286e-02  1.38888889e-01 -2.97619048e-02
      -9.92063492e-03  4.68388844e-04]
     [ 4.68388844e-04 -2.97619048e-02 -2.45251583e-02  1.19047619e-02
      -9.92063492e-03  5.35714286e-02 -2.97619048e-02  1.38888889e-01
       6.42076980e-02 -8.40489678e-03]
     [ 1.19047619e-02 -2.45251583e-02 -2.97619048e-02  4.68388844e-04
      -2.97619048e-02  5.35714286e-02 -9.92063492e-03  6.42076980e-02
       1.38888889e-01 -8.40489678e-03]
     [ 1.19047619e-03  1.19047619e-02  1.19047619e-02  1.19047619e-03
       4.68388844e-04 -8.40880570e-18  4.68388844e-04 -8.40489678e-03
      -8.40489678e-03  2.30158730e-02]]
    (15, 10)

```python

```