Formulating the Crop Yield Problem using QUBO and solving using VQE and QAOA approach

Quantum Computing has the potential to revolutionize computing, as it can solve problems that are not possible to solve on a classical computer. This extra ability that quantum computers have is called quantum advantage.

## Initialization

To ensure the demonstrations have the required Python modules and libraries, run the following cell before continuing.

```python
# Import auxiliary libraries
import numpy as np

# Import Qiskit
from qiskit import IBMQ, Aer
from qiskit.algorithms import QAOA, VQE, NumPyMinimumEigensolver
from qiskit.algorithms.optimizers import COBYLA
from qiskit.utils import QuantumInstance, algorithm_globals
from qiskit.providers.aer.noise.noise_model import NoiseModel

from qiskit_optimization import QuadraticProgram
from qiskit_optimization.algorithms import MinimumEigenOptimizer
from qiskit_optimization.converters import QuadraticProgramToQubo

import qiskit.test.mock as Fake
```

## Table of Contents

The notebook is structured as follows:

1. Initialization
2. Table of Contents
3. Quadratic Problems
4. Crop-Yield Problem as a Quadratic Problem
5. Solving the Crop-Yield Problem using Quantum Computing
6. Simulating a Real Quantum Computer for the Crop-Yield Problem

## Quadratic Problems

Some computational problems can be formulated into quadratic equations such that the minimum of the quadratic equations is the optimal solution, if any exist. These problems are encountered in finance, agriculture, operations and production management, and economics.

---

Quadratic problems take on the following structure. Given a vector of $n$ variables $x\in\mathbb{R}^n$, the quadratic function to minimize is as follows.

$$
\begin{align}
\text{minimize}\quad & f\left(x\right)=\frac{1}{2}x^\top{}\mathbf{Q}x + c^\top{}x &\\
\text{subject to}\quad & \mathbf{A}x\leq{}b&\\
& x^\top{}\mathbf{Q}_ix + c_{i}^\top{}x\leq{}r_i,\quad&\forall{}i\in[1,k_q]\\
& l_i\leq{}x_i\leq{}u_i,\quad&\forall{}i\in[1,k_l]\\
\end{align}
$$

$\mathbf{Q}$, $\mathbf{Q}_i$, and $\mathbf{A}$ are $n\times{}n$ symmetric matrices. $c$ and $c_i$ are $n\times{}1$ column vectors. $\mathbf{Q}_i$, $\mathbf{A}$, $c_i$, $l_i$, and $u_i$ define constraints on the variables in $x$. The quadratic equation at the core of the quadratic problem is found by multiplying out the matrices in the minimization function. Though '$\leq{}$' is used in the constraint equations above, any identity relationship may be used for any number of constraints: i.e. "$<$", "$=$", "$>$", "$\geq$", or "$\leq$".

A valid solution to the quadratic must satisfy all conditions for the problem. Examples of some constraints are given below. The first two are linear constraints whereas the third example is a quadratic constraint.

$$ x_1 + x_4 \leq{} 10$$

$$ x_2 - 3x_6 = 10$$

$$x_1x_2 - 4x_3x_4 + x_5 \leq{} 15 $$

Qiskit has Python code that allows you to implement a quadratic problem as a `QuadraticProgram` instance. Though our definition above used matrices to define the coefficients, `QuadraticProgram` allows you to define the objective (function overwhich to minimize) directly. To illustrate how to use `QuadraticProgram`, we will use the following quadratic problem definition, with three integer variables.

$$\begin{align}
\text{minimize}\quad{} & f(x)=(x_1)^2 + (x_2)^2 - x_1x_2 - 6x_3 \\
\text{subject to}\quad{} & x_1 + x_2 = 2             \\
                         & x_2x_3 \geq{} 1           \\
                         & -2 \leq{} x_2 \leq{} 2    \\
                         & -2 \leq{} x_3 \leq{} 4    \\
\end{align}$$

The figure below shows the constraints on $x_1$ and $x_3$, with some simplifcation. The shaded area denotes valid values for $x_1$ and $x_3$, within which $f(x)$ must be minimized.

In the following code, the above quadratic problem is defined as a `QuadraticProgram` instance. Have a look at the [Qiskit documentation for `QuadraticProgram`](https://qiskit.org/documentation/stubs/qiskit.optimization.QuadraticProgram.html), as it can be very useful in helping your understanding of its interface.

The quadratic to minimize, called an objective, is implemented using dictionaries. This allows you, the developer, to explicitly define coefficients for specific variables and terms. The keys in the dictionaries are the variable names identifying a term in $f(x)$. For example, `("x_1","x_2")` is for $x_1x_2$. The values for each item are the coefficients for said terms. Terms that are subtracted in $f(x)$ must have a negative coefficient.

```python
quadprog = QuadraticProgram(name="example 1")
quadprog.integer_var(name="x_1", lowerbound=0, upperbound=4)
quadprog.integer_var(name="x_2", lowerbound=-2, upperbound=2)
quadprog.integer_var(name="x_3", lowerbound=-2, upperbound=4)
quadprog.minimize(
    linear={"x_3": -6},
    quadratic={("x_1", "x_1"): 1, ("x_2", "x_2"): 1, ("x_1", "x_2"): -1},
)
quadprog.linear_constraint(linear={"x_1": 1, "x_2": 1}, sense="=", rhs=2)
quadprog.quadratic_constraint(quadratic={("x_2", "x_3"): 1}, sense=">=", rhs=1)
```

    <qiskit_optimization.problems.quadratic_constraint.QuadraticConstraint at 0x7f79180558b0>

A `QuadraticProgram` can have three types of variables: binary, integer, and continuous. The Qiskit implementation of the algorithms we are going to use currently only support binary and integer variables. There are other algorithms that allow for the simulation of continuous variables, but they are not covered in this notebook. If you want to know more about them though, have a look at this Qiskit tutorial on [algorithms to solve mixed-variable quadratic problems](https://qiskit.org/documentation/tutorials/optimization/5_admm_optimizer.html).

We can visualize the `QuadraticProgram` as an LP string, a portable text-based format used representating the model as a **L**inear **P**rogramming problem.

```python
print(quadprog.export_as_lp_string())
```

    \ This file has been generated by DOcplex
    \ ENCODING=ISO-8859-1
    \Problem name: example 1

    Minimize
     obj: - 6 x_3 + [ 2 x_1^2 - 2 x_1*x_2 + 2 x_2^2 ]/2
    Subject To
     c0: x_1 + x_2 = 2
     q0: [ x_2*x_3 ] >= 1

    Bounds
           x_1 <= 4
     -2 <= x_2 <= 2
     -2 <= x_3 <= 4

    Generals
     x_1 x_2 x_3
    End

Any optimization problem that can be represented as a single second-order equation, in that the greatest exponent of any term is 2, can be transformed into a quadratic problem or program of the form given above. The above example is arbitrary and does not necessarily represent a given real-world problem. The main problem of this notebook focuses on optimizing the yield of a farm, though only the problem definition need be changed to apply this technique to other quadratic problem applications.

## Crop-Yield Problem as a Quadratic Problem

To show how to solve your quadratic program using a quantum computer, we will use two algorithms to solve the crop-yield problem. It is a common need to optimize the crops and management of a farm to reduce risk while increasing profits. One of the big challenges is how to produce enough food for everyone. The problem here focuses not on profits but on the tonnage of crops harvested. Imagine you have a farm with three hectares of land suitable for farming. You need to choose which crops to plant from a selection of four. Furthermore, you also need to determine how many hectares of each you should plant. The four crops you can plant are wheat, soybeans, maize, and a push-pull crop. The fourth cannot be sold once harvested but it can help increase the yield of the other crops.

            Our beautiful three hectare farm

            Wheat

            Soybeans

            Maize

            Push-Pull

There are three types of farming methods we can use: monocropping, intercropping, and push-pull farming. These are shown below. Monocropping is where only one crop is farmed. This is can make the farm susceptible to disease and pests as the entire yield would be affected. In some instances, growing two different plants nearby each other will increase the yield of both, though sometimes it can decrease the yield. Intercropping is the process where different plants are chosen to _increase_ the yield. Push-Pull crops are pairs of plants that repel pests and attract pests respectively. Integrating them into a larger farm increases the yield of harvested food but with the cost of not necessarily being able to use the harvest of Push-Pull crops as part of the total yield. This is because the Push-Pull crop may not be usable or even edible.

            Monocropping

            Intercropping

            Push-Pull farming

---
Only in certain cases can quadratic programming problems be solved easily using classical problems. In their general sense, they are NP-Hard; a class of problems that is difficult to solve using classical computational methods. In fact, the best classical method to solve these problems involves heuristics, a technique that finds an approximate solution. Quantum Computers have been shown to provide significant speed-up and better scaling for some heuristic problems. The crop-yield problem is a combinatorial problem, in that the solution is a specific combination of input parameters. Though the problem shown here is small enough to solve classically, larger problems become intractable on a classical computer owing to the number of combinations of which to optimize.

Solving the above problem using quantum computing involves three components:

1. Defining the problem
2. Defining the algorithm
3. Executing the algorithm on a backend

Many problems in Qiskit follow this structure as the algorithm you use can typically be swapped for another without significantly redefining your problem. Execution on different backends is the easiest, as long as the device has sufficient resource. The first component is given below, with the second and third in sections 1.5 and 1.6.

### Define the Crop-Yield problem

The following problem is defined for you but the `QuadraticProgram` is not implemented. Your task at the end of this section is to implement the `QuadraticProgram` for the given crop-yield model.

Your farm has three hectares available, $3~ha$, with each crop taking up $0~ha$ or $1~ha$. We define the yield of the farm as a quadratic function where the influence of each crop on eachother is represented by the quadratic coefficients. The variables in this quadratic are the number of hectares of the crop to be planted and the objective function to maximize is the yield of usable crops in tons. Here is the mathematical model for the problem. In this scenario, all crops increase the yield of other crops. However, the problem to solve is which crops to use to achieve the maximum yield.

----
#### Non-graphical notation
Here is a non-graphical representation of the above model, if you are struggling to interpret the above graphic.

$$
\begin{align}
    \text{maximize} \quad & 2(\operatorname{Wheat}) + \operatorname{Soybeans} + 4(\operatorname{Maize}) \\
    & + 2.4(\operatorname{Wheat}\times\operatorname{Soybeans}) \\ & + 4(\operatorname{Wheat}\times\operatorname{Maize})\\
    &+ 4(\operatorname{Wheat}\times\operatorname{PushPull}) \\ & + 2(\operatorname{Soybeans}\times\operatorname{Maize}) \\
                          & + (\operatorname{Soybeans}\times\operatorname{PushPull}) \\ & + 5(\operatorname{Maize}\times\operatorname{PushPull})
\end{align}
$$

$$
\begin{align}
\text{subject to} \quad & \operatorname{Wheat} + \operatorname{Soybeans} + \operatorname{Maize} + \operatorname{PushPull} \leq{} 3\\
& 0\leq{}\operatorname{Wheat}\leq{}1\\
& 0\leq{}\operatorname{Soybeans}\leq{}1\\
& 0\leq{}\operatorname{Maize}\leq{}1\\
& 0\leq{}\operatorname{PushPull}\leq{}1
\end{align}
$$

Your first exercise is to create a `QuadraticProgram` that represents the above model. Write your implementation in the `cropyield_quadratic_program` function below. Remember to use the example as a guide, and to look at the [QuadraticProgram documentation](https://qiskit.org/documentation/tutorials/optimization/1_quadratic_program.html?highlight=quadraticprogram) and [Qiskit reference](https://qiskit.org/documentation/stubs/qiskit.optimization.QuadraticProgram.html?highlight=quadraticprogram#qiskit.optimization.QuadraticProgram).

**Note:** Ensure your variables are named `Wheat`, `Soybeans`, `Maize,` and `PushPull`. This is necessary for the grader to work.

```python
def cropyield_quadratic_program():
    cropyield = QuadraticProgram(name="Crop Yield")
    ##############################
    # Put your implementation here
    cropyield.integer_var(name="Wheat", lowerbound=0, upperbound=1)
    cropyield.integer_var(name="Soybeans", lowerbound=0, upperbound=1)
    cropyield.integer_var(name="Maize", lowerbound=0, upperbound=1)
    cropyield.integer_var(name="PushPull", lowerbound=0, upperbound=1)

    cropyield.maximize(
        linear={"Wheat":2, "Soybeans":1, "Maize":4},
        quadratic={("Wheat","Soybeans"):2.4, ("Wheat","Maize"):4, ("Wheat","PushPull"):4, ("Soybeans","Maize"):2, ("Soybeans","PushPull"):1, ("Maize","PushPull"):5},
    )

    cropyield.linear_constraint(linear={"Wheat":1,"Soybeans":1,"Maize":1,"PushPull":1}, sense="<=", rhs=3)

    #
    ##############################
    return cropyield
```

### Converting QuadraticPrograms

If we want to estimate how many qubits this quadratic program requires, we can convert it to an Ising Model and print the `num_qubits` parameter. An [ising model](https://qiskit.org/documentation/apidoc/qiskit.optimization.applications.ising.html?highlight=ising) is a special system model type that is well suited for quantum computing. Though we will not be using an ising model explicitly, the algorithms and Qiskit classes we are using do this conversion internally.

```python
cropyield = cropyield_quadratic_program()

# Estimate the number of qubits required
ising_operations, _ = (
    QuadraticProgramToQubo()
    .convert(
        cropyield,
    )
    .to_ising()
)
print(f"Number of qubits required is {ising_operations.num_qubits}")
```

    Number of qubits required is 6

Even though quadratic programs are widely used in Qiskit, the algorithms we are going to use require binary variables. Qiskit provides an automated method for converting our integer variables into binary variables. The binary-only form is called a _Quadratic Unconstrained Binary Optimization_ problem, or `QUBO`. The conversion is done using `QuadraticProgramToQUBO` from the Qiskit optimization module. Every integer variable, and their associated constraints, are transformed into binary variables.

Run the following code to see how the QUBO version of the cropyield problem looks. Notice how the quadratic becomes longer and more variables are added. This is to account for the bits in each variable, including the constraints. When we run our quantum algorithm to solve this QuadraticProgram, it is converted to a QUBO instance within the Qiskit algorith implementation, implicitly.

```python
QuadraticProgramToQubo().convert(cropyield)
```

    \ This file has been generated by DOcplex
    \ ENCODING=ISO-8859-1
    \Problem name: Crop Yield

    Minimize
     obj: - 160.400000000000 Wheat@0 - 159.400000000000 Soybeans@0
          - 162.400000000000 Maize@0 - 158.400000000000 PushPull@0
          - 158.400000000000 c0@int_slack@0 - 316.800000000000 c0@int_slack@1 + [
          52.800000000000 Wheat@0^2 + 100.800000000000 Wheat@0*Soybeans@0
          + 97.600000000000 Wheat@0*Maize@0 + 97.600000000000 Wheat@0*PushPull@0
          + 105.600000000000 Wheat@0*c0@int_slack@0
          + 211.200000000000 Wheat@0*c0@int_slack@1 + 52.800000000000 Soybeans@0^2
          + 101.600000000000 Soybeans@0*Maize@0
          + 103.600000000000 Soybeans@0*PushPull@0
          + 105.600000000000 Soybeans@0*c0@int_slack@0
          + 211.200000000000 Soybeans@0*c0@int_slack@1 + 52.800000000000 Maize@0^2
          + 95.600000000000 Maize@0*PushPull@0
          + 105.600000000000 Maize@0*c0@int_slack@0
          + 211.200000000000 Maize@0*c0@int_slack@1 + 52.800000000000 PushPull@0^2
          + 105.600000000000 PushPull@0*c0@int_slack@0
          + 211.200000000000 PushPull@0*c0@int_slack@1
          + 52.800000000000 c0@int_slack@0^2
          + 211.200000000000 c0@int_slack@0*c0@int_slack@1
          + 211.200000000000 c0@int_slack@1^2 ]/2 + 237.600000000000
    Subject To

    Bounds
     0 <= Wheat@0 <= 1
     0 <= Soybeans@0 <= 1
     0 <= Maize@0 <= 1
     0 <= PushPull@0 <= 1
     0 <= c0@int_slack@0 <= 1
     0 <= c0@int_slack@1 <= 1

    Binaries
     Wheat@0 Soybeans@0 Maize@0 PushPull@0 c0@int_slack@0 c0@int_slack@1
    End

## Solving the Crop-Yield Problem using a Quantum Computer

There are three ways to _run_ a quantum algorithm using Qiskit:
1. on a simulator locally on your own machine
2. on a simulator hosted in the cloud by IBM
3. on an actual quantum computer accessible through IBM Quantum.

All of these are called backends. In all cases, the _backend_ can easily be swapped for another as long as the simulator or device has appropriate resources (number of qubits etc.). In the code below, we show how to access different backends. We demonstrate this using the local Aer QASM simulator from Qiskit. The Aer QASM simulator models the physical properties of a real quantum computer so you, researchers, and developers can test their quantum computing code and algorithms before running on real devices.

```python
# We will use the Aer provided QASM simulator
backend = Aer.get_backend("qasm_simulator")

# Given we are using a simulator, we will fix the algorithm seed to ensure our results are reproducible
algorithm_globals.random_seed = 271828
```

We would like to compare our quantum solution to that obtained classically. Secondly, we also want to try different algorithms. The following three subsections show how these different methods for solving the Crop-Yield problem are implemented in Qiskit. The two algorithms used are the [_Quantum Approximate Optimization Algorithm_](https://qiskit.org/documentation/stubs/qiskit.algorithms.QAOA.html?highlight=qaoa#qiskit.algorithms.QAOA) `QAOA` and the [_Variational Quantum Eigensolver_](https://qiskit.org/documentation/stubs/qiskit.algorithms.VQE.html?highlight=vqe#qiskit.algorithms.VQE) `VQE`.

Both of these algorithms are hybrid, in that they use a classical _optimizer_ to alter parameters that affect the quantum computation. The VQE algorithm is used to find the lowest eigenvalue of a matrix, which can describe a system to optimize. The QAOA also finds the lowest eigenvalue, but achieves this is a different way to VQE. Both are very popular algorithms, with varying applications and strengths.

### Classical Solution

The classical solution to the crop-yield problem can easily be found using Numpy and Qiskit. The QUBO problem can be solved by finding the minimum eigenvalue of its underlying matrix representation. Fortunately, we don't have to know what this matrix looks like. We only need to pass it to a `MinimumEigensolver` and `MinimumEigenOptimizer`.

The optimizer translates the provided problem into a parameterised representation which is then passed to the solver. By optimizing the paramters, the solver will eventually give the minimum eigenvalue for the parameterized representation and thus the solution to the original problem. Here we use a classical solver from NumPy, the `NumPyMinimumEigensolver`.

```python
def get_classical_solution_for(quadprog: QuadraticProgram):
    # Create solver
    solver = NumPyMinimumEigensolver()

    # Create optimizer for solver
    optimizer = MinimumEigenOptimizer(solver)

    # Return result from optimizer
    return optimizer.solve(quadprog)
```

If we execute the classical method for our crop-yield problem, we get a valid solution that maximises the yield.

```python
# Get classical result
classical_result = get_classical_solution_for(cropyield)

# Format and print result
print("Solution found using the classical method:\n")
print(f"Maximum crop-yield is {classical_result.fval} tons")
print(f"Crops used are: ")

_crops = [v.name for v in cropyield.variables]
for cropIndex, cropHectares in enumerate(classical_result.x):
    print(f"\t{cropHectares} ha of {_crops[cropIndex]}")
```

    Solution found using the classical method:

    Maximum crop-yield is 19.0 tons
    Crops used are:
    	1.0 ha of Wheat
    	0.0 ha of Soybeans
    	1.0 ha of Maize
    	1.0 ha of PushPull

----
The VQE algorithm and QAOA are iterative, meaning that they incorporate a classical-quantum loop which repeats certain computations, _hopefully_ converging to a valid solution. In each iteration, or evaluation, the quantum backend will execute the quantum operations 2048 times. Each shot is quite fast, so we do not have to worry about a significant increase in processing time by using more shots.

### QAOA Solution

To solve our problem using QAOA, we need only replace the classical_solver with a `QAOA` class instance. Now that we are running a quantum algorithm, we need to tell the solver where to execute the quantum component. We use a `QuantumInstance` to store the backend information. The QAOA is an iterative algorithm, and thus is run multiple times with different internal parameters. The parameters are tuned classically during the optimization step of the computation by `optimizer`. If we leave `optimizer` as `None`, our algorithms will use the default optimization algorithm. To determine how many iterations there are, we define a callback function that runs for each iteration and stores the number of evaluations thus far. At the end of our algorithm execution, we return the result and the number of iterations.

```python
def get_QAOA_solution_for(
    quadprog: QuadraticProgram, quantumInstance: QuantumInstance, optimizer=None,
):
    _eval_count = 0

    def callback(eval_count, parameters, mean, std):
        nonlocal _eval_count
        _eval_count = eval_count

    # Create solver
    solver = QAOA(
        optimizer=optimizer, quantum_instance=quantumInstance, callback=callback,
    )

    # Create optimizer for solver
    optimizer = MinimumEigenOptimizer(solver)

    # Get result from optimizer
    result = optimizer.solve(quadprog)
    return result, _eval_count
```

```python
# Create a QuantumInstance
simulator_instance = QuantumInstance(
    backend=backend,
    seed_simulator=algorithm_globals.random_seed,
    seed_transpiler=algorithm_globals.random_seed,
)

# Get QAOA result
qaoa_result, qaoa_eval_count = get_QAOA_solution_for(cropyield, simulator_instance)

# Format and print result
print("Solution found using the QAOA method:\n")
print(f"Maximum crop-yield is {qaoa_result.fval} tons")
print(f"Crops used are: ")
for cropHectares, cropName in zip(qaoa_result.x, qaoa_result.variable_names):
    print(f"\t{cropHectares} ha of {cropName}")

print(f"\nThe solution was found within {qaoa_eval_count} evaluations of QAOA.")
```

    Solution found using the QAOA method:

    Maximum crop-yield is 19.0 tons
    Crops used are:
    	1.0 ha of Wheat
    	0.0 ha of Soybeans
    	1.0 ha of Maize
    	1.0 ha of PushPull

    The solution was found within 3 evaluations of QAOA.

### VQE Solution

The `VQE` algorithm works in a very similar way to the `QAOA`. Not only in a mathematical modelling and algorithmic perspective, but also programmatically. There is a quantum solver and a classical optimizer. The `VQE` instance is also iterative, and so we can measure how many iterations are needed to find a solution to the Crop-Yield problem.

```python
def get_VQE_solution_for(
    quadprog: QuadraticProgram, quantumInstance: QuantumInstance, optimizer=None,
):
    _eval_count = 0

    def callback(eval_count, parameters, mean, std):
        nonlocal _eval_count
        _eval_count = eval_count

    # Create solver and optimizer
    solver = VQE(
        optimizer=optimizer, quantum_instance=quantumInstance, callback=callback
    )

    # Create optimizer for solver
    optimizer = MinimumEigenOptimizer(solver)

    # Get result from optimizer
    result = optimizer.solve(quadprog)
    return result, _eval_count
```

And we should get the exact same answer as before.

```python
# Create a QuantumInstance
simulator_instance = QuantumInstance(
    backend=backend,
    seed_simulator=algorithm_globals.random_seed,
    seed_transpiler=algorithm_globals.random_seed,
)

# Get VQE result
vqe_result, vqe_eval_count = get_VQE_solution_for(cropyield, simulator_instance)

# Format and print result
print("Solution found using the VQE method:\n")
print(f"Maximum crop-yield is {vqe_result.fval} tons")
print(f"Crops used are: ")
for cropHectares, cropName in zip(vqe_result.x, vqe_result.variable_names):
    print(f"\t{cropHectares} ha of {cropName}")

print(f"\nThe solution was found within {vqe_eval_count} evaluations of VQE")
```

    Solution found using the VQE method:

    Maximum crop-yield is 19.0 tons
    Crops used are:
    	1.0 ha of Wheat
    	0.0 ha of Soybeans
    	1.0 ha of Maize
    	1.0 ha of PushPull

    The solution was found within 25 evaluations of VQE

### Scaling of the Quantum Solution vs Classical

When we created our quadratic program for the crop-yield problem, we saw that the Ising Model required 6 qubits. We had constrained our problem such that we could only plant up to 1 hectare per crop. However, we could change the model so that we can plot 3 hectares per crop, upto our maximum available farm area of 3 hectares.

Furthermore, what if we had more land to farm? We know that this problem is NP-Hard and thus classical solutions are mostly found using heuristics. This is the core reason why quantum computers are promising to solve these kinds of problems. But what are the resource requirements for the quantum solution, with a larger farm and crops that can be planted in more hectares?

To illustrate this, we've provided a function that returns the number of qubits required by the underlying Ising Model for the Crop-Yield Problem. We then see the estimated number of qubits needed for different problem parameters. Feel free to modify the variables.

```python
# Function to estimate the number of qubits required
def estimate_number_of_qubits_required_for(max_hectares_per_crop, hectares_available):
    return int(
        4 * np.ceil(np.log2(max_hectares_per_crop + 1))
        + np.ceil(np.log2(hectares_available + 1))
    )

# Our new problem parameters
hectares_available = 10
max_hectares_per_crop = 10

# Retrieving the number of qubits required
number_of_qubits_required = estimate_number_of_qubits_required_for(
    max_hectares_per_crop=max_hectares_per_crop, hectares_available=hectares_available
)

print(
    f"Optimizing a {hectares_available} ha farm with each crop taking up to {max_hectares_per_crop} ha each,",
    f"the computation is estimated to require {number_of_qubits_required} qubits.",
)
```

    Optimizing a 10 ha farm with each crop taking up to 10 ha each, the computation is estimated to require 20 qubits.

Thus it can be seen that, the number of qubits required is related to the constraints in the quadratic program and how the integer variables are converted to binary variables. In fact, the scaling of the number of qubits, as a function of the hectares available, is logarithmic in nature; owing to this conversion.


# Josephus Problem

## Introduction

The Josephus Problem is introduced in this [Numberphile video (youtu.be/uCsD3ZGzMgE)](https://youtu.be/uCsD3ZGzMgE).

There is a circle of $41$ soldiers, and they will sequentially kill the next living soldier to their right, until only one soldier remains.  If you want to remain alive at the end, which position in the circle should you pick?

```python
# Symbols
from sympy import *
from sympy.plotting import plot_parametric
import numpy as np
import scipy
import scipy.special

init_printing(use_unicode=True)

x,y,z,t,mu,nu=symbols('x y z t mu nu')
k,m,n=symbols('k m n', integer=True)
f,g,h=symbols('f g h', cls=Function)

R=S.Reals
C=S.Complexes

i=sqrt(-1)
```

```python
# Plotting Setup
%config InlineBackend.figure_formats = ['svg']
%matplotlib inline
```

```python
from matplotlib import animation, rc
from IPython.display import HTML
import matplotlib.pyplot as plt
plt.rcParams['figure.autolayout'] = False
plt.rcParams['figure.figsize'] = 3, 3
```

## Representation

```python
import networkx as nx
```

The Josephus problem state will be defined as a tuple $(i,G)$ where $i$ is the node about to kill, and $G$ is a directed graph with edges from killers to victims.

```python
def setup(n):
    G = nx.empty_graph(n, create_using = nx.DiGraph)
    return (0,G)
```

It follows from our definition that a node is alive if it has no edges directed into it.

```python
def is_alive(i, G):
    return not any(True for _ in G.predecessors(i))
def next_alive(i, G):
    l = G.order()
    j = (i + 1) % l
    while j != i:
        if is_alive(j, G):
            return j
        j = (j + 1) % l
    return i
```

Such a state will be drawn as the digraph, with the next killer marked.

```python
def draw(jp):
    G = jp[1]
    ca = lambda v: ('c' if v == jp[0]
                    else ('y' if is_alive(v, G)
                          else 'm'))
    colors = [ca(v) for v in G]
    nx.draw_circular(G, with_labels=True, node_color=colors, label='X')
```

To advance the problem state, we add an edge from the current killer to the next living node, and then advance the current killer to the new, next killer.

```python
def next(jp):
    G = jp[1].copy()
    i = jp[0]
    l = G.order()
    a = next_alive(i, G)
    if i == a:
        return (i, G)
    G.add_edge(i, a, weight = G.degree)
    return (next_alive(a, G), G)
```

The problem is solved, when only one node remains alive.

```python
def finish(jp):
    i = -1
    while jp[0] != i:
        i = jp[0]
        jp = next(jp)
    return jp
```

## Sample runs

```python
draw(finish(setup(2)))
```

```python
draw(finish(setup(3)))
```

```python
draw(finish(setup(5)))
```

```python
draw(finish(setup(6)))
```

```python
draw(finish(setup(7)))
```

```python
draw(finish(setup(8)))
```

```python
draw(finish(setup(15)))
```

```python
draw(finish(setup(16)))
```

## Original problem

Now we can see that our solution to the $41$-soldier problem shows the living soldier is number $18$ in a $0$-based index, which matches the video's $19$ in a $1$-based index.

```python
jp = setup(41)
plt.rcParams['figure.figsize'] = 6, 6
draw(jp)
```

```python
jp = finish(jp)
draw(jp)
```

## Sequence

Now we see the pattern for the survivor index for circles of various numbers of soldiers.

```python
def survivor(i):
    return finish(setup(i))[0]
def seq(n):
    return [[i+1, survivor(i+1)] for i in range(n)]
Matrix(seq(41))
```

```python

```

```python

```


# Numerical Solution of the Helmholtz Equation using the Finite Element Method

This notebook illustrates the numerical solution of the wave equation for harmonic excitation using the so called [Finite Element Method](https://en.wikipedia.org/wiki/Finite_element_method) (FEM). The method aims at an approximate solution by subdividing the area of interest into smaller parts with simpler geometry, linking these parts together and applying methods from the calculus of variations to solve the problem numerically. The FEM is a well established method for the numerical approximation of the solution of partial differential equations (PDEs). The solutions of PDEs are often known analytically only for rather simple geometries. FEM based simulations allow to gain insights into other more complex cases.

## Problem Statement

The inhomogeneous linear [wave equation](https://en.wikipedia.org/wiki/Wave_equation) is given as

\begin{equation}
\Delta p(\mathbf{x}, t) - \frac{1}{c^2} \frac{\partial^2}{\partial t^2} p(\mathbf{x}, t) = - q(\mathbf{x}, t) ,
\end{equation}

where $p(\mathbf{x}, t)$ denotes the sound pressure at position $\mathbf{x}$, $c$ the speed of sound and $q(\mathbf{x}, t)$ the inhomogeneity.
For an harmonic excitation $q(\mathbf{x}, t) = \Re \{ Q(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} \}$ with frequency $\omega = 2 \pi f$ we choose the Ansatz $p(\mathbf{x}, t) = \Re \{ P(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} \}$ for the sound pressure.
Introduction of the complex quantities into the wave equation yields

\begin{equation}
\Delta P(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} + \frac{\omega^2}{c^2} P(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} = - Q(\mathbf{x}, \omega) \mathrm{e}^{\mathrm{j} \omega t} ,
\end{equation}

and canceling out the $\mathrm{e}^{\mathrm{j} \omega t}$ terms yields the [Helmholtz equation](https://en.wikipedia.org/wiki/Helmholtz_equation)

\begin{equation}
\Delta P(\mathbf{x}, \omega) + \frac{\omega^2}{c^2} P(\mathbf{x}, \omega) = - Q(\mathbf{x}, \omega) .
\end{equation}

We aim for a numerical solution of the Helmholtz equation on the domain $V$ with respect to the homogeneous Dirichlet boundary condition

\begin{equation}
P(\mathbf{x}, \omega) = 0 \qquad \text{for } x \in \partial V
\end{equation}

or the homogeneous Neumann boundary condition

\begin{equation}
\frac{\partial}{\partial n} P(\mathbf{x}, \omega) = 0 \qquad \text{for } x \in \partial V ,
\end{equation}

where $\partial V $ denotes the boundary of $V$.

## Variational Formulation

The FEM is based on expressing the partial differential equation (PDE) to be solved in its [variational](https://en.wikipedia.org/wiki/Calculus_of_variations) or weak form.
The first step towards this formulation is to multiply the Helmholtz equation by the test function $V(\mathbf{x}, \omega)$

\begin{equation}
\Delta P(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega)  + \frac{\omega^2}{c^2} P(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) = - Q(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) ,
\end{equation}

followed by integration over the domain $V$

\begin{equation}
\int_V \Delta P(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) \mathrm{d}x  + \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) \mathrm{d}x = - \int_V Q(\mathbf{x}, \omega) \cdot V(\mathbf{x}, \omega) \mathrm{d}x ,
\end{equation}

where $\mathrm{d}x$ denotes a suitably chosen differential element for integration.
Application of [Green's first identity](https://en.wikipedia.org/wiki/Green%27s_identities) yields

\begin{equation}
{-} \int_V \nabla P(\mathbf{x}, \omega) \cdot \nabla V(\mathbf{x}, \omega) \mathrm{d}x  + \int_{\partial V} V(\mathbf{x}, \omega) \frac{\partial}{\partial n}  P(\mathbf{x}, \omega) \mathrm{d}s + \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x = - \int_V Q(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x .
\end{equation}

This way the differential order of the first integral is lowered which is advisable for application of the FEM.
The second integral vanishes as

* the variation formulation requires $V(\mathbf{x}, \omega) = 0$ on $\partial V$ where $P(\mathbf{x}, \omega)$ is known - here by the pure Dirichlet boundary condition - or
* due to a pure homogeneous Neumann boundary condition $\frac{\partial}{\partial n} P(\mathbf{x}, \omega)$ on $\partial V$.

This results in the variational/weak formulation of the Helmholtz equation

\begin{equation}
{-} \int_V \nabla P(\mathbf{x}, \omega) \cdot \nabla V(\mathbf{x}, \omega) \mathrm{d}x  + \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x = - \int_V Q(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x .
\end{equation}

It is common to express the integral equation above in terms of the bilinear $a(P, V)$ and linear $L(V)$ forms

\begin{equation}
a(P, V) = \frac{\omega^2}{c^2} \int_V P(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x - \int_V \nabla P(\mathbf{x}, \omega) \cdot \nabla V(\mathbf{x}, \omega) \mathrm{d}x ,
\end{equation}

\begin{equation}
L(V) = - \int_V Q(\mathbf{x}, \omega) V(\mathbf{x}, \omega) \mathrm{d}x ,
\end{equation}

where

\begin{equation}
a(P, V) = L(V) .
\end{equation}

## Numerical Solution

The numerical solution of the variational problem is based on [FEniCS](https://fenicsproject.org/), an open-source framework for numerical solution of PDEs.
Its high-level Python interface `dolfin` is used in the following to define the problem and compute its solution.
The implementation is based on the variational formulation derived above.
It is common in the FEM to denote the solution of the problem by $u$ and the test function by $v$.
The definition of the problem in FEniCS is very close to the mathematical formulation of the problem.
We limit ourselves to real-valued $P(\mathbf{x}, \omega)$ due to the assumption of Dirichlet or Neumann boundary conditions.

For the subsequent examples the solution of the inhomogeneous wave equation for a point source $Q(\mathbf{x}) = \delta(\mathbf{x}-\mathbf{x_s})$ at position $\mathbf{x_s}$ is computed using the FEM.
A function is defined for this purpose, accompanied by a plotting routine for the resulting sound field.

```python
import dolfin
import mshr
import matplotlib.pyplot as plt
%matplotlib inline

def FEM_Helmholtz(mesh, frequency, xs, neumann_bc=True, c=343):
    '''numerical solution of the Helmholtz equation using the FEM'''

    # squared wavenumber
    k2 = dolfin.Constant(2*dolfin.pi*frequency/c)**2

    # define function space
    V = dolfin.FunctionSpace(mesh, "CG", 2)

    # define boundary conditions
    if neumann_bc:
        bcs = None
    else:
        bcs = dolfin.DirichletBC(V, dolfin.Constant(0), "on_boundary")

    # define variational problem
    u = dolfin.TrialFunction(V)
    v = dolfin.TestFunction(V)

    a = k2 * dolfin.inner(u, v) * dolfin.dx - dolfin.inner(dolfin.nabla_grad(u), dolfin.nabla_grad(v)) * dolfin.dx
    L = dolfin.Constant(0) * v * dolfin.dx

    A, b = dolfin.assemble_system(a, L, bcs)

    # define inhomogenity
    delta = dolfin.PointSource(V, xs, -1)  # to account for negative sign in inhomogeneity
    delta.apply(b)

    # compute solution
    u = dolfin.Function(V)
    dolfin.solve(A, u.vector(), b)

    return u

def plot_soundfield(u):
    '''plot solution of FEM-based simulation'''
    fig = plt.figure(figsize=(10,10))
    fig = dolfin.plot(u)
    plt.title(r'$P(\mathbf{x}, \omega)$')
    plt.xlabel(r'$x$ / m')
    plt.ylabel(r'$y$ / m')
    plt.colorbar(fig, fraction=0.038, pad=0.04);
```

### Sound Field in a Rectangular Room with Sound-Hard Boundaries

The two-dimensional sound field in a rectangular room whose height is very small compared to the wavelength and with rigid boundaries (Neumann boundary condition) is computed for the frequency $f=1000$ Hz and source position $x_s = (1.2,3.2)$ m.

```python
# define geometry and mesh
mesh = dolfin.RectangleMesh(dolfin.Point(0,0), dolfin.Point(5,4), 200, 200, "right/left")

# compute solution
u = FEM_Helmholtz(mesh, 1000, dolfin.Point(1.2,3.2))

# plot sound field
plot_soundfield(u)
plot_soundfield(abs(u))
plt.title(r'$|P(\mathbf{x}, \omega)|$');
```

### Sound Field in Two Coupled Rectangular Rooms

The two-dimensional sound field in two coupled rectangular rooms with sound-hard boundaries (Neumann boundary condition) is computed for the frequency $f=1000$ Hz and source position $x_s = (2,0.5)$ m. First the geometry is defined and plotted, for which the mesh is generated with a low number of elements for ease of illustration. A higher resolution is then used for the simulations.

```python
# define geometry and mesh
domain = mshr.Rectangle(dolfin.Point(0, 0), dolfin.Point(3,4)) + \
         mshr.Rectangle(dolfin.Point(3, 1.5), dolfin.Point(3.5, 2.5)) + \
         mshr.Rectangle(dolfin.Point(3.5, 0), dolfin.Point(6, 4))
mesh2 = mshr.generate_mesh(domain, 20)

dolfin.plot(mesh2);
```

```python
# compute solution
mesh2 = mshr.generate_mesh(domain, 100)
u = FEM_Helmholtz(mesh2, 1000, dolfin.Point(2, .5))

# plot sound field
plot_soundfield(u)
plot_soundfield(abs(u))
plt.title(r'$|P(\mathbf{x}, \omega)|$');
```

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT).


```python
import sympy as sp
import warnings
warnings.filterwarnings('ignore')
sp.init_printing()
```

```python
N = sp.Symbol("N")
x = sp.IndexedBase("x")
y = sp.IndexedBase("y")
z = sp.IndexedBase("z")
i = sp.Symbol("i")
j = sp.Symbol("j")
k = sp.Symbol("k")
dx = x[i] - x[j]
dy = y[i] - y[j]
dz = z[i] - z[j]
r = sp.sqrt(dx**2 + dy**2 + dz**2)
eps = sp.Symbol("ε")
sig = sp.Symbol("σ")
```

```python
#energy = (1/2)*sp.Sum(sp.Sum(4 * eps * ((sig/r)**12 - (sig/r)**6), (i, 1, N)), (j, 1, N))
# It is easier to work with just the summand
energy = 4 * eps * ((sig/r)**12 - (sig/r)**6)
```

```python
energy
```

$\displaystyle 4 ε \left(\frac{σ^{12}}{\left(\left({x}_{i} - {x}_{j}\right)^{2} + \left({y}_{i} - {y}_{j}\right)^{2} + \left({z}_{i} - {z}_{j}\right)^{2}\right)^{6}} - \frac{σ^{6}}{\left(\left({x}_{i} - {x}_{j}\right)^{2} + \left({y}_{i} - {y}_{j}\right)^{2} + \left({z}_{i} - {z}_{j}\right)^{2}\right)^{3}}\right)$

```python
r.diff(x[k])
```

$\displaystyle \frac{\left(2 \delta_{i k} - 2 \delta_{j k}\right) \left({x}_{i} - {x}_{j}\right)}{2 \sqrt{\left({x}_{i} - {x}_{j}\right)^{2} + \left({y}_{i} - {y}_{j}\right)^{2} + \left({z}_{i} - {z}_{j}\right)^{2}}}$

```python
energy.diff(x[k])
```

$\displaystyle 4 ε \left(- \frac{6 σ^{12} \left(2 \delta_{i k} - 2 \delta_{j k}\right) \left({x}_{i} - {x}_{j}\right)}{\left(\left({x}_{i} - {x}_{j}\right)^{2} + \left({y}_{i} - {y}_{j}\right)^{2} + \left({z}_{i} - {z}_{j}\right)^{2}\right)^{7}} + \frac{3 σ^{6} \left(2 \delta_{i k} - 2 \delta_{j k}\right) \left({x}_{i} - {x}_{j}\right)}{\left(\left({x}_{i} - {x}_{j}\right)^{2} + \left({y}_{i} - {y}_{j}\right)^{2} + \left({z}_{i} - {z}_{j}\right)^{2}\right)^{4}}\right)$

```python
energy.diff(y[k])

```

$\displaystyle 4 ε \left(- \frac{6 σ^{12} \left(2 \delta_{i k} - 2 \delta_{j k}\right) \left({y}_{i} - {y}_{j}\right)}{\left(\left({x}_{i} - {x}_{j}\right)^{2} + \left({y}_{i} - {y}_{j}\right)^{2} + \left({z}_{i} - {z}_{j}\right)^{2}\right)^{7}} + \frac{3 σ^{6} \left(2 \delta_{i k} - 2 \delta_{j k}\right) \left({y}_{i} - {y}_{j}\right)}{\left(\left({x}_{i} - {x}_{j}\right)^{2} + \left({y}_{i} - {y}_{j}\right)^{2} + \left({z}_{i} - {z}_{j}\right)^{2}\right)^{4}}\right)$

```python
energy.diff(z[k])
```

$\displaystyle 4 ε \left(- \frac{6 σ^{12} \left(2 \delta_{i k} - 2 \delta_{j k}\right) \left({z}_{i} - {z}_{j}\right)}{\left(\left({x}_{i} - {x}_{j}\right)^{2} + \left({y}_{i} - {y}_{j}\right)^{2} + \left({z}_{i} - {z}_{j}\right)^{2}\right)^{7}} + \frac{3 σ^{6} \left(2 \delta_{i k} - 2 \delta_{j k}\right) \left({z}_{i} - {z}_{j}\right)}{\left(\left({x}_{i} - {x}_{j}\right)^{2} + \left({y}_{i} - {y}_{j}\right)^{2} + \left({z}_{i} - {z}_{j}\right)^{2}\right)^{4}}\right)$

```python
import sympy as sp
import warnings
warnings.filterwarnings('ignore')
sp.init_printing()
eps = sp.Symbol("ε")
sig = sp.Symbol("σ")
rad = sp.Symbol("r")
energyRad = 4 * eps * ((sig/rad)**12 - (sig/rad)**6)
energyRad.diff(rad)
```

$\displaystyle 4 ε \left(\frac{6 σ^{6}}{r^{7}} - \frac{12 σ^{12}}{r^{13}}\right)$

```python

```

```python

```


# Regressione usando la teoria Bayesiana

La regressione lineare che abbiamo visto fino ad ora usa modello di tipo deterministico per analizzare e predire dati, ma è anche possibile usare modelli di tipo probabilistico per effettuare una regressione lineare, vediamone alcuni modelli, per maggiori info visitate questo __[link](https://scikit-learn.org/stable/modules/linear_model.html#bayesian-regression)__.

All'interno di questi modelli si usa il Teorema di Bayes che stabilisce la seguente formula:

\begin{equation}
    \Large P(H_{0}|E) = \frac{P(E|H_{0})P(H_{0})}{P(E)}
\end{equation}

Dove $P(H_{0}|E)$ è detta probabilità di $H_{0}$, dato un evento $E$, $P(H_{0})$ è detta probabilità a priori di $H_{0}$, $P(E)$ è detta probabilità di osservare $E$ detta ***probabilita marginale***, mentre $P(E|H_{0})$ è la **funzione verosomigliante** ovvero una funzione di probabilità condizionata in cui **il primo argomento($E$) è fisso, mentre il secondo ($H_{0}$) è variabile**.

In questi tipi di modelli si introduce una distribuzione a priori non informativa sugli iperparametri che dovranno essere poi determinati grazie ai dati attraverso il teorema precedente, per fare un esempio ipotizziamo di avere dei dati che noi pensiamo siano distribuiti secondo una gaussiana, allora il modello dovrà determinare:

\begin{equation}
    \Large p(y | X,w,\alpha) = \mathcal{N}(Xw, \alpha)
\end{equation}

Ovvero una distribuzione probabilistica di tipo gaussiana centrata in $Xw$ e con una varianza $\alpha$ che deve essere determinata dai dati.

**Tra i vantaggi abbiamo che:**

- questi modelli si adattano ai dati per definizione
- possiamo introdurre termini di regoralizzazione per renderli più robusti

**Tra gli svantaggi abbiamo che:**

- per fare inferenza statistica si usa il teorema di Bayes che è computazionalmente molto oneroso in termini di risorse.

## Bayesian Ridge Regression

Questo modello di regressione bayesiano usa un termine di regoralizzazione $l_2$, come nella Ridge Regression, dal punto di vista probabilistico questo modello usa una distribuzione gaussiana e determina i parametri usando:

\begin{equation}
    \Large p(w|\lambda) = \mathcal{N}(w|0, \lambda^{-1}I_p)
\end{equation}

Per essere precisi al fine di determinare i pesi si utilizza la **Maximum a posteriori (MAP) usando la log marginal likelihood** ovvero:

\begin{equation}
\Large W = max_{w} \quad exp[-\frac{(y-X\beta)^{T}(y-X\beta)}{2\sigma^{2}} -\frac{\parallel \beta \parallel^{2}_{2}}{2\tau^2}]\\
\Large W = min_{w} (y-X\beta)^{T}(y-X\beta) +  \lambda \parallel \beta \parallel^{2}_{2}
\end{equation}

Sono presenti altri parametri nell'implementazione dell'algoritmo, quello che ci serve sapere è che gli $\alpha$ e i $\beta$ sono dei parametri per la definizione di una __[gamma distribution](https://en.wikipedia.org/wiki/Gamma_distribution)__ che è una distribuzione continua e che tali parametri sono definiti dai dati a partire da dei valori iniziali e quelli che non sono informativi, ovvero forniscono una vaga informazione sulla variabile sono di solito settati piccoli in modo da non influire sulla determinazione della distribuzione, per maggiori dettagli consultate __[scikit](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.BayesianRidge.html#sklearn.linear_model.BayesianRidge)__.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.linear_model import BayesianRidge
from sklearn.datasets import load_boston
from sklearn.model_selection import train_test_split

#load the dataset
data = load_boston()
df = pd.DataFrame(data.data, columns = data.feature_names)
df['MEDV'] = data.target
#prepare the data
X_train, X_test, y_train, y_test = train_test_split(df.LSTAT.values, df.MEDV.values,
                                                    random_state=0, test_size = 0.2)
bayesian_ridge = BayesianRidge(compute_score=True)
bayesian_ridge.fit(X_train.reshape(-1, 1), y_train)

#setting plot
fig, ax = plt.subplots(figsize=(8, 6))
ax.scatter(X_train, y_train, label = "train", color = "blue")
ax.scatter(X_test, y_test, label = "test", color = "green")
line, = ax.plot(X_train, bayesian_ridge.predict(X_train.reshape(-1, 1)), label = "fit", color = "red")
plt.legend()
plt.show()

print('Bayesian Regression parameters estimated:')
print(f'Estimated precision of noise: {bayesian_ridge.alpha_}')
print(f'Estimated precision of weights: {bayesian_ridge.lambda_}')
print(f'Estimated variance-covariance matrix of the weights: {bayesian_ridge.sigma_}')
print(f'Number of iterations applied to obtain estimated parameters: {bayesian_ridge.n_iter}')
print(f'Value of the log marginal likelihood (to be maximized) at each iteration: {bayesian_ridge.scores_}')
print(f'Coefficients obtained: {bayesian_ridge.coef_} ')
print(f'R^2 score on training set: {bayesian_ridge.score(X_train.reshape(-1, 1), y_train)},'
      f'test set: {bayesian_ridge.score(X_test.reshape(-1, 1), y_test)}')
```

```python
#use bayesian regression for multiple features
X, Y = df.drop('MEDV', axis = 1).values, df.MEDV.values
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, random_state=0, test_size = 0.2)

bayesian_ridge = BayesianRidge(compute_score=True)
bayesian_ridge.fit(X_train, y_train)

print('Bayesian Regression parameters estimated:')
print(f'Estimated precision of noise: {bayesian_ridge.alpha_}')
print(f'Estimated precision of weights: {bayesian_ridge.lambda_}')
print(f'Estimated variance-covariance matrix of the weights:\n {bayesian_ridge.sigma_}')
print(f'Number of iterations applied to obtain estimated parameters: {bayesian_ridge.n_iter}')
print(f'Value of the log marginal likelihood (to be maximized) at each iteration: {bayesian_ridge.scores_}')
print(f'Coefficients obtained: {bayesian_ridge.coef_}')
print(f'R^2 score on training set: {bayesian_ridge.score(X_train, y_train)},'
      f'test set: {bayesian_ridge.score(X_test, y_test)}')
```

    Bayesian Regression parameters estimated:
    Estimated precision of noise: 0.04852260984263468
    Estimated precision of weights: 0.6102776668911499
    Estimated variance-covariance matrix of the weights:
     [[ 1.37807667e-03 -4.21686237e-05  1.14783119e-04  1.33038925e-03
       5.23898493e-04 -5.03102014e-04  2.54259490e-05  8.10729775e-04
      -7.08264350e-04  1.73865830e-06  4.19197193e-06  9.65422950e-07
      -4.28286306e-04]
     [-4.21686237e-05  2.11934550e-04  9.46457792e-05 -3.36739425e-05
       2.26167868e-04 -1.19769254e-03  3.40602275e-05 -1.23376941e-03
       1.15702942e-04 -1.22646117e-05  6.28935842e-04 -9.28306016e-07
      -5.52116221e-05]
     [ 1.14783119e-04  9.46457792e-05  3.84978972e-03 -3.29431563e-03
      -5.88290917e-03  2.96787024e-03 -4.79759035e-05  3.86353396e-03
       1.23200302e-03 -1.19937394e-04 -6.99517992e-04  8.49353980e-06
      -3.29102032e-04]
     [ 1.33038925e-03 -3.36739425e-05 -3.29431563e-03  5.54104395e-01
       3.22722392e-03 -1.40269180e-02 -2.10841800e-04  4.20558065e-03
      -5.26855992e-03  2.82056262e-04  6.89665918e-03 -1.46206232e-04
       5.23290591e-04]
     [ 5.23898493e-04  2.26167868e-04 -5.88290917e-03  3.22722392e-03
       1.50306746e+00  1.40723837e-02 -1.51436902e-03  2.06812830e-02
      -4.63635763e-03 -9.52089831e-05  1.54729761e-02  8.47550865e-05
      -8.11699603e-04]
     [-5.03102014e-04 -1.19769254e-03  2.96787024e-03 -1.40269180e-02
       1.40723837e-02  1.88532573e-01 -1.00276279e-03  1.23364142e-02
      -3.96165695e-03  1.36181246e-04  7.05598456e-03  1.40262994e-04
       1.23707593e-02]
     [ 2.54259490e-05  3.40602275e-05 -4.79759035e-05 -2.10841800e-04
      -1.51436902e-03 -1.00276279e-03  1.92886918e-04  1.11553215e-03
       1.41681895e-05 -2.21449764e-06  6.89317812e-05 -3.43306916e-06
      -2.82745377e-04]
     [ 8.10729775e-04 -1.23376941e-03  3.86353396e-03  4.20558065e-03
       2.06812830e-02  1.23364142e-02  1.11553215e-03  4.24687506e-02
       5.06790571e-04  1.39860154e-05 -6.06660615e-03  8.08883058e-06
      -1.11213750e-04]
     [-7.08264350e-04  1.15702942e-04  1.23200302e-03 -5.26855992e-03
      -4.63635763e-03 -3.96165695e-03  1.41681895e-05  5.06790571e-04
       4.84860106e-03 -2.19995676e-04 -1.29297181e-03  2.71447407e-05
      -3.82643525e-05]
     [ 1.73865830e-06 -1.22646117e-05 -1.19937394e-04  2.82056262e-04
      -9.52089831e-05  1.36181246e-04 -2.21449764e-06  1.39860154e-05
      -2.19995676e-04  1.54552120e-05 -3.29135378e-05  2.00490745e-07
       2.10006857e-06]
     [ 4.19197193e-06  6.28935842e-04 -6.99517992e-04  6.89665918e-03
       1.54729761e-02  7.05598456e-03  6.89317812e-05 -6.06660615e-03
      -1.29297181e-03 -3.29135378e-05  1.72110542e-02 -3.21893047e-05
      -4.28030248e-04]
     [ 9.65422950e-07 -9.28306016e-07  8.49353980e-06 -1.46206232e-04
       8.47550865e-05  1.40262994e-04 -3.43306916e-06  8.08883058e-06
       2.71447407e-05  2.00490745e-07 -3.21893047e-05  8.87238476e-06
       3.82467045e-05]
     [-4.28286306e-04 -5.52116221e-05 -3.29102032e-04  5.23290591e-04
      -8.11699603e-04  1.23707593e-02 -2.82745377e-04 -1.11213750e-04
      -3.82643525e-05  2.10006857e-06 -4.28030248e-04  3.82467045e-05
       2.91326441e-03]]
    Number of iterations applied to obtain estimated parameters: 300
    Value of the log marginal likelihood (to be maximized) at each iteration: [-1353.62507777 -1227.46832923 -1226.12557601 -1225.95851367
     -1225.93824383 -1225.93569    -1225.9353617  -1225.93531916
     -1225.93531363 -1225.93531291 -1225.93496052]
    Coefficients obtained: [-0.11428353  0.04907884 -0.0518813   1.62176048 -1.34219915  3.4597405
     -0.01454935 -1.18479245  0.21385904 -0.01261243 -0.91358884  0.00882926
     -0.52904135]
    R^2 score on training set: 0.7648922821629467,test set: 0.5623483354170234

## Logistic Regression

La **logistic regression sebbene il nome non è in genere usato per fare la regressione, ma per fare classificazione**, la base su cui si fonda la  logistic regression è la funzione logistica che ha la seguente formula:

\begin{equation}
    \Large f(x) = C \frac{1}{1 + e^{-k(x-x_0)}}
\end{equation}

dove $C$ rappresenta il valore massimo della funzione, $k$ rappresenta la ripidità con cui sale la curva, $x_0$ rappresenta il punto in cui la funzione assume valore $f(x_0) = \frac{1}{2}$.

```python
import numpy as np

def logistica(X, C = 1, k = 1, x0 = 0):
    return C /(1 + np.exp(-k *(X-x0)))

X = np.linspace(-10,10,100)

fig, axs = plt.subplots(1, 4, figsize = (18,6))
fig.suptitle("Funzione logistica per diversi parametri")

axs[0].plot(X, logistica(X))
axs[0].set_title("C = 1, k = 1, x0 = 0")
axs[0].grid(True)

axs[1].plot(X, logistica(X, C = 2))
axs[1].set_title("C = 2, k = 1, x0 = 0")
axs[1].grid(True)

axs[2].plot(X, logistica(X, k = 10))
axs[2].set_title("C = 1, k = 10, x0 = 0")
axs[2].grid(True)

axs[3].plot(X, logistica(X, x0 = 5))
axs[3].set_title("C = 1, k = 1, x0 = 5")
axs[3].grid(True)

plt.show()
```

Ora che abbiamo capito come è fatta la funzione logistica, vediamo di capire cosa fa la logistic regression guardando la formula usata per l'optimizzazione in scikit:

\begin{equation}
    \Large \min_{w,c} \frac{1-\rho}{2} w^{T}w + \rho \parallel w \parallel_{1} + C \sum_{i=1}^{n} log(exp(-y_i (X_{i}^{T} w + c)) +1)
\end{equation}

In questa equazione possiamo vedere che abbiamo sia il termine di regoralizzazione $l2$, che $l1$, il cui peso nella formula ci dice in quale misura stiamo usando le regoralizzazioni, abbiamo poi un valore $C$ iperparametro dell'equazione che dobbiamo determinare per capire in che misura vogliamo che la regoralizzazione sia determinante ed notare bene **usiamo il log sulle funzione logistiche, questo perché in questa maniera si riduce la possibilità di avere numeri enormi e computazionalmente è più veloce**.

Per valutare la qualità del modello nella classificazione si possono utilizzare diverse metriche, per capire come funzionano in genere bisogna capire cosa si intendono con True Positive(TP), False Postive(FP), True Negative(TN), False Negative(FN). Al fine di capire come funziona cerchiamo di rappresentarli usando una __[confusion matrix](https://scikit-learn.org/stable/modules/model_evaluation.html#confusion-matrix)__ che è qui rappresentata in maniera generica:

| Actual/Predicted |      True      |      False     |   |   |
|:----------------:|:--------------:|:--------------:|---|---|
|       True       |  True Positive | False Positive |   |   |
|       False      | False Negative |  True Negative |   |   |

In questa matrice possiamo notare che il caso ideale sarebbe se i valori sono tutti sulla diagonale poiché il modello dovrebbe essere in grado di classificare con accuratezza assoluta il dataset. In genere però per aver una singola metrica che contenga tutte le informazioni riguardo alla sua capacità di classificazione in maniera quantitativa, tra le più usate abbiamo:
- precision : $ \Large \frac{tp}{tp+tn} $
- accuracy : $ \Large \frac{\sum_{i = 0}^{N-1} I(y_{pred} = y_{true})}{N} $ con $I$ funzione di identità che fornisce il valore 1 solo se la condizione è soddisfatta e $N$è il numero di sample
- recall : $ \Large \frac{tp}{tp + fn} $
- f1 score: $ 2* \Large \frac{precision * recall}{precision + recall} $
Esistono molte altre metriche per la classificazione per maggiori info consultate __[scikit](https://scikit-learn.org/stable/modules/classes.html#module-sklearn.metrics)__.

La logistic regression può essere usata sia in condizioni di classificazione binaria che in condizione di classificazione di classi multiple , vediamo degli esempi in tal caso.

```python
from sklearn.linear_model import LogisticRegressionCV
from sklearn.metrics import classification_report
from mlxtend.plotting import plot_decision_regions

#load data
df = pd.read_csv('../data/diabetes2.csv')
print('Dataset of diabetes used:')
display(df)
#prepare data
X_train, X_test, y_train, y_test = train_test_split(df.drop('Outcome', axis = 1).values,
                                                     df.Outcome.values, random_state=0, test_size = 0.2)
target_names = ['class 0', 'class 1']
#get the best classifier given training data
logistic = LogisticRegressionCV(class_weight= 'balanced', cv = 10, max_iter = 1e4).fit(X_train, y_train)
print(f'Logisitc regression score using 10 folds:\n {logistic.scores_}')
print('-'*80)
print(f'Training accuracy score: {logistic.score(X_train, y_train)}, Test accuracy score:{logistic.score(X_test, y_test)}')
print('-'*80)
print(classification_report(y_test, logistic.predict(X_test), target_names=target_names))
#create output dataframe
output = {'Probability class 0': logistic.predict_proba(X_test)[:,0],
          'Probability class 1': logistic.predict_proba(X_test)[:,1],
          'Predicted': logistic.predict(X_test),
          'True': y_test}
test_results = pd.DataFrame(data = output, index = np.arange(0, y_test.shape[0]))
print('Dataframe of the results with the classes predicted, true and probability associated')
display(test_results)
```

    Dataset of diabetes used:

      Pregnancies
      Glucose
      BloodPressure
      SkinThickness
      Insulin
      BMI
      DiabetesPedigreeFunction
      Age
      Outcome

      0
      6
      148
      72
      35
      0
      33.6
      0.627
      50
      1

      1
      1
      85
      66
      29
      0
      26.6
      0.351
      31
      0

      2
      8
      183
      64
      0
      0
      23.3
      0.672
      32
      1

      3
      1
      89
      66
      23
      94
      28.1
      0.167
      21
      0

      4
      0
      137
      40
      35
      168
      43.1
      2.288
      33
      1

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      763
      10
      101
      76
      48
      180
      32.9
      0.171
      63
      0

      764
      2
      122
      70
      27
      0
      36.8
      0.340
      27
      0

      765
      5
      121
      72
      23
      112
      26.2
      0.245
      30
      0

      766
      1
      126
      60
      0
      0
      30.1
      0.349
      47
      1

      767
      1
      93
      70
      31
      0
      30.4
      0.315
      23
      0

768 rows × 9 columns

    Logisitc regression score using 10 folds:
     {1: array([[0.74193548, 0.72580645, 0.70967742, 0.70967742, 0.74193548,
            0.72580645, 0.72580645, 0.72580645, 0.72580645, 0.72580645],
           [0.72580645, 0.67741935, 0.69354839, 0.72580645, 0.74193548,
            0.75806452, 0.75806452, 0.75806452, 0.75806452, 0.75806452],
           [0.75806452, 0.74193548, 0.77419355, 0.79032258, 0.80645161,
            0.80645161, 0.80645161, 0.80645161, 0.80645161, 0.80645161],
           [0.75806452, 0.74193548, 0.74193548, 0.74193548, 0.72580645,
            0.74193548, 0.74193548, 0.74193548, 0.74193548, 0.74193548],
           [0.67213115, 0.68852459, 0.6557377 , 0.68852459, 0.68852459,
            0.68852459, 0.68852459, 0.68852459, 0.68852459, 0.68852459],
           [0.67213115, 0.73770492, 0.75409836, 0.78688525, 0.78688525,
            0.78688525, 0.78688525, 0.78688525, 0.78688525, 0.78688525],
           [0.81967213, 0.80327869, 0.78688525, 0.7704918 , 0.75409836,
            0.75409836, 0.75409836, 0.75409836, 0.75409836, 0.75409836],
           [0.70491803, 0.70491803, 0.68852459, 0.68852459, 0.68852459,
            0.67213115, 0.67213115, 0.67213115, 0.67213115, 0.67213115],
           [0.75409836, 0.78688525, 0.78688525, 0.78688525, 0.80327869,
            0.80327869, 0.80327869, 0.80327869, 0.80327869, 0.80327869],
           [0.72131148, 0.73770492, 0.73770492, 0.73770492, 0.73770492,
            0.73770492, 0.73770492, 0.73770492, 0.73770492, 0.73770492]])}
    --------------------------------------------------------------------------------
    Training accuracy score: 0.754071661237785, Test accuracy score:0.7662337662337663
    --------------------------------------------------------------------------------
                  precision    recall  f1-score   support

         class 0       0.87      0.78      0.82       107
         class 1       0.59      0.74      0.66        47

        accuracy                           0.77       154
       macro avg       0.73      0.76      0.74       154
    weighted avg       0.79      0.77      0.77       154

    Dataframe of the results with the classes predicted, true and probability associated

      Probability class 0
      Probability class 1
      Predicted
      True

      0
      0.028167
      0.971833
      1
      1

      1
      0.745021
      0.254979
      0
      0

      2
      0.842474
      0.157526
      0
      0

      3
      0.243233
      0.756767
      1
      1

      4
      0.806568
      0.193432
      0
      0

      ...
      ...
      ...
      ...
      ...

      149
      0.708021
      0.291979
      0
      1

      150
      0.864217
      0.135783
      0
      0

      151
      0.233446
      0.766554
      1
      1

      152
      0.833437
      0.166563
      0
      0

      153
      0.738377
      0.261623
      0
      0

154 rows × 4 columns

Per vedere come funziona con più classi potete vedere questo __[esempio di scikit](https://scikit-learn.org/stable/auto_examples/linear_model/plot_logistic_multinomial.html#sphx-glr-auto-examples-linear-model-plot-logistic-multinomial-py)__ in cui potete anche vedere anche al differenza dei due tipi di algoritmi di classificazione OnevsRest and Multinomial per maggiori info andate __[qui](https://scikit-learn.org/stable/modules/multiclass.html?highlight=onevsrest)__.

## Polynomial Regression

Fino ad ora abbiamo visto metodi lineari che riuscivano a classificare bene, ma in molti casi potrebbe capitare che il dataset presenta **non linearità, in tal caso dobbiamo mappare il dataset in un nuovo spazio in cui possiamo usare i metodi lineari**, scikit in tal caso fornisce come soluzione la trasformazione di dati attraverso una funzione polinomiale oppure un applicazione di una __[kernel ridge regression](https://scikit-learn.org/stable/modules/kernel_ridge.html)__ simile alla futura Support Vector Machine, ma con piccole differenze.
Cerchiamo di creare un problema per cui sia necessario tale situazione e vediamo come applicare questi strumenti usando inoltre la __[pipeline scikit](https://scikit-learn.org/stable/modules/compose.html#combining-estimators)__.

```python
from sklearn.preprocessing import PolynomialFeatures

x = np.arange(20).reshape(10,2)
print(f'Dati originali:\n{x}')

poly = PolynomialFeatures(degree=2)
print(f'Dati tansformati usando un polinomio di grado 2:\n {poly.fit_transform(x)}')

poly = PolynomialFeatures(degree=2, interaction_only= True)
print(f'Dati tansformati usando un polinomio di grado 2 contando solo i termini'
      f'misti, le identità e il termine di grado nullo:\n{poly.fit_transform(x)}')

```

    Dati originali:
    [[ 0  1]
     [ 2  3]
     [ 4  5]
     [ 6  7]
     [ 8  9]
     [10 11]
     [12 13]
     [14 15]
     [16 17]
     [18 19]]
    Dati tansformati usando un polinomio di grado 2:
     [[  1.   0.   1.   0.   0.   1.]
     [  1.   2.   3.   4.   6.   9.]
     [  1.   4.   5.  16.  20.  25.]
     [  1.   6.   7.  36.  42.  49.]
     [  1.   8.   9.  64.  72.  81.]
     [  1.  10.  11. 100. 110. 121.]
     [  1.  12.  13. 144. 156. 169.]
     [  1.  14.  15. 196. 210. 225.]
     [  1.  16.  17. 256. 272. 289.]
     [  1.  18.  19. 324. 342. 361.]]
    Dati tansformati usando un polinomio di grado 2 contando solo i terminimisti, le identità e il termine di grado nullo:
    [[  1.   0.   1.   0.]
     [  1.   2.   3.   6.]
     [  1.   4.   5.  20.]
     [  1.   6.   7.  42.]
     [  1.   8.   9.  72.]
     [  1.  10.  11. 110.]
     [  1.  12.  13. 156.]
     [  1.  14.  15. 210.]
     [  1.  16.  17. 272.]
     [  1.  18.  19. 342.]]

Possiamo quindi vedere come i dati originali nella forma $[x_0, x_1]$ sono stati trasformati nella forma $[1, x_0, x_1, x_{0}^2, x_{0}x_{1}, x_{1}^2]$ mentre con il caso `intercation_only = True` ritorna $[1, x_0, x_1, x_{0}x_{1}]$, vediamo ora come usarlo nel caso di un dataset non lineare.

```python
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline

x = np.arange(100)
y = 2 * x**3 + 3 * x**2 + 5 * x + 4

# generate points used to plot
x_plot = np.linspace(0, 100, 100)

# create matrix versions of these arrays
X = x[:, np.newaxis]
X_plot = x_plot[:, np.newaxis]

colors = ['blue', 'green', 'red']
fig, axs = plt.subplots(figsize = (12,8))
plt.scatter(x, y, color='navy', s=30, marker='o', label="training points")

#polynomial features
for count, degree in enumerate([1, 2, 3]):
    model = make_pipeline(PolynomialFeatures(degree), LinearRegression())
    model.fit(X, y)
    y_plot = model.predict(X_plot)
    plt.plot(x_plot, y_plot, color=colors[count],
             label="degree %d" % degree)

plt.legend()

plt.show()
```

Abbiamo quindi capito come applicare trasformazioni lineari, in futuro scopriremo che questo approcio è molto sconveniente dal punto di vista computazionale a causa della dimensione sempre maggiore da tenere in conto, per questo in futuro vedremo le __[SVM](https://scikit-learn.org/stable/modules/svm.html#svm)__ per ovviare a questi problemi.

---

COMPLIMENTI AVETE FINITO IL NOTEBOOK 4 DI MACHINE LEARNING!


```python
# -*- coding: utf-8 -*-
"""
Created on Thu Sep 16 20:23:07 2021

@author: gansa001
"""
from sympy import *
from sympy.plotting import plot
import matplotlib.pyplot as plt
import numpy as np
import math
```

### 1. Write a computer program to calculate the Lagrange interpolation polynomial Pn(x) to f(x) such that: Pn(xj)=f(xj)  j=0,1,2,……,n

```python
def delta(j,a,x,N):
    '''
    Parameters
    ----------
    j : int.
        the jth lagrange polynomial
    a : sympy symbol
        the dependent term in the lagrange polynomial
    x : list
        x values from the original function
    N : int

    Returns
    -------
    The lagrange polynomial for any given j

    '''
    num=1
    den=1
    for i in range(N):
        if i!=j:
            num*=(a-x[i])
            den*=(x[j]-x[i])
    return (num/den)

def Lagrange(x,y):
    '''

    Parameters
    ----------
    x : list
        dependent variable values
    y : list
        independent variable values

    Returns
    -------
    f: The lagrangian approximation of the function

    '''
    N=len(x)
    a=Symbol('a')
    f=0
    for j in range(len(x)):
        answer= (delta(j,a,x,N))
        f+=y[j]*answer
    return (simplify(f))
```

#### a)	Apply it to some elementary functions or discrete data sets.

```python
'''
Exponential function from range -1 to 1 and 3 points to confirm the values from class.
'''
x=[-1,0,1]
y=[math.exp(i) for i in x]

print('x = ',x)
print('y = ',y)
```

    x =  [-1, 0, 1]
    y =  [0.36787944117144233, 1.0, 2.718281828459045]

```python
##function call
Lagrange(x,y)
```

$\displaystyle 0.543080634815244 a^{2} + 1.1752011936438 a + 1.0$

```python
##plotting
a=Symbol('a')

p=plot(Lagrange(x,y), (E**a),(a,-1,1),show=False)
p[1].line_color='r'
p[0].label='Pn(x)'
p[1].label='f(x)'
p.legend=True
p.show()
```

```python
'''
Quartic function from range -1 to 1 and 3 points.
'''
x=[-1,0,1]
y=[i**4 for i in x]

print('x = ',x)
print('y = ',y)
```

    x =  [-1, 0, 1]
    y =  [1, 0, 1]

```python
Lagrange(x,y)
```

$\displaystyle a^{2}$

```python
p=plot(Lagrange(x,y), (a**4),(a,-1,1),show=False)
p[1].line_color='r'
p[0].label='Pn(x)'
p[1].label='f(x)'
p.legend=True
p.show()
```

#### b)	Apply it to function $$ f(x) = \frac{1}{1+25x^2} $$  in the interval [-1,1] with equally spaced interpolation points and n=8, 16, 32.  Plot Pn(x) and f(x) in [-1,1] interval to see the interpolation error.

```python
n=[8,16,32]

for npoints in n:
    print('********** NUMBER OF POINTS = ',npoints,' **********')
    print()
    x=list(np.linspace(-1,1,npoints))
    y=[(1/(1+25*(i**2))) for i in x]

    print('Lagrange= ',Lagrange(x,y))

    p=plot(Lagrange(x,y), (1/(1+25*(a**2))),(a,-1,1),show=False)
    p[1].line_color='r'
    p[0].label='Pn(x)'
    p[1].label='f(x)'
    p.legend=True
    p.show()

    print('\n\n')

```

#### c)	Do the same as in b, but use the roots of the Chebychev polynomial Tn+1(x) as interpolation points. Compare results with those obtained in (b).

```python
for npoints in n:
    print('********** NUMBER OF POINTS = ',npoints,' **********')
    x = [math.cos(((math.pi/2)*(2*j+1))/npoints) for j in range(npoints)]
    print('Chebyshev roots= ',x)
    print()
    y=[(1/(1+25*(i**2))) for i in x]

    print('Lagrange= ',Lagrange(x,y))

    p=plot(Lagrange(x,y), (1/(1+25*(a**2))),(a,-1,1),show=False)
    p[1].line_color='r'
    p[0].label='Pn(x)'
    p[1].label='f(x)'
    p.legend=True
    p.show()

    print('\n\n')
```

### Comparing errors

```python
for npoints in n:
    print('********** NUMBER OF POINTS = ',npoints,' **********')
    x = [math.cos(((math.pi/2)*(2*j+1))/npoints) for j in range(npoints)]
    print('Chebyshev roots= ',x)
    print()
    x1=list(np.linspace(-1,1,npoints))
    y=[(1/(1+25*(i**2))) for i in x]

    print('Lagrange= ',Lagrange(x,y))

    p=plot(Lagrange(x,y),Lagrange(x1,y), (1/(1+25*(a**2))),(a,-1,1),show=False)
    p[1].line_color='r'
    p[2].line_color='m'
    p[1].label='Pn(x)'
    p[0].label='Tn(x)'
    p[2].label='f(x)'
    p.legend=True
    p.show()

    print('\n\n')
```

# The plots above show that using the Chebyshev roots provide a much smaller error when doing Lagragian interpolation as opposed to using equally spaced points in a given range.


# day22: Principal Components Analysis

# Objectives

* Understand PCA on both toy and "real" image datasets
* Get familiar with how to perform transformations to low-dim. embeddings.
* Get familiar with how to "reconstruct" images

# Outline

* [Part 1: PCA for toy data with F=2 features](#part1)
* [Part 2: PCA with K=10 for image data with many features](#part2)
* [Part 3: PCA where we select K for image data with many features](#part3)

# Takeaways

* Understanding how we add/remove the *mean* vector is important
* Understanding that PCA is a nested model
* * The component vectors for K=4 are the first four components of K=5, K=6, or K=10 or ...)
* Selecting the number of components can be done in several ways:
* * Using the fraction of explained variance criteria, as discussed here
* * * Can be done very fast by reusing eigenvalues used at training time anyway
* * Use an application specific performance metric

```python
import numpy as np
import pandas as pd
```

```python
import sklearn.datasets
```

```python
import sklearn.decomposition
```

```python
# import plotting libraries
import matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
plt.style.use('seaborn') # pretty matplotlib plots

import seaborn as sns
sns.set('notebook', font_scale=1.25, style='whitegrid')
```

```python
import matplotlib.colors as mcolors
```

```python

```

# Setup: Create Small Toy Dataset with F=2

We will have N=21 examples, with 2 features $x_1$ and $x_2$

```python
random_state = np.random.RandomState(0)

# Create N training x values between -5 and 5
N = 21
F = 2

x1_N = np.linspace(-5, 5, N) + random_state.uniform(low=-0.2, high=0.2, size=N)

B = 1.5
x2_N = 1.0 * x1_N + random_state.uniform(low=-B, high=B, size=N)

x_NF = np.zeros((N,F))
x_NF[:,0] = x1_N
x_NF[:,1] = x2_N

# Plot the training data
plt.plot(x1_N, x2_N, 'ks')
plt.title('Training Data')
plt.xlabel('x1');
plt.ylabel('x2');
plt.gca().set_aspect('equal', 'box')
```

# Review: Math of Principal Components Analysis (PCA)

We'll assume feature vectors  $x \in \mathbf{R}^F$.

We want to find a principal components analysis model with $K$ components, which has parameters:
* $m \in \mathbb{R}^F$ : mean vector
* $W \in \mathbb{R}^{F \times K}$ : weight vectorss for each component

This model makes projections to the latent "codes" $z_n \in \mathbb{R}^K$ like this:
$$
z_n = \mathbf{W}^T (\mathbf{x}_n - \mathbf{m})
$$

And then makes reconstructions like this:
$$
\hat{x}_n(\mathbf{m}, \mathbf{W}) = \mathbf{W} z_n + \mathbf{m}
$$

The training objective is:
$$
\min_{\mathbf{m} \in \mathbb{R}^F, \mathbf{W} \in \mathbb{R}^{F\times K}}
\quad \sum_{n=1}^N \sum_{f=1}^F (\mathbf{x}_{nf} - \hat{\mathbf{x}}_{nf}(\mathbf{m}, \mathbf{W}) )^2
$$

# Part 1: PCA for F=2 data

## Fit a pca model with K=1 component to our data

```python
pca = sklearn.decomposition.PCA(n_components=1)
```

```python
pca.fit(x_NF)
```

    PCA(n_components=1)

This fitting process estimates two parameters, $m$ and $W$, and stores them as attributes of `pca`

## Inspecting the learned Mean vector $m$

This vector is accessible as the `mean_` attribute

```python
m_F = pca.mean_
print(m_F.shape)
print("")
print(m_F)
```

    (2,)

    [0.04018504 0.13658959]

## Exercise 1a: Can you convince yourself that this is the same as the training set's empirical mean vector?

Compute the mean of each feature in the training set `x_NF`

```python
# TODO
```

## Inspecting the learned weights $W$

This vector is accessible as the `components_` attribute of our learned `pca` model

```python
w_KF = pca.components_
print(w_KF.shape)
print("")
print(w_KF)

```

    (1, 2)

    [[0.69601341 0.71802879]]

## Exercise 1b: Can you convince yourself that this weight vector is a unit vector?

That is, is the sum-of-squares magnitude of each component vector (each row) equal to one?

```python
# TODO compute sum of squares of each row of w_KF
```

## Calling `transform`: Obtaining the low-dimensional embeddings

```python
z_NK = pca.transform(x_NF)
```

```python
print(z_NK.shape)
print("")
print(z_NK)
```

    (21, 1)

    [[-6.52423243]
     [-6.4504965 ]
     [-5.11980459]
     [-5.87207543]
     [-4.10995134]
     [-4.34687772]
     [-2.03157546]
     [-1.97845129]
     [-1.46165685]
     [-1.40616067]
     [ 0.62968306]
     [ 0.50286399]
     [ 1.47389739]
     [ 1.19917502]
     [ 2.71280757]
     [ 3.41699812]
     [ 4.09659534]
     [ 5.96711016]
     [ 6.07911096]
     [ 6.14379828]
     [ 7.07924238]]

### Compute the reconstructions

Each training example (index $n$) can be reconstructed as:

$$
\hat{x}_n(\mathbf{m}, \mathbf{W}) = \mathbf{W} z_n + \mathbf{m}
$$

If we convert this to matrix-vector arithmetic for the full dataset, we can do:

$$
\hat{X} = \mathbf{Z} \mathbf{W} + \mathbf{m}
$$

where $\hat{X}$ is an $N \times F$ matrix, where each row is equal to $\hat{x}_n(\mathbf{m}, \mathbf{W})$

```python
xhat_NF = np.dot(z_NK, w_KF) + m_F[np.newaxis,:]
```

```python
print(xhat_NF.shape)
print()
print("First 5 rows of this array")
print(xhat_NF[:5])
```

    (21, 2)

    First 5 rows of this array
    [[-4.5007682  -4.54799709]
     [-4.449447   -4.49505257]
     [-3.52326759 -3.53957748]
     [-4.04685818 -4.07972959]
     [-2.82039619 -2.81447377]]

### Visualize the reconstructions

```python
# Plot the training data
plt.plot(x1_N, x2_N, 'ks', label='Training Data')

# Plot the reconstructed data
plt.plot(xhat_NF[:,0], xhat_NF[:,1], 'r.', label='Reconstructed Data', mew=5)

# Draw line between training example and reconstruction
for n in range(N):
    xy_22 = np.zeros((2,2))
    xy_22[0] = x_NF[n,:]
    xy_22[1] = xhat_NF[n,:]
    plt.plot(xy_22[:,0], xy_22[:,1], 'r-')

plt.legend(bbox_to_anchor=(1.75, 0.5))
plt.xlabel('x1');
plt.ylabel('x2');
plt.gca().set_aspect('equal', 'box')
```

## Show that we know how to perform the PCA transform ourselves

```python
def my_transform(x_NF, w_KF, m_F):
    ''' Perform transformation to get low-dim embeddings

    Args
    ----
    x_NF : 2D array, shape (N,F)
    w_KF : 2D array, shape (K,F)
    m_F : 1D array, shape (F,)

    Returns
    -------
    z_NK : 2D array, shape (n_examples, n_components) = (N,K)
        Each row is a low-dim. "embedding" of the corresponding row in x_NF
    '''

    # Use the equation defining z in terms of x and m and W
    z_NF = np.dot(x_NF - m_F[np.newaxis,:], w_KF.T)
    return z_NF
```

```python
z_NK = my_transform(x_NF, w_KF, m_F)
```

```python
print("Shape of z_NK")
print(z_NK.shape)
print()
print("First 5 rows of z_NK")
print(z_NK[:5])
```

    Shape of z_NK
    (21, 1)

    First 5 rows of z_NK
    [[-6.52423243]
     [-6.4504965 ]
     [-5.11980459]
     [-5.87207543]
     [-4.10995134]]

## Discussion 1c: Are these z_NK values from `my_transform` the same as produced by `pca.transform`?

```python
# TODO discuss
```

## Now try with K=2 PCA for our F=2 dataset

```python
pca = sklearn.decomposition.PCA(n_components=2)
pca.fit(x_NF)
```

    PCA(n_components=2)

Print out the transformed embedding features:

```python
pca.transform(x_NF)
```

    array([[-6.52423243, -0.66808798],
           [-6.4504965 ,  0.0494726 ],
           [-5.11980459, -0.6066986 ],
           [-5.87207543,  0.78661395],
           [-4.10995134, -0.29266499],
           [-4.34687772,  0.75715317],
           [-2.03157546, -0.90685279],
           [-1.97845129, -0.00897906],
           [-1.46165685,  0.22647118],
           [-1.40616067,  0.5457974 ],
           [ 0.62968306, -0.50382783],
           [ 0.50286399,  0.16903618],
           [ 1.47389739, -0.05406408],
           [ 1.19917502,  1.10777136],
           [ 2.71280757, -0.13916024],
           [ 3.41699812, -0.11644356],
           [ 4.09659534, -0.11612761],
           [ 5.96711016, -0.78036674],
           [ 6.07911096, -0.22292293],
           [ 6.14379828,  0.46189491],
           [ 7.07924238,  0.31198566]])

## Discussion 1d: What do you notice about the first column of z here when K=2?

How does it compare with the old z?

```python
# TODO discuss
```

## Exercise 1e: Show the learned weight vectors

Print out the learned weight vectors for K=2 and show your self they are "unit" vectors.

```python
# TODO access and print weights from the `pca` object
```

## Discussion 1f: How do the learned weight vectors for K=2 compare to the old weights for K=1?

What is different? What is the same? Do any vectors point in the same direction?

```python
# TODO discuss
```

# Part 2: PCA for 8x8 digit images

Let us load a dataset of 8x8 pixel images of numerical digits (1, 2, ... 8, 9, 0) that comes from sklearn

```python
x_all_LF, y_all_L = sklearn.datasets.load_digits(n_class=10, return_X_y=True)
x_all_LF = np.asarray(x_all_LF, dtype=np.float64) / 16.0 # Make each pixel intensity between 0 and 1

F = 64 # number of features (flattened 8x8 pixel grayscale image)

L = 1797 # all examples
N = 1697 # train set size
T =  100 # test set size

shuffled_rows_L = np.random.RandomState(0).permutation(L)
train_rows_N = shuffled_rows_L[:N]
test_rows_T = shuffled_rows_L[N:]

x_tr_NF = x_all_LF[train_rows_N].copy()
y_tr_N = y_all_L[train_rows_N].copy()

x_te_TF = x_all_LF[test_rows_T].copy()
y_te_T = y_all_L[test_rows_T].copy()
```

```python
x_tr_NF.shape
```

    (1697, 64)

```python
x_te_TF.shape
```

    (100, 64)

## Show the first image in training set

```python
plt.imshow(x_tr_NF[0].reshape((8,8)), interpolation='nearest', vmin=0, vmax=1, cmap='gray')
plt.colorbar();
plt.gcf().set_size_inches((3,3));
```

## Show fifth row image in training set

```python
plt.imshow(x_tr_NF[4].reshape((8,8)), interpolation='nearest', vmin=0, vmax=1, cmap='gray')
plt.colorbar();
plt.gcf().set_size_inches((3,3));
```

# Fit PCA with 10 components to this image dataset

We'll fit this to the entire training set.

```python
K = 10
```

```python
digit_pca = sklearn.decomposition.PCA(n_components=K)
```

```python
digit_pca.fit(x_tr_NF)
```

    PCA(n_components=10)

Unpack the mean parameter

```python
m_F = digit_pca.mean_
```

Unpack the weights (K x F matrix)

```python
w_KF = digit_pca.components_
```

```python
w_KF.shape
```

    (10, 64)

## Show the mean image

```python
plt.imshow(m_F.reshape((8,8)), interpolation='nearest', vmin=0, vmax=1, cmap='gray')
plt.colorbar();
plt.gcf().set_size_inches((3,3));
```

## Show the weight vectors for the first 10 components (each one reshaped as an image)

```python
W = 2
H = 2
fig, axgrid = plt.subplots(nrows=2, ncols=5, figsize=(5*W, 2*H))
for k in range(10):
    cur_ax = axgrid.flatten()[k]
    im_handle = cur_ax.imshow(w_KF[k].reshape((8,8)), interpolation='nearest', vmin=-.5, vmax=.5, cmap='RdBu_r')
    cur_ax.set_xticks([]);
    cur_ax.set_yticks([]);
    cur_ax.set_title("Comp k=%d" % (k+1))
    if k == K - 1:
        plt.colorbar(im_handle, fraction=0.04, shrink=1.0);
```

## Discussion 2a: Are these interpretable? Do you see evidence of the "digits" here?

Look for evidence of strong vertical lines, or certain holes, etc. What are these components capturing?

```python
# TODO discuss
```

## Create reconstructed images

```python
x_tr_NF.shape
```

    (1697, 64)

```python
z_tr_NK = my_transform(x_tr_NF, w_KF, m_F)
```

```python
z_tr_NK.shape
```

    (1697, 10)

```python
xhat_tr_NF = np.dot(z_tr_NK, w_KF) + m_F[np.newaxis,:]
```

```python
xhat_tr_NF.shape
```

    (1697, 64)

## Show the original and reconstructed images from training set

We'll look at *first five* training examples.

```python
row_ids = [0, 1, 2, 3, 4]

W = 2
H = 2
fig, axgrid = plt.subplots(nrows=2, ncols=len(row_ids), figsize=(5*W, 2*H))

# Top row: original images (rows of x_tr_NF)
for col_id, row_id in enumerate(row_ids):
    cur_ax = axgrid.flatten()[col_id]
    im_handle = cur_ax.imshow(x_tr_NF[row_id].reshape((8,8)),
                              interpolation='nearest', vmin=0.0, vmax=1.0, cmap='gray')
    if col_id == 0:
        cur_ax.set_ylabel("Original Image")
    cur_ax.set_title("Im. %d" % (row_id))
    cur_ax.set_xticks([]);
    cur_ax.set_yticks([]);

# Bottom row: reconstructed images (rows of xhat_tr_NF)

for col_id, row_id in enumerate(row_ids):
    cur_ax = axgrid[1].flatten()[col_id]
    im_handle = cur_ax.imshow(xhat_tr_NF[row_id].reshape((8,8)),
                              interpolation='nearest', vmin=0.0, vmax=1.0, cmap='gray')
    if col_id == 0:
        cur_ax.set_ylabel("Reconstruction\nwith K=10")
    cur_ax.set_xticks([]);
    cur_ax.set_yticks([]);
plt.colorbar(im_handle, fraction=0.05);
plt.tight_layout()

```

## Show more original and reconstructed images from training set: All 9s

We'll look at *five other* training examples, selected because they are all 9s

```python
row_ids = np.hstack(np.flatnonzero(y_tr_N== 9)[-5:])
W = 2
H = 2
fig, axgrid = plt.subplots(nrows=2, ncols=len(row_ids), figsize=(5*W, 2*H))

# Top row: original images (rows of x_tr_NF)
for col_id, row_id in enumerate(row_ids):
    cur_ax = axgrid.flatten()[col_id]
    if col_id == 0:
        cur_ax.set_ylabel("Original Image")
    cur_ax.set_title("Image %d" % (row_id))
    im_handle = cur_ax.imshow(x_tr_NF[row_id].reshape((8,8)),
                              interpolation='nearest', vmin=0.0, vmax=1.0, cmap='gray')
    cur_ax.set_xticks([]);
    cur_ax.set_yticks([]);

# Bottom row: reconstructed images (rows of xhat_tr_NF)

for col_id, row_id in enumerate(row_ids):
    cur_ax = axgrid[1].flatten()[col_id]
    if col_id == 0:
        cur_ax.set_ylabel("Reconstruction\nwith K=10")
    im_handle = cur_ax.imshow(xhat_tr_NF[row_id].reshape((8,8)),
                              interpolation='nearest', vmin=0.0, vmax=1.0, cmap='gray')
    cur_ax.set_xticks([]);
    cur_ax.set_yticks([]);
plt.colorbar(im_handle, fraction=0.05);
plt.tight_layout()

```

## Discussion 2b: How good are these reconstructions?

Across the two plots above, what do you notice?

Remember that we are storing only 10 values for each image, not 64, which is a 6x savings!

Given that, do you think the reconstructions are pretty good? What visual aspects aren't covered very well?

## Exercise 2c: Show reconstructions for the first five *test* examples

Step 1: Compute the low-dimensional embedding for the test set x_te_TF

```python
z_te_TK = None # TODO transform x_te_TF to obtain z_te_TK by calling pca.transform
```

```python
xhat_te_TF = np.zeros((T,F)) # TODO reconstruct the data by multiplying weights times z, plus adding the mean
```

```python
row_ids = [0, 1, 2, 3, 4] # row ids for our test set
W = 2
H = 2
fig, axgrid = plt.subplots(nrows=2, ncols=len(row_ids), figsize=(5*W, 2*H))

# Top row: original images (rows of x_tr_NF)
for col_id, row_id in enumerate(row_ids):
    cur_ax = axgrid.flatten()[col_id]
    if col_id == 0:
        cur_ax.set_ylabel("Original Image")
    cur_ax.set_title("Im. %d" % (row_id))
    im_handle = cur_ax.imshow(x_te_TF[row_id].reshape((8,8)),
                              interpolation='nearest', vmin=0.0, vmax=1.0, cmap='gray')
    cur_ax.set_xticks([]);
    cur_ax.set_yticks([]);

# Bottom row: reconstructed images (rows of xhat_tr_NF)
for col_id, row_id in enumerate(row_ids):
    cur_ax = axgrid[1].flatten()[col_id]
    if col_id == 0:
        cur_ax.set_ylabel("Reconstruction\nwith K=10")
    im_handle = cur_ax.imshow(xhat_te_TF[row_id].reshape((8,8)),
                              interpolation='nearest', vmin=0.0, vmax=1.0, cmap='gray')
    cur_ax.set_xticks([]);
    cur_ax.set_yticks([]);
plt.colorbar(im_handle, fraction=0.05);
plt.tight_layout()

```

## Discussion 2d: Is the PCA model good at reconstructing the test examples it has never seen before?

Why or why not? Do you notice any differences compared to the training data?

# Part 3: PCA with ALL components and selecting K

## First, fit PCA that learns all possible components

Remember that in general, PCA is a *nested* model.

The weights for K=5 are just the first 5 rows of the weights for K=6.

The weights for K=9 are just the first 9 rows of the weights for K=10 or K=15, etc.

```python
all_pca = sklearn.decomposition.PCA().fit(x_tr_NF)
```

# Determine how each additional component explains observed variance better

```python
plt.plot(np.cumsum(all_pca.explained_variance_ratio_), 'bs-');
plt.xlabel('number of components K');
plt.ylabel('fraction of variance explained');
```

## Exercise 3a: What K value will give you at least 80% of the variance? 95%?

Specify a concrete K value, by carefully interpreting the plot above.

Often, we use these heuristics to choose a K that strikes a balance between "good" reconstruction and small size.

```python

```

## Exercise 3b: Visualize reconstructions for the first 5 training set images with K=30.

```python
# TODO compute z
```

```python
# TODO compute xhat given z
```

```python
# TODO make plot of reconstructions xhat
```

## Exercise 3c: Visualize reconstructions for the first 5 *test* set images with K=30.

```python
# TODO compute z
```

```python
# TODO compute xhat given z
```

```python
# TODO make plot of reconstructions xhat
```

```python

```

# Part 4: Understanding what we mean by "explained variance"

A goal of PCA is to find a low-dimensional reconstruction of our data that contains all the possible "information" of our original data.

One way to define this precisely is to first look at squared error "reconstruction cost" of using just the empirical mean of our training data (this is like PCA with $K=0$)

\begin{align}
\text{cost}_{K=0}(X, m) &= \frac{1}{N} \sum_{n=1}^N \sum_{f=1}^F (x_{nf} - m_f)^2 \\
&= \frac{1}{N} \sum_{n=1}^N \sum_{f=1}^F \tilde{x}_{nf}^2, \qquad \tilde{x}_{nf} = x_{nf} - m_f
\end{align}

Note that this cost is precisely equal to the *empirical variance* of the training set, since variance is the average squared error from the mean.

This is just a property of the data itself. We can't change it. But it gives us a place to start when quantifying the quality of a reconstruction.

Now, consider a reconstruction that uses the mean $m$ as well as a good $K$-dimensional weight component matrix $W$.

We should be able to do better than the baseline above, right?

Let's write the cost when we have $K>0$ principal components:

\begin{align}
\text{cost}_K(X, m, W) &= \frac{1}{N}  \sum_{n=1}^N \sum_{f=1}^F (x_{nf} - m_f - \sum_{k=1}^K W_{kf} \tilde{x}_{nf} )^2 \\
\\
&= \frac{1}{N}  \sum_{n=1}^N \sum_{f=1}^F (\tilde{x}_{nf} - \sum_{k=1}^K W_{kf} \tilde{x}_{nf} )^2 \\
\end{align}

Now, this cost will have units that depend on the scale of the original data. However, we can look at the *ratio* of this cost to the original mean-only cost above.

\begin{align}
\text{frac-unexplained-variance} = \frac{\text{cost}_K(X, m, W)}{\text{cost}_{K=0}(X,m)}
    &= \frac
        { \frac{1}{N}  \sum_{n=1}^N \sum_{f=1}^F (\tilde{x}_{nf} - \sum_{k=1}^K W_{kf} \tilde{x}_{nf} )^2 }
        { \frac{1}{N}  \sum_{n=1}^N \sum_{f=1}^F \tilde{x}_{nf}^2 }
\end{align}

Notice that as $K \rightarrow 0$, this ratio approaches 1, while as $K \rightarrow F$ this ratio approaches 0 on the training set. It turns out that this ratio will *monotonically decrease* as we increase $K$ and iteratively find the optimal weights $W$ for that K value.

We can thus interpret this fraction as a measure of the *unexplained* variance of our original dataset. If the fraction is close to 1., most of our data is not well approximated by our PCA reconstructions. If the fraction is close to 0, we are doing a good job (most of the variance is accounted for).

### Surprising result: We can compute this fraction simply using sums of eigenvalues we already have!

Now we can show a surprising result. The denominator can be computed as the sum of all eigenvalues of the covariance of $X$. And the numerator can be computed as a partial version of this sum.

$$
\text{frac-unexplained-variance} = \frac{\text{cost}_K(X, m, W)}{\text{cost}_{K=0}(X,m)}
    = \frac
        { \sum_{\ell={K+1}}^F \lambda_{\ell} }
        { \sum_{f={1}}^F \lambda_{f} }
$$

where $\lambda_k$ is the $k$-th largest eigenvalue of the $F \times F$ covariance matrix of the training data

## Computing the denominator (PCA with K=0 reconstruction cost) in 3 ways

### Method 1: Directly compute reconstruction cost with K=0 model

```python
m_1F = np.mean(x_tr_NF, axis=0)[np.newaxis,:]
```

```python
costK0_v1 = np.sum(np.mean(np.square(x_tr_NF - m_1F), axis=0))
```

```python
costK0_v1
```

    4.686065871686287

### Method 2: Compute cost by measuring empirical variance of each feature and adding up

```python
cov_FF = np.cov(x_tr_NF.T, bias=1)
```

```python
var_F = np.diag(cov_FF)
```

```python
costK0_v2 = np.sum(var_F)
```

```python
costK0_v2
```

    4.6860658716862815

### Method 3: Compute cost as sum of eigenvalues of covariance matrix

```python
lam_F, W_FF = np.linalg.eig(cov_FF)
```

Get eigenvalues in sorted order from big to small

```python
sorted_ids = np.argsort(-1 * lam_F)
```

```python
lam_F = lam_F[sorted_ids]
W_FF = W_FF[:, sorted_ids]
```

```python
lam_F[:10]
```

    array([0.69601696, 0.64133098, 0.55282934, 0.39584765, 0.26867765,
           0.2312219 , 0.20297662, 0.17096015, 0.15846922, 0.14244372])

```python
costK0_v3 = np.sum(lam_F)
```

```python
costK0_v3
```

    4.686065871686285

Verify all three methods are the same!

```python
costK0_v1
```

    4.686065871686287

```python
costK0_v2
```

    4.6860658716862815

```python
costK0_v3
```

    4.686065871686285

## Computing the numerator (PCA with K-components reconstruction cost) in 2 ways

```python
K = 3
```

```python
xtilde_NF = x_tr_NF - m_1F
```

### Method 1: Direct reconstruction cost calculation

```python
W_KF = W_FF[:, :K].T.copy()
```

Compute xhat given x in 2 steps:

* project each x to z
* then project each z back to xhat

```python
# Step 1: x to z
z_tr_NK = np.dot(xtilde_NF, W_KF.T)
```

```python
# Step 2: z to xhat
xhat_tr_NF = np.dot(z_tr_NK, W_KF) + m_1F
```

Now compute the total reconstruction cost

```python
costK_v1 = np.sum(np.mean(np.square(x_tr_NF - xhat_tr_NF), axis=0))
```

```python
costK_v1
```

    2.7958885969409755

### Method 2: Calculation just summing all eigenvalues except the top-K

```python
costK_v2 = np.sum(lam_F[K:])
```

```python
costK_v2
```

    2.795888596940978

```python
np.allclose(costK_v1, costK_v2)
```

    True

## Bringing it all together: Fraction of variance explained

Thus, a measure of our reconstruction quality on the training set is given by the fraction of variance that is explained by our $K$-component PCA

Note, we switch here to using fraction-of-explained-variance (rather than unexplained).

$$
\text{frac-explained-variance} = 1 - \frac{\text{cost}_K(X, m, W)}{\text{cost}_{K=0}(X,m)}
    = \frac
        { \sum_{\ell={1}}^K \lambda_{\ell} }
        { \sum_{f={1}}^F \lambda_{f} }
$$

```python
K
```

    3

### Version 1: Ratio of direct reconstruction costs (PCA with K to PCA with 0 components)

```python
frac_explained_var = 1 - (costK_v1 / costK0_v1)
```

```python
frac_explained_var
```

    0.4033612259200122

### Version 2: Ratio of sums of eigenvectors

```python
frac_explained_var_v2 = np.sum(lam_F[:K]) / np.sum(lam_F)
```

```python
frac_explained_var_v2
```

    0.40336122592001156

### Version 3: Check against sklearn implementation

```python
np.sum(all_pca.explained_variance_ratio_[:K])
```

    0.4033612259200119

```python

```

```python

```


```python
from Integrais import Integrais as it
from EDO import EDO
from sympy import var, Lambda, cos, sin, tan, exp, log, ln, sqrt, solve
x = var('x')
y = var('y')
```

```python
# Questão 01
x0 = 0
y0 = 1
dy = Lambda((x, y), x*y**2 - y)
print('f(x, y) = ', dy(x,y))
I = (0, 2)
N = 8
Y = EDO(dy, y0, x0)
y1, y_steps, x_steps = Y.euler(I, N, dp=6)
print(f'Solução Aproximada: y(2) = {y1}')
```

    f(x, y) =  x*y**2 - y
    Solução Aproximada: y(2) = 0.22829

```python
# Questão 02
x0 = 0
y0 = 0
dy = Lambda((x, y), 2*x + y)
print('f(x, y) = ', dy(x,y))
I = (0, 0.6)
N = 3
Y = EDO(dy, y0, x0)
y1, y_steps, x_steps = Y.runge_kutta2(I, N, dp=7)
print(f'Solução Aproximada: y(0.6) = {y1}')
```

    f(x, y) =  2*x + y
    Solução Aproximada: y(0.6) = 0.431696

```python
# Questão 3
f = Lambda(x, sqrt(4 + x**2))
a = 0
b = 3
display(f)

Q1 = it(f, a, b)
Q1.simpson3_8(n=6)
```

$\displaystyle \left( x \mapsto \sqrt{x^{2} + 4} \right)$

    *************** Regra de 3/8 de Simpson ***************
    Portanto, o valor da integral é:
    7.7978, com |E| < 0.0009

    7.7978

```python
# Questão 4
f = Lambda(x, x**3 * exp(2*x))
a = 0
b = 2
display(f)

Q1 = it(f, a, b)
Q1.simpson1_3(n=8)
```

$\displaystyle \left( x \mapsto x^{3} e^{2 x} \right)$

    *************** Regra de 1/3 de Simpson ***************
    Portanto, o valor da integral é:
    116.7145, com |E| < 2.0095

    ********************  AVISO  ********************
    Para garantia de um |E| < 0.01, utilize n = 32.0

    116.7145