# Variational Quantum Eigensolvers

# ANALYSING TRADE-OFFS IN SYMMETRY-PRESERVING ANSATZ CIRCUITS FOR THE SCALABLE VARIATIONAL QUANTUM EIGENSOLVER ALGORITHM.

VQE - hybrid classical-quantum algorithm that can find the ground state energy of various quantum systems.

Has applications in quantum chemistry, where resources to classically simulate molecular wavefunctions increase exponentially with size of molecule.

True speedups in computational chemistry are still far off and dependant on development of large fault-tolerant quantum computers, VQEs can already be used to determine ground state energy of smaller molecules to a high degree of accuracy

$$\newcommand{\ket}[1]{\left|{#1}\right\rangle}$$
$$\newcommand{\bra}[1]{\left\langle{#1}\right|}$$
$$\newcommand{\braket}[2]{\left\langle{#1}\middle|{#2}\right\rangle}$$

```python
from typing import Optional, Union, Callable, cast
import matplotlib.pyplot as plt
import numpy as np

from qiskit import IBMQ, BasicAer, Aer
from qiskit.providers.aer import StatevectorSimulator
from qiskit.utils import QuantumInstance

from qiskit_nature.results import ElectronicStructureResult

from qiskit_nature.drivers import UnitsType, Molecule, QMolecule
from qiskit_nature.drivers.second_quantization import PySCFDriver, ElectronicStructureDriverType, ElectronicStructureMoleculeDriver
from qiskit_nature.problems.second_quantization import ElectronicStructureProblem
from qiskit_nature.converters.second_quantization import QubitConverter
from qiskit_nature.mappers.second_quantization import ParityMapper, JordanWignerMapper

from qiskit_nature.circuit.library import HartreeFock, UCC, UCCSD

from qiskit.opflow.primitive_ops import Z2Symmetries

from qiskit_nature.algorithms import GroundStateEigensolver
from qiskit_nature.algorithms import VQEUCCFactory
from qiskit.algorithms import NumPyMinimumEigensolver, VQE
from qiskit.algorithms.optimizers import SLSQP, COBYLA

```

### Overview

System is represented by a complex-valued wavefunction, and energy is associated with Hamiltonian $\mathbb{H}$, a matrix with eigenvalues that represent possible energies of the system.

System's groudn state is eigenstate of the Hamiltonian with lowest eigenvalue.

$\mathcal{H}\ket{\psi_g} = E_g\ket{\psi_g}$

$E_g = \frac{ \bra{\psi_g}|\mathcal{H}|\ket{\psi_g} }{ \braket{\psi_g}{\psi_g} }$

$E_g \leq \frac{ \bra{\psi}|\mathcal{H}|\ket{\psi} }{ \braket{\psi}{\psi} }$

VQE leverages this inequality by parameterizing quantum state used to evaluate energy expectation, and varationally updating these parameters on a classical computer to find tight upper bound to the ground state.

### Algorithm

* Parameterized quantum circuit prepares trialground state using some initial parameters values
* A quantum measurement circuit estimates energy expectation in this trial state.
* Classical optimization algorithm updates parameters to idealy decrease energy expectation of the next trail state.
* Algorithm is iterates until energy expectation converges to some value, which is taken as the approx ground state energy.

### Challenges

* Number of parameters required t prepare arbitrary quantum states increases problematically with system size, challenging the classical optimizer.

### Solutions

* reduce number of parameters, hence reduce algorithmic search space.

### Classical Optimizers

Responsible for determining parameter updates, thus for generating the new variational trial states.

* Gradient Descent - identifying energy expectation as const function to be minimized, but has tendency of being stuch in local minima and has relatively high number of required quantum circuit evaluations. (poor optimizer for VQE).

Popular optimizers for noise-free models i.e statevector simulations on classical computers. .e.g:
* Nelder-Mead method
* Sequential Least Squares Programming (SLSQP)
* Constrained Optimization by Linear Optimization (COBYLA) - being more efficient at the expense of fewer(just one) energy expectation evaluations per iteration.

VQE on physical quantum coputers incorporating noise models often involves more nuanced optimization schemes.

### Hamiltonian mapping

How wavefunction of a molecule can be mapped onto qubit states of a quantum computer.

General Hamiltonian for a quantum molecular system: (very intuitive)

$$\mathcal{\hat{H}} = - \sum_i \frac{\nabla^2_i}{2} - \sum_{i, I} \frac{Z_I}{|r_i - R_I|} + \sum_{i\neq j} \frac{1}{2|r_i - r_j|}$$

where: H = kinetic energy - electron-nucleus interaction + electron-electron interaction

Where:

$i$ and $j$ denote electrons,

$I$ denotes nuclei,

$Z$ is the atomic number of nuclei $I$,

$r$ is position of the electron $i$, $j$,

$R$ is position of each nucleus, which is fixed in space in accordance with the Bohr-Oppenheimer approximation.

The $\textbf{second quantization}$ Hamiltonian in which electrons are treated as excitations in an electron field, will ultimately prove more useful in our effort to encode probelm onto quantum copmuter:

$\begin{align}
\mathcal{\hat{H}} = \sum_{p,q} h_{pq}a^\dagger_p a_q + \frac{1}{2}\sum_{p,q,r,s}h_{pqrs}a^\dagger_p a^\dagger_q a_r a_s
\end{align}$

The $a^\dagger$ operators (creation operator) in each term "excite" electron to orbitals $p$ and $(p, q)$ respectively. The $a$ operator (annihilation operator) in each term $de-excites$ electron from orbitals $q$ and $(r, s)$ respectively.

The $h$ terms are coefficients known as the one an two-electron integrals and can be calculated relatively eficciently  in terms of a small set of orbital basis states on a classical computer.

We will restrict ourselves to single- and double-electron excitations to obtain a workable approximation of the more complicated true physical system.

There are 3 common mappings that can be used to produce a Hamiltonian for distinguishable fermions i.e. qubits from a Hmailtonian for indistinguishable fermions i.e. electrons:
* Jordan-Wigner mapping
* Parity mapping
* Bravyi-Kitaev mapping

All the mappings produce a Hamitonian of the form:

$\begin{equation}
\mathcal{\hat{H}} = \sum_j \alpha_j (\prod_i \hat{\sigma^j_i})
\end{equation}$

This is to say the Hamiltonian will be a linear combination of products of Pauli matrices (with $i$ denoting qubit being acted upon) which can be executed on a quantum computer.

Given a trial state, the energy expectation may be evaluated by measuring this superposition of Pauli operators, as below:

$E(\vec{\theta}) = \sum^N_j \alpha_j \bra{\psi(\vec{\theta})} \prod_i \sigma_i^j \ket{\psi(\vec{\theta})}$

### Parameterized Wavefunction Ansatz

Among most common wavefunction ansatz is the Unitary Coupled-CLuster Single and Double excitation (UCCSD) ansatz, used in foundational VQE paper by Perruzo et al.

UCCSD is constructed by applying an exponentiated single and double-electron excitation operator to an initial state, commonly chossen to be the Hartree-Fock mean-field wavefunction (an unentangled state that decently approximates the ground state):

$
\begin{align}
\ket{\psi(\theta)} = e^({\vec{T} - \vec{T}^\dagger}) \ket{\phi}\\\\
\hat{T} = \sum_{i\in virt, j\in occ} t^j_i \hat{a}_i^\dagger + \sum_{i, j \in virt, k, l \in occ} t^{kl}_{ij} \hat{a}_i^\dagger \hat{a}_j^\dagger \hat{a}_k \hat{a}_l
\end{align}
$

Where:

"virt" denotes unoccupied orbitals,

"occ" denotes occupied orbitals,

$a\dagger$ creation operators excite electrons and $a$ annihilation operators de-excite electrons to/from orbiatls,

t coefficients are the tunable parameters that are fed into classical optimizer.

The $T$ operator is then converted via one of the 3 mappings into as effective Hamiltoniann operator on qubits, which may be subsequently executed on a quantum copmuter.

### Chemistry-Inspired Ansatz

* UCCSD - UCC wit Singles and Doubles
* UpCCD - Unitary Pair UCCD
* OO-UCC - Orbital Optimized UCC
* DUCC - Double UCC

### Hardware Efficient Ansatz

* Symmetry Preserving State Preparation
* Qubit Coupled Cluster Method

### Energy

The hartree (symbol: Eh or Ha), also known as the Hartree energy, is the unit of energy in the Hartree atomic units system, named after the British physicist Douglas Hartree. Its CODATA recommended value is Eh = 4.3597447222071(85)×10−18 J[1] = 27.211386245988(53) eV.[2]

The hartree energy is approximately the electric potential energy of the hydrogen atom in its ground state and, by the virial theorem, approximately twice its ionization energy; the relationships are not exact because of the finite mass of the nucleus of the hydrogen atom and relativistic corrections.

The hartree is usually used as a unit of energy in atomic physics and computational chemistry: for experimental measurements at the atomic scale, the electronvolt (eV) or the reciprocal centimetre (cm−1) are much more widely used.

(Courtesy: Wikipedia: https://en.wikipedia.org/wiki/Hartree)

### Building a VQE in Qiskit

VQE in Python using Qiskit, as open-source SDK for working with quantum computers, to find ground state energy and corresponding interatomic spearaton for several small molecules: diatomic hydrogen ($H_2$), lithium hydride ($LiH$), ionic helium hydride ($HeH+$) and hydroxide ($OH-$)

---
* First the molecule structure was specified as a string in xyz coordinates, lithium hydroxide woudl specify as "Li 0.0 0.0 0.0; H 0.0 0.0 d" where d is the interatomic distance.

For two-atom molecules, the z-coordinate of the second atom was varied to determine ground state energy as a function of interatomic distance.

* A PySCF driver was initialized with this molecule specification, creating a data structure representing the molecule along with several useful calculated attributes, including the nuclear repulsion energy - a quantity later added to the VQE-determined electron energy to obtain the total molecular energy.

The PySCF molecular data structure was then provided as input to the ElectronicStructureProblem class in Qiskit-Nature module. This class determines the Hamiltoninan of the molecule in terms of second-quantization operators, calculating the one- and two-electron integral coefficients discussed previously and returning the electronic operator in the form:

Where:

"+" is the excitation operator

"-" is the de-excitation operator

"I" is the identity operator

"N" is the number operator (de-excitation followed by excitation)

This operator was then converted into a qubit operator using the class qiskit_nature.converters.second_quantization,QubitConverter, with the mapping type set to the Jordan-Wigner transformation, yielding a summation of Pauli operator products:

Note that the number of qubits needed to execute algorithm s equal to number of second quantization operators, which is defined as the number of molecular spin orbitals considered.

* Next,the Hartree-Fock initial state preparation circuit and the UCCSD variational unitary transformation circuit. (with HF as its initial state) were retrieved from a library of optimized circuits in the qiskit_nature.circuits.library module, being sure to pass the Jordan-Wigner QubitConverter object created earlier as an argument to each circuit for consistency.

Finally, these were all supplied to the VQE class in the module qiskit.algorithms, which simulated the UCCSD variational circuit repeatedly on a Qiskit Aer statevector backend to solve for the minimum eigenvalue of the qubit Hamiltonina, using the SLSQP classical optimizer to calculate parameter updates.

This algorithm was repeated for intermolecular separations ranging between 0.2 and 2 Angstroms (~0.1 nanometres), storing the minimum energy as determined by the VQE at each iteration.

```python
state_sim = StatevectorSimulator()

from qiskit_nature.transformers.second_quantization.electronic import FreezeCoreTransformer

from qiskit_nature.algorithms import ExcitedStatesEigensolver, VQEUCCFactory

# vqe_energies = []

class MolecularVQE:
    def __init__(self):
        # H2
        self.molecule_name = "H 0.0 0.0 0.0; H 0.0 0.0 "
        self.backend = QuantumInstance(state_sim)
        self.optimizer = SLSQP(maxiter=500)
        self.vqe = VQE(
            ansatz = None,
            quantum_instance = self.backend,
            optimizer = self.optimizer
        )

    def get_qubit_op(self, dist, mapper="parity"):
        # Use PySCF, a classical computational chemistry software
        # package, to compute the on-body and two-body integrals in
        # electronic-orbital basis, necessary to form the Fermionic operator

        driver = PySCFDriver(
            atom = self.molecule_name + str(dist),
            # unit = UnitsType.ANGSTROM,
            # charge = 0,
            # spin = 0,
            # basis = "sto3g"
        )

        #molecule = Molecule(
        #    geometry=[
        #        ["H", [0.0, 0.0, 0.0]],
        #        ["H", [0.0, 0.0, 0.735]]
        #    ],
        #    charge=0,
        #    multiplicity=1
        #)

        #driver = ElectronicStructureMoleculeDriver(
        #    basis = "sto3g",
        #    driver_type=ElectronicStructureDriverType.PYSCF
        #)

        qmolecule = driver.run()    # returns ElectronicStructureDriverResult
        transformer = FreezeCoreTransformer()
        qmolecule = transformer.transform(qmolecule)

        es_problem = ElectronicStructureProblem(driver)

        if mapper == "jw":
            qubit_converter = QubitConverter(mapper=JordanWignerMapper())
        elif mapper == "parity":
            qubit_converter = QubitConverter(mapper=ParityMapper(), two_qubit_reduction=True)

        return (es_problem, qubit_converter)

    def run(self):
        numpy_solver = NumPyMinimumEigensolver()

        # distances = np.arange(0.3, 2.0, 0.05)
        distances = np.arange(0.65, 1.0, 0.05)
        exact_energies = []
        vqe_energies = []

        n = len(distances)
        i = 1

        for dist in distances:
            print("Distance {}/{}".format(i, n))
            i += 1

            es_problem, qubit_converter = self.get_qubit_op(dist)
            second_q_ops = es_problem.second_q_ops()

            # Hamiltonian
            main_op = second_q_ops[0]

            #mine z2sym = Z2Symmetries.find_Z2_symmetries(second_q_ops)
            qubit_op = qubit_converter.convert(
                main_op,
                num_particles = es_problem.num_particles,
                #sector_locator = es_problem.symmetry_sector_locator(None, qubit_converter)
            )

            # aux_ops = qubit_converter.convert_match(
            #     second_q_ops[1:]
            # )

            # q_molecule_transformed = cast(QMolecule, es_problem.molecule_data_transformed)
            # num_molecular_orbitals = q_molecule_transformed.num_molecular_orbitals
            # num_particles = (q_molecule_transformed.num_alpha, q_molecule_transformed.num_beta)
            # num_spin_orbitals = 2 * num_molecular_orbitals

            num_particles = es_problem.num_particles
            num_spin_orbitals = es_problem.num_spin_orbitals

            # initial state is Hartree-Fock state
            initial_state = HartreeFock(num_spin_orbitals, num_particles, qubit_converter)

            # UCCSD ansatz for unitary update
            ansatz = UCCSD()
            ansatz.qubit_converter = qubit_converter
            ansatz.num_particles = num_particles
            ansatz.num_spin_orbitals = num_spin_orbitals
            ansatz.initial_state = initial_state

            self.vqe.ansatz = ansatz
            solver = self.vqe

            # solver = VQEUCCFactory(
            #     quantum_instance=self.backend,
            #     optimizer=self.optimizer,
            #     ansatz=ansatz,
            #     initial_state=initial_state
            # )

            uccsd_excited_states_calculation = ExcitedStatesEigensolver(qubit_converter, solver)
            print(uccsd_excited_states_calculation)
            # uccsd_ground_excited_states_properties = uccsd_excited_states_calculation.solve(es_problem)

            print("Computing the minimum eigenvalue...")

            # Approximate minimum eigensolver using VQE
            vqe_result = solver.compute_minimum_eigenvalue(qubit_op)
            print(vqe_result)
            # print(np.real(vqe_result.eigenvalue) + nuclear_repulsion_energy)

            # vqe_energies.append(np.real(vqe_result.eigenvalue) + nuclear_repulsion_energy)

            vqe_energies.append(np.real(vqe_result.eigenvalue))

        return (distances, vqe_energies, exact_energies)
        # return self.vqe.ansatz
```

### Results

Demonstrate close correspondence between experimentally determined ground state energies and the ground state energies determined using VQEs. The ground state interatmoic distance was dtermined by finding the minimum on the plot of VQE ground state energy against interatomic distance.

Find a theoretical curve of ground stte energy against interatomic distance - determined by diagonalizing the molecule Hamiltonian directly using NumpyMinimumEigensolver - check if it coincides with VQE curve.

```python
vqe = MolecularVQE()
res = vqe.run()

# res.draw("mpl")

```

    Distance 1/7
    <qiskit_nature.algorithms.excited_states_solvers.excited_states_eigensolver.ExcitedStatesEigensolver object at 0x7f71584dc580>
    Computing the minimum eigenvalue...
    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 13,
        'eigenstate': array([-1.06023981e-08-9.02056208e-17j,  9.95433931e-01+5.85952419e-16j,
           -9.54530748e-02-1.51164432e-16j, -3.37124193e-08-5.55111512e-17j]),
        'eigenvalue': (-1.9440235703061848+0j),
        'optimal_parameters': {   ParameterVectorElement(t[1]): 3.371241934912606e-08,
                                  ParameterVectorElement(t[2]): 0.09559862244051313,
                                  ParameterVectorElement(t[0]): 1.060239817902326e-08},
        'optimal_point': array([1.06023982e-08, 3.37124193e-08, 9.55986224e-02]),
        'optimal_value': -1.9440235703061848,
        'optimizer_evals': None,
        'optimizer_time': 0.23056364059448242}
    Distance 2/7
    <qiskit_nature.algorithms.excited_states_solvers.excited_states_eigensolver.ExcitedStatesEigensolver object at 0x7f7159220970>
    Computing the minimum eigenvalue...
    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 13,
        'eigenstate': array([-1.47916226e-08-1.73472348e-16j,  9.94507056e-01+1.88466006e-16j,
           -1.04669551e-01-3.28431483e-17j,  5.84318919e-09-1.66533454e-16j]),
        'eigenvalue': (-1.8921568981821482+0j),
        'optimal_parameters': {   ParameterVectorElement(t[0]): 1.4791622764614418e-08,
                                  ParameterVectorElement(t[2]): 0.10486162098941848,
                                  ParameterVectorElement(t[1]): -5.843189119363774e-09},
        'optimal_point': array([ 1.47916228e-08, -5.84318912e-09,  1.04861621e-01]),
        'optimal_value': -1.8921568981821482,
        'optimizer_evals': None,
        'optimizer_time': 0.10698914527893066}
    Distance 3/7
    <qiskit_nature.algorithms.excited_states_solvers.excited_states_eigensolver.ExcitedStatesEigensolver object at 0x7f7159387940>
    Computing the minimum eigenvalue...
    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 10,
        'eigenstate': array([ 4.79615422e-09-2.77555756e-17j,  9.93414622e-01+4.12029416e-17j,
           -1.14574816e-01+7.01568604e-18j,  3.96386964e-08-5.55111512e-17j]),
        'eigenvalue': (-1.8426866818468055+0j),
        'optimal_parameters': {   ParameterVectorElement(t[2]): 0.11482698684986334,
                                  ParameterVectorElement(t[1]): -3.9638696375370774e-08,
                                  ParameterVectorElement(t[0]): -4.796154203779538e-09},
        'optimal_point': array([-4.79615420e-09, -3.96386964e-08,  1.14826987e-01]),
        'optimal_value': -1.8426866818468055,
        'optimizer_evals': None,
        'optimizer_time': 0.09701395034790039}
    Distance 4/7
    <qiskit_nature.algorithms.excited_states_solvers.excited_states_eigensolver.ExcitedStatesEigensolver object at 0x7f7159395a90>
    Computing the minimum eigenvalue...
    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 10,
        'eigenstate': array([ 5.13520705e-09-1.24900090e-16j,  9.92133184e-01+4.72899907e-16j,
           -1.25186839e-01-7.08391401e-17j,  1.78507571e-16-5.55111512e-17j]),
        'eigenvalue': (-1.7956191802702903+0j),
        'optimal_parameters': {   ParameterVectorElement(t[0]): -5.135206700088415e-09,
                                  ParameterVectorElement(t[1]): 0.0,
                                  ParameterVectorElement(t[2]): 0.12551614970340444},
        'optimal_point': array([-5.1352067e-09,  0.0000000e+00,  1.2551615e-01]),
        'optimal_value': -1.7956191802702903,
        'optimizer_evals': None,
        'optimizer_time': 0.14327788352966309}
    Distance 5/7
    <qiskit_nature.algorithms.excited_states_solvers.excited_states_eigensolver.ExcitedStatesEigensolver object at 0x7f71e8de7550>
    Computing the minimum eigenvalue...
    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 9,
        'eigenstate': array([-7.01156592e-17-2.77555756e-17j,  9.90570747e-01-2.33737371e-16j,
           -1.37002175e-01+8.38896345e-18j, -5.45935204e-09-1.11022302e-16j]),
        'eigenvalue': (-1.7509229979339356+0j),
        'optimal_parameters': {   ParameterVectorElement(t[0]): 0.0,
                                  ParameterVectorElement(t[1]): 5.459352019457762e-09,
                                  ParameterVectorElement(t[2]): 0.13743441561618416},
        'optimal_point': array([0.00000000e+00, 5.45935202e-09, 1.37434416e-01]),
        'optimal_value': -1.7509229979339356,
        'optimizer_evals': None,
        'optimizer_time': 0.08663034439086914}
    Distance 6/7
    <qiskit_nature.algorithms.excited_states_solvers.excited_states_eigensolver.ExcitedStatesEigensolver object at 0x7f71593ac550>
    Computing the minimum eigenvalue...
    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 9,
        'eigenstate': array([ 9.12272917e-18-8.32667268e-17j,  9.88839385e-01-2.73095698e-16j,
           -1.48985474e-01+9.12272845e-18j, -1.16929881e-08-5.55111512e-17j]),
        'eigenvalue': (-1.708534808574801+0j),
        'optimal_parameters': {   ParameterVectorElement(t[2]): 0.14954221632965023,
                                  ParameterVectorElement(t[1]): 1.1692988179978664e-08,
                                  ParameterVectorElement(t[0]): 0.0},
        'optimal_point': array([0.00000000e+00, 1.16929882e-08, 1.49542216e-01]),
        'optimal_value': -1.708534808574801,
        'optimizer_evals': None,
        'optimizer_time': 0.07053995132446289}
    Distance 7/7
    <qiskit_nature.algorithms.excited_states_solvers.excited_states_eigensolver.ExcitedStatesEigensolver object at 0x7f715937b7f0>
    Computing the minimum eigenvalue...
    {   'aux_operator_eigenvalues': None,
        'cost_function_evals': 9,
        'eigenstate': array([ 1.25030027e-08-1.38777878e-17j,  9.86828383e-01+2.60800385e-17j,
           -1.61770650e-01-6.85990279e-17j,  9.47715835e-17-1.66533454e-16j]),
        'eigenvalue': (-1.6683680051111445+0j),
        'optimal_parameters': {   ParameterVectorElement(t[1]): 0.0,
                                  ParameterVectorElement(t[0]): -1.2503002380773462e-08,
                                  ParameterVectorElement(t[2]): 0.1624846741472046},
        'optimal_point': array([-1.25030024e-08,  0.00000000e+00,  1.62484674e-01]),
        'optimal_value': -1.6683680051111445,
        'optimizer_evals': None,
        'optimizer_time': 0.10727286338806152}

```python
plt.plot(res[0], res[1], c="r", label="VQE")

# plt.rc("text", usetext=True)
plt.title(r"$H2$: Minimum energy vs. Interatomic distance")
plt.ylabel("Ground state energy (Hartree)")
plt.xlabel("Interatomic distance  (A)")
plt.legend()
# plt.show()

idx = res[1].index(min(res[1]))
dist, min_energy = res[0][res[1].index(min(res[1]))], min(res[1])
print("Min Distance: {}\n\n".format(dist))
print("Min Energy: {}".format(min_energy))
```

```python

```

```python

```


_Lambda School Data Science — Regression 1_

# Understanding Ordinary Least Squares

#### Objectives
- understand how ordinary least squares regression minimizes the sum of squared errors
- understand how linear algebra can solve ordinary least squares regression
- get and interpret coefficients of a linear model
- visualize a line of best fit in 2D, and hyperplane in 3D
- use regression metrics: MSE, RMSE, MAE, R^2

#### Extra Links
- [Statistics 101: Simple Linear Regression](https://www.youtube.com/watch?v=ZkjP5RJLQF4) (20 minute video)
- [_An Introduction to Statistical Learning_](http://www-bcf.usc.edu/~gareth/ISL/ISLR%20Seventh%20Printing.pdf), Chapter 3.1, Simple Linear Regression, & 3.2, Multiple Linear Regression
- Priceonomics, [The Discovery of Statistical Regression](https://priceonomics.com/the-discovery-of-statistical-regression/)
- Priceonomics, [Why the Father of Modern Statistics Didn’t Believe Smoking Caused Cancer](https://priceonomics.com/why-the-father-of-modern-statistics-didnt-believe/)
- Harvard Business Review, [When to Act on a Correlation, and When Not To](https://hbr.org/2014/03/when-to-act-on-a-correlation-and-when-not-to)
- [xkcd 552: Correlation](https://www.explainxkcd.com/wiki/index.php/552:_Correlation)
- [xkcd 1725: Linear Regression](https://www.explainxkcd.com/wiki/index.php/1725:_Linear_Regression)

## What is Linear Regression?

[Linear Regression](https://en.wikipedia.org/wiki/Linear_regression) is a statistical model that seeks to describe the relationship between some y variable and one or more x variables.

In the simplest case, linear regression seeks to fit a straight line through a cloud of points. This line is referred to as the "regression line" or "line of best fit." This line tries to summarize the relationship between our X and Y in a way that enables us to use the equation for that line to make predictions.

### Synonyms for "y variable"
- Dependent Variable
- Response Variable
- Outcome Variable
- Predicted Variable
- Measured Variable
- Explained Variable
- Label
- Target

### Synonyms for "x variable"
- Independent Variable
- Explanatory Variable
- Regressor
- Covariate
- Feature

## Simple Linear Regresion

#### Making Predictions

Say that we were trying to create a model that captured the relationship between temperature outside and ice cream sales. In Machine Learning our goal is often different that of other flavors of Linear Regression Analysis, because we're trying to fit a model to this data with the intention of making **predictions** on new data (in the future) that we don't have yet.

#### What are we trying to predict?

So if we had measured ice cream sales and the temprature outside on 11 different days, at the end of our modeling **what would be the thing that we would want to predict? - Ice Cream Sales or Temperature?**

We would probably want to be measuring temperature with the intention of using that to **forecast** ice cream sales. If we were able to successfully forecast ice cream sales from temperature, this might help us know beforehand how much ice cream to make or how many cones to buy or on which days to open our store, etc. Being able to make predictions accurately has a lot of business implications. This is why making accurate predictions is so valuable (And in large part is why data scientists are paid so well).

#### y variable intuition

We want the thing that we're trying to predict to serve as our **y** variable. This is why it's sometimes called the "predicted variable." We call it the "dependent" variable because our prediction for how much ice cream we're going to sell "depends" on the temperature outside.

#### x variable intuition

All other variables that we use to predict our y variable (we're going to start off just using one) we call our **x** variables. These are called our "independent" variables because they don't *depend* on y, they "explain" y. Hence they are also referred to as our "explanatory" variables.

## Example, reprised: Predict presidential election voting

#### Douglas Hibbs, [Background Information on the ‘Bread and Peace’ Model of Voting in Postwar US Presidential Elections](https://douglas-hibbs.com/background-information-on-bread-and-peace-voting-in-us-presidential-elections/)

> Aggregate two-party vote shares going to candidates of the party holding the presidency during the postwar era are well explained by just two fundamental determinants:

> (1) Positively by weighted-average growth of per capita real disposable personal income over the term.
> (2) Negatively by cumulative US military fatalities (scaled to population) owing to unprovoked, hostile deployments of American armed forces in foreign wars.

#### Data sources
- 1952-2012: Douglas Hibbs, [2014 lecture at Deakin University Melbourne](http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf), Slide 40
- 2016, Vote Share: [The American Presidency Project](https://www.presidency.ucsb.edu/statistics/elections)
- 2016, Recent Growth in Personal Incomes: [The 2016 election economy: the "Bread and Peace" model final forecast](https://angrybearblog.com/2016/11/the-2016-election-economy-the-bread-and-peace-model-final-forecast.html)
- 2016, US Military Fatalities: Assumption that Afghanistan War fatalities in 2012-16 occured at the same rate as 2008-12

```python
import pandas as pd

columns = ['Year','Incumbent Party Candidate','Other Candidate','Incumbent Party Vote Share']

data = [[1952,"Stevenson","Eisenhower",44.6],
        [1956,"Eisenhower","Stevenson",57.76],
        [1960,"Nixon","Kennedy",49.91],
        [1964,"Johnson","Goldwater",61.34],
        [1968,"Humphrey","Nixon",49.60],
        [1972,"Nixon","McGovern",61.79],
        [1976,"Ford","Carter",48.95],
        [1980,"Carter","Reagan",44.70],
        [1984,"Reagan","Mondale",59.17],
        [1988,"Bush, Sr.","Dukakis",53.94],
        [1992,"Bush, Sr.","Clinton",46.55],
        [1996,"Clinton","Dole",54.74],
        [2000,"Gore","Bush, Jr.",50.27],
        [2004,"Bush, Jr.","Kerry",51.24],
        [2008,"McCain","Obama",46.32],
        [2012,"Obama","Romney",52.00],
        [2016,"Clinton","Trump",48.2]]

votes = pd.DataFrame(data=data, columns=columns)
```

```python
columns = ['Year','Average Recent Growth in Personal Incomes']

data = [[1952,2.40],
        [1956,2.89],
        [1960, .85],
        [1964,4.21],
        [1968,3.02],
        [1972,3.62],
        [1976,1.08],
        [1980,-.39],
        [1984,3.86],
        [1988,2.27],
        [1992, .38],
        [1996,1.04],
        [2000,2.36],
        [2004,1.72],
        [2008, .10],
        [2012, .95],
        [2016, .10]]

growth = pd.DataFrame(data=data, columns=columns)
```

```python
"""
Fatalities denotes the cumulative number of American military
fatalities per millions of US population the in Korea, Vietnam,
Iraq and Afghanistan wars during the presidential terms
preceding the 1952, 1964, 1968, 1976 and 2004, 2008 and
2012 elections.

http://www.douglas-hibbs.com/HibbsArticles/HIBBS-PRESVOTE-SLIDES-MELBOURNE-Part1-2014-02-26.pdf
"""

columns = ['Year','US Military Fatalities per Million']

data = [[1952,190],
        [1956,  0],
        [1960,  0],
        [1964,  1],
        [1968,146],
        [1972,  0],
        [1976,  2],
        [1980,  0],
        [1984,  0],
        [1988,  0],
        [1992,  0],
        [1996,  0],
        [2000,  0],
        [2004,  4],
        [2008, 14],
        [2012,  5],
        [2016,  5]]

deaths = pd.DataFrame(data=data, columns=columns)
```

### Acquire new features

#### Shark attack data source
- https://www.sharkattackfile.net/incidentlog.htm (Download the Excel file manually, because web crawlers are blocked)

#### Economic data sources
- Unemployment: https://fred.stlouisfed.org/series/UNRATE
- GDP: https://fred.stlouisfed.org/series/GDPC1
- GDP change: https://fred.stlouisfed.org/series/A191RL1Q225SBEA

```python
from google.colab import files
files.upload()
```

<input type="file" id="files-f1cb8b7c-ab0d-4a58-9593-71c31adb1fa8" name="files[]" multiple disabled />
<output id="result-f1cb8b7c-ab0d-4a58-9593-71c31adb1fa8">
 Upload widget is only available when the cell has been executed in the
 current browser session. Please rerun this cell to enable.
 </output>

    {}

```python
sharks = pd.read_excel('GSAF5 (1).xls')[['Year', 'Country']]
sharks = (sharks
          .where(sharks.Country=='USA')
          .groupby('Year')
          .count()
          .reset_index()
          .rename(columns={'Country': 'Shark Attacks'}))
```

```python
url = 'https://fred.stlouisfed.org/graph/fredgraph.csv?bgcolor=%23e1e9f0&chart_type=line&drp=0&fo=open%20sans&graph_bgcolor=%23ffffff&height=450&mode=fred&recession_bars=on&txtcolor=%23444444&ts=12&tts=12&width=1168&nt=0&thu=0&trc=0&show_legend=yes&show_axis_titles=yes&show_tooltip=yes&id=UNRATE&scale=left&cosd=1948-01-01&coed=2019-04-01&line_color=%234572a7&link_values=false&line_style=solid&mark_type=none&mw=3&lw=2&ost=-99999&oet=99999&mma=0&fml=a&fq=Monthly&fam=avg&fgst=lin&fgsnd=2009-06-01&line_index=1&transformation=lin&vintage_date=2019-05-30&revision_date=2019-05-30&nd=1948-01-01'
unemployment = pd.read_csv(url, parse_dates=['DATE'])

# Annual average unemployment, only using the first 10 months of the year
# (because presidential elections are in November)
unemployment = (unemployment
                .where(unemployment.DATE.dt.month <= 10)
                .set_index('DATE')
                .resample('A')
                .mean()
                .reset_index()
                .rename(columns={'DATE': 'Year', 'UNRATE': 'Unemployment Rate'}))

unemployment['Year'] = unemployment['Year'].dt.year
```

```python
url = 'https://fred.stlouisfed.org/graph/fredgraph.csv?bgcolor=%23e1e9f0&chart_type=line&drp=0&fo=open%20sans&graph_bgcolor=%23ffffff&height=450&mode=fred&recession_bars=on&txtcolor=%23444444&ts=12&tts=12&width=1168&nt=0&thu=0&trc=0&show_legend=yes&show_axis_titles=yes&show_tooltip=yes&id=GDPC1&scale=left&cosd=1947-01-01&coed=2019-01-01&line_color=%234572a7&link_values=false&line_style=solid&mark_type=none&mw=3&lw=2&ost=-99999&oet=99999&mma=0&fml=a&fq=Quarterly&fam=avg&fgst=lin&fgsnd=2009-06-01&line_index=1&transformation=lin&vintage_date=2019-05-30&revision_date=2019-05-30&nd=1947-01-01'

gdp = pd.read_csv(url, parse_dates=['DATE'])
gdp = (gdp
       .where(gdp.DATE.dt.month==7)
       .rename(columns={'DATE': 'Year', 'GDPC1': 'GDP Q3'}))

gdp['Year'] = gdp['Year'].dt.year
```

```python
url = 'https://fred.stlouisfed.org/graph/fredgraph.csv?bgcolor=%23e1e9f0&chart_type=line&drp=0&fo=open%20sans&graph_bgcolor=%23ffffff&height=450&mode=fred&recession_bars=on&txtcolor=%23444444&ts=12&tts=12&width=1168&nt=0&thu=0&trc=0&show_legend=yes&show_axis_titles=yes&show_tooltip=yes&id=A191RL1Q225SBEA&scale=left&cosd=1947-04-01&coed=2019-01-01&line_color=%234572a7&link_values=false&line_style=solid&mark_type=none&mw=3&lw=2&ost=-99999&oet=99999&mma=0&fml=a&fq=Quarterly&fam=avg&fgst=lin&fgsnd=2009-06-01&line_index=1&transformation=lin&vintage_date=2019-05-30&revision_date=2019-05-30&nd=1947-04-01'

gdp_change = pd.read_csv(url, parse_dates=['DATE'])
gdp_change = (gdp_change
              .where(gdp_change.DATE.dt.month==7)
              .rename(columns={'DATE': 'Year', 'A191RL1Q225SBEA': 'GDP Change Q3'}))

gdp_change['Year'] = gdp_change['Year'].dt.year
```

### Merge data

```python
df = (votes
      .merge(growth)
      .merge(deaths)
      .merge(sharks)
      .merge(unemployment)
      .merge(gdp)
      .merge(gdp_change))
```

### Engineer new feature

```python
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million
      Shark Attacks
      Unemployment Rate
      GDP Q3
      GDP Change Q3

      0
      1952
      Stevenson
      Eisenhower
      44.60
      2.40
      190
      11
      3.08
      2564.401
      2.9

      1
      1956
      Eisenhower
      Stevenson
      57.76
      2.89
      0
      9
      4.10
      2925.035
      -0.4

      2
      1960
      Nixon
      Kennedy
      49.91
      0.85
      0
      25
      5.38
      3274.029
      2.0

      3
      1964
      Johnson
      Goldwater
      61.34
      4.21
      1
      14
      5.21
      3954.121
      6.4

      4
      1968
      Humphrey
      Nixon
      49.60
      3.02
      146
      15
      3.59
      4825.799
      3.1

      5
      1972
      Nixon
      McGovern
      61.79
      3.62
      0
      9
      5.67
      5415.712
      3.8

      6
      1976
      Ford
      Carter
      48.95
      1.08
      2
      18
      7.68
      5965.265
      2.2

      7
      1980
      Carter
      Reagan
      44.70
      -0.39
      0
      10
      7.14
      6688.794
      -0.5

      8
      1984
      Reagan
      Mondale
      59.17
      3.86
      0
      21
      7.56
      7686.059
      3.9

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      2.27
      0
      26
      5.53
      8891.435
      2.4

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      0.38
      0
      24
      7.51
      9732.979
      4.0

      11
      1996
      Clinton
      Dole
      54.74
      1.04
      0
      27
      5.41
      11096.976
      3.6

      12
      2000
      Gore
      Bush, Jr.
      50.27
      2.36
      0
      52
      3.98
      13178.419
      0.5

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      1.72
      4
      35
      5.57
      14464.984
      3.8

      14
      2008
      McCain
      Obama
      46.32
      0.10
      14
      60
      5.55
      15667.032
      -2.1

      15
      2012
      Obama
      Romney
      52.00
      0.95
      5
      64
      8.13
      16220.667
      0.5

      16
      2016
      Clinton
      Trump
      48.20
      0.10
      5
      65
      4.91
      17706.705
      1.9

```python
# True Incumbent =
# The Incumbent Party Candidate this election is the same as the Incumbent Party Candidate 1 election ago,
# OR, the Incumbent Party Candidate this election is the same as the Other Candidate 1 election ago
df['True Incumbent'] = ((df['Incumbent Party Candidate'] == df['Incumbent Party Candidate'].shift(1)) |
                        (df['Incumbent Party Candidate'] == df['Other Candidate'].shift(1)))

# Change the data type of this feature from boolean (True/False) to integer (1/0)
df['True Incumbent'] = df['True Incumbent'].astype(int)
```

```python
df.groupby('True Incumbent')['Incumbent Party Vote Share'].mean()
```

    True Incumbent
    0    50.347778
    1    53.493750
    Name: Incumbent Party Vote Share, dtype: float64

```python
df
```

      Year
      Incumbent Party Candidate
      Other Candidate
      Incumbent Party Vote Share
      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million
      Shark Attacks
      Unemployment Rate
      GDP Q3
      GDP Change Q3
      True Incumbent

      0
      1952
      Stevenson
      Eisenhower
      44.60
      2.40
      190
      11
      3.08
      2564.401
      2.9
      0

      1
      1956
      Eisenhower
      Stevenson
      57.76
      2.89
      0
      9
      4.10
      2925.035
      -0.4
      1

      2
      1960
      Nixon
      Kennedy
      49.91
      0.85
      0
      25
      5.38
      3274.029
      2.0
      0

      3
      1964
      Johnson
      Goldwater
      61.34
      4.21
      1
      14
      5.21
      3954.121
      6.4
      0

      4
      1968
      Humphrey
      Nixon
      49.60
      3.02
      146
      15
      3.59
      4825.799
      3.1
      0

      5
      1972
      Nixon
      McGovern
      61.79
      3.62
      0
      9
      5.67
      5415.712
      3.8
      1

      6
      1976
      Ford
      Carter
      48.95
      1.08
      2
      18
      7.68
      5965.265
      2.2
      0

      7
      1980
      Carter
      Reagan
      44.70
      -0.39
      0
      10
      7.14
      6688.794
      -0.5
      1

      8
      1984
      Reagan
      Mondale
      59.17
      3.86
      0
      21
      7.56
      7686.059
      3.9
      1

      9
      1988
      Bush, Sr.
      Dukakis
      53.94
      2.27
      0
      26
      5.53
      8891.435
      2.4
      0

      10
      1992
      Bush, Sr.
      Clinton
      46.55
      0.38
      0
      24
      7.51
      9732.979
      4.0
      1

      11
      1996
      Clinton
      Dole
      54.74
      1.04
      0
      27
      5.41
      11096.976
      3.6
      1

      12
      2000
      Gore
      Bush, Jr.
      50.27
      2.36
      0
      52
      3.98
      13178.419
      0.5
      0

      13
      2004
      Bush, Jr.
      Kerry
      51.24
      1.72
      4
      35
      5.57
      14464.984
      3.8
      1

      14
      2008
      McCain
      Obama
      46.32
      0.10
      14
      60
      5.55
      15667.032
      -2.1
      0

      15
      2012
      Obama
      Romney
      52.00
      0.95
      5
      64
      8.13
      16220.667
      0.5
      1

      16
      2016
      Clinton
      Trump
      48.20
      0.10
      5
      65
      4.91
      17706.705
      1.9
      0

### Plot univariate correlations
[Seaborn tutorial: Visualizing linear relationships](https://seaborn.pydata.org/tutorial/regression.html)

```python
%matplotlib inline
import seaborn as sns

target = 'Incumbent Party Vote Share'
features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million',
            'Shark Attacks',
            'Unemployment Rate',
            'GDP Q3',
            'GDP Change Q3',
            'True Incumbent']

for feature in features:
    sns.lmplot(x=feature, y=target, data=df)
```

We can see from the scatterplot that these data points seem to follow a somewhat linear relationship for the "Average Recent Growth in Personal Incomes" feature. This means that we could probably summarize their relationship well by fitting a line of best fit to these points. Lets do it.

## The Equation for a Line

A common equation for a line is:

\begin{align}
y = mx + b
\end{align}

Where $m$ is the slope of our line and $b$ is the y-intercept.

If we want to plot a line through our cloud of points we figure out what these two values should be. Linear Regression seeks to **estimate** the slope and intercept values that describe a line that best fits the data points.

## The Anatomy of Linear Regression

- Intercept: The $b$ value in our line equation $y=mx+b$
- Slope: The $m$ value in our line equation $y=mx+b$. These two values together define our regression line.

- $\hat{y}$ : A prediction
- Line of Best Fit (Regression Line)
- Predicted (fitted) Values: Points on our regression line
- Observed Values: Points from our dataset
- Error: The distance between predicted and observed values.

```python
x = df['Average Recent Growth in Personal Incomes']
y = df['Incumbent Party Vote Share']

m = 0
b = y.mean()
y_pred = m*x + b
```

```python
from sklearn.metrics import mean_absolute_error, r2_score
import matplotlib.pyplot as plt

def plot_preds(x, y, y_pred):
    plt.scatter(x, y, label='y_true')
    plt.plot(x, y_pred, label='y_pred')
    plt.legend()

    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

plot_preds(x, y, y_pred)
plt.title('Mean Baseline');
```

```python
m = 4
b = 45
y_pred = m*x + b
plot_preds(x, y, y_pred)
plt.title('Guessing & Checking Manually');
```

## R Squared:  $R^2$

One final attribute of linear regressions that we're going to talk about today is a measure of goodness of fit known as $R^2$ or R-squared. $R^2$ is a statistical measure of how close the data are fitted to our regression line. A helpful interpretation for the $R^2$ is the percentage of the dependent variable that is explained by the model.

In other words, the $R^2$ is the percentage of y that is explained by the x variables included in the model. For this reason the $R^2$ is also known as the "coefficient of determination," because it explains how much of y is explained (or determined) by our x varaibles. We won't go into the calculation of $R^2$ today, just know that a higher $R^2$ percentage is nearly always better and indicates a model that fits the data more closely.

## Residual Error

The residual error is the distance between points in our dataset and our regression line.

```python
def regression_residuals(df, feature, target, m, b):
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    plt.scatter(x, y, label='y_true')
    plt.plot(x, y_pred, label='y_pred')
    plt.legend()

    # Plot residual errors
    for x, y1, y2 in zip(x, y, y_pred):
        plt.plot((x, x), (y1, y2), color='grey')

    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

regression_residuals(df, feature='Average Recent Growth in Personal Incomes',
                     target='Incumbent Party Vote Share', m=3, b=46)
```

```python
from ipywidgets import interact, fixed

interact(regression_residuals,
         df=fixed(df),
         feature=fixed('Average Recent Growth in Personal Incomes'),
         target=fixed('Incumbent Party Vote Share'),
         m=(-10,10,0.5),
         b=(40,60,0.5));
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=10.0, min=-10.0, step=0.5), FloatSlider(valu…

## More Formal Notation

We have talked about a line of regression being represented like a regular line $y=mx+b$ but as we get to more complicated versions we're going to need to extend this equation. So lets establish the proper terminology.

**X** - Independent Variable, predictor variable, explanatory variable, regressor, covariate

**Y** - Response variable, predicted variable, measured vairable, explained variable, outcome variable

$\beta_0$ - "Beta Naught" or "Beta Zero", the intercept value. This is how much of y would exist if X were zero. This is sometimes represented by the letter "a" but I hate that. So it's "Beta 0" during my lecture.

$\beta_1$ - "Beta One" The primary coefficient of interest. This values is the slope of the line that is estimated by "minimizing the sum of the squared errors/residuals" - We'll get to that.

$\epsilon$ - "Epsilon" The "error term", random noise, things outside of our model that affect y.

## Minimizing the Sum of the Squared Error

The most common method of estimating our $\beta$ parameters  is what's known as "Ordinary Least Squares" (OLS). (There are different methods of arriving at a line of best fit). OLS estimates the parameters that minimize the squared distance between each point in our dataset and our line of best fit.

\begin{align}
SSE = \sum(y_i - \hat{y})^2
\end{align}

```python
from matplotlib.patches import Rectangle
import numpy as np
from sklearn.metrics import mean_squared_error

def regression_squared_errors(df, feature, target, m, b):
    x = df[feature]
    y = df[target]
    y_pred = m*x + b

    fig = plt.figure(figsize=(7,7))
    ax = plt.axes()
    ax.scatter(x, y, label='y_true')
    ax.plot(x, y_pred, label='y_pred')
    ax.legend()

    # Plot square errors
    xmin, xmax = ax.get_xlim()
    ymin, ymax = ax.get_ylim()
    scale = (xmax-xmin)/(ymax-ymin)
    for x, y1, y2 in zip(x, y, y_pred):
        bottom_left = (x, min(y1, y2))
        height = abs(y1 - y2)
        width = height * scale
        ax.add_patch(Rectangle(xy=bottom_left, width=width, height=height, alpha=0.1))

    mse = mean_squared_error(y, y_pred)
    rmse = np.sqrt(mse)
    mae = mean_absolute_error(y, y_pred)
    r2 = r2_score(y, y_pred)
    print('Mean Squared Error:', mse)
    print('Root Mean Squared Error:', rmse)
    print('Mean Absolute Error:', mae)
    print('R^2:', r2)

regression_squared_errors(df, feature='Average Recent Growth in Personal Incomes',
           target='Incumbent Party Vote Share', m=3, b=46)
```

```python
interact(regression_squared_errors,
         df=fixed(df),
         feature=fixed('Average Recent Growth in Personal Incomes'),
         target=fixed('Incumbent Party Vote Share'),
         m=(-10,10,0.5),
         b=(40,60,0.5));
```

    interactive(children=(FloatSlider(value=0.0, description='m', max=10.0, min=-10.0, step=0.5), FloatSlider(valu…

## Hypotheses

```python
b = 46
ms = np.arange(-10,10,0.5)
sses = []

for m in ms:
    predictions = m * df[feature] + b
    errors = predictions - df[target]
    square_errors = errors ** 2
    sse = square_errors.sum()
    sses.append(sse)

hypotheses = pd.DataFrame({'Slope': ms})
hypotheses['Intercept'] = b
hypotheses['Sum of Square Errors'] = sses

hypotheses
```

      Slope
      Intercept
      Sum of Square Errors

      0
      -10.0
      46
      3062.3534

      1
      -9.5
      46
      2924.4034

      2
      -9.0
      46
      2790.4534

      3
      -8.5
      46
      2660.5034

      4
      -8.0
      46
      2534.5534

      5
      -7.5
      46
      2412.6034

      6
      -7.0
      46
      2294.6534

      7
      -6.5
      46
      2180.7034

      8
      -6.0
      46
      2070.7534

      9
      -5.5
      46
      1964.8034

      10
      -5.0
      46
      1862.8534

      11
      -4.5
      46
      1764.9034

      12
      -4.0
      46
      1670.9534

      13
      -3.5
      46
      1581.0034

      14
      -3.0
      46
      1495.0534

      15
      -2.5
      46
      1413.1034

      16
      -2.0
      46
      1335.1534

      17
      -1.5
      46
      1261.2034

      18
      -1.0
      46
      1191.2534

      19
      -0.5
      46
      1125.3034

      20
      0.0
      46
      1063.3534

      21
      0.5
      46
      1005.4034

      22
      1.0
      46
      951.4534

      23
      1.5
      46
      901.5034

      24
      2.0
      46
      855.5534

      25
      2.5
      46
      813.6034

      26
      3.0
      46
      775.6534

      27
      3.5
      46
      741.7034

      28
      4.0
      46
      711.7534

      29
      4.5
      46
      685.8034

      30
      5.0
      46
      663.8534

      31
      5.5
      46
      645.9034

      32
      6.0
      46
      631.9534

      33
      6.5
      46
      622.0034

      34
      7.0
      46
      616.0534

      35
      7.5
      46
      614.1034

      36
      8.0
      46
      616.1534

      37
      8.5
      46
      622.2034

      38
      9.0
      46
      632.2534

      39
      9.5
      46
      646.3034

```python
hypotheses.plot(x='Slope', y='Sum of Square Errors',
                title=f'Intercept={b}');
```

## Scikit-learn

#### Jake VanderPlas, [Python Data Science Handbook, Chapter 5.2, Introducing Scikit-Learn](https://jakevdp.github.io/PythonDataScienceHandbook/05.02-introducing-scikit-learn.html), Scikit-Learn's Estimator API

> Most commonly, the steps in using the Scikit-Learn estimator API are as follows (we will step through a handful of detailed examples in the sections that follow).

> 1. Choose a class of model by importing the appropriate estimator class from Scikit-Learn.
> 2. Choose model hyperparameters by instantiating this class with desired values.
> 3. Arrange data into a features matrix and target vector following the discussion above.
> 4. Fit the model to your data by calling the `fit()` method of the model instance.
> 5. Apply the Model to new data: For supervised learning, often we predict labels for unknown data using the `predict()` method.

```python
from sklearn.linear_model import LinearRegression
model = LinearRegression()

features = ['Average Recent Growth in Personal Incomes']
target = 'Incumbent Party Vote Share'
X = df[features]
y = df[target]

model.fit(X, y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

```python
model.intercept_, model.coef_,
```

    (46.499209757741625, array([2.97417709]))

```python
model.predict([[0]])
```

    array([46.49920976])

```python
model.predict([[1]])
```

    array([49.47338685])

```python
model.predict([[1]]) - model.predict([[0]])
```

    array([2.97417709])

```python
model.predict([[2]])
```

    array([52.44756393])

```python
model.predict([[2]]) - model.predict([[1]])
```

    array([2.97417709])

```python
m = model.coef_[0]
b = model.intercept_

y_pred = m*x + b
plot_preds(x, y, y_pred)
```

```python
y_pred = model.predict(X)
plot_preds(x, y, y_pred)
```

## Linear Algebra!

The same result that is found by minimizing the sum of the squared errors can be also found through a linear algebra process known as the "Least Squares Solution:"

Before we can work with this equation in its linear algebra form we have to understand how to set up the matrices that are involved in this equation.

### The $\beta$ vector

The $\beta$ vector represents all the parameters that we are trying to estimate, our $y$ vector and $X$ matrix values are full of data from our dataset. The $\beta$ vector holds the variables that we are solving for: $\beta_0$ and $\beta_1$

Now that we have all of the necessary parts we can set them up in the following equation:

\begin{align}
y = X \beta + \epsilon
\end{align}

Since our $\epsilon$ value represents **random** error we can assume that it will equal zero on average.

\begin{align}
y = X \beta
\end{align}

The objective now is to isolate the $\beta$ matrix. We can do this by pre-multiplying both sides by "X transpose" $X^{T}$.

\begin{align}
X^{T}y =  X^{T}X \beta
\end{align}

Since anything times its transpose will result in a square matrix, if that matrix is then an invertible matrix, then we should be able to multiply both sides by its inverse to remove it from the right hand side. (We'll talk tomorrow about situations that could lead to $X^{T}X$ not being invertible.)

\begin{align}
(X^{T}X)^{-1}X^{T}y =  (X^{T}X)^{-1}X^{T}X \beta
\end{align}

Since any matrix multiplied by its inverse results in the identity matrix, and anything multiplied by the identity matrix is itself, we are left with only $\beta$ on the right hand side:

\begin{align}
(X^{T}X)^{-1}X^{T}y = \hat{\beta}
\end{align}

We will now call it "beta hat" $\hat{\beta}$ because it now represents our estimated values for $\beta_0$ and $\beta_1$

### Lets calculate our $\beta$ coefficients with numpy!

```python

```

# Multiple Regression

Simple or bivariate linear regression involves a single $x$ variable and a single $y$ variable. However, we can have many $x$ variables. A linear regression model that involves multiple x variables is known as **Multiple** Regression (NOT MULTIVARIATE).

```python
from mpl_toolkits import mplot3d

def viz3D(fitted_model, df, features, target='', num=100):
    """
    Visualize model predictions in 3D, for regression or binary classification

    Parameters
    ----------
    fitted_model : scikit-learn model, already fitted
    df : pandas dataframe, which was used to fit model
    features : list of strings, name of features 1 & 2
    target : string, name of target
    num : int, number of grid points for each feature

    References
    ----------
    https://jakevdp.github.io/PythonDataScienceHandbook/04.12-three-dimensional-plotting.html
    https://scikit-learn.org/stable/auto_examples/tree/plot_iris.html
    """
    feature1, feature2 = features
    x1 = np.linspace(df[feature1].min(), df[feature1].max(), num)
    x2 = np.linspace(df[feature2].min(), df[feature2].max(), num)
    X1, X2 = np.meshgrid(x1, x2)
    X = np.c_[X1.flatten(), X2.flatten()]
    if hasattr(fitted_model, 'predict_proba'):
        predicted = fitted_model.predict_proba(X)[:,1]
    else:
        predicted = fitted_model.predict(X)
    Z = predicted.reshape(num, num)

    fig = plt.figure()
    ax = plt.axes(projection='3d')
    ax.plot_surface(X1, X2, Z, cmap='viridis')
    ax.set_xlabel(feature1)
    ax.set_ylabel(feature2)
    ax.set_zlabel(target)
    plt.show()
```

```python
model = LinearRegression()

features = ['Average Recent Growth in Personal Incomes',
            'US Military Fatalities per Million']
target = 'Incumbent Party Vote Share'
X = df[features]
y = df[target]
model.fit(X, y)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

```python
pd.Series(model.coef_, features)
```

    Average Recent Growth in Personal Incomes    3.406214
    US Military Fatalities per Million          -0.053752
    dtype: float64

```python
df[features].describe()
```

      Average Recent Growth in Personal Incomes
      US Military Fatalities per Million

      count
      17.000000
      17.000000

      mean
      1.791765
      21.588235

      std
      1.419812
      55.767440

      min
      -0.390000
      0.000000

      25%
      0.850000
      0.000000

      50%
      1.720000
      0.000000

      75%
      2.890000
      5.000000

      max
      4.210000
      190.000000

```python
pd.Series(model.coef_, features).plot.barh(color='grey');
```

```python
%matplotlib notebook
viz3D(model, df, features, target)
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

# Dimensionality in Linear Regression!

Muliple Regression is simply an extension of the bivariate case. The reason why we see the bivariate case demonstrated so often is simply because it's easier to graph and all of the intuition from the bivariate case is the same as we keep on adding explanatory variables.

As we increase the number of $x$ values in our model we are simply fitting a n-1-dimensional plane to an n-dimensional cloud of points within an n-dimensional hypercube.

# Interpreting Coefficients

One of Linear Regression's strengths is that the parameters of the model (coefficients) are readily interpretable and useful. Not only do they describe the relationship between x and y but they put a number on just how much x is associated with y. We should be careful to not speak about this relationshiop in terms of causality because these coefficients are in fact correlative measures. We would need a host of additional techniques in order to estimate a causal effect using linear regression (econometrics).

\begin{align}
\hat{\beta} = \frac{Cov(x,y)}{Var(y)}
\end{align}

Going back to the two equations for the two models that we have estimated so far, lets replace their beta values with their actual values to see if we can make sense of how to interpret these beta coefficients.

## Bivariate Model

$y_i = \beta_0 + \beta_1temperature + \epsilon$

$sales_i = -596.2 + 24.69temperature + \epsilon$

What might $\beta_0$ in this model represent? It represents the level of sales that we would have if temperature were 0. Since this is negative one way of interpreting it is that it's so cold outside that you would have to pay people to eat ice cream. A more appropriate interpretation is probably that the ice cream store owner should close his store down long before the temperature reaches 0 degrees farenheit (-17.7 celsius). The owner can compare his predicted sales with his costs of doing business to know how warm the weather has to get before he should open his store.

What might the $beta_1$ in this model reprsent? it represents the increase in sales for each degree of temperature increase. For every degree that the temperature goes up outside he has $25 more in sales.

## Multiple Regression Model

$y_i = \beta_0 + \beta_1age_i + \beta_2weight_i + \epsilon$

$BloodPressure_i = 30.99+ .86age_i + .33weight_i + \epsilon$

The interpretation of coefficients in this example are similar. The intercept value repesents the blood pressure a person would have if they were 0 years old and weighed 0 pounds. This not a super useful interpretation. If we look at our data it is unlikely that we have any measurements like these in the dataset. This means that our interpretation of our intercept likely comes from extrapolating the regression line (plane). Coefficients having straightforward interpretations is a strength of linear regression if we're careful about extrapolation and only interpreting our data within the context that it was gathered.

The interpretation of our other coefficients can be a useful indicator for how much a person similar to those in our dataset's blood pressure will go up on average with each additional year of age and pound of weight.

# Basic Model Validation

One of the downsides of relying on $R^2$ too much is that although it tells you when you're fitting the data well, it doesn't tell you when you're *overfitting* the data. The best way to tell if you're overfitting the data is to get some data that your model hasn't seen yet, and evaluate how your predictions do. This is essentially what "model validation" is.

# Why is Linear Regression so Important?

## Popularity

Linear Regression is an extremely popular technique that every data scientist **needs** to understand. It's not the most advanced technique and there are supervised learning techniques that will obtain a higher accuracy, but where it lacks in accuracy it makes up for it in interpretability and simplicity.

## Interpretability

Few other models possess coefficients that are so directly linked to their variables with a such a clear interpretation. Tomorrow we're going to learn about ways to make them even easier to interpret.

## Simplicity

A linear regression model can be communicated just by writing out its equation. It's kind of incredible that such high dimensional relationships can be described from just a linear combination of variables and coefficients.


##### Copyright 2020 The TensorFlow Authors.

```python
#@title Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
```

# Barren plateaus

    View on TensorFlow.org

    Run in Google Colab

    View source on GitHub

    Download notebook

In this example you will explore the result of McClean, 2019 that says not just any quantum neural network structure will do well when it comes to learning. In particular you will see that a certain large family of random quantum circuits do not serve as good quantum neural networks, because they have gradients that vanish almost everywhere. In this example you won't be training any models for a specific learning problem, but instead focusing on the simpler problem of understanding the behaviors of gradients.

## Setup

```python
try:
    %tensorflow_version 2.x
except Exception:
    pass
```

Install TensorFlow Quantum:

Note: This may require restarting the Colab runtime (*Runtime > Restart Runtime*).

```python
!pip install tensorflow-quantum
```

Now import TensorFlow and the module dependencies:

```python
import tensorflow as tf
import tensorflow_quantum as tfq

import cirq
import sympy
import numpy as np

# visualization tools
%matplotlib inline
import matplotlib.pyplot as plt
from cirq.contrib.svg import SVGCircuit

np.random.seed(1234)
```

## 1. Summary

Random quantum circuits with many blocks that look like this ($R_{P}(\theta)$ is a random Pauli rotation):

Where if $f(x)$ is defined as the expectation value w.r.t. $Z_{a}Z_{b}$ for any qubits $a$ and $b$, then there is a problem that $f'(x)$  has a mean very close to 0 and does not vary much. You will see this below:

## 2. Generating random circuits

The construction from the paper is straightforward to follow. The following implements a simple function that generates a random quantum circuit—sometimes referred to as a *quantum neural network* (QNN)—with the given depth on a set of qubits:

```python
def generate_random_qnn(qubits, symbol, depth):
    """Generate random QNN's with the same structure from McClean et al."""
    circuit = cirq.Circuit()
    for qubit in qubits:
        circuit += cirq.Ry(np.pi / 4.0)(qubit)

    for d in range(depth):
        # Add a series of single qubit rotations.
        for i, qubit in enumerate(qubits):
            random_n = np.random.uniform()
            random_rot = np.random.uniform(
            ) * 2.0 * np.pi if i != 0 or d != 0 else symbol
            if random_n > 2. / 3.:
                # Add a Z.
                circuit += cirq.Rz(random_rot)(qubit)
            elif random_n > 1. / 3.:
                # Add a Y.
                circuit += cirq.Ry(random_rot)(qubit)
            else:
                # Add a X.
                circuit += cirq.Rx(random_rot)(qubit)

        # Add CZ ladder.
        for src, dest in zip(qubits, qubits[1:]):
            circuit += cirq.CZ(src, dest)

    return circuit

generate_random_qnn(cirq.GridQubit.rect(1, 3), sympy.Symbol('theta'), 2)
```

The authors investigate the gradient of a single parameter $\theta_{1,1}$. Let's follow along by placing a `sympy.Symbol` in the circuit where $\theta_{1,1}$ would be. Since the authors do not analyze the statistics for any other symbols in the circuit, let's replace them with random values now instead of later.

## 3. Running the circuits

Generate a few of these circuits along with an observable to test the claim that the gradients don't vary much. First, generate a batch of random circuits. Choose a random *ZZ* observable and batch calculate the gradients and variance using TensorFlow Quantum.

### 3.1 Batch variance computation

Let's write a helper function that computes the variance of the gradient of a given observable over a batch of circuits:

```python
def process_batch(circuits, symbol, op):
    """Compute the variance of a batch of expectations w.r.t. op on each circuit that
    contains `symbol`. Note that this method sets up a new compute graph every time it is
    called so it isn't as performant as possible."""

    # Setup a simple layer to batch compute the expectation gradients.
    expectation = tfq.layers.Expectation()

    # Prep the inputs as tensors
    circuit_tensor = tfq.convert_to_tensor(circuits)
    values_tensor = tf.convert_to_tensor(
        np.random.uniform(0, 2 * np.pi, (n_circuits, 1)).astype(np.float32))

    # Use TensorFlow GradientTape to track gradients.
    with tf.GradientTape() as g:
        g.watch(values_tensor)
        forward = expectation(circuit_tensor,
                              operators=op,
                              symbol_names=[symbol],
                              symbol_values=values_tensor)

    # Return variance of gradients across all circuits.
    grads = g.gradient(forward, values_tensor)
    grad_var = tf.math.reduce_std(grads, axis=0)
    return grad_var.numpy()[0]
```

### 3.1 Set up and run

Choose the number of random circuits to generate along with their depth and the amount of qubits they should act on. Then plot the results.

```python
n_qubits = [2 * i for i in range(2, 7)
           ]  # Ranges studied in paper are between 2 and 24.
depth = 50  # Ranges studied in paper are between 50 and 500.
n_circuits = 200
theta_var = []

for n in n_qubits:
    # Generate the random circuits and observable for the given n.
    qubits = cirq.GridQubit.rect(1, n)
    symbol = sympy.Symbol('theta')
    circuits = [
        generate_random_qnn(qubits, symbol, depth) for _ in range(n_circuits)
    ]
    op = cirq.Z(qubits[0]) * cirq.Z(qubits[1])
    theta_var.append(process_batch(circuits, symbol, op))

plt.semilogy(n_qubits, theta_var)
plt.title('Gradient Variance in QNNs')
plt.xlabel('n_qubits')
plt.ylabel('$\\partial \\theta$ variance')
plt.show()
```

This plot shows that for quantum machine learning problems, you can't simply guess a random QNN ansatz and hope for the best. Some structure must be present in the model circuit in order for gradients to vary to the point where learning can happen.

## 4. Heuristics

An interesting heuristic by Grant, 2019 allows one to start very close to random, but not quite. Using the same circuits as McClean et al., the authors propose a different initialization technique for the classical control parameters to avoid barren plateaus. The initialization technique starts some layers with totally random control parameters—but, in the layers immediately following, choose parameters such that the initial transformation made by the first few layers is undone. The authors call this an *identity block*.

The advantage of this heuristic is that by changing just a single parameter, all other blocks outside of the current block will remain the identity—and the gradient signal comes through much stronger than before. This allows the user to pick and choose which variables and blocks to modify to get a strong gradient signal. This heuristic does not prevent the user from falling in to a barren plateau during the training phase (and restricts a fully simultaneous update), it just guarantees that you can start outside of a plateau.

### 4.1 New QNN construction

Now construct a function to generate identity block QNNs. This implementation is slightly different than the one from the paper. For now, look at the behavior of the gradient of a single parameter so it is consistent with McClean et al, so some simplifications can be made.

To generate an identity block and train the model, generally you need $U1(\theta_{1a}) U1(\theta_{1b})^{\dagger}$ and not $U1(\theta_1) U1(\theta_1)^{\dagger}$. Initially $\theta_{1a}$ and $\theta_{1b}$ are the same angles but they are learned independently. Otherwise, you will always get the identity even after training. The choice for the number of identity blocks is empirical. The deeper the block, the smaller the variance in the middle of the block. But at the start and end of the block, the variance of the parameter gradients should be large.

```python
def generate_identity_qnn(qubits, symbol, block_depth, total_depth):
    """Generate random QNN's with the same structure from Grant et al."""
    circuit = cirq.Circuit()

    # Generate initial block with symbol.
    prep_and_U = generate_random_qnn(qubits, symbol, block_depth)
    circuit += prep_and_U

    # Generate dagger of initial block without symbol.
    U_dagger = (prep_and_U[1:])**-1
    circuit += cirq.resolve_parameters(
        U_dagger, param_resolver={symbol: np.random.uniform() * 2 * np.pi})

    for d in range(total_depth - 1):
        # Get a random QNN.
        prep_and_U_circuit = generate_random_qnn(
            qubits,
            np.random.uniform() * 2 * np.pi, block_depth)

        # Remove the state-prep component
        U_circuit = prep_and_U_circuit[1:]

        # Add U
        circuit += U_circuit

        # Add U^dagger
        circuit += U_circuit**-1

    return circuit

generate_identity_qnn(cirq.GridQubit.rect(1, 3), sympy.Symbol('theta'), 2, 2)
```

### 4.2 Comparison

Here you can see that the heuristic does help to keep the variance of the gradient from vanishing as quickly:

```python
block_depth = 10
total_depth = 5

heuristic_theta_var = []

for n in n_qubits:
    # Generate the identity block circuits and observable for the given n.
    qubits = cirq.GridQubit.rect(1, n)
    symbol = sympy.Symbol('theta')
    circuits = [
        generate_identity_qnn(qubits, symbol, block_depth, total_depth)
        for _ in range(n_circuits)
    ]
    op = cirq.Z(qubits[0]) * cirq.Z(qubits[1])
    heuristic_theta_var.append(process_batch(circuits, symbol, op))

plt.semilogy(n_qubits, theta_var)
plt.semilogy(n_qubits, heuristic_theta_var)
plt.title('Heuristic vs. Random')
plt.xlabel('n_qubits')
plt.ylabel('$\\partial \\theta$ variance')
plt.show()
```

This is a great improvement in getting stronger gradient signals from (near) random QNNs.