```julia
"""Tutorial: Second-Order Moller--Plesset Perturbation Theory (MP2)"""

__author__    = ["D. Menendez", "Dominic A. Sirianni"]
__credit__    = ["D. Menendez", "Dominic A. Sirianni", "Daniel G. A. Smith"]

__copyright__ = "(c) 2014-2020, The Psi4Julia Developers"
__license__   = "BSD-3-Clause"
__date__      = "2020-07-30"
```

    "2020-07-30"

# Second-Order Moller-Plesset Perturbation Theory (MP2)

Moller-Plesset perturbation theory [also referred to as many-body perturbation theory (MBPT)] is an adaptation of the more general Rayleigh-Schrodinger perturbation theory (RSPT), applied to problems in molecular electronic structure theory.  This tutorial will provide a brief overview of both RSPT and MBPT, before walking through an implementation of second-order Moller-Plesset perturbation theory (specifically referred to as MP2) which uses conventional, 4-index ERIs.

### I. Overview of Rayleigh-Schrodinger Perturbation Theory
Given the Hamiltonian operator $\hat{H}$ for a system, perturbation theory solves the Schrodinger equation for that system by rewriting $\hat{H}$ as

\begin{equation}
\hat{H} = \hat{H}{}^{(0)} + \lambda\hat{V},
\tag{[Szabo:1996], pp. 322, Eqn. 6.3}
\end{equation}

were $\hat{H}{}^{(0)}$ is the Hamiltonian operator corresponding to a solved problem which resembles $\hat{H}$, and $\hat{V}$ is the *perturbation* operator, defined as $\hat{V} = \hat{H} - \hat{H}{}^{(0)}$.  Then the Schrodinger equation for the system becomes

\begin{equation}
\hat{H}\mid\Psi_n\rangle = (\hat{H}{}^{(0)} + \lambda\hat{V})\mid\Psi_n\rangle = E_n\mid\Psi_n\rangle.
\tag{[Szabo:1996], pp. 322, Eqn. 6.2}
\end{equation}

The energies $E_n$ and wavefunctions $\mid\Psi_n\rangle$ will both be functions of $\lambda$; they can therefore be written as a Taylor series expansion about $\lambda = 0$ ([Szabo:1996], pp. 322, Eqns. 6.4a & 6.4b):

\begin{align}
E_n &= E_n^{(0)} + \lambda E_n^{(1)} + \lambda^2E_n^{(2)} + \ldots;\tag{[Szabo:1996], pp. 322, Eqn. 6.4a}\\
\mid\Psi_n\rangle &= \mid\Psi_n^{(0)}\rangle + \lambda\mid\Psi_n^{(1)}\rangle + \lambda^2\mid\Psi_n^{(2)}\rangle + \ldots,\tag{[Szabo:1996], pp. 322, Eqn. 6.4b}\\
\end{align}

in practice, these perturbation expansions may be truncated to a given power of $\lambda$.  Substituting the perturbation series above back into the Schrodinger equation yields

\begin{equation*}
(\hat{H}{}^{(0)} + \lambda\hat{V})(\mid\Psi_n^{(0)}\rangle + \lambda\mid\Psi_n^{(1)}\rangle + \lambda^2\mid\Psi_n^{(2)}\rangle + \ldots) = (E_n^{(0)} + \lambda E_n^{(1)} + \lambda^2E_n^{(2)} + \ldots)(\mid\Psi_n^{(0)}\rangle + \lambda\mid\Psi_n^{(1)}\rangle + \lambda^2\mid\Psi_n^{(2)}\rangle + \ldots),
\end{equation*}

which by equating powers of $\lambda$ ([Szabo:1996], pp. 323, Eqns. 6.7a-6.7d) gives expressions for the $E_n^{(i)}$and $\mid\Psi_n^{(i)}\rangle$. Note that for $\lambda^0$, $E_n^{(0)}$ and $\mid\Psi_n^{(0)}\rangle$ are known, as they are the solution to the zeroth-order problem $\hat{H}{}^{(0)}$.  For $\lambda^1$ and $\lambda^2$, the expressions for $E_n^{(1)}$ and $E_n^{(2)}$ are given by

\begin{align}
\lambda^1:\;\;\;\;E_n^{(1)} &= \langle\Psi_n^{(0)}\mid\hat{V}\mid\Psi_n^{(0)}\rangle,\tag{[Szabo:1996], pp. 323, Eqn. 6.8b}\\
\lambda^2:\;\;\;\;E_n^{(2)} &= \sum_{\mu\neq n}\frac{\mid\langle\Psi_{n}^{(0)}\mid\hat{V}\mid\Psi_{\mu}^{(0)}\rangle\mid^2}{E_n^{(0)} - E_{\mu}^{(0)}}\tag{[Szabo:1996], pp. 324, Eqn. 6.12}\\
\end{align}

### II. Overview of Moller-Plesset Perturbation Theory

The exact electronic Hamiltonian for an N-electron molecule with a given atomic configuration is (in atomic units):

\begin{equation}
\hat{H}_{elec} = \sum_i\hat{h}(i) + \sum_{i<j}\frac{1}{r_{ij}}.\tag{[Szabo:1996], pp. 43, Eqn. 2.10}
\end{equation}

Moller-Plesset perturbation theory seeks to solve the molecular electronic Schrodinger equation with the above Hamiltonian using the techniques of Rayleigh-Schroding perturbation theory, by selecting the zeroth-order Hamiltonian and wavefunctions to be those from Hartree-Fock theory:

\begin{align}
\hat{H}{}^{(0)}  &= \sum_i \hat{f}(i) = \sum_i \hat{h}(i) + \upsilon^{HF}(i)\tag{[Szabo:1996], pp. 350, Eqn. 6.59}\\
\hat{V} &= \hat{H} - \hat{H}{}^{(0)} = \sum_{i<j} \frac{1}{r_{ij}} - \upsilon^{HF}(i).\tag{[Szabo:1996], pp. 350, Eqn. 6.60}\\
\end{align}

With these choices of $\hat{H}{}^{(0)}$, $\hat{V}$, and $\mid\Psi_n^{(0)}\rangle$, the ground-state zeroth-order energy is given by $E_0^{(0)} = \sum_i\epsilon_i$ ([Szabo:1996], pp. 351, Eqn. 6.67).  Then, the first-order ground state energy may be computed by $E_0^{(1)} = \langle\Psi_0^{HF}\mid\hat{V}\mid\Psi_0^{HF}\rangle$ to find that the total ground-state energy through first order, $E_0 = E_0^{(0)} + E_0^{(1)}$, is exactly the Hartree-Fock energy. ([Szabo:1996], pp. 351, Eqn. 6.69)  Therefore, the first correction to the Hartree-Fock energy will occur at second-order in the perturbation series, i.e., with $E_0^{(2)}$; truncating the perturbation series at second order is commonly referred to as MP2.  The second order correction to the ground state energy will be given by

\begin{equation}
E_0^{(2)} = \sum_{\mu\neq 0}\frac{\left|\langle\Psi_{0}^{HF}\mid\hat{V}\mid\Psi_{\mu}^{HF}\rangle\right|^2}{E_0^{(0)} - E_{\mu}^{(0)}}\tag{[Szabo:1996], pp. 351, Eqn. 6.70}
\end{equation}

For brevity, we will now drop the "HF" from all zeroth-order wavefunctions.  This summation is over the eigenstate index $\mu$, each associated with a different eigenstate of the zeroth-order Hamiltonian.  For a single-determinant wavefunction constructed from spin orbitals, the summation over the eigenstate index $\mu\neq 0$ therefore refers to determinants which are constructed from *different* spin orbitals than the ground state determinant.  To distinguish such determinants, we will denote MOs occupied in the ground state with indices $i,\,j,\,\ldots$, and MOs which are unoccupied in the ground state with indices $a,\,b,\,\ldots\,$.  Then a determinant where orbital $a$ is substituted for orbital $i$ is denoted $\mid\Psi_i^a\rangle$, and so on.  Before substituting this new notation into the above energy expression, however, we may immediately recognize that many terms $\langle\Psi_{0}\mid\hat{V}\mid\Psi_{\mu}\rangle$ will not contribute to the second order energy:

| Term         | Determinant                               |  Contribution to $E_0^{(2)}$                            |
|--------------|-------------------------------------------|---------------------------------------------------------|
| Singles      | $\mid\Psi_i^a\rangle$                     | 0; Brillouin's Theorem                                  |
| Doubles      | $\mid\Psi_{ij}^{ab}\rangle$               | Survive                                                 |
| Higher-order | $\mid\Psi_{ijk\ldots}^{abc\ldots}\rangle$ | 0; $\hat{V}$ is a two-particle operator            |

Hence we see that only doubly-substituted determinants $\mid\Psi_{ij}^{ab}\rangle$ will contribute to $E_0^{(2)}$.  From Hartree-Fock theory, we know that

\begin{equation}
\langle\Psi_0\mid r_{ij}^{-1}\mid\Psi_{ij}^{ab}\rangle = [ia\| jb],\tag{[Szabo:1996], pp. 72, Tbl. 2.6}
\end{equation}

where $[ia\| jb] = [ia\mid jb] - [ib\mid ja]$ is an antisymmetrized two-electron integral, and the square brackets "$[\ldots ]$" indicate that we are employing chemists' notation.  What about the energies of these doubly substituted determinants?  Recognizing that the difference between the energies of the newly- and formerly-occupied orbitals in each substitution must modify the total energy of the ground state determinant,

\begin{equation}
E_{ij}^{ab} = E_0 - (\epsilon_i - \epsilon_a + \epsilon_j - \epsilon_b).
\end{equation}

Substituting these expressions into the one for the second-order energy, we have that

\begin{equation}
E_0^{(2)} = \sum_{i<j}\sum_{a<b} \frac{\left|\,[ia\|jb]\,\right|^2}{\epsilon_i - \epsilon_a + \epsilon_j - \epsilon_b}.\tag{[Szabo:1996], pp. 352, Eqn. 6.71}
\end{equation}

So far, our discussion has used spin-orbitals instead of the more familiar spatial orbitals.  Indeed, significant speedups are achieved when using spatial orbitals.  Integrating out the spin variable $\omega$ from $E_0^{(2)}$ yields two expressions; one each for the interaction of particles with the same spin (SS) and opposite spin (OS):

\begin{align}
E_{\rm 0,\,SS}^{(2)} = \sum_{ij}\sum_{ab}\frac{(ia\mid jb)[(ia\mid jb) - (ib\mid ja)]}{\epsilon_i - \epsilon_a + \epsilon_j - \epsilon_b}, \quad
E_{\rm 0,\,OS}^{(2)} = \sum_{ij}\sum_{ab}\frac{(ia\mid jb)(ia\mid jb)}{\epsilon_i - \epsilon_a + \epsilon_j - \epsilon_b},
\end{align}

where these spin-free expressions make use of integrals in chemists' notation over spatial orbitals. (Rearranged from [Szabo:1996], pp. 352, Eqn. 6.74)  Note that an exchange integral arises between particles of the same spin; this is because the motions of particles with identical spins are correlated due to the requirement that $\left|\Psi\right|^2$ remain invariant to the exchange of the spatial and spin coordinates of any pair of electrons.  Finally, the total MP2 correction energy $E_0^{(2)} = E_{\rm 0,\,SS}^{(2)} + E_{\rm 0,\,OS}^{(2)}$.

### Implementation of Conventional MP2

Let's begin by importing Psi4, NumPy and TensorOperations, and setting memory and output file options:

```julia
# ==> Import statements & Global Options <==
using PyCall: pyimport
psi4 = pyimport("psi4")
np   = pyimport("numpy") # used only to cast to Psi4 arrays
using TensorOperations: @tensor

psi4.set_memory(Int(2e9))
numpy_memory = 2
psi4.core.set_output_file("output.dat", false)
```

      Memory set to   1.863 GiB by Python driver.

Next, we can define our molecule and Psi4 options.  Notice that we are using `scf_type pk` to indicate that we wish to use conventional, full 4-index ERIs, and that we have specified `mp2_type conv` so that the MP2 algorithm we check against also uses the conventional ERIs.

```julia
# ==> Molecule & Psi4 Options Definitions <==
mol = psi4.geometry("""
O
H 1 1.1
H 1 1.1 2 104
symmetry c1
""")

psi4.set_options(Dict("basis"         => "6-31g",
                      "scf_type"      => "pk",
                      "mp2_type"      => "conv",
                      "e_convergence" => 1e-8,
                      "d_convergence" => 1e-8))
```

Since MP2 is a perturbation on the zeroth-order Hartree-Fock description of a molecular system, all of the relevant information (Fock matrix, orbitals, orbital energies) about the system can be computed using any Hartree-Fock program.  We could use the RHF program that we wrote in tutorial 3a, but we could just as easily use Psi4 to do our dirty work.  In the cell below, use Psi4 to compute the RHF energy and wavefunction, and store them using the `return_wfn=True` keyword argument to `psi4.energy()`:

```julia
# Get the SCF wavefunction & energies
scf_e, scf_wfn = psi4.energy("scf", return_wfn=true)
```

    (-75.95252904632221, PyObject <psi4.core.RHF object at 0x14859e110>)

In the expression for $E_0^{(2)}$, the two summations are over occupied and virtual indices, respectively.  Therefore, we'll need to get the number of occupied orbitals and the total number of orbitals.  Additionally, we must obtain the MO energy eigenvalues; again since the sums are over occupied and virtual orbitals, it is good to separate the occupied orbital energies from the virtual orbital energies.  From the SCF wavefunction you generated above, get the number of doubly occupied orbitals, number of molecular orbitals, and MO energies:

```julia
# ==> Get orbital information & energy eigenvalues <==
# Number of Occupied orbitals & MOs
ndocc = scf_wfn.nalpha()
nmo = scf_wfn.nmo()

# Get orbital energies, cast into NumPy array, and separate occupied & virtual
eps = np.asarray(scf_wfn.epsilon_a())
e_ij = eps[1:ndocc]
e_ab = eps[ndocc+1:end];
```

Unlike the orbital information, Psi4 does not return the ERIs when it does a computation.  Fortunately, however, we can just build them again using the `psi4.core.MintsHelper()` class.  Recall that these integrals will be generated in the AO basis; before using them in the $E_0^{(2)}$ expression, we must transform them into the MO basis.  To do this, we first need to obtain the orbital coefficient matrix, **C**.  In the cell below, generate the ERIs for our molecule, get **C** from the SCF wavefunction, and obtain occupied- and virtual-orbital slices of **C** for future use.

```julia
# ==> ERIs <==
# Create instance of MintsHelper class
mints = psi4.core.MintsHelper(scf_wfn.basisset())

# Memory check for ERI tensor
I_size = nmo^4 * 8.e-9
println("\nSize of the ERI tensor will be $I_size GB.")
memory_footprint = I_size * 1.5
if I_size > numpy_memory
    psi4.core.clean()
    throw(OutOfMemoryError("Estimated memory utilization ($memory_footprint GB) exceeds " *
                           "allotted memory limit of $numpy_memory GB."))
end

# Build ERI Tensor
I = np.asarray(mints.ao_eri())

# Get MO coefficients from SCF wavefunction
C = np.asarray(scf_wfn.Ca())
Cocc = C[:, 1:ndocc]
Cvirt = C[:, ndocc+1:end];
```

    Size of the ERI tensor will be 0.00022848800000000003 GB.

In order to transform the four-index integrals from the AO to the MO basis, we must perform the following contraction:

$$(i\,a\mid j\,b) = C_{\mu i}C_{\nu a}(\mu\,\nu\mid|\,\lambda\,\sigma)C_{\lambda j}C_{\sigma b}$$

Again, here we are using $i,\,j$ as occupied orbital indices and $a,\, b$ as virtual orbital indices.  We could carry out the above contraction all in one step using either `@tensor` or explicit loops:

~~~julia
# Naive Algorithm for ERI Transformation
@tensor I_mo[i,a,j,b] := Cocc[p,i] * Cvirt[q,a] * I[p,q,r,s] * Cocc[r,j] * Cvirt[s,b]
~~~

Notice that the transformation from AO index to occupied (virtual) MO index requires only the occupied (virtual) block of the **C** matrix; this allows for computational savings in large basis sets, where the virtual space can be very large.  This algorithm, while efficient with `@tensor`, has horrendous scaling if the search of an optimal contraction fails.  We will enforce a better contraction. Examining the contraction more closely, we see that there are 8 unique indices, and thus the step above scales as ${\cal O}(N^8)$.  With this algorithm, a twofold increase of the number of MO's would result in $2^8 = 256\times$ expense to perform.  We can, however, refactor the above contraction such that

$$(i\,a\mid j\,b) = \left[C_{\mu i}\left[C_{\nu a}\left[C_{\lambda j}\left[C_{\sigma b}(\mu\,\nu\mid|\,\lambda\,\sigma)\right]\right]\right]\right],$$

where we have now written the transfomation as four ${\cal O}(N^5)$ steps instead of one ${\cal O}(N^8)$ step. This is a savings of $\frac{4}{n^3}$, and is responsible for the feasibility of the MP2 method for application to any but very small systems and/or basis sets.  We may carry out the above ${\cal O}(N^5)$ algorithm by carrying out one index transformation at a time, and storing the result in a temporary array.  In the cell below, transform the ERIs from the AO to MO basis, using our smarter algorithm:

```julia
# ==> Transform I -> I_mo @ O(N⁵) <==
I_mo = @tensor begin
   I_mo[i,q,r,s] := Cocc[p,i]     * I[p,q,r,s]
   I_mo[i,a,r,s] := Cvirt[q,a]    * I_mo[i,q,r,s]
   I_mo[i,a,j,s] :=                 I_mo[i,a,r,s] * Cocc[r,j]
   I_mo[i,a,j,b] :=                 I_mo[i,a,j,s] * Cvirt[s,b]
end
nothing
```

We note here that we can use infrastructure in Psi4 to carry out the above integral transformation; this entails obtaining the occupied and virtual blocks of **C** Psi4-side, and then using the built-in `MintsHelper` function `MintsHelper.mo_eri()` to transform the integrals.  Just to check your work above, execute the next cell to see this tech in action:

```julia
# ==> Compare our Imo to MintsHelper <==
Co = scf_wfn.Ca_subset("AO","OCC")
Cv = scf_wfn.Ca_subset("AO","VIR")
MO = np.asarray(mints.mo_eri(Co, Cv, Co, Cv))
println("Do our transformed ERIs match Psi4's? ", np.allclose(I_mo, np.asarray(MO)))
```

    Do our transformed ERIs match Psi4's? true

Now we have all the pieces needed to compute $E_0^{(2)}$.  This could be done by writing explicit loops over occupied and virtual indices Julia side, e.g.,

```julia
# Compute SS & OS MP2 Correlation
mp2_corr = let mp2_ss_corr = 0.0, mp2_os_corr = 0.0
   nvirt = nmo - ndocc
   for i in 1:ndocc, a in 1:nvirt, j in 1:ndocc, b in 1:nvirt
       numerator = I_mo[i,a,j,b] * (I_mo[i, a, j, b] - I_mo[i, b, j, a])
       mp2_ss_corr += numerator / (e_ij[i] + e_ij[j] - e_ab[a] - e_ab[b])
       mp2_os_corr += I_mo[i,a,j,b]^2 / (e_ij[i] + e_ij[j] - e_ab[a] - e_ab[b])
   end
   mp2_ss_corr + mp2_os_corr
end

# Total MP2 Energy
MP2_E = scf_e + mp2_corr

# ==> Compare to Psi4 <==
psi4.compare_values(psi4.energy("mp2"), MP2_E, 6, "MP2 Energy")
```

    	MP2 Energy........................................................PASSED

    true

In this method it is very clear what is going on and is easy to program. Julia has the distinct advantage loops are as fast as the same block written in a compiled language like C, C++, or Fortran.

However, we will provide an alternative formulation using tensor contractions. It should be clear how to contract the four-index integrals $(i\,a\mid j\,b)$ and $(i\,a\mid j\,b)$ with one another, but what about the energy eigenvalues $\epsilon$?  We can use a Julia trick called *broadcasting* to construct a four-index array of all possible energy denominators, which can then be contracted with the full I_mo arrays.  To do this, we'll use the function `reshape()`:
~~~julia
# Prepare 4d energy denominator array
e_denom  = reshape(e_ij,  1, 1, 1, :)      # Diagonal of 4d array are occupied orbital energies
e_denom -= reshape(e_ab', 1, 1, :)         # all combinations of (e_ij - e_ab)
e_denom += e_ij                            # all combinations of [(e_ij - e_ab) + e_ij]
e_denom -= e_ab'                           # All combinations of full denominator
e_denom  = premutedims(e_denom, (1,2,4,3)) # permute 3rd and 4th dims to have (nocc,nvirt,nocc,nvirt) shape
e_denom  = inv.(e_denom)                   # Take reciprocal for contracting with numerator
~~~
In the cell below, compute the energy denominator using `reshape()` and contract this array with the four-index ERIs to compute the same-spin and opposite-spin MP2 correction using `sum()`. Then, add these quantities to the SCF energy computed above to obtain the total MP2 energy.

Hint: For the opposite-spin correlation, use `permutedims()` to obtain the correct ordering of the indices in the exchange integral.

```julia
#using Einsum: @einsum
# ==> Compute MP2 Correlation & MP2 Energy <==
# Compute energy denominator array
e_denom = reshape(e_ij,1,1,1,:) .- reshape(e_ab',1,1,:) .+ (e_ij .- e_ab')
e_denom = permutedims(e_denom, (1,2,4,3)) # 3 ↔ 4
e_denom = inv.(e_denom)

# check
#using Test
#nvirt = nmo - ndocc
#for i in 1:ndocc, a in 1:nvirt, j in 1:ndocc, b in 1:nvirt
#    @test e_denom[i,a,j,b] ≈ 1 / (e_ij[i] + e_ij[j] - e_ab[a] - e_ab[b])
#end

# Compute SS & OS MP2 Correlation with sum()
bctd_mp2_os_corr = sum(I_mo .* I_mo .* e_denom)
I_mo_swap = permutedims(I_mo,(3,2,1,4)) # 1 ↔ 3
bctd_mp2_ss_corr = sum(I_mo .* (I_mo .- I_mo_swap) .* e_denom)

# Compare broadcasted and loop MP2
@assert bctd_mp2_os_corr + bctd_mp2_ss_corr ≈ mp2_corr

# Total MP2 Energy
MP2_E = scf_e + bctd_mp2_os_corr + bctd_mp2_ss_corr
```

    -76.09464888642944

```julia
# ==> Compare to Psi4 <==
psi4.compare_values(psi4.energy("mp2"), MP2_E, 6, "MP2 Energy")
```

    	MP2 Energy........................................................PASSED

    true

## References

1. Original paper: "Note on an Approximation Treatment for Many-Electron Systems"
	> [[Moller:1934:618](https://journals.aps.org/pr/abstract/10.1103/PhysRev.46.618)] C. Møller and M. S. Plesset, *Phys. Rev.* **46**, 618 (1934)
2. The Laplace-transformation in MP theory: "Minimax approximation for the decomposition of energy denominators in Laplace-transformed Møller–Plesset perturbation theories"
    > [[Takasuka:2008:044112](http://aip.scitation.org/doi/10.1063/1.2958921)] A. Takatsuka, T. Siichiro, and W. Hackbusch, *J. Phys. Chem.*, **129**, 044112 (2008)
3. Equations taken from:
	> [[Szabo:1996](https://books.google.com/books?id=KQ3DAgAAQBAJ&printsec=frontcover&dq=szabo+%26+ostlund&hl=en&sa=X&ved=0ahUKEwiYhv6A8YjUAhXLSCYKHdH5AJ4Q6AEIJjAA#v=onepage&q=szabo%20%26%20ostlund&f=false)] A. Szabo and N. S. Ostlund, *Modern Quantum Chemistry: Introduction to Advanced Electronic Structure Theory*. Courier Corporation, 1996.
4. Algorithms taken from:
	> [Crawford:prog] T. D. Crawford, "The Second-Order Møller–Plesset Perturbation Theory (MP2) Energy."  Accessed via the web at http://github.com/CrawfordGroup/ProgrammingProjects.


```python
# This cell is mandatory in all Dymos documentation notebooks.
missing_packages = []
try:
    import openmdao.api as om
except ImportError:
    if 'google.colab' in str(get_ipython()):
        !python -m pip install openmdao[notebooks]
    else:
        missing_packages.append('openmdao')
try:
    import dymos as dm
except ImportError:
    if 'google.colab' in str(get_ipython()):
        !python -m pip install dymos
    else:
        missing_packages.append('dymos')
try:
    import pyoptsparse
except ImportError:
    if 'google.colab' in str(get_ipython()):
        !pip install -q condacolab
        import condacolab
        condacolab.install_miniconda()
        !conda install -c conda-forge pyoptsparse
    else:
        missing_packages.append('pyoptsparse')
if missing_packages:
    raise EnvironmentError('This notebook requires the following packages '
                           'please install them and restart this notebook\'s runtime: {",".join(missing_packages)}')
```

(examples:the_brachistochrone)=
# The Brachistochrone

```{admonition} Things you'll learn through this example
- How to define a basic Dymos ODE system.
- How to test the partials of your ODE system.
- Adding a Trajectory object with a single Phase to an OpenMDAO Problem.
- Imposing boundary conditions on states with simple bounds via `fix_initial` and `fix_final`.
- Using the Phase.interpolate` method to set a linear guess for state and control values across the Phase.
- Checking the validity of the result through explicit simulation via the `Trajectory.simulate` method.
```

The brachistochrone is one of the most well-known optimal control problems.
It was originally posed as a challenge by Johann Bernoulli.

```{admonition} The brachistochrone problem
_Given two points A and B in a vertical plane, find the path AMB
down which a movable point M must by virtue of its weight fall from
A to B in the shortest possible time._

-  Johann Bernoulli, Acta Eruditorum, June 1696
```

We seek to find the optimal shape of a wire between two points (A and B) such that a bead sliding
without friction along the wire moves from point A to point B in minimum time.

```python
import numpy as np

import matplotlib.pyplot as plt
from matplotlib.patches import FancyArrowPatch, Arc

LW = 2

fig, ax = plt.subplots(1, 1, figsize=(5, 5))
ax.axis('off')

ax.set_xlim(-1, 11)
ax.set_ylim(-1, 11)

circle = plt.Circle((0, 10), radius=0.1, fc='k')
ax.add_patch(circle)
plt.text(0.2, 10.2, 'A')

circle = plt.Circle((10, 5), radius=0.1, fc='k')
ax.add_patch(circle)
plt.text(10.2, 5.2, 'B')

# Choose a to suite, compute b
a = 0.1
b = -0.5 - 10*a
c = 10

def y_wire(x):
    return a*x**2 + b*x + c, 2*a*x + b

x = np.linspace(0, 10, 100)
y, _ = y_wire(x)
plt.plot(x, y, 'b-')

# Add the bead to the wire
x = 3
y, dy_dx = y_wire(x)
plt.plot(x, y, 'ro', ms=10)

# Draw and label the gravity vector
gvec = FancyArrowPatch((x, y), (x, y-2), arrowstyle='->', mutation_scale=10, linewidth=LW, color='k')
lv_line = plt.Line2D((x, x), (y, y-2), visible=False)  # Local vertical
ax.add_patch(gvec)
plt.text(x - 0.5, y-1, 'g')

# Draw and label the velocity vector
dx = 2
dy = dy_dx * dx
vvec = FancyArrowPatch((x, y), (x+dx, y+dy), arrowstyle='->', mutation_scale=10, linewidth=LW, color='k')
ax.add_patch(vvec)
plt.text(x+dx-0.25, y+dy-0.25, 'v')

# Draw angle theta
vvec_line = plt.Line2D((x, x+dx), (y, y+dy), visible=False)
# angle_plot = get_angle_plot(lv_line, vvec_line, color='k', origin=(x, y), radius=3)
# ax.add_patch(angle_plot)
ax.text(x+0.25, y-1.25, r'$\theta$')

# Draw the axes
x = 0
y = 2
dx = 5
dy = 0
xhat = FancyArrowPatch((x, y), (x+dx, y+dy), arrowstyle='->', mutation_scale=10, linewidth=LW, color='k')
ax.add_patch(xhat)
plt.text(x+dx/2.0-0.5, y+dy/2.0-0.5, 'x')

dx = 0
dy = 5
yhat = FancyArrowPatch((x, y), (x+dx, y+dy), arrowstyle='->', mutation_scale=10, linewidth=LW, color='k')
ax.add_patch(yhat)
plt.text(x+dx/2.0-0.5, y+dy/2.0-0.5, 'y')

plt.ylim(1, 11)
plt.xlim(-0.5, 10.5)

plt.savefig('brachistochrone_fbd.png')

plt.show()
```

## State variables

In this implementation, three _state_ variables are used to define the configuration of the system at any given instant in time.

- **x**: The horizontal position of the particle at an instant in time.
- **y**: The vertical position of the particle at an instant in time.
- **v**: The speed of the particle at an instant in time.

## System dynamics

From the free-body diagram above, the evolution of the state variables is given by the following ordinary differential equations (ODE).

\begin{align}
    \frac{d x}{d t} &= v \sin(\theta) \\
    \frac{d y}{d t} &= -v \cos(\theta) \\
    \frac{d v}{d t} &= g \cos(\theta)
\end{align}

## Control variables

This system has a single control variable.

- **$\theta$**: The angle between the gravity vector and the tangent to the curve at the current instant in time.

## The initial and final conditions

In this case, starting point **A** is given as _(0, 10)_.
The point moving along the curve will begin there with zero initial velocity.

The initial conditions are:

\begin{align}
    x_0 &= 0 \\
    y_0 &= 10 \\
    v_0 &= 0
\end{align}

The end point **B** is given as _(10, 5)_.
The point will end there, but the velocity at that point is not constrained.

The final conditions are:

\begin{align}
    x_f &= 10 \\
    y_f &= 5 \\
    v_f &= \mathrm{free}
\end{align}

## Defining the ODE as an OpenMDAO System

In Dymos, the ODE is an OpenMDAO System (a Component, or a Group of components).
The following ExplicitComponent computes the state rates for the brachistochrone problem.

More detail on the workings of an ExplicitComponent can be found in the OpenMDAO documentation.  In summary:

- **initialize**:  Called at setup, and used to define options for the component.  **ALL** Dymos ODE components should have the property `num_nodes`, which defines the number of points at which the outputs are simultaneously computed.
- **setup**: Used to add inputs and outputs to the component, and declare which outputs (and indices of outputs) are dependent on each of the inputs.
- **compute**: Used to compute the outputs, given the inputs.
- **compute_partials**: Used to compute the derivatives of the outputs w.r.t. each of the inputs analytically.  This method may be omitted if finite difference or complex-step approximations are used, though analytic is recommended.

```python
import numpy as np
import openmdao.api as om

class BrachistochroneODE(om.ExplicitComponent):

    def initialize(self):
        self.options.declare('num_nodes', types=int)
        self.options.declare('static_gravity', types=(bool,), default=False,
                             desc='If True, treat gravity as a static (scalar) input, rather than '
                                  'having different values at each node.')

    def setup(self):
        nn = self.options['num_nodes']

        # Inputs
        self.add_input('v', val=np.zeros(nn), desc='velocity', units='m/s')

        if self.options['static_gravity']:
            self.add_input('g', val=9.80665, desc='grav. acceleration', units='m/s/s',
                           tags=['dymos.static_target'])
        else:
            self.add_input('g', val=9.80665 * np.ones(nn), desc='grav. acceleration', units='m/s/s')

        self.add_input('theta', val=np.ones(nn), desc='angle of wire', units='rad')

        self.add_output('xdot', val=np.zeros(nn), desc='velocity component in x', units='m/s',
                        tags=['dymos.state_rate_source:x', 'dymos.state_units:m'])

        self.add_output('ydot', val=np.zeros(nn), desc='velocity component in y', units='m/s',
                        tags=['dymos.state_rate_source:y', 'dymos.state_units:m'])

        self.add_output('vdot', val=np.zeros(nn), desc='acceleration magnitude', units='m/s**2',
                        tags=['dymos.state_rate_source:v', 'dymos.state_units:m/s'])

        self.add_output('check', val=np.zeros(nn), desc='check solution: v/sin(theta) = constant',
                        units='m/s')

        # Setup partials
        arange = np.arange(self.options['num_nodes'])
        self.declare_partials(of='vdot', wrt='theta', rows=arange, cols=arange)

        self.declare_partials(of='xdot', wrt='v', rows=arange, cols=arange)
        self.declare_partials(of='xdot', wrt='theta', rows=arange, cols=arange)

        self.declare_partials(of='ydot', wrt='v', rows=arange, cols=arange)
        self.declare_partials(of='ydot', wrt='theta', rows=arange, cols=arange)

        self.declare_partials(of='check', wrt='v', rows=arange, cols=arange)
        self.declare_partials(of='check', wrt='theta', rows=arange, cols=arange)

        if self.options['static_gravity']:
            c = np.zeros(self.options['num_nodes'])
            self.declare_partials(of='vdot', wrt='g', rows=arange, cols=c)
        else:
            self.declare_partials(of='vdot', wrt='g', rows=arange, cols=arange)

    def compute(self, inputs, outputs):
        theta = inputs['theta']
        cos_theta = np.cos(theta)
        sin_theta = np.sin(theta)
        g = inputs['g']
        v = inputs['v']

        outputs['vdot'] = g * cos_theta
        outputs['xdot'] = v * sin_theta
        outputs['ydot'] = -v * cos_theta
        outputs['check'] = v / sin_theta

    def compute_partials(self, inputs, partials):
        theta = inputs['theta']
        cos_theta = np.cos(theta)
        sin_theta = np.sin(theta)
        g = inputs['g']
        v = inputs['v']

        partials['vdot', 'g'] = cos_theta
        partials['vdot', 'theta'] = -g * sin_theta

        partials['xdot', 'v'] = sin_theta
        partials['xdot', 'theta'] = v * cos_theta

        partials['ydot', 'v'] = -cos_theta
        partials['ydot', 'theta'] = v * sin_theta

        partials['check', 'v'] = 1 / sin_theta
        partials['check', 'theta'] = -v * cos_theta / sin_theta ** 2
```

```{admonition} "Things to note about the ODE system"
- There is no input for the position states ($x$ and $y$).  The dynamics aren't functions of these states, so they aren't needed as inputs.
- While $g$ is an input to the system, since it will never change throughout the trajectory, it can be an option on the system.  This way we don't have to define any partials w.r.t. $g$.
- The output `check` is an _auxiliary_ output, not a rate of the state variables.  In this case, optimal control theory tells us that `check` should be constant throughout the trajectory, so it's a useful output from the ODE.
```

## Testing the ODE

Now that the ODE system is defined, it is strongly recommended to test the analytic partials before using it in optimization.
If the partials are incorrect, then the optimization will almost certainly fail.
Fortunately, OpenMDAO makes testing derivatives easy with the `check_partials` method.
The `assert_check_partials` method in `openmdao.utils.assert_utils` can be used in test frameworks to verify the correctness of the partial derivatives in a model.

The following is a test method which creates a new OpenMDAO problem whose model contains the ODE class.
The problem is setup with the `force_alloc_complex=True` argument to enable complex-step approximation of the derivatives.
Complex step typically produces derivative approximations with an error on the order of 1.0E-16, as opposed to ~1.0E-6 for forward finite difference approximations.

```python
import numpy as np
import openmdao.api as om

num_nodes = 5

p = om.Problem(model=om.Group())

ivc = p.model.add_subsystem('vars', om.IndepVarComp())
ivc.add_output('v', shape=(num_nodes,), units='m/s')
ivc.add_output('theta', shape=(num_nodes,), units='deg')

p.model.add_subsystem('ode', BrachistochroneODE(num_nodes=num_nodes))

p.model.connect('vars.v', 'ode.v')
p.model.connect('vars.theta', 'ode.theta')

p.setup(force_alloc_complex=True)

p.set_val('vars.v', 10*np.random.random(num_nodes))
p.set_val('vars.theta', 10*np.random.uniform(1, 179, num_nodes))

p.run_model()
cpd = p.check_partials(method='cs', compact_print=True)
```

```python
from dymos.utils.testing_utils import assert_check_partials

assert_check_partials(cpd)
```

## Solving the problem with Legendre-Gauss-Lobatto collocation in Dymos

The following script fully defines the brachistochrone problem with Dymos and solves it.  In this section we'll walk through each step.

```python
import openmdao.api as om
import dymos as dm
from dymos.examples.plotting import plot_results
from dymos.examples.brachistochrone import BrachistochroneODE
import matplotlib.pyplot as plt

#
# Initialize the Problem and the optimization driver
#
p = om.Problem(model=om.Group())
p.driver = om.ScipyOptimizeDriver()
p.driver.declare_coloring()

#
# Create a trajectory and add a phase to it
#
traj = p.model.add_subsystem('traj', dm.Trajectory())

phase = traj.add_phase('phase0',
                       dm.Phase(ode_class=BrachistochroneODE,
                                transcription=dm.GaussLobatto(num_segments=10)))

#
# Set the variables
#
phase.set_time_options(fix_initial=True, duration_bounds=(.5, 10))

phase.add_state('x', fix_initial=True, fix_final=True)

phase.add_state('y', fix_initial=True, fix_final=True)

phase.add_state('v', fix_initial=True, fix_final=False)

phase.add_control('theta', continuity=True, rate_continuity=True,
                  units='deg', lower=0.01, upper=179.9)

phase.add_parameter('g', units='m/s**2', val=9.80665)

#
# Minimize time at the end of the phase
#
phase.add_objective('time', loc='final', scaler=10)

p.model.linear_solver = om.DirectSolver()

#
# Setup the Problem
#
p.setup()

#
# Set the initial values
#
p['traj.phase0.t_initial'] = 0.0
p['traj.phase0.t_duration'] = 2.0

p.set_val('traj.phase0.states:x', phase.interp('x', ys=[0, 10]))
p.set_val('traj.phase0.states:y', phase.interp('y', ys=[10, 5]))
p.set_val('traj.phase0.states:v', phase.interp('v', ys=[0, 9.9]))
p.set_val('traj.phase0.controls:theta', phase.interp('theta', ys=[5, 100.5]))

#
# Solve for the optimal trajectory
#
dm.run_problem(p)

# Check the results
print(p.get_val('traj.phase0.timeseries.time')[-1])
```

```python
from openmdao.utils.assert_utils import assert_near_equal

assert_near_equal(p.get_val('traj.phase0.timeseries.time')[-1], 1.8016, tolerance=1.0E-3)
```

```python
# Generate the explicitly simulated trajectory
exp_out = traj.simulate()

plot_results([('traj.phase0.timeseries.states:x', 'traj.phase0.timeseries.states:y',
               'x (m)', 'y (m)'),
              ('traj.phase0.timeseries.time', 'traj.phase0.timeseries.controls:theta',
               'time (s)', 'theta (deg)')],
             title='Brachistochrone Solution\nHigh-Order Gauss-Lobatto Method',
             p_sol=p, p_sim=exp_out)

plt.show()
```

(examples:brachistochrone:explicit_shooting)=
## Solving the problem with single shooting in Dymos

The following script fully defines the brachistochrone problem with Dymos and solves it using a single explicit shooting method.

The code is nearly identical to that using the collocation approach.
Key differences are shown when defining the transcription, specifying how to constraint the final state values of `x` and `y` states, and providing initial guesses for all states.

```python
import openmdao.api as om
import dymos as dm
from dymos.examples.plotting import plot_results
from dymos.examples.brachistochrone import BrachistochroneODE
import matplotlib.pyplot as plt

#
# Initialize the Problem and the optimization driver
#
p = om.Problem(model=om.Group())
p.driver = om.ScipyOptimizeDriver()

# We'll try to use coloring, but OpenMDAO will tell us that it provides no benefit.
p.driver.declare_coloring()

#
# Create a trajectory and add a phase to it
#
traj = p.model.add_subsystem('traj', dm.Trajectory())

phase = traj.add_phase('phase0',
                       dm.Phase(ode_class=BrachistochroneODE,
                                transcription=dm.ExplicitShooting(num_segments=10,
                                                                  num_steps_per_segment=10,
                                                                  method='rk4')))

#
# Set the variables
#
phase.set_time_options(fix_initial=True, duration_bounds=(.5, 10))

# Note, we cannot use fix_final=True with the shooting method
# because the final value of the states are
# not design variables in the transcribed optimization problem.
phase.add_state('x', fix_initial=True)

phase.add_state('y', fix_initial=True)

phase.add_state('v', fix_initial=True)

phase.add_control('theta', continuity=True, rate_continuity=True,
                  units='deg', lower=0.01, upper=179.9)

phase.add_parameter('g', units='m/s**2', val=9.80665)

#
# Minimize time at the end of the phase
#
phase.add_objective('time', loc='final', scaler=10)

#
# Add boundary constraints for x and y since we could not use `fix_final=True`
#
phase.add_boundary_constraint('x', loc='final', equals=10)
phase.add_boundary_constraint('y', loc='final', equals=5)

p.model.linear_solver = om.DirectSolver()

#
# Setup the Problem
#
p.setup()

#
# Set the initial values
#
p['traj.phase0.t_initial'] = 0.0
p['traj.phase0.t_duration'] = 2.0

# Only the initial values of the states are design variables,
# so the phase interp method is not used on states.
p.set_val('traj.phase0.states:x', 0.0)
p.set_val('traj.phase0.states:y', 10.0)
p.set_val('traj.phase0.states:v', 0.0)
p.set_val('traj.phase0.controls:theta', phase.interp('theta', ys=[5, 100.5]))

#
# Solve for the optimal trajectory
#
dm.run_problem(p)

# Check the results
print(p.get_val('traj.phase0.timeseries.time')[-1])
```

```python
assert_near_equal(p.get_val('traj.phase0.timeseries.time')[-1], 1.8016, tolerance=1.0E-3)
```

```python
plot_results([('traj.phase0.timeseries.states:x', 'traj.phase0.timeseries.states:y',
               'x (m)', 'y (m)'),
              ('traj.phase0.timeseries.time', 'traj.phase0.timeseries.controls:theta',
               'time (s)', 'theta (deg)')],
             title='Brachistochrone Solution\nExplicit Shooting Method',
             p_sol=p)

plt.show()
```

Note that the shooting methods provide timeseries values at the start and end of each segment.
Due to plans to enable variable step integration in the future, Dymos cannot provide outputs for each individual step since the number of steps will not be known when setting up the problem.

Because control values and rates may not be continuous across segment bounds, depending on the settings used, there may be discontinuities in ODE outputs and therefore timeseries values are provided on each side of each segment boundary.


# Jupyter CheatSheet

```python

```

```python

```

```python

```

```python

```

```python

```

```python
# Code
a, b = 3, 4
a + b
# print("a+b", a+b)
```

    7

[Markdown basic syntax](https://www.markdownguide.org/basic-syntax/)

# Markdown Heading 1
## Heading 1.1
### Heading 1.1.1

Bold text: **Bold Text**

Italic text: *Italic Text*

## LaTex Equations
\begin{equation}
e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i
\end{equation}

## HTML Table
| index | number |
|------|------|
| 0 | 11.2 |
| 1 | 36.8 |

# Image

# Link Cells
Firstly, define the destination in the cell you want to link with a `HTML` anchor tag and give it an `Id`.
For exmaple, I have added ```<a id='another_cell'></a>``` in the cell above.

Secondly, create the internal hyperlink to the destination created above using Markdown syntax in another cell and run it.

[Image Cell](#another_cell)

# Lists
- item1
- itme2
- itme3

1. item1
2. item2
3. item3

```python
def f(x):
    """a docstring"""
    return x**2

f(4)
```

    16

def f(x):
    """a docstring"""
    return x**2
# Blockquotes

> This is a quotation....

# jupyter notebook to Gitbook

gitbook also supports Markdown syntax. we can use export jupyter notebook to `.md` and add it to gitbook to avoid duplicated work and the user-unfriendly UI.

`$ jupyter nbconvert py_examples.ipynb --to md`

```python

```


```
#hide
#skip
! [ -e /content ] && pip install -Uqq self-supervised
```

```
#default_exp vision.supcon
```

# SupCon

> **SupCon**: [Supervised Contrastive Learning](https://arxiv.org/pdf/2004.11362.pdf)

```
#export
_all_ = ['UnsupMethod']
```

```
#export
from fastai.vision.all import *
from self_supervised.augmentations import *
from self_supervised.layers import *
```

## Algorithm

#### SupCon

SupCon introduces a generalized form for contrastive losses, and shows that self-supervised loss from SimCLR, N-Pair loss and triplet margin loss are special cases. In this repo, we leverage this general form and formalize a loss that can transition from self-supervised contrastive loss to supervised contrastive loss:

$$
\begin{equation}
\begin{split}
\mathcal{L} & = \hspace{30mm} \mathcal{L}^{unsup} \hspace{16mm}+ \hspace{30mm} \lambda\mathcal{L}^{sup} \\
            \\
            & = - \sum_{i \in I} \log \frac{\exp \left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{j(i)} / \tau\right)}{\sum_{a \in A(i)} \exp \left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{a} / \tau\right)} + \lambda \sum_{i \in I} \frac{-1}{|P(i)|} \sum_{p \in P(i)} \log \frac{\exp \left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{p} / \tau\right)}{\sum_{a \in A(i)} \exp \left(\boldsymbol{z}_{i} \cdot \boldsymbol{z}_{a} / \tau\right)}
\end{split}
\end{equation}
$$

We use supervised signal as regularization which has an associated weight $\lambda$, this allows to pretrain with a dataset mixed with labelled and unlabelled data. This regularization can help to learn more generic features which in turn can help with the downstream task. In `SupCon` callback you can choose to use all samples (`UnsupMethod.All`) for unsupervised loss or only use the ones doesn't have a label (`UnsupMethod.All`). Supervised loss will use the samples with labels.

Therefore, positive samples come form two disjoint categories:

    (1) Other view of the anchor sample after augmentation (self-supervised case)
    (2) All views of the samples that have the same class id with anchor, including the other view of the same sample (supervised case)

*Note that self-supervised and unsupervised are used interchangably in this context*

SimCLR model consists of an `encoder` and a `projector (MLP)` layer. The definition of this module is fairly simple as below.

```
#export
class SupConModel(Module):
    "Compute predictions of concatenated xi and xj"
    def __init__(self,encoder,projector): self.encoder,self.projector = encoder,projector
    def forward(self,x): return self.projector(self.encoder(x))
```

Instead of directly using `SupConModel` by passing both an `encoder` and a `projector`, `create_simclr_model` function can be used by minimally passing a predefined `encoder` and the expected input channels.

```
#export
def create_supcon_model(encoder, hidden_size=256, projection_size=128, bn=False, nlayers=2):
    "Create SupCon model"
    n_in  = in_channels(encoder)
    with torch.no_grad(): representation = encoder(torch.randn((2,n_in,128,128)))
    projector = create_mlp_module(representation.size(1), hidden_size, projection_size, bn=bn, nlayers=nlayers)
    apply_init(projector)
    return SupConModel(encoder, projector)
```

You can use `self_supervised.layers` module to create an encoder. It supports all **timm** and **fastai** models available out of the box.

We define number of input channels with `n_in`, projector/mlp's hidden size with `hidden_size`,  projector/mlp's final projection size with `projection_size` and projector/mlp's number of layers with `nlayers`.

```
encoder = create_encoder("tf_efficientnet_b0_ns", n_in=3, pretrained=False, pool_type=PoolingType.CatAvgMax)
model = create_supcon_model(encoder, hidden_size=2048, projection_size=128, nlayers=2)
out = model(torch.randn((2,3,224,224))); out.shape
```

    torch.Size([2, 128])

## SupCon Callback

The following parameters can be passed;

- **aug_pipelines** list of augmentation pipelines List[Pipeline] created using functions from `self_supervised.augmentations` module. Each `Pipeline` should be set to `split_idx=0`. You can simply use `get_supcon_aug_pipelines` utility to get aug_pipelines.
- **temp** temperature scaling for cross entropy loss (defaults to paper's best value)

SupCon algorithm uses 2 views of a given image, and `SupCon` callback expects a list of 2 augmentation pipelines in `aug_pipelines`.

You can simply use helper function `get_supcon_aug_pipelines()` which will allow augmentation related arguments such as size, rotate, jitter...and will return a list of 2 pipelines, which then can be passed to the callback. This function uses `get_multi_aug_pipelines` which then `get_batch_augs`. For more information you may refer to `self_supervised.augmentations` module.

Also, you may choose to pass your own list of aug_pipelines which needs to be List[Pipeline, Pipeline] where Pipeline(..., split_idx=0). Here, `split_idx=0` forces augmentations to be applied in training mode.

```
#export
@delegates(get_multi_aug_pipelines)
def get_supcon_aug_pipelines(size, **kwargs): return get_multi_aug_pipelines(n=2, size=size, **kwargs)
```

```
aug_pipelines = get_supcon_aug_pipelines(size=28, rotate=False, jitter=False, bw=False, blur=False, stats=None, cuda=False)
aug_pipelines
```

    [Pipeline: RandomResizedCrop -> RandomHorizontalFlip,
     Pipeline: RandomResizedCrop -> RandomHorizontalFlip]

```
#export
mk_class('UnsupMethod', **{o:o.lower() for o in ['All', 'Only']},
         doc="Whether to use all (sup+unsup) or only unsup data in a batch for unsup loss")
```

```
#export
class SupCon(Callback):
    order,run_valid = 9,True
    def __init__(self, aug_pipelines, unsup_class_id, unsup_method=UnsupMethod.All,
                 reg_lambda=1.,
                 temp=0.07,
                 print_augs=False):
        assert_aug_pipelines(aug_pipelines)
        self.aug1, self.aug2 = aug_pipelines
        if print_augs: print(self.aug1), print(self.aug2)
        store_attr('unsup_class_id,unsup_method,reg_lambda,temp')

    def before_fit(self):
        self.learn.loss_func = self.lf

    def before_batch(self):
        xi,xj = self.aug1(self.x), self.aug2(self.x)
        self.learn.xb = (torch.cat([xi, xj]),)

    def _remove_diag(self, x):
        bs = x.shape[0]
        return x[~torch.eye(bs).bool()].reshape(bs,bs-1)

    def unsup_lf(self, pred, yb):
        "Self-Supervised contrasitve loss with all or only unlabelled batch samples"
        targ = torch.cat([yb,yb])
        unsup_mask = (targ == self.unsup_class_id)
        if self.unsup_method == UnsupMethod.All:
            pass
        elif self.unsup_method == UnsupMethod.Only:
            pred = pred[unsup_mask]
        else:
            raise Exception(f"{self.unsup_method} is not a valid UnsupMethod")

        if len(pred) == 0: return 0

        pred = F.normalize(pred, dim=1)
        bs = pred.shape[0]
        targ = torch.arange(bs, device=pred.device).roll(bs//2)
        sim  = self._remove_diag(pred @ pred.T) / self.temp
        targ = self._remove_diag(torch.eye(targ.shape[0], device=pred.device)[targ]).nonzero()[:,-1]
        return F.cross_entropy(sim, targ)

    def sup_lf(self, pred, yb):
        "Supervised contrasitve loss with labelled batch samples"
        targ = torch.cat([yb,yb])
        unsup_mask = (targ == self.unsup_class_id)
        pred = pred[~unsup_mask]
        targ = targ[~unsup_mask]

        if len(pred) == 0: return 0

        # exclude anchor from loss calc
        ohe_labels = (targ[...,None] == targ[None, ...]).float()
        pred = F.normalize(pred, dim=1)
        sim  = self._remove_diag(pred @ pred.T) / self.temp
        targ = self._remove_diag(ohe_labels)

        return (F.cross_entropy(sim, targ, reduction='none')/targ.sum(1)).mean()

    def lf(self, pred, *yb):
        unsup_loss = self.unsup_lf(pred, *yb)
        sup_loss   = self.sup_lf(pred, *yb)
        return unsup_loss + self.reg_lambda*sup_loss

    @torch.no_grad()
    def show(self, n=1):
        bs = self.learn.x.size(0)//2
        x1,x2  = self.learn.x[:bs], self.learn.x[bs:]
        idxs = np.random.choice(range(bs),n,False)
        x1 = self.aug1.decode(x1[idxs].to('cpu').clone()).clamp(0,1)
        x2 = self.aug2.decode(x2[idxs].to('cpu').clone()).clamp(0,1)
        images = []
        for i in range(n): images += [x1[i],x2[i]]
        return show_batch(x1[0], None, images, max_n=len(images), nrows=n)
```

### Tests

```
# No unsupervised, all samples are labelled, but use all for unsup loss
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="all")
yb = torch.tensor([1,1,2,2])
pred = torch.randn((yb.shape[0]*2,128))
```

```
loss1 = supcon.unsup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
loss2 = torch.mean(tensor([nll[i,idx] for i, idx in enumerate([3,4,5,6,0,1,2,3])]))
assert torch.isclose(loss1,loss2)
```

```
loss1 = supcon.sup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
ohe = supcon._remove_diag(tensor([[1,1,0,0,1,1,0,0],
                                 [1,1,0,0,1,1,0,0],
                                 [0,0,1,1,0,0,1,1],
                                 [0,0,1,1,0,0,1,1],
                                 [1,1,0,0,1,1,0,0],
                                 [1,1,0,0,1,1,0,0],
                                 [0,0,1,1,0,0,1,1],
                                 [0,0,1,1,0,0,1,1]]))
loss2 = (tensor([(row[idxs.bool()].sum()/idxs.sum()) for row, idxs in zip(nll, ohe)])).mean()
assert torch.isclose(loss1, loss2)
```

```
# No unsupervised, all samples are labelled, use only unlabelled for unsup loss
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="only")
yb = torch.tensor([1,1,2,2])
pred = torch.randn((yb.shape[0]*2,128))
```

```
loss1 = supcon.unsup_lf(pred, yb)
assert loss1 == 0
```

```
loss1 = supcon.sup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
ohe = supcon._remove_diag(tensor([[1,1,0,0,1,1,0,0],
                                 [1,1,0,0,1,1,0,0],
                                 [0,0,1,1,0,0,1,1],
                                 [0,0,1,1,0,0,1,1],
                                 [1,1,0,0,1,1,0,0],
                                 [1,1,0,0,1,1,0,0],
                                 [0,0,1,1,0,0,1,1],
                                 [0,0,1,1,0,0,1,1]]))
loss2 = (tensor([(row[idxs.bool()].sum()/idxs.sum()) for row, idxs in zip(nll, ohe)])).mean()
assert torch.isclose(loss1,loss2)
```

```
# All samples are unlabelled, but use all for unsup loss
yb = torch.tensor([0,0,0,0])
pred = torch.randn((yb.shape[0]*2,128))
```

```
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="all")
loss1 = supcon.unsup_lf(pred, yb)
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="only")
loss2 = supcon.unsup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
loss3 = torch.mean(tensor([nll[i,idx] for i, idx in enumerate([3,4,5,6,0,1,2,3])]))
assert torch.isclose(loss1, loss2) and torch.isclose(loss2, loss3)
```

```
loss1 = supcon.sup_lf(pred, yb)
assert loss1 == 0
```

```
# Mixed samples, but use all for unsup loss
yb = torch.tensor([1,1,2,0])
pred = torch.randn((yb.shape[0]*2,128))
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="all")
```

```
loss1 = supcon.unsup_lf(pred, yb)
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
loss2 = torch.mean(tensor([nll[i,idx] for i, idx in enumerate([3,4,5,6,0,1,2,3])]))
assert torch.isclose(loss1, loss2)
```

```
supcon = SupCon([Pipeline([],0),Pipeline([],0)], unsup_class_id=0, unsup_method="only")
loss1 = supcon.unsup_lf(pred, yb)
assert loss1 == 0 # log(1) -> 0, there is no negative sample
```

```
loss1 = supcon.sup_lf(pred, yb)
targ = torch.cat([yb,yb])
unsup_mask = (targ == supcon.unsup_class_id)
pred = pred[~unsup_mask]
nll = -(supcon._remove_diag(F.normalize(pred) @ F.normalize(pred).T)/supcon.temp).log_softmax(1)
ohe = supcon._remove_diag(tensor([[1,1,0,1,1,0],
                                  [1,1,0,1,1,0],
                                  [0,0,1,0,0,1],
                                  [1,1,0,1,1,0],
                                  [1,1,0,1,1,0],
                                  [0,0,1,0,0,1]]))

loss2 = (tensor([(row[idxs.bool()].sum()/idxs.sum()) for row, idxs in zip(nll, ohe)])).mean()
assert torch.isclose(loss1, loss2)
```

## SupConMOCO Callback [Experimental]

The following parameters can be passed;

- **aug_pipelines** list of augmentation pipelines List[Pipeline] created using functions from `self_supervised.augmentations` module. Each `Pipeline` should be set to `split_idx=0`. You can simply use `get_supcon_aug_pipelines` utility to get aug_pipelines.
- **temp** temperature scaling for cross entropy loss (defaults to paper's best value)

SupCon algorithm uses 2 views of a given image, and `SupCon` callback expects a list of 2 augmentation pipelines in `aug_pipelines`.

You can simply use helper function `get_supcon_aug_pipelines()` which will allow augmentation related arguments such as size, rotate, jitter...and will return a list of 2 pipelines, which then can be passed to the callback. This function uses `get_multi_aug_pipelines` which then `get_batch_augs`. For more information you may refer to `self_supervised.augmentations` module.

Also, you may choose to pass your own list of aug_pipelines which needs to be List[Pipeline, Pipeline] where Pipeline(..., split_idx=0). Here, `split_idx=0` forces augmentations to be applied in training mode.

```
#export
class SupConMOCO(Callback):
    order,run_valid = 9,True
    def __init__(self, aug_pipelines, unsup_class_id, unsup_method=UnsupMethod.All,
                 K=4096,
                 m=0.999,
                 reg_lambda=1.,
                 temp=0.07,
                 print_augs=False):
        assert_aug_pipelines(aug_pipelines)
        self.aug1, self.aug2 = aug_pipelines
        if print_augs: print(self.aug1), print(self.aug2)
        store_attr('unsup_class_id,unsup_method,K,m,reg_lambda,temp')

    def before_fit(self):
        "Create key encoder and init queue"
        if (not hasattr(self, "encoder_k")) and (not hasattr(self, "queue")):
            # init key encoder
            self.encoder_k = deepcopy(self.learn.model).to(self.dls.device)
            for param_k in self.encoder_k.parameters(): param_k.requires_grad = False

            # init queue
            nf = self.learn.model.projector[-1].out_features
            self.emb_queue = torch.randn(self.K, nf).to(self.dls.device)
            self.emb_queue = nn.functional.normalize(self.emb_queue, dim=1)
            self.label_queue = torch.zeros(self.K).to(self.dls.device) + self.unsup_class_id
            self.queue_ptr = 0
        else:
            warnings.warn("Key encoder and queue are already defined, keeping them.")

        self.learn.loss_func = self.lf

    def before_batch(self):
        "Generate query and key for the current batch"
        q_img,k_img = self.aug1(self.x), self.aug2(self.x.clone())
        self.learn.xb = (q_img,)
        with torch.no_grad():
            self.encoder_k.eval()
            self.learn.yb = (F.normalize(self.encoder_k(k_img)), self.y) # query and labels

    @torch.no_grad()
    def _momentum_update_key_encoder(self):
        for param_q, param_k in zip(self.learn.model.parameters(), self.encoder_k.parameters()):
            param_k.data = param_k.data * self.m + param_q.data * (1. - self.m)

    @torch.no_grad()
    def _dequeue_and_enqueue(self):
        bs = self.x.size(0)
        key_embs, key_labels = self.yb
        assert self.K % bs == 0  # for simplicity
        self.emb_queue[self.queue_ptr:self.queue_ptr+bs, :] = key_embs
        self.label_queue[self.queue_ptr:self.queue_ptr+bs] = key_labels
        self.queue_ptr = (self.queue_ptr + bs) % self.K  # move pointer

    def unsup_lf(self, pred, key_embs, key_labels):
        "Self-Supervised contrasitve loss with all or only unlabelled batch samples"
        query_embs = F.normalize(pred, dim=1)
        queue_embs, queue_labels = self.emb_queue, self.label_queue

        if self.unsup_method == UnsupMethod.All:
            key_embs = torch.cat([key_embs, queue_embs])

        elif self.unsup_method == UnsupMethod.Only:
            query_embs = query_embs[(key_labels == self.unsup_class_id)]
            key_embs   = key_embs[(key_labels == self.unsup_class_id)]
            queue_embs = queue_embs[(queue_labels == self.unsup_class_id)]

            key_embs   = torch.cat([key_embs, queue_embs])

        else:
            raise Exception(f"{self.unsup_method} is not a valid UnsupMethod")

        if len(query_embs) == 0: return 0

        labels = torch.arange(len(query_embs), device=pred.device)
        sim  = query_embs @ key_embs.T / self.temp
        return F.cross_entropy(sim, labels)

    def sup_lf(self, pred, key_embs, key_labels):
        "Supervised contrasitve loss with labelled batch samples"
        query_embs = F.normalize(pred, dim=1)
        queue_embs, queue_labels = self.emb_queue,  self.label_queue

        query_embs = query_embs[(key_labels != self.unsup_class_id)]
        key_embs   = key_embs[(key_labels != self.unsup_class_id)]
        queue_embs = queue_embs[(queue_labels != self.unsup_class_id)]

        key_embs   = torch.cat([key_embs, queue_embs])

        key_labels   = key_labels[(key_labels != self.unsup_class_id)]
        queue_labels = queue_labels[(queue_labels != self.unsup_class_id)]

        labels   = torch.cat([key_labels, queue_labels])

        if len(query_embs) == 0: return 0

        # exclude anchor from loss calc
        ohe_labels = (key_labels[...,None] == labels[None, ...]).float()
        sim  = query_embs @ key_embs.T / self.temp
        return (F.cross_entropy(sim, ohe_labels, reduction='none')/ohe_labels.sum(1)).mean()

    def lf(self, pred, *yb):
        unsup_loss = self.unsup_lf(pred, *yb)
        sup_loss   = self.sup_lf(pred, *yb)
        return unsup_loss + self.reg_lambda*sup_loss

    def after_step(self):
        "Update momentum (key) encoder and queue"
        self._momentum_update_key_encoder()
        self._dequeue_and_enqueue()

    @torch.no_grad()
    def show(self, n=1):
        bs = self.learn.x.size(0)//2
        x1,x2  = self.learn.x[:bs], self.learn.x[bs:]
        idxs = np.random.choice(range(bs),n,False)
        x1 = self.aug1.decode(x1[idxs].to('cpu').clone()).clamp(0,1)
        x2 = self.aug2.decode(x2[idxs].to('cpu').clone()).clamp(0,1)
        images = []
        for i in range(n): images += [x1[i],x2[i]]
        return show_batch(x1[0], None, images, max_n=len(images), nrows=n)
```

### Tests

```
from fastai.test_utils import *
```

```
class ContrastiveModel(Module):
    def __init__(self):
        self.encoder   = nn.Parameter(tensor([1.]))
        self.projector = nn.Linear(1,5, bias=False)
        self.projector.weight.data.zero_()
        self.projector.weight.data += 1
        self.projector = nn.Sequential(self.projector)

    def forward(self, x): return self.projector(x*self.encoder)
```

```
# No unsupervised, all samples are labelled, but use all for unsup loss
supcon = SupConMOCO([Pipeline([noop],0),Pipeline([noop],0)], unsup_class_id=0, unsup_method="all", K=8, m=0.999, reg_lambda=1.0, temp=0.07)
yb = torch.tensor([1,1,2,2])
pred = torch.randn((yb.shape[0]*2,128))
```

```
learner = synth_learner(cbs=supcon, data=synth_dbunch(a=0,b=0,bs=4), model=ContrastiveModel())
```

```
learner.sup_con_moco.aug1, learner.sup_con_moco.aug2
```

    (Pipeline: , Pipeline: )

```
learner.sup_con_moco.__dict__['__stored_args__']
```

    {'unsup_class_id': 0,
     'unsup_method': 'all',
     'K': 8,
     'm': 0.999,
     'reg_lambda': 1.0,
     'temp': 0.07}

```
learner('before_fit')
```

```
assert learner.sup_con_moco.emb_queue.shape == (8,5)
assert torch.all(learner.sup_con_moco.label_queue == torch.zeros(8))
assert not any(list(o.requires_grad for o in learner.sup_con_moco.encoder_k.parameters()))
assert torch.all(learner.sup_con_moco.encoder_k.projector[0].weight == 1)
```

```
b = tensor([1,1,-1,1]).reshape(-1,1),tensor([1,1,2,2])
learner._split(b)
learner('before_batch')
```

```
key_embs, labels = learner.sup_con_moco.yb
assert torch.equal(F.normalize(learner.sup_con_moco.encoder_k(b[0])), key_embs)
assert torch.equal(labels, b[1])
```

```
learner.model.encoder.data += 0.1 # pseudo param update 1.0 -> 1.1
learner.model.projector[0].weight.data += 0.1 # pseudo param update 1.0 -> 1.1
```

```
learner('after_step')
```

```
assert torch.equal(learner.sup_con_moco.emb_queue[:4], key_embs)
```

```
newval = 1*supcon.m + 1.1*(1-supcon.m)
assert torch.all(learner.sup_con_moco.encoder_k.encoder.data == newval) and torch.all(learner.sup_con_moco.encoder_k.projector[0].weight.data==newval)
```

```
b = tensor([-1,-1,1,-1]).reshape(-1,1),tensor([1,1,2,2])
learner._split(b)
learner('before_batch')
```

```
key_embs, labels = learner.sup_con_moco.yb
assert torch.equal(F.normalize(learner.sup_con_moco.encoder_k(b[0])), key_embs)
assert torch.equal(labels, b[1])
```

```
learner('after_step')
```

```
assert torch.equal(learner.sup_con_moco.emb_queue[-4:], key_embs)
assert torch.equal(learner.sup_con_moco.label_queue, tensor([1,1,2,2,1,1,2,2]).float())
```

```
newval = newval*supcon.m + 1.1*(1-supcon.m)
assert torch.all(learner.sup_con_moco.encoder_k.encoder.data == newval) and torch.all(learner.sup_con_moco.encoder_k.projector[0].weight.data==newval)
```

```
pred = F.normalize(learner.model(learner.x))
loss1 = learner.sup_con_moco.unsup_lf(pred, *learner.yb)
key_embs, labels = learner.yb
logits = pred @ torch.cat([key_embs,learner.sup_con_moco.emb_queue]).T / learner.sup_con_moco.temp
loss2 = F.cross_entropy(logits, tensor([0,1,2,3]))
assert loss1 == loss2
```

```
learner.sup_con_moco.unsup_method = UnsupMethod.Only
learner.sup_con_moco.unsup_class_id = 1
loss1 = learner.sup_con_moco.unsup_lf(pred, *learner.yb)
logits = pred[labels==1] @ torch.cat([key_embs[labels==1],learner.sup_con_moco.emb_queue[learner.sup_con_moco.label_queue==1]]).T / learner.sup_con_moco.temp
loss2 = F.cross_entropy(logits, tensor([0,1]))
assert loss1 == loss2
```

```
learner.sup_con_moco.unsup_class_id = 0
pred = F.normalize(learner.model(learner.x))
loss1 = learner.sup_con_moco.sup_lf(pred, *learner.yb)
logits = pred @ torch.cat([key_embs,learner.sup_con_moco.emb_queue]).T / learner.sup_con_moco.temp
ohe_labels = tensor([[1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1,],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1]])
loss2 = (F.cross_entropy(logits, ohe_labels, reduction='none') / ohe_labels.sum(1)).mean()
assert loss1 == loss2
```

```
learner.sup_con_moco.unsup_class_id = 2
pred = F.normalize(learner.model(learner.x))
loss1 = learner.sup_con_moco.sup_lf(pred, *learner.yb)
logits = pred[labels != 2] @ torch.cat([key_embs[labels != 2],learner.sup_con_moco.emb_queue[learner.sup_con_moco.label_queue != 2]]).T / learner.sup_con_moco.temp
ohe_labels = tensor([[1., 1., 1., 1., 1., 1.],
                     [1., 1., 1., 1., 1., 1.]])
loss2 = (F.cross_entropy(logits, ohe_labels, reduction='none') / ohe_labels.sum(1)).mean()
assert loss1 == loss2
```

```
key_embs, labels = learner.yb
learner.sup_con_moco.unsup_class_id = 3
learner.xb = (torch.cat([learner.x, tensor([[0]])]),)
key_embs, labels = torch.cat([learner.y[0], learner.y[0][:1]]), torch.cat([learner.y[1],  tensor([3])])
learner.yb = (key_embs, labels)
pred = F.normalize(learner.model(learner.x))
loss1 = learner.sup_con_moco.sup_lf(pred, *learner.yb)
logits = pred[labels != 3] @ torch.cat([key_embs[labels != 3],learner.sup_con_moco.emb_queue[learner.sup_con_moco.label_queue != 3]]).T / learner.sup_con_moco.temp
ohe_labels = tensor([[1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1,],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1]])
loss2 = (F.cross_entropy(logits, ohe_labels, reduction='none') / ohe_labels.sum(1)).mean()
assert loss1 == loss2
```

```
key_embs, labels = learner.yb
learner.sup_con_moco.unsup_class_id = 3

learner.xb = (torch.cat([learner.x[:2],tensor([[0]]), learner.x[2:]]),)
key_embs, labels = torch.cat([learner.y[0][:2], learner.y[0][:1], learner.y[0][2:]]), torch.cat([learner.y[1][:2],  tensor([3]), learner.y[1][2:]])

learner.yb = (key_embs, labels)
pred = F.normalize(learner.model(learner.x))

loss1 = learner.sup_con_moco.sup_lf(pred, *learner.yb)
logits = pred[labels != 3] @ torch.cat([key_embs[labels != 3],learner.sup_con_moco.emb_queue[learner.sup_con_moco.label_queue != 3]]).T / learner.sup_con_moco.temp
ohe_labels = tensor([[1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [1., 1., 0, 0, 1., 1., 0, 0, 1., 1., 0, 0],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1,],
                     [0,  0,  1, 1, 0,  0,  1, 1, 0,  0,  1, 1]])
loss2 = (F.cross_entropy(logits, ohe_labels, reduction='none') / ohe_labels.sum(1)).mean()
assert loss1 == loss2
```

### Example Usage

```
path = untar_data(URLs.IMAGEWANG_160)
items = get_image_files(path)
```

```
items = np.random.choice(items, size=1000)
```

```
tds = Datasets(items, [[PILImage.create, ToTensor, RandomResizedCrop(112, min_scale=1.)],
                       [parent_label, Categorize()]], splits=RandomSplitter()(items))
dls = tds.dataloaders(bs=5, after_item=[ToTensor(), IntToFloatTensor()], device='cpu')
```

```
unsup_class_id = dls.vocab.o2i['unsup']
```

```
fastai_encoder = create_encoder('xresnet18', n_in=3, pretrained=False)
model = create_supcon_model(fastai_encoder, hidden_size=2048, projection_size=128)
aug_pipelines = get_supcon_aug_pipelines(size=28, rotate=False, jitter=False, bw=False, blur=False, stats=None, cuda=False)
learn = Learner(dls, model, cbs=[SupCon(aug_pipelines,
                                        unsup_class_id,
                                        unsup_method=UnsupMethod.All, reg_lambda=1.0, temp=0.07,
                                        print_augs=True),ShortEpochCallback(0.001)])
```

    Pipeline: RandomResizedCrop -> RandomHorizontalFlip
    Pipeline: RandomResizedCrop -> RandomHorizontalFlip

Also, with `show_one()` method you can inspect data augmentations as a sanity check. You can use existing augmentation functions from `augmentations` module.

```
b = dls.one_batch()
learn._split(b)
learn('before_batch')
axes = learn.sup_con.show(n=5)
```

```
learn.fit(1)
```

<style>
    /* Turns off some styling */
    progress {
        /* gets rid of default border in Firefox and Opera. */
        border: none;
        /* Needs to be in here for Safari polyfill so background images work as expected. */
        background-size: auto;
    }
    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {
        background: #F44336;
    }
</style>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>epoch</th>
      <th>train_loss</th>
      <th>valid_loss</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>0</td>
      <td>00:02</td>
    </tr>
  </tbody>
</table>

```
learn.recorder.losses
```

    [TensorCategory(1.7556)]

```
fastai_encoder = create_encoder('xresnet18', n_in=3, pretrained=False)
model = create_supcon_model(fastai_encoder, hidden_size=2048, projection_size=128)
aug_pipelines = get_supcon_aug_pipelines(size=28, rotate=False, jitter=False, bw=False, blur=False, stats=None, cuda=False)
learn = Learner(dls, model, cbs=[SupConMOCO(aug_pipelines,
                                            unsup_class_id,
                                            unsup_method=UnsupMethod.All, K=25, reg_lambda=1.0, temp=0.07,
                                            print_augs=True),ShortEpochCallback(0.001)])
```

    Pipeline: RandomResizedCrop -> RandomHorizontalFlip
    Pipeline: RandomResizedCrop -> RandomHorizontalFlip

```
learn.fit(1)
```

<style>
    /* Turns off some styling */
    progress {
        /* gets rid of default border in Firefox and Opera. */
        border: none;
        /* Needs to be in here for Safari polyfill so background images work as expected. */
        background-size: auto;
    }
    .progress-bar-interrupted, .progress-bar-interrupted::-webkit-progress-bar {
        background: #F44336;
    }
</style>

<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: left;">
      <th>epoch</th>
      <th>train_loss</th>
      <th>valid_loss</th>
      <th>time</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <td>0</td>
      <td>00:06</td>
    </tr>
  </tbody>
</table>

## Export -

```
#hide
from nbdev.export import notebook2script
notebook2script()
```

    Converted 01 - augmentations.ipynb.
    Converted 02 - layers.ipynb.
    Converted 03 - distributed.ipynb.
    Converted 10 - simclr.ipynb.
    Converted 11 - moco.ipynb.
    Converted 12 - byol.ipynb.
    Converted 13 - swav.ipynb.
    Converted 14 - barlow_twins.ipynb.
    Converted 15 - dino.ipynb.
    Converted 16 - supcon.ipynb.
    Converted 20 - clip.ipynb.
    Converted 21 - clip-moco.ipynb.
    Converted 70 - vision.metrics.ipynb.
    Converted 90 - models.vision_transformer.ipynb.
    Converted index.ipynb.

```

```


Sveučilište u Zagrebu

Fakultet elektrotehnike i računarstva

# Strojno učenje

http://www.fer.unizg.hr/predmet/su

Ak. god. 2015./2016.

# Bilježnica 8: Stroj potpornih vektora (SVM)

(c) 2015 Jan Šnajder

Verzija: 0.3 (2015-12-11)

```python
import scipy as sp
import scipy.stats as stats
import matplotlib.pyplot as plt
import pandas as pd
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

### Sadržaj:

* Uvod

* Problem maksimalne margine

* Optimizacija uz ograničenja

* Metoda Lagrangeovih multiplikatora

* Dualna formulacija problema maksimalne margine

* Meka margina

* Gubitak zglobnice

* Jezgreni trik

* Mercerove jezgre

* Optimizacija hiperparametara

# Uvod

* Vrlo učinkovit **diskriminativan model**

* Podsjetnik: Algoritam strojnog učenja definiran je
  1. modelom
  * pogreškom
  * optimizacijskim postupkom

* Model:

$$
    h(\mathbf{x}) = \mathbf{w}^\intercal\boldsymbol{\phi}(\mathbf{x})
$$

* Dakle, to je poopćeni linearan model bez aktivacijske funkcije $f$

* Gornja definicija modela je tzv. **primarna formulacija**

* Postoji i **dualna formulacija**:
  * Umjesto da značajke novog primjera množimo težinama $\mathbf{w}$, možemo za novi primjer izračunati koliko je sličan primjerima iz skupa za učenje i na temelju toga odrediti klasifikaciju
  * Model efektivno postaje **neparametarski**!

* U dualnoj formulaciji možemo iskoristiti tzv. **jezgreni trik**, koji nam omogućava jeftino preslikavanje primjera u prostor više dimenzije (a time i nelinearnost modela)
  * SVM je jedna vrsta tzv. **jezgrenog stroja** (engl. *kernel machine*)

* Model je jednostavan, no definicija pogreške i optimizacijskog postupka su nešto složeniji

* Osnovna ideja: primjere dviju klasa razdvojiti tako da je prostor između njih što veći $\Rightarrow$ tzv. **maksimalna margina**

* Opravdanje: generalizacija će biti najbolja onda kada granicu između klasa povućemo točno po sredini margine

* Do sada smo optimizacijski problem definirali tako da smo
  * krenuli od log-izglednosti pa izveli MLE (naivan Bayes)
  * krenuli od funkcije gubitka pa izveli funkciju pogreške (linearna regresija) i napravili analitičku minimizaciju
  * krenuli od funkcije pogreške pa izveli funkciju gubitka (logistička regresija, perceptron) i iskoristili je za gradijentni spust

* Kod SVM-a, krenut ćemo odmah od onoga što u konačnici želimo dobiti: maksimalnu marginu $\Rightarrow$ to je ono što SVM optimizira

* Ići ćemo "klasičnim pristupom": formalizirati to kao problem **kvadratnog programiranja**

* Naknadno ćemo iz toga izvesti funkciju gubitka (i funkciju pogreške), ali samo radi usporedbe s drugim algoritmima

> **Sinopsis:**
>
> * Prvo ćemo se fokusirati na **linearan model** i **linearno odvojive probleme** (tvrda granica)
  * Matematika: Lagrangeovi multiplikatori
>
> * Zatim ćemo proširiti **linearan model** tako da može raditi s **linearno neodvojivim problemima** (meka granica)
>
> * Na kraju ćemo proširiti na **nelinearan model** (jezgreni trik)
>   * Matematika: Mercerove jezgre

# Problem maksimalne margine

* Model:
$$
h(\mathbf{x}) = \mathbf{w}^\intercal x + w_0
$$

* Oznake primjera za učenje: $y\in\{-1,+1\}$

* Granica između klasa: hiperravnina $h(\mathbf{x})=0$

* Predikcija klase: $y=\mathrm{sgn}(h(\mathbf{x}))$

* Pretpostavimo da su primjeri iz $\mathcal{D}$ **linearno odvojivi**

* Onda postoji $\mathbf{w}$ i $w_0$ takvi da
$$
\begin{align*}
h(\mathbf{x}^{(i)}) \geq 0 & \quad\text{za svaki $y^{(i)}=+1$}\\
h(\mathbf{x}^{(i)}) < 0 & \quad\text{za svaki $y^{(i)}=-1$}\\
\end{align*}
$$

* Kraće, postoje $\mathbf{w}$ i $w_0$ takvi da
$$
\forall(\mathbf{x}^{(i)},y^{(i)}) \in \mathcal{D}.\ y^{(i)}h(\mathbf{x}^{(i)})\geq 0
$$

* Postoji beskonačno mnogo rješenja za $\mathbf{w}$ i $w_0$ (prostor inačica je beskonačan)

* No nas zanima rješenje **maksimalne margine** $\Rightarrow$ induktivna pristranost preferencijom

* **Margina = udaljenost hiperravnine do najbližeg primjera**

* Ako maksimiziramo marginu, onda će hiperravnina prolaziti točno na pola puta između dva primjera

#### Formulacija optimizacijskog problema

* Predznačena udaljenost primjera od hiperravnine je

$$
d = \frac{h(\mathbf{x})}{\|\mathbf{w}\|}
$$

* Nas zanimaju samo hiperravnine koje ispravno klasificiraju primjere. U tom slučaju **apsolutna** udaljenost primjera do hiperravnine je:

$$
\frac{y^{(i)}h(\mathbf{x})}{\|\mathbf{w}\|} =
\frac{y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0)}{\|\mathbf{w}\|}
$$

* Po definiciji, margina je udaljenost hiperravnine do najbližeg primjera:

$$
\frac{1}{\|\mathbf{w}\|}\mathrm{min}_i\big\{y^{(i)}(\mathbf{w}^\intercal\mathbf{x} + w_0)\big\}
$$

* Tu udaljenost želimo maksimizirati:

$$
\mathrm{argmax}_{\mathbf{w},w_0}\Big\{\frac{1}{\|\mathbf{w}\|}\mathrm{min}_i\big\{y^{(i)}(\mathbf{w}^\intercal\mathbf{x} + w_0)\big\}\Big\}
$$

* Ako je $\mathcal{D}$ linearno odvojiv, onda postoji samo jedna takva margina

#### Pojednostavljenje optimizacijskog problema

* Gornji problem teško je riješiti izravno (min unutar max)

* Vektor $(\mathbf{w},w_0)$ možemo pomnožiti s proizvoljnom konstantom, a da to ne utječe na udaljenosti između primjera i hiperravnine:

$$
d=\frac{h(\mathbf{x})}{\|\mathbf{w}\|}=
\frac{\color{red}{\alpha}\mathbf{w}^\intercal\mathbf{x}+\color{red}{\alpha}w_0}{\|\color{red}{\alpha}\mathbf{w}\|}=
\frac{\color{red}{\alpha}(\mathbf{w}^\intercal\mathbf{x}+w_0)}{\color{red}{\alpha}\|\mathbf{w}\|}=
\frac{\mathbf{w}^\intercal\mathbf{x}+w_0}{\|\mathbf{w}\|}
$$

* Kako bismo pojednostavili problem, možemo definirati da za primjer $\mathbf{x}^{(i)}$ koji je najbliži margini vrijedi

$$
y^{(i)}(\mathbf{w}^\intercal\mathbf{x}+w_0)=1
$$

* Svi ostali primjeri jednako su blizu margine ili su od nje još udaljeniji:

$$
y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0) \geq 1, \qquad i=1,\dots,N
$$

* Za primjere za koje $y^{(i)}h(\mathbf{x})=1$ kažemo da su ograničenja **aktivna**, dok su za ostale primjere ograničenja neaktivna
  * (Primjeri za koje su ograničenja aktivna zovemo potpornim vektorima, ali o tome više kasnije)

* Uvijek će postojati barem **dva** aktivna ograničenja

* [Skica: maksimalna margina]

* Dakle, umjesto
$$
\mathrm{argmax}_{\mathbf{w},w_0}\Big\{\frac{1}{\|\mathbf{w}\|}\underbrace{\mathrm{min}_i\big\{y^{(i)}(\mathbf{w}^\intercal\mathbf{x} + w_0)\big\}}_{=1}\Big\}
$$
mi sada maksimiziramo
$$
\mathrm{argmax}_{\mathbf{w},w_0}\frac{1}{\|\mathbf{w}\|}
$$
uz ograničenja
$$
y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0) \geq 1, \qquad i=1,\dots,N
$$

* Maksimizator od $\frac{1}{\|\mathbf{w}\|}$ ekvivalentan je minimizatoru od $\|\mathbf{w}\|=\sqrt{\mathbf{w}^\intercal\mathbf{w}}$, a taj je ekvivalentan minimizatoru od $\|\mathbf{w}\|^2$. Još ćemo pomnožiti s $\frac{1}{2}$ radi kasnije matematičke jednostavnosti

* Konačna formulacija optimizacijskog problema maksimalne margine:

> $\mathrm{argmin}_{\mathbf{w},w_0}\frac{1}{2}\|\mathbf{w}\|^2$

> tako da $\quad y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0) \geq 1, \quad i=1,\dots,N$

* Naš optimizacijski problem sveo se na ciljnu funkciju koju želimo optimirati i ograničenja koja pritom moramo poštovati: tipičan problem **konveksne optimizacije uz ograničenja**, točnije **kvadratnog programiranja**
  * "Programiranje" = matematičko programiranje = (matematička) optimizacija

# Optimizacija uz ograničenja

* Optimizacijski problem $\Rightarrow$ Optimizacija uz ograničenja $\Rightarrow$ Konveksni optimizacijski problem $\Rightarrow$ Kvadratno programiranje

* Optimizacijski problem uz ograničenja (standardni oblik):

$$
\begin{align*}
\text{minimizirati} &\quad f(\mathbf{x})\\
\text{uz ograničenja} &\quad g_i(\mathbf{x})\leq0,\quad i=1,\dots,m\\
&\quad h_i(\mathbf{x}) = 0,\quad i=1,\dots,p
\end{align*}
$$

* $f : \mathbb{R}^n\to\mathbb{R}$ je **ciljna funkcija** (engl. *objective function*
* $h_i : \mathbb{R}^n\to\mathbb{R}$ su **ograničenja
jednakosti** (engl. *equality constraints*)
* $g_i:\mathbb{R}^n\to\mathbb{R}$ su **ograničenja nejednakosti** (engl. *inequality constraints*)

* **NB:** Sva ograničenja (ne)jednakosti mogu se se svesti na standardni oblik

* Tražimo minimum koji zadovoljava sva ograničenja
* Točke koje zadovoljavaju rješenja zovemo **ostvarivim
točkama** (engl. *feasible points*)

* Konveksni optimizacijski problem:

$$
\begin{align*}
\text{minimizirati} &\quad f(\mathbf{x})\ \color{red}{\text{$\Rightarrow$ konveksna}}\\
\text{uz ograničenja} &\quad g_i(\mathbf{x})\leq0,\quad i=1,\dots,m\\
&\quad \mathbf{a}_i^\intercal\mathbf{x} - b_i = 0,\quad i=1,\dots,p
\end{align*}
$$

* Kvadratni program:

$$
\begin{align*}
\text{minimizirati} &\quad \frac{1}{2}\mathbf{x}^\intercal P\mathbf{x} + \mathbf{q}^\intercal\mathbf{x} + r\ \color{red} {\text{$\Rightarrow$ kvadratna funkcija}}\\
\text{uz ograničenja} &\quad G\mathbf{x}\leq\mathbf{h}\\
&\quad A\mathbf{x}=\mathbf{b}\\
\end{align*}
$$

# Metoda Lagrangeovih multiplikatora

* Kvadratni program može se riješiti metodom **Langrangeovih multiplikatora**

* Lagrangeovi multiplikatori mogu se koristiti općenito za optimizaciju s ograničenjima (problem ne mora biti konveksan)

* Ideja: preformulirati ograničen problem tako da se u ciljnu funkciju ugrade ograničenja

* Alternative: **metode unutarnje točke** (engl. *interior-point methods*, *barrier methods*) ili **metode s kaznom** (engl. *penalty methods*)

#### Lagrangeova funkcija

* **Lagrangeova funkcija** izravno ugrađuje ograničenja u ciljnu funkciju

* [Skica ideje]

* Početni problem:

\begin{align*}
\text{minimizirati} &\quad f(\mathbf{x})\\
\text{uz ograničenja} &\quad g_i(\mathbf{x})\leq0,\quad i=1,\dots,m\\
 &\quad h_i(\mathbf{x}) = 0,\quad i=1,\dots,p
\end{align*}

* Lagrangeova funkcija:

$$
\begin{equation}
L(\mathbf{x},\color{red}{\boldsymbol\alpha},\color{red}{\boldsymbol\beta}) = f(\mathbf{x}) + \sum_{i=1}^m\color{red}{\alpha_i} g(\mathbf{x}) + \sum_{i=1}^p\color{red}{\beta_i}
h(\mathbf{x})
\end{equation}
$$

* Dobili smo neograničeni problem optimizacije: rješenje izvornoga ograničenog problema jest točka koja mimimizira $\mathbf{x}$ i maksimizira $\boldsymbol\alpha$ (sedlo)

* Vrijednosti $\alpha_i$ i $\beta_i$ su **Lagrangeovi multiplikatori** (množitelji) za ograničenja nejednakosti odnosno jednakosti

* Za vrijednosti $\alpha_i$ vrijede takozvani **Karush-Kuhn-Tuckerovi (KKT)** uvjeti:

$$
\begin{align}
\alpha_i &\geq 0,\quad i=1,\dots,m\\
\alpha_i g_i(\mathbf{x}) &= 0,\quad i=1,\dots,m
\end{align}
$$

#### Lagrangeova dualnost

* Načelo dualnosti u teoriji optimizacije:
  * **Primarni problem** (engl. *primal problem*): minimizacija funkcije $f(\mathbf{x})$
  * **Dualni problem**: nalaženje donje granice primarnog problema ("minimum ne može biti manji od $\mathbf{x}$")

* [Skica: primal-dual sedlo]

* Općenito, rješenja primarnog i dualnog problema se ne preklapaju već postoji **dualni procjep**

* Uz određene uvjete, kod konveksne optimizacije dualni procjep jednak je nuli $\Rightarrow$ **jaka dualnost**

* To znači da je rješenje dualnog problema ujedno i rješenje primarnog problema

* Dakle, ako nam je tako pogodnije, možemo rješavati dualni problem umjesto primarnog

* U nastavku promatramo Lagrangeovu dualnost

* Lagrangeova funkcija (BSO: samo s ograničenjima jednakosti):

$$
L(\mathbf{x},\boldsymbol\alpha) = f(\mathbf{x}) + \sum_i\alpha_i h_i(\mathbf{x})
$$

* To je fukcija od $\mathbf{x}$ (**primarne varijable**) i $\boldsymbol\alpha$ (**dualne varijable**)

* Optimum:
$$
L(\mathbf{x}^*,\boldsymbol\alpha^*) = \min_{\mathbf{x},\boldsymbol\alpha} L(\mathbf{x},\boldsymbol\alpha)
$$

* Vrijednost za $\mathbf{x}^*$ nalazimo rješavanjem sustava:

$$
\begin{equation}
\nabla f(\mathbf{x}) + \nabla \sum_i\alpha_i h_i(\mathbf{x}) = 0
\end{equation}
$$

* To rješenje ne mora dovesti do uklanjanja dualnih varijabli! Općenito, dobit ćemo rješenje koje minimizira $\mathbf{x}$ za neki zadani $\boldsymbol\alpha$:
$$
\tilde{L}(\boldsymbol\alpha)= \min_{\mathbf{x}}L(\mathbf{x},\boldsymbol\alpha) = \min_{\mathbf{x}}\Big(f(\mathbf{x}) + \sum_i
\alpha_i h(\mathbf{x})\Big)
$$
$\Rightarrow$ **Dualna Lagrangeova funkcija**

* Sigurno vrijedi:
$$
\tilde{L}(\boldsymbol\alpha) \leq L(\mathbf{x}^*,\boldsymbol\alpha)
$$
$\Rightarrow$ Dualna Lagrangeova funkcija je **donja ograda** primarnog problema

* U točki $\boldsymbol\alpha^*$ vrijedi $\tilde{L}(\boldsymbol\alpha^*)=L(\mathbf{x}^*,\boldsymbol\alpha^*)$

* Kako bismo pronašli $\boldsymbol\alpha^*$, moramo maksimizirati
donju ogradu, tj. riješiti sljedeći konveksni problem:

$$
\begin{align*}
\text{maksimizirati} &\quad \tilde{L}(\boldsymbol\alpha)\\
\text{uz ograničenja} &\quad \alpha_i\geq 0,\quad i=1,\dots,p
\end{align*}
$$

* **NB:** minimizacija ciljne funkcije $\Leftrightarrow$ maksimizacija dualne funkcije

# Dualna formulacija problema maksimalne margine

* Optimizacijski problem maksimalne margine:

> $\mathrm{argmin}_{\mathbf{w},w_0}\frac{1}{2}\|\mathbf{w}\|^2$

> tako da $\quad y^{(i)}(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0) \geq 1, \quad i=1,\dots,N$

* Odgovarajuća Lagrangeova funkcija:

$$
L(\mathbf{w},w_0,\color{red}{\boldsymbol\alpha})=\frac{1}{2}\|\mathbf{w}\|^2 -
\sum_{i=1}^N\color{red}{\alpha_i}\Big\{y^{(i)}\big(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0\big)-1\Big\}
$$

*  $\color{red}{\boldsymbol\alpha=(\alpha_1,\dots,\alpha_N)}$ je vektor Lagrangeovih multiplikatora, po jedan za svako ograničenje

* Prelazimo na **dualnu formulaciju** problema jer je ta formulacija jednostavnija (optimirat ćemo samo po $\boldsymbol\alpha$, a postoje i neke druge prednosti)

* Minimizator $(\mathbf{w}^*, w_0^*)$: deriviranje po $\mathbf{w}$ odnosno $w_0$ i izjednačavanje s nulom:
$$
\begin{align}
\mathbf{w} &= \sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^{(i)}\\
0 &= \sum_{i=1}^N\alpha_i y^{(i)}
\end{align}
$$

* Dualna Lagrangeova funkcija:

$$
\begin{align*}
\tilde{L}(\boldsymbol\alpha) &=
\frac{1}{2}\|\mathbf{w}\|^2 -\sum_{i=1}^N\alpha_i\Big\{y^{(i)}\big(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0\big)-1\Big\}\\
&=
\frac{1}{2}\|\mathbf{w}\|^2
-\sum_{i=1}^N\alpha_i y^{(i)}\mathbf{w}^\intercal\mathbf{x}^{(i)}
\color{gray}{\underbrace{-w_0 \sum_{i=1}^N\alpha_iy^{(i)}}_{=0}} +
\sum_{i=1}^N\alpha_i\\
&= \nonumber
\frac{1}{2}\sum_{i=1}^N\alpha_i y^{(i)}(\mathbf{x}^{(i)})^\intercal\sum_{j=1}^N\alpha_j y^{(j)}\mathbf{x}^{(j)}
-
\sum_{i=1}^N\alpha_i y^{(i)}(\mathbf{x}^{(i)})^\intercal\sum_{j=1}^N\alpha_j y^{(j)}\mathbf{x}^{(j)}
+ \sum_{i=1}^N\alpha_i\\
&=
\sum_{i=1}^N\alpha_i -
\frac{1}{2}\sum_{i=1}^N
\sum_{j=1}^N
\alpha_i
\alpha_j
y^{(i)}
y^{(j)}
(\mathbf{x}^{(i)})^\intercal
\mathbf{x}^{(j)}
\end{align*}
$$

* Dobili smo **dualni optimizacijski problem**:

> **Maksimizirati** izraz
> \begin{align}
\sum_{i=1}^N\alpha_i -
\frac{1}{2}\sum_{i=1}^N
\sum_{j=1}^N
\alpha_i
\alpha_j
y^{(i)}
y^{(j)}
(\mathbf{x}^{(i)})^\intercal
\mathbf{x}^{(j)}
\end{align}
> tako da:
> \begin{align*}
\alpha_i &\geq 0,\quad i=1,\dots,N \\
\sum_{i=1}^N\alpha_i y^{(i)} &= 0
\end{align*}

* Također vrijedi KKT-uvjet:
$$
\alpha_i\big(y^{(i)} h(\mathbf{x}^{(i)})-1\big) = 0
$$

* Ovo je i dalje problem kvadratnog programiranja, međutim broj varijabli se promijenio
  * Primarni problem: $n+1$ varijabli
  * Dualni problem: $N$ varijabli

* Dakle, prijelaz u dualni problem se računalno isplati ako $N\ll n$

* Općenito, složenost kvadratnog programiranja od $n$ varijabli je $\mathcal{O}(n^3)$
  * Posebni algoritmi su efikasniji: **slijedna minimalna optimizacija (SMO)** ima $\mathcal{O}(n^2)$

#### Model

* Već smo izračunali:
$$
\begin{align}
\mathbf{w} &= \sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^{(i)}
\end{align}
$$

* Uvrštavanjem:
$$
\begin{equation}
h(\mathbf{x})=
\underbrace{\mathbf{w}^\intercal\mathbf{x}+w_0}_{\text{Primarno}} =
\underbrace{\sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^\intercal\mathbf{x}^{(i)} + w_0}_{\text{Dualno}}
\end{equation}
$$

* Kako bismo klasificirali primjer $\mathbf{x}$, računamo skalarni produkt između $\mathbf{x}$ i svih primjera $\mathbf{x}^{(i)}$ iz skupa $\mathcal{D}$, pomnožen s težinom $\alpha_i$ i predznakom $\mathcal{y}^(i)$

* Računanje skalarnog produkta $\mathbf{x}^\intercal\mathbf{x}^{(i)}$ je zapravo računanje sličnosti između vektora $\mathbf{x}$ i $\mathbf{x}^{(i)}$, budući da:
$$
\mathbf{x}^\intercal\mathbf{y} = \sum_{i=1}^n x_i y_i
$$
(produkt će biti to veći što se vektori podudaraju u više komponenenata)

* Dakle, umjesto da pohranjujemo težine $\mathbf{w}$, trebamo pohraniti primjere i njihove oznake
  * Umjesto $h(\mathbf{x}|\mathbf{w})$ imamo $h(\mathbf{x}|\boldsymbol\alpha,\mathcal{D})$

* Složenost modela sada ovisi o broju primjera $\Rightarrow$ **neparametarski model**

* Zaključak: ako se trenira tako da se rješava primarni problem, SVM je parametarski model, a ako se rješava dualni problem, onda je neparametarski

#### Potporni vektori

* Iz KKT-uvjeta
$$
\alpha_i\big(y^{(i)} h(\mathbf{x}^{(i)})-1\big) = 0
$$
slijedi da za svaki primjer $\mathbf{x}^{(i)}$ iz $\mathcal{D}$ vrijedi
$$
\alpha_i=0
$$
ili
$$
y^{(i)}h(\mathbf{x}^{(i)})=1
$$

* To znači da je se u izrazu za model pojavljuju samo vektori koji leže točno na ravnini maksimalne margine $\Rightarrow$ **potporni vektori** (engl. *support vectors*)

* Svi ostali vektori za koje $\alpha_i=0$ uopće ne utječu na izlaz modela i možemo ih zanemariti kada radimo predikciju

* Alternativni pogled: hiperravnina (u primarnom problemu) definirana je linearnom kombinacijom potpornih vektora (u dualnom problemu)

```python
seven_X = sp.array([[2,1], [2,3], [1,2], [3,2], [5,2], [5,4], [6,3]])
seven_y = sp.array([1, 1, 1, 1, -1, -1, -1])
```

```python
plot_problem(seven_X, seven_y)
```

```python
from sklearn.svm import SVC

svc = SVC(kernel='linear')
svc.fit(seven_X, seven_y)
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0, degree=3, gamma=0.0,
      kernel='linear', max_iter=-1, probability=False, random_state=None,
      shrinking=True, tol=0.001, verbose=False)

```python
plot_problem(seven_X, seven_y, svc.predict)
```

```python
svc.predict(sp.array([1, 1]))
```

    array([1])

```python
svc.predict(sp.array([6, 1]))
```

    array([-1])

```python
svc.predict(sp.array([3.5, 2]))
```

    array([1])

```python
svc.decision_function(seven_X)
```

    array([[ 1.99980469],
           [ 1.99921875],
           [ 2.99921875],
           [ 0.99980469],
           [-0.99960937],
           [-1.00019531],
           [-1.99960938]])

```python
svc.support_
```

    array([4, 5, 3], dtype=int32)

```python
svc.dual_coef_
```

    array([[  4.99707031e-01,   1.46484375e-04,  -4.99853516e-01]])

**NB:** Koeficijenti iz `dual_coef_` ne odgovaraju vrijednostima $\alpha_i$ već vrijednostima $-\alpha_i y^{(i)}$

```python
svc.support_vectors_
```

    array([[ 5.,  2.],
           [ 5.,  4.],
           [ 3.,  2.]])

#### Rekonstrukcija primarnog problema

* Nakon što je model naučen, možemo rekonstruirati primarne varijable, odnosno težine $\mathbf{w}$ i $w_0$

* Težine $\mathbf{w}$:
$$
\begin{align}
\mathbf{w} &= \sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^{(i)}
\end{align}
$$

* Pomak $w_0$: za potporne vektore $\mathbf{x}^{(i)}\in S$ vrijedi

$$
\begin{align*}
y^{(i)} h(\mathbf{x}) &= 1\\
y^{(i)} \Big(\sum_{j\in S} \alpha_j y^{(j)}(\mathbf{x}^{(i)})^\intercal\mathbf{x}^{(j)} + w_0\Big) &= y^{(i)} y^{(i)}\\
w_0 &= y^{(i)} - \sum_{j\in S} \alpha_j y^{(j)}(\mathbf{x}^{(i)})^\intercal\mathbf{x}^{(j)}\\
\end{align*}
$$

* $w_0$ možemo izračunati na temelju jednog primjera $\mathbf{x}^{(i)}$. Međutim, radi numeričke stabilnosti, bolje je uprosječiti izračun nad svim potpornim vektorima:

$$
w_0 = \frac{1}{|S|}\sum_{i\in S} \Big( y^{(i)} - \sum_{j\in S} \alpha_j y^{(j)}(\mathbf{x}^{(i)})^\intercal\mathbf{x}^{(j)}\Big)
$$

```python
def w(X, y, dc, sv):
    s = 0
    for alpha, i in zip(dc, sv):
        s += -alpha * X[i]
    return s

def w0(X, y, dc, sv):
    s = 0
    for i in sv:
        r = 0
        for alpha, j in zip(dc, sv) :
            r += -alpha * sp.dot(X[i], X[j])
        s += y[i] - r
    return s / float(len(sv))
```

```python
w(seven_X, seven_y, svc.dual_coef_[0], svc.support_)
```

    array([ -9.99707031e-01,  -2.92968750e-04])

```python
svc.coef_
```

    array([[ -9.99707031e-01,  -2.92968750e-04]])

```python
w0(seven_X, seven_y, svc.dual_coef_[0], svc.support_)
```

    3.9995117187499982

```python
svc.intercept_
```

    array([ 3.99951172])

```python
def h_primal(x, w, w0):
    return sp.dot(w, x) + w0

def h_dual(x, X, y, dc, sv):
    xs = [ -alpha * sp.dot(x, X[i]) for alpha, i in zip(dc, sv) ]
    return sum(xs) + w0(X, y, dc, sv)
```

```python
svc.decision_function(seven_X)
```

    array([[ 1.99980469],
           [ 1.99921875],
           [ 2.99921875],
           [ 0.99980469],
           [-0.99960937],
           [-1.00019531],
           [-1.99960938]])

```python
map(lambda x: h_primal(x ,svc.coef_, svc.intercept_), seven_X)
```

    [array([ 1.99980469]),
     array([ 1.99921875]),
     array([ 2.99921875]),
     array([ 0.99980469]),
     array([-0.99960938]),
     array([-1.00019531]),
     array([-1.99960938])]

```python
map(lambda x: h_dual(x, seven_X, seven_y, svc.dual_coef_[0], svc.support_), seven_X)
```

    [1.9998046874999988,
     1.9992187499999989,
     2.999218749999998,
     0.99980468749999929,
     -0.99960937500000036,
     -1.0001953124999989,
     -1.9996093750000021]

#### Je li ovo jedini način za treniranje SVM-a?

* Nije. Mogli smo ostati na primarnom problemu, i riješiti ga npr. stohastičkim gradijentnim spustom uz penalizaciju (algoritam *Pegasos*)

* Postoji niz SVM solvera
  * https://cseweb.ucsd.edu/~akmenon/ResearchExam.pdf
  * https://mitpress.mit.edu/sites/default/files/titles/content/9780262026253_sch_0001.pdf

* Međutim, prednost dualne formulacije je da omogućava **jezgreni trik**

# Meka margina

$$
\begin{align}
y^{(i)}\big(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0\big) \geq 1 - \xi_i, \quad i=1,\dots,N
\end{align}
$$

$$
\xi_i \geq 0, i=1,\dots,N
$$

* Ciljna funkcija:

$$
\begin{equation}
\frac{1}{2}\|\mathbf{w}\|^2+ C\sum_{i=1}^N\xi_i
\end{equation}
$$

> $$
\mathrm{argmin}_{\mathbf{w},w_0}
\Big\{\frac{1}{2}\|\mathbf{w}\|^2+ C\sum_{i=1}^N\xi_i \Big\}
$$
> Uz ograničenja:
> $$
\begin{align*}
y^{(i)}\big(\mathbf{w}^\intercal\mathbf{x}^{(i)}+w_0\big) &\geq 1 - \xi_i, \quad& i=1,\dots,N\\
\xi_i&\geq 0, & i=1,\dots,N\\
\end{align*}
$$

* Pripadna Lagrangeova funkcija:

$$
L(\mathbf{w}, w_0,\boldsymbol\xi,\color{red}{\boldsymbol\alpha},\color{red}{\boldsymbol\beta})=
\frac{1}{2}\|\mathbf{w}\|^2+ C\sum_{i=1}^N \xi_i
- \sum_{i=1}^N\color{red}{\alpha_i}\big(y^{(i)} h(\mathbf{x})-1 + \xi_i\big)
- \sum_{i=1}^N\color{red}{\beta_i}\xi_i
$$

* Minimizacija po $\mathbf{w}$, $w_0$ i $\xi_i$:

$$
\begin{align*}
\mathbf{w} &= \sum_{i=1}^N \alpha_i y^{(i)}\mathbf{x}^{(i)}\\
\sum_{i=1}^N\alpha_i y^{(i)} &= 0\\
\alpha_i &= C - \beta_i
\end{align*}
$$

* Dualna Lagrangeova funkcija:

$$
\tilde{L}(\boldsymbol\alpha) =
\begin{align}
\sum_{i=1}^N\alpha_i -
\frac{1}{2}\sum_{i=1}^N
\sum_{j=1}^N
\alpha_i
\alpha_j
y^{(i)}
y^{(j)}
(\mathbf{x}^{(i)})^\intercal
\mathbf{x}^{(j)}
\end{align}
$$

* Dualni optimizacijski problem:

> **Maksimizirati** izraz
> \begin{align}
\sum_{i=1}^N\alpha_i -
\frac{1}{2}\sum_{i=1}^N
\sum_{j=1}^N
\alpha_i
\alpha_j
y^{(i)}
y^{(j)}
(\mathbf{x}^{(i)})^\intercal
\mathbf{x}^{(j)}
\end{align}
> tako da:
> \begin{align*}
0 \leq \alpha_i &\leq C,\quad i=1,\dots,N \\
\sum_{i=1}^N\alpha_i y^{(i)} &= 0, \quad i=1,\dots,N
\end{align*}

```python
X1, y1 = sp.append(seven_X, [[3,3]], axis=0), sp.append(seven_y, -1)
X2, y2 = sp.append(seven_X, [[2,2]], axis=0), sp.append(seven_y, -1)
```

```python
def predict(model,x) :
    #h = sp.dot(model.coef_,x)+model.intercept_
    h = model.decision_function(x)
    if h >= -1 and h <= 1 : return 0.5
    else : return max(-1,min(1,h))
    return h
```

```python
from sklearn.metrics import hinge_loss

for C in [1e-2,1,1e2] :
    svc = SVC(C=C,kernel='linear')
    svc.fit(X1,y1)
    plot_problem(X1,y1,lambda x: predict(svc,x)); show()
    print svc.support_vectors_
    print "margin = ", 2/sp.linalg.norm(svc.coef_)
    print "loss = ", hinge_loss(y1,svc.decision_function(X1))
```

```python
for C in [1e-2, 1, 1e2] :
    svc = SVC(C=C, kernel='linear')
    svc.fit(X2, y2)
    plot_problem(X2, y2, lambda x: predict(svc,x)); show()
    print svc.support_vectors_
    print "margin = ", 2/sp.linalg.norm(svc.coef_)
    print "loss = ", hinge_loss(y1,svc.decision_function(X2))
```

# Gubitak zglobnice

TODO (v. skriptu)

# Jezgreni trik

TODO (v. skriptu)

# Mercerove jezgre

TODO (v. skriptu)

# Optimizacija hiperparametara

TODO (v. skriptu)

# Sažetak

TODO

```python
def plot_problem(X, y, h=None, surfaces=True) :
    '''
    Plots a two-dimensional labeled dataset (X,y) and, if function h(x) is given,
    the decision boundaries (surfaces=False) or decision surfaces (surfaces=True)
    '''
    assert X.shape[1] == 2, "Dataset is not two-dimensional"
    if h!=None :
        # Create a mesh to plot in
        r = 0.02  # mesh resolution
        x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
        y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
        xx, yy = np.meshgrid(np.arange(x_min, x_max, r),
                             np.arange(y_min, y_max, r))
        XX=np.c_[xx.ravel(), yy.ravel()]
        try:
            Z_test = h(XX)
            if shape(Z_test) == () :
                # h returns a scalar when applied to a matrix; map explicitly
                Z = sp.array(map(h,XX))
            else :
                Z = Z_test
        except ValueError:
            # can't apply to a matrix; map explicitly
            Z = sp.array(map(h,XX))
        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        if surfaces :
            plt.contourf(xx, yy, Z, cmap=plt.cm.Pastel1)
        else :
            plt.contour(xx, yy, Z)
    # Plot the dataset
    scatter(X[:,0],X[:,1],c=y, cmap=plt.cm.Paired,marker='o',s=50);
```