# Phase Transitions in Magnets
### Christina Lee
### Category: Prerequisites
### Physics Prerequisites:  Statistical Mechanics

### Monte Carlo Physics Series
* [Monte Carlo: Calculation of Pi](../Numerics_Prog/Monte-Carlo-Pi.ipynb)
* [Monte Carlo Markov Chain](../Numerics_Prog/Monte-Carlo-Markov-Chain.ipynb)
* [Monte Carlo Ferromagnet](../Prerequisites/Monte-Carlo-Ferromagnet.ipynb)
* [Phase Transitions](../Prerequisites/Phase-Transitions.ipynb)

In Monte Carlo Ferromagnets, we only looked at the state of a magnet at one temperature.  While I displayed how it looked at both a high temperature and a low-temperature situation, that had to be done manually.  Today we are going to automate our study of temperature's effect.

First off, what's the theory?

The standard apparatus for phase transitions is Landau Theory.  In Landau theory, we assume that we have an order parameter.

Order parameters have
* local physical meaning
* global observational consequences

For example, in magnets we use the local magnetization $m$; in freezing fluids, the difference in density $\rho$.  The value of an order parameter should undergo a jump across a phase transition, usually from a zero to a non-zero value.

To construct Landau theory, we postulate that we can create a Free Energy for the system that is
* continuous in the order parameter
* respects the symmetries of the system

Given those constraints, we get
\begin{equation}
F=f_0(T) + \alpha_0 (T-T_c) m^2+ \frac{1}{2} \beta m^4 - h m + ...
\end{equation}
From this simple equation, we can derive many of the qualitative properties of phase transitions.  I won't repeat that here, but you can look it up in a Statistical Mechanics textbook.

Non-zero order parameters also represent symmetry breaking.  In an unmagnetized magnet, all directions are equivalent, but once it magnetizes, it chooses a preferred direction.  That direction is arbitrary up until the magnet spontaneously chooses it.

Back in the early days of cars, streets didn't have lanes and a specified side for cars to drive on.  Driving was symmetric.   When two cars approached each other on the same road, the drivers had to figure out how to avoid a collision, either by everyone going to their left or everyone going to their right.  Eventually, all the cars in a specific area picked up one convention. In the United States, we choose the right side of the road.  In Japan, they chose left.  Neither is wrong; it was pure chance.  The solution breaks the symmetry of the problem.

### Specific Heat and Magnetic Susceptibility

In addition to magnetization and energy, today I introduce the physical observables specific heat and magnetic susceptibility.  These numbers describe how the old quantities change with temperature.

Let's take a look at the first one, Specific Heat:
\begin{equation}
C_h=\frac{d \langle E \rangle}{dT}
\end{equation}

While the above is the formal definition, we can calculate a more convenient expression by plugging in the Maxwell-Boltzmann Distribution:

\begin{equation}
\frac{ d
\frac{\sum_{i} E_{i} e^{-\beta E_i}}{ \sum_j e^{-\beta E_j} }
}{dT}=\beta^2 \left( \langle E^2 \rangle - \langle E \rangle ^2 \right)
\end{equation}

This equation we use to calculate the specific heat, as it uses the values calculated at only one temperature, eliminating the need for a numerical derivative.  For each temperature, we will need to calculate both the expectation of the Energy and the expectation of the Energy Squared.

While in general, a material may be difficult to describe, near phase transitions certain properties diverge according to corresponding critical exponents.  These fall out of the phenomenology of Landau theory, which you can find more about in a textbook.

These critical exponents obey universality.  Phase transitions in the same class will follow the same critical exponents.  Determining the exponent for the 2D square lattice through our simulations also determines the exponent for situations like the mixing of two materials or the percolation problem.

We define our specific heat critical exponent this way,
\begin{equation}
C_h \propto \left( T- T_c \right)^{-\alpha_{\pm}}.
\end{equation}

The divergent scaling of most properties around a critical temperature depends on the divergence of the correlation length $\xi$, the size of fluctuations in the system.  At the critical temperature, fluctuations exist at every length scale.   Once we get close enough to the transition temperature for the fluctuations to become greater than our finite simulation lattice, our computed numbers no longer agree with infinite-lattice-assumed theory.  The divergence gets modified by some function of the ratio between the correlation length and our system size,
\begin{equation}
C_h \propto \left( T - T_c \right)^{-\alpha_{\pm}} g\left( \frac{L}{\xi (T)} \right).
\end{equation}

Magnetic Susceptibility fulfills the same general role as specific heat, but for magnetization instead of energy.
\begin{equation}
\chi =\frac{d \langle M \rangle}{dT}=\beta^2 \left( \langle M^2 \rangle - \langle M \rangle ^2 \right)
\end{equation}

Again, this property also has a corresponding critical exponent.
\begin{equation}
\chi \propto \left( T - T_c \right)^{-\gamma_{\pm}}
\end{equation}

## Exact Solution

The 2D Square Ising model possesses an exact solution, first discovered by Lars Onsager in 1944, but since rewritten in a variety of different ways.  I won't go over how to achieve the solution, but I will show the results for comparison.

The critical temperature:
\begin{equation}
J \beta_c= \frac{\ln \left( \sqrt{2}+1 \right)}{2} \approx .4406
\end{equation}

\begin{equation}
T_c \approx 2.269 J
\end{equation}

The magnetization curve when $T<T_c$ also $\beta > \beta_c$
\begin{equation}
M=\left(1-\sinh^{-4} \left(2 \beta J \right) \right)^{1/8}
\end{equation}

And finally, the much more complicated expression for internal energy per site,

\begin{equation}
k=\sinh (2 \beta J)^{-2}
\end{equation}

\begin{equation}
U=-J \coth (2 \beta J) \left(
1 + \frac{2}{\pi} \left( 2 \tanh^2 (2 \beta J) - 1 \right)
\int_0^{\pi/2} \frac{1}{\sqrt{1-4k (1+k)^{-2} \sin^2 (\theta)}}
d \theta
\right)
\end{equation}

If we can achieve these results to perfect accuracy on pen and paper, why do we even bother simulating them in a computer?  If you try some other lattices and coupling constants, you might soon find out why.  We want to make sure our code is running well on something where we know the results before venturing into unknown territory.

Load our packages.

`Lattices.jl` is the same class I used in the previous post.

```julia
push!(LOAD_PATH,"../Packages")
using Lattices;
using Statistics
using QuadGK
using DelimitedFiles
using Plots
pyplot()
```

    Plots.PyPlotBackend()

```julia
# The critical inverse temperature
betac=log(1+sqrt(2))/2

# The exact magnetization
# Note: Does not take arrays
function Mexact(beta::Float64)
    if beta>betac
        return (1-sinh(2*beta*J)^(-4.0))^(1.0 ./8.0)
    else
        return 0
    end
end

# The exact energy
# Note: Does not take arrays
function Eexact(beta::Float64)
    k=sinh(2*beta*J)^(-2.0)

    # the inside of the integral.  Define a function so we can use quadgk
    insides(θ)=(1-4*k*(1+k)^(-2)*sin(θ)^2)^(-0.5)
    #we run a numerical integration
    integ=quadgk(insides,0,π/2)[1]

    return -J*coth(2*beta*J)*(1+2/π*(2*tanh(2*beta*J)^2-1)*integ )
end
```

    Eexact (generic function with 1 method)

Here we set up our lattice.

If looking for a mind bender, look at a `"Checkerboard"`, with an anti-ferromagnetic coupling constant, like `J=-1`.  Otherwise, just stick with a square.

```julia
## Define l here
l=10;

lt=MakeLattice("Square",l);
S=ones(Int8,l,l);  #Our spins
dt=1/(lt.N);
```

Same as last time.

These functions calculate properties of our lattice.

```julia
# The energy contribution of just one site
function dE(i::Int)
    Eii=0;
    for j in 1:lt.nnei
        Eii+=S[lt.neigh[i,j]];
    end
    Eii*=-J*S[i];  # we are computing J sz_i sz_j for one i
    return Eii;
end
# The energy of the entire lattice
function E()
    Evar=0;
    for k in 1:lt.N
        Evar+=.5*dE(k);
    end
    return Evar;
end
# The magnetization of the entire lattice
function M()
    Mvar=0;
    for k in 1:lt.N
        Mvar+=S[k];
    end
    return Mvar;
end
"defined functions"
```

    "defined functions"

### Adjustable Parameters

```julia
beta0=.1;
betaf=1;
dbeta = .1 ; # coarse grained for faster computation
# dbeta=.025; # fine grained if you have a faster computer

J=1;
t=100000;
nskip=10;   # don't measure every sweep= better decorrelation
"Parameters set"
```

    "Parameters set"

```julia
nmeas=Int64(t/nskip); # how many times we will measure
betas=collect(beta0:dbeta:betaf)
"done"
```

    "done"

### Exact Solution Data Points

```julia
Mex=zeros(length(betas))
Eex=zeros(length(betas))
for ii in 1:length(betas)
    Mex[ii]=Mexact(betas[ii])
    Eex[ii]=Eexact(betas[ii])
end
```

I took what we ran last time and wrapped it into a function, `MCMCMagnet`.  We aren't looking at spin configurations this time; we only want the final measureables, which get returned at the end.

```julia
function MCMCMagnet(beta::Float64)
    tm=1; #Our measurement time step

    Ma=Array{Int32}(undef,nmeas); # our magnetization measurements
    Ea=Array{Int32}(undef,nmeas); # our energy measurements
    Ma2=Array{Int32}(undef,nmeas); # magnetization squared
    Ea2=Array{Int32}(undef,nmeas); # energy squared

    for ti in 1:t
        for j in 1:lt.N
            i = rand(1:lt.N); #Choosing a random site
            de=dE(i);
            if(de>0 || rand()<exp(2*beta*de) )
                S[i]=-S[i]; #Switch the sign
            end
        end
        if isapprox(mod(ti,nskip),0)
            Ma[tm]=M();
            Ma2[tm]=Ma[tm]^2;

            Ea[tm]=E();
            Ea2[tm]=Ea[tm]^2;

            tm+=1;

        end
    end
    Mave=mean(Ma);
    Mstd=std(Ma)/lt.N;
    Eave=mean(Ea);
    Estd=std(Ea)/lt.N;

    E2ave=mean(Ea2);
    M2ave=mean(Ma2);

    Ch=beta^2*(E2ave-Eave^2)/lt.N;
    χ=beta*(M2ave-Mave^2)/lt.N;

    return Mave/lt.N,Mstd,Eave/lt.N,Estd,Ch,χ
end
```

    MCMCMagnet (generic function with 1 method)

# The Temperature Loop

This cell looks pretty simple: initialization and one for loop.  But this is where everything ties together.

```julia
Mm=zeros(length(betas))
Mstd=zeros(length(betas))
Ee=zeros(length(betas))
Estd=zeros(length(betas))
Ch=zeros(length(betas))
χ=zeros(length(betas))
for ii in 1:length(betas)
    Mm[ii],Mstd[ii],Ee[ii],Estd[ii],Ch[ii],χ[ii]=MCMCMagnet(betas[ii])
    println("beta: ",betas[ii],"\tM: ",Mm[ii],"\tE: ",Ee[ii])
end
```

    beta: 0.1	M: -0.001804	E: -0.203736
    beta: 0.2	M: -0.000968	E: -0.428008
    beta: 0.3	M: 0.0013059999999999999	E: -0.706624
    beta: 0.4	M: -0.00808	E: -1.186944
    beta: 0.5	M: 0.32823399999999997	E: -1.7440280000000001
    beta: 0.6	M: 0.973302	E: -1.908696
    beta: 0.7	M: 0.9904219999999999	E: -1.964804
    beta: 0.8	M: 0.9960840000000001	E: -1.984992
    beta: 0.9	M: 0.9984439999999999	E: -1.99392
    beta: 1.0	M: 0.999256	E: -1.9971

That was the temperature loop for the MCMC method.

### Storing and Restoring Data

Since running these simulations take slightly longer than some of my previous posts, I've outputed data files of the results of three different sizes lattices (l=10, l=20, l=50), for runs of length 100,000. Feel free to take a look at them, or make your own data :)

```julia
# if you want to write a new set of data
writedlm(string("PhaseTransitionsL",l,"t",t,"dbeta",dbeta,".dat"),[betas Mm Mstd Ee Estd Ch χ])
```

```julia
temp=readdlm("PhaseTransitionsL50t100000dbeta0.025.dat")
betas50=temp[:,1];
Mm50=temp[:,2];
Mstd50=temp[:,3];
Ee50=temp[:,4];
Estd50=temp[:,5];
Ch50=temp[:,6];
χ50=temp[:,7];

temp=readdlm("PhaseTransitionsL10t100000dbeta0.025.dat")
betas10=temp[:,1];
Mm10=temp[:,2];
Mstd10=temp[:,3];
Ee10=temp[:,4];
Estd10=temp[:,5];
Ch10=temp[:,6];
χ10=temp[:,7];

temp=readdlm("PhaseTransitionsL20t100000dbeta0.025.dat")
betas20=temp[:,1];
Mm20=temp[:,2];
Mstd20=temp[:,3];
Ee20=temp[:,4];
Estd20=temp[:,5];
Ch20=temp[:,6];
χ20=temp[:,7];
```

# Plotting Section

```julia
plot(betas,Mex,label="Exact",linewidth=20,color=:black)
plot!(betas10,abs.(Mm10),label="l=10",linewidth=15)
plot!(betas20,abs.(Mm20),label="l=20",linewidth=10)
plot!(betas50,abs.(Mm50),label="l=50",linewidth=5)

vline!([betac],label="")
plot!(xlabel="Inverse Temperature β",ylabel="Magnetization",
        title="2D Square Ferromagnet Phase Transition",legend=:left)
```

```julia
plot(betas,Eex,label="Exact",linewidth=20,color=:black)
plot!(betas10,Ee10,label="l=10",linewidth=15)
plot!(betas20,Ee20,label="l=20",linewidth=10)
plot!(betas50,Ee50,label="l=50",linewidth=5)

vline!([betac],label="")
plot!(xlabel="Inverse Temperature β",ylabel="Energy",
        title="2D Square Ferromagnet Phase Transition")
```

```julia
plot(betas50,Ch50,label="l=50",linewidth=5)
plot!(betas20,Ch20,label="l=20",linewidth=5)
plot!(betas10,Ch10,label="l=10",linewidth=5)

vline!([betac],label="")
plot!(xlabel="Inverse Temperature β",ylabel="Specific Heat",
    title="Specific Heat Divergence with Finite Size Scaling")
```

I presented two different equivalent formulas for the specific heat.  Let's check to make sure they're equivalent.

For the formal defination, we'll take a numeric derivative with respect to $\beta$,
\begin{equation}
\frac{d \langle E \rangle}{dT} \approx \frac{ \Delta \langle E \rangle}{\Delta \beta}\frac{d \beta}{d T}
\end{equation}

Note: I corrected the scaling by hand.

```julia
Ch2=(Ee50[1:end-1]-Ee50[2:end]).*betas50[2:end].^2
plot(betas50[2:end],Ch2*35,label="dE/dT")
plot!(betas50,Ch50,label="beta2(<E2>-<E>2)")

plot!(xlabel="Inverse Temperature β",ylabel="Specific Heat",
    title="Comparing Specific Heat Calculations")
```

```julia
plot(betas10,χ10,label="l=10")
plot!(betas20,χ20,label="l=20")
plot!(betas50,χ50,label="l=50")

vline!([betac],label="",linecolor=:black)

plot!(xlabel="Inverse Temperature β",ylabel="Susceptibility",
    title="Magnetic Susceptibility with Finite Size Scaling")
```

```julia
M2=(Mm50[2:end]-Mm50[1:end-1]).*(betas50[1:end-1]).^2
plot(betas50[1:end-1],M2*750,label="dM/dT")
plot!(betas50,χ50,label="(<M2>-<M>2)beta2")
plot!(xlabel="Inverse Temperature β",ylabel="Susceptibility",
    title="Comparing Methods of Computing Susceptibility")
```

That's all for now.  Plenty more to talk about on this subject, but hopefully that can keep you occupied.
My plots are saved in M4/Images/PhaseTransitions .


# **Terra**

```python
import numpy as np

# Importing standard Qiskit libraries
from qiskit import QuantumCircuit, transpile, Aer, IBMQ, execute
from qiskit.tools.jupyter import *
from qiskit.visualization import *
from ibm_quantum_widgets import *
from qiskit.providers.aer import QasmSimulator

# Loading your IBM Quantum account(s)
provider = IBMQ.load_account()
```

```python
# Import registers
from qiskit import QuantumRegister, ClassicalRegister

qr = QuantumRegister(2, 'my_Q')
cr = ClassicalRegister(2, 'my_C')
qc = QuantumCircuit(qr,cr)
qc.draw()
```

```python
qc = QuantumCircuit(QuantumRegister(2, 'my_Q'), ClassicalRegister(2, 'my_C'))
```

```python
#Import the register classes
from qiskit import QuantumRegister, ClassicalRegister
#Create the quantum and classical registers, each with labels
qr1 = QuantumRegister(2, name='qr1')
cr1 = ClassicalRegister(2, name='cr1')
#Create the quantum circuit using the registers
qc1 = QuantumCircuit(qr1, cr1)
#Draw the circuit
qc1.draw()
```

```python
#Create two Quantum and Classical registers
qr2 = QuantumRegister(2, name='qr2')
cr2 = ClassicalRegister(2, name='cr2')
#Create a second circuit using the registers created above
qc2 = QuantumCircuit(qr2, cr2)
#Draw the second quantum circuit
qc2.draw()
```

```python
#Concatenate the two previous circuits to create a new circuit
qc_combined = qc1 + qc2
#Draw the concatenated circuit
qc_combined.draw()
```

```python
#Import the random_circuit class
from qiskit.circuit.random import random_circuit
#Construct the random circuit with the number of qubits = 3
#with a depth = 2, and include the measurement operator for each qubit
qc = random_circuit(3, 2, measure=True)
#Draw the circuit
qc.draw()
```

```python
# Import the random circuit class
from qiskit.circuit.random import random_circuit
#Create two random circuits, each with 2 qubit registers and random #gate operator counts.
qc1 = random_circuit(2,2)
qc2 = random_circuit(2,4)
#Concatenate the two random circuits
qc = qc1 + qc2
#Draw the circuit
qc.draw()
```

```python
#Define function to print circuit properties:
def print_circuit_props(qc):
    width = qc.width()
    depth = qc.depth()
    num_operators = qc.count_ops()
    circuit_size = qc.size()
    print('Width = ',width)
    print('Depth = ', depth)
    print('Circuit size = ',circuit_size)
    print('Number of operators = ', num_operators)
```

```python
#Pass our quantum circuit to print out the circuit properties
print_circuit_props(qc)
```

    Width =  2
    Depth =  6
    Circuit size =  7
    Number of operators =  OrderedDict([('crz', 3), ('rzz', 1), ('cy', 1), ('rx', 1), ('u1', 1)])

```python
qc = QuantumCircuit(3)
qc.ccx(0,1,2)
qc.draw()
```

```python
#Print out the circuit properties
print_circuit_props(qc)
```

    Width =  3
    Depth =  1
    Circuit size =  1
    Number of operators =  OrderedDict([('ccx', 1)])

```python
#Print out the circuit properties
print_circuit_props(qc.decompose())
```

    Width =  3
    Depth =  11
    Circuit size =  15
    Number of operators =  OrderedDict([('cx', 6), ('t', 4), ('tdg', 3), ('h', 2)])

## **Customization and Parameterization of Circuits**

```python
#Create a custom two-qubit composite gate
#Create the quantum register
qr = QuantumRegister(2, name='qr_c')
#Generate quantum circuit which will make up the composite gate
comp_qc = QuantumCircuit(qr, name='my-composite')
#Add any gates you wish to your composite gate
comp_qc.h(0)
comp_qc.cx(0, 1)
#Create the composite instructions by converting
#the QuantumCircuit to a list of Instructions
composite_inst = comp_qc.to_instruction()
#Draw the circuit which will represent the composite gate
comp_qc.draw()
```

```python
#Create your 2-qubit circuit to generate your composite gate
qr2 = QuantumRegister(3, 'qr')
#Create a quantum circuit using the quantum register
qc = QuantumCircuit(qr2)
#Add any arbitrary gates that would represent the function
#of the composite gate
qc.h(0)
qc.cx(0,1)
qc.cx(0,2)
#Draw the composite circuit
qc.draw()
```

```python
#Append your composite gate to the specified qubits.
qc.append(composite_inst, [qr2[0], qr2[1]])
#Draw the complete circuit
qc.draw()
```

```python
#Import the Parameter object
from qiskit.circuit import Parameter
#Construct the Parameter set to Theta
param_theta = Parameter('θ')
#Create a two-qubit quantum circuit and add some gates
qc = QuantumCircuit(2)
qc.h(0)
qc.cx(0, 1)
#Include a rotation gate which we wish to apply
#the Parameter value
qc.rz(param_theta,0)
qc.rz(param_theta,1)
#Draw the circuit
qc.draw()
```

```python
import numpy as np
#Bind the parameters with a value, in this case 2π
qc = qc.bind_parameters({param_theta: 2*np.pi})
#Draw the circuit with the set parameter values
qc.draw()
```

## **Generating Pulse Schedules on Quantum Hardware**

```python
#Import pulse classes
from qiskit.pulse import DriveChannel, Play, Schedule
from qiskit.pulse.library import Waveform
#Import some helpful utils
from qiskit.scheduler.utils import measure_all
```

```python
#Import numpy and generate the sin sample values
import numpy as np
x = np.linspace(0,2*np.pi,64)
data = np.sin(x)
#Generate our Waveform
wave_form = Waveform(data, name="sin_64_pulse")
#Draw the generated sample pulse
wave_form.draw()
```

```python
#Import the Gaussian Square pulse from Pulse Library
from qiskit.pulse.library import GaussianSquare
#Create a Gaussian Square pulse:
#Args: duration, amp, sigma, width, name
gaussian_square_pulse = GaussianSquare(128,1,2, width=112, risefall_sigma_ratio=None, name="gaussian square", limit_amplitude=None)
gaussian_square_pulse.draw()
```

## **Generating and executing Schedules**

```python
#Create the first schedule with our Gaussian Square pulse
schedule_1 = Schedule(name='Schedule 1')
schedule_1 = schedule_1.insert(0, Play(gaussian_square_pulse, DriveChannel(0)))
#Draw the schedule
schedule_1.draw()
```

```python
#Create a second schedule with our sample pulse
schedule_2 = Schedule(name='Schedule 2')
schedule_2 = schedule_2.insert(0, Play(wave_form, DriveChannel(0)))
#Draw the schedule
schedule_2.draw()
```

```python
#Let's create a third schedule
#Where we add the first schedule and second schedules
#And shift the second to the right by a time of 5 after the first
schedule_3 = schedule_1.insert(schedule_1.duration+5, schedule_2)
schedule_3.draw()
```

```python
#We could have also combined the two using the append operator
#The two schedules are appended immediately after one another
schedule_3_append = schedule_1.append(schedule_2)
schedule_3_append.draw()
```

## **Scheduling existing quantum circuits**

```python
qc = QuantumCircuit(1, 1)
qc.h(0)
qc.measure(0,0)
#Draw the circuit
qc.draw()
```

```python
#Import transpile and schedule
from qiskit import transpile, schedule
#Set the backend to ibmq_armonk
backend = provider.get_backend('ibmq_armonk')
#Transpile the circuit using basis gates from the specified backend
transpiled_qc = transpile(qc, backend)
#Draw the transpiled circuit
transpiled_qc.draw()
```

```python
#Create the circuit schedule using the transpiled circuit
circuit_schedule = schedule(transpiled_qc, backend)
#Draw the circuit
circuit_schedule.draw()
```

```python
#Draw the circuit with a shorter time range to ease visibility
circuit_schedule.draw(time_range=[0, 500])
```

```python
#Create a 2-qubit circuit
qc2 = QuantumCircuit(2, 2)
#Apply a Hadamard to the first qubit
qc2.h(0)
#Apply a CNOT gate where the Source is qubit 0, and Target qubit 1
qc2.cx(0, 1)
#Add measurement gates to all qubits
qc2.measure([0, 1], [0, 1])
#Draw the circuit
qc2.draw()
```

```python
#Import the test backend
from qiskit.test.mock import FakeAlmaden
#Construct the backend
backend = FakeAlmaden()
#Transpile the circuit to the test backend and its basis states
transpiled_qc2 = transpile(qc2, backend)
#Draw the transpiled circuit
transpiled_qc2.draw()
```

```python
#Create the circuit from the transpiled circuit results
circuit_schedule2 = schedule(transpiled_qc2, backend)
#Draw the 2-qubit circuit schedule with range of 2000 time steps
circuit_schedule2.draw(time_range=[0, 2000])
```

## **Leveraging Provider information**

```python
#Import the IBMQ interface
from qiskit import IBMQ

#Save account ONLY needed if running on a local system for first time
#Uncomment below if running for the first time on a local machine.
#IBMQ.save_account('API_TOKEN')

#Load the account which was saved on local system using save_account.
#Note: this is handled each time a Qiskit Notebook is loaded on IQX.
IBMQ.load_account()
```

    ibmqfactory.load_account:WARNING:2022-01-18 06:45:09,200: Credentials are already in use. The existing account in the session will be replaced.

    <AccountProvider for IBMQ(hub='ibm-q', group='open', project='main')>

```python
#Indicate a hub to link account to:
IBMQ.get_provider(hub='ibm-q')
#Indicate a project which your account is associated with
IBMQ.get_provider(project='main')
```

    <AccountProvider for IBMQ(hub='ibm-q', group='open', project='main')>

```python
#Create the Provider object using the IBMQ interface
provider = IBMQ.get_provider(group='open')
#Query the list of backends available to your account
provider.backends()
```

    [<IBMQSimulator('ibmq_qasm_simulator') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_armonk') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_santiago') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_bogota') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_belem') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_quito') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_statevector') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_mps') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_extended_stabilizer') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQSimulator('simulator_stabilizer') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_manila') from IBMQ(hub='ibm-q', group='open', project='main')>]

```python
#Filter the list of backends to include only non-simulator,
#and operational (meaning, not offline or under maintenance)
provider.backends(simulator=False, operational=True)
```

    [<IBMQBackend('ibmq_armonk') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_bogota') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_belem') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_quito') from IBMQ(hub='ibm-q', group='open', project='main')>,
     <IBMQBackend('ibmq_manila') from IBMQ(hub='ibm-q', group='open', project='main')>]

```python
#Select a specific device from the provider
backend = provider.get_backend('ibmq_lima')
```

```python
#Import the least_busy function
from qiskit.providers.ibmq import least_busy

#Identify the least busy devices
#smaller than 6 qubits and not a simulator
small_devices = provider.backends(filters=lambda x: x.configuration().n_qubits < 6 and not x.configuration().simulator)

#Identify the least busy devices
#larger than 6 qubits and not a simulator
large_devices = provider.backends(filters=lambda x: x.configuration().n_qubits > 6 and not x.configuration().simulator)

#Print the least busy devices
print('The least busy small devices: {}'.format(least_busy(small_devices)))
print('The least busy large devices: {}'.format(least_busy(large_devices)))
```

    The least busy small devices: ibmq_bogota
    Traceback (most recent call last):
      Input In [54] in <module>
        print('The least busy large devices: {}'.format(least_busy(large_devices)))
      File /opt/conda/lib/python3.8/site-packages/qiskit/providers/ibmq/__init__.py:142 in least_busy
        raise IBMQError('Unable to find the least_busy '
    IBMQError: 'Unable to find the least_busy backend from an empty list.'

    Use %tb to get the full traceback.

<button class='button iqx-button'>Search for solution online</button>

## **Quantum backend components**

```python
#Set ibmq_lima as the backend, or whichever backend you wish
backend = provider.get_backend('ibmq_lima')
#Confirm this is the backend selected by querying for its name,
backend.name()
```

    'ibmq_lima'

```python
#View the status of the backend
status = backend.status()
is_operational = status.operational
jobs_in_queue = status.pending_jobs
print('Number of pending jobs in the queue: ', jobs_in_queue)
```

    Number of pending jobs in the queue:  56

```python
#View the configuration of the backend
backend.configuration()
backend
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>

```python
# Display the number of qubits from the backend properties
backend.properties().qubits
```

    [[Nduv(datetime.datetime(2022, 1, 18, 5, 18, 10, tzinfo=tzlocal()), T1, us, 116.61434014337499),
      Nduv(datetime.datetime(2022, 1, 18, 5, 18, 44, tzinfo=tzlocal()), T2, us, 190.7616020143424),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), frequency, GHz, 5.029752763617062),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), anharmonicity, GHz, -0.33574155258829724),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_error, , 0.02200000000000002),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas0_prep1, , 0.028800000000000048),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas1_prep0, , 0.0152),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_length, ns, 5351.11111111111)],
     [Nduv(datetime.datetime(2022, 1, 18, 5, 18, 10, tzinfo=tzlocal()), T1, us, 101.8379516834072),
      Nduv(datetime.datetime(2022, 1, 18, 5, 19, 40, tzinfo=tzlocal()), T2, us, 116.31418889990404),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), frequency, GHz, 5.127683591792908),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), anharmonicity, GHz, -0.3183491230373718),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_error, , 0.020000000000000018),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas0_prep1, , 0.03380000000000005),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas1_prep0, , 0.0062),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_length, ns, 5351.11111111111)],
     [Nduv(datetime.datetime(2022, 1, 18, 5, 18, 10, tzinfo=tzlocal()), T1, us, 71.26662201204402),
      Nduv(datetime.datetime(2022, 1, 18, 5, 18, 44, tzinfo=tzlocal()), T2, us, 91.7610492310778),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), frequency, GHz, 5.247344147376312),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), anharmonicity, GHz, -0.3336036607926792),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_error, , 0.016199999999999992),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas0_prep1, , 0.025000000000000022),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas1_prep0, , 0.0074),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_length, ns, 5351.11111111111)],
     [Nduv(datetime.datetime(2022, 1, 18, 5, 18, 10, tzinfo=tzlocal()), T1, us, 73.76839752646555),
      Nduv(datetime.datetime(2022, 1, 18, 5, 18, 44, tzinfo=tzlocal()), T2, us, 83.67843926069811),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), frequency, GHz, 5.302254227477037),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), anharmonicity, GHz, -0.33124193185343576),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_error, , 0.02410000000000001),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas0_prep1, , 0.036599999999999966),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas1_prep0, , 0.0116),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_length, ns, 5351.11111111111)],
     [Nduv(datetime.datetime(2022, 1, 18, 5, 18, 10, tzinfo=tzlocal()), T1, us, 23.889774178193875),
      Nduv(datetime.datetime(2022, 1, 18, 5, 19, 40, tzinfo=tzlocal()), T2, us, 25.6247146620734),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), frequency, GHz, 5.092155017911517),
      Nduv(datetime.datetime(2022, 1, 18, 5, 47, 18, tzinfo=tzlocal()), anharmonicity, GHz, -0.3344696566823515),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_error, , 0.05190000000000006),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas0_prep1, , 0.08440000000000003),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), prob_meas1_prep0, , 0.0194),
      Nduv(datetime.datetime(2022, 1, 18, 5, 17, 31, tzinfo=tzlocal()), readout_length, ns, 5351.11111111111)]]

```python
#Print out the frequency of qubit (0)
print('Frequency of first qubit is: '+ str(backend.properties().frequency(0)))
#Print out the readout error of qubit (0)
print('Readout error of first qubit is: '+ str(backend.properties().readout_error(0)))
```

    Frequency of first qubit is: 5029752763.617063
    Readout error of first qubit is: 0.02200000000000002

```python
#Run a few jobs on this backend to generate jobs on the backend
qc = QuantumCircuit(1)
qc.h(0)
qc.measure_all()
qc.draw()
```

```python
# Launch the job watcher widget
%qiskit_job_watcher

for i in range(0,3):
    result = execute(qc, backend, shots=1024).result()
```

    Accordion(children=(VBox(layout=Layout(max_width='710px', min_width='710px')),), layout=Layout(max_height='500…

    <IPython.core.display.Javascript object>

```python
#List out the last 3 jobs we ran on the device
for executed_job in backend.jobs(limit=3):
    print('Job id: '
          + str(executed_job.job_id()) + ', '
          + str(executed_job.creation_date) + ', '
          + str(executed_job.status()))
```

    Job id: 61e6676184e3d23cb536100d, <bound method IBMQJob.creation_date of <qiskit.providers.ibmq.job.ibmqjob.IBMQJob object at 0x7f9dfd384cd0>>, JobStatus.DONE
    Job id: 61e66743658f81014d6fef02, <bound method IBMQJob.creation_date of <qiskit.providers.ibmq.job.ibmqjob.IBMQJob object at 0x7f9dfd695f70>>, JobStatus.DONE
    Job id: 61e6673082ce86dee0b42d52, <bound method IBMQJob.creation_date of <qiskit.providers.ibmq.job.ibmqjob.IBMQJob object at 0x7f9dfd14cf70>>, JobStatus.DONE

## **Understanding the Job component**

```python
#From the previous output of executed jobs, enter its job id.
job = backend.retrieve_job(executed_job.job_id())
```

```python
#Print the job instance status
job.status()
```

    <JobStatus.DONE: 'job has successfully run'>

```python
job.result()
```

    Result(backend_name='ibmq_lima', backend_version='1.0.27', qobj_id='c54368c0-7559-42a8-8d18-141ea4251bfa', job_id='61e6673082ce86dee0b42d52', success=True, results=[ExperimentResult(shots=1024, success=True, meas_level=2, data=ExperimentResultData(counts={'0x0': 548, '0x1': 476}), header=QobjExperimentHeader(clbit_labels=[['meas', 0]], creg_sizes=[['meas', 1]], global_phase=0.7853981633974483, memory_slots=1, metadata={}, n_qubits=5, name='circuit-134', qreg_sizes=[['q', 5]], qubit_labels=[['q', 0], ['q', 1], ['q', 2], ['q', 3], ['q', 4]]))], date=2022-01-18 07:07:42+00:00, status=Successful completion, status=QobjHeader(backend_name='ibmq_lima', backend_version='1.0.27'), execution_id='52c2bc72-782d-11ec-9015-b02628f7f59e', time_taken=3.0157341957092285, error=None, client_version={'qiskit': '0.34.1'})

```python
job.backend()
```

    VBox(children=(HTML(value="<h1 style='color:#ffffff;background-color:#000000;padding-top: 1%;padding-bottom: 1…

    <IBMQBackend('ibmq_lima') from IBMQ(hub='ibm-q', group='open', project='main')>

```python
import qiskit.tools.jupyter
%qiskit_version_table
```

Version Information

Qiskit Software	Version

qiskit-terra	0.19.1

qiskit-aer	0.10.2

qiskit-ignis	0.7.0

qiskit-ibmq-provider	0.18.3

qiskit	0.34.1

qiskit-nature	0.3.0

qiskit-finance	0.3.0

qiskit-optimization	0.3.0

qiskit-machine-learning	0.3.0

System information

Python version	3.8.12

Python compiler	GCC 9.4.0

Python build	default, Oct 12 2021 21:59:51

OS	Linux

CPUs	8

Memory (Gb)	31.400043487548828

Tue Jan 18 07:18:31 2022 UTC


*This notebook is part of  course materials for CS 345: Machine Learning Foundations and Practice at Colorado State University.
Original versions were created by Asa Ben-Hur.
The content is availabe [on GitHub](https://github.com/asabenhur/CS345).*

*The text is released under the [CC BY-SA license](https://creativecommons.org/licenses/by-sa/4.0/), and code is released under the [MIT license](https://opensource.org/licenses/MIT).*

<a href="https://colab.research.google.com/github//asabenhur/CS345/blob/master/notebooks/module07_02_logistic_regression.ipynb">

</a>

```python
import numpy as np
import matplotlib.pyplot as plt
%autosave 0
%matplotlib inline
```

    Autosave disabled

# Logistic Regression

Although its name would suggest otherwise, logistic regression is a classification method.
As we go along, it may become clear why the word regression is in there.

### Recap:  linear classifiers

In earlier notebooks we discussed the use of a linear function to make predictions using a linear function:

$$
f(\mathbf{x}) = \mathbf{w}^\top \mathbf{x} + b.
$$

The class associated with $\mathbf{x}$ is decided according to the sign of the discriminant function $f(\mathbf{x})$.

For example:

```python
from sklearn.datasets import make_classification
X,y = make_classification(n_samples=100, n_features=2, n_informative=2, n_redundant=0, n_repeated=0, n_classes=2, n_clusters_per_class=1, class_sep=0.3, random_state=1)
plt.style.use('default')
plt.scatter(X[:,0], X[:,1], c=1-y, alpha=0.5, s=20, cmap='magma');

w = np.array([-0.5, 0.1])
b = 0
delta = 0.01
xmin=-1.25;xmax=0.75;ymin=-0.75;ymax=1.5
xs = np.arange(xmin, xmax, delta)
ys = np.arange(ymin, ymax, delta)
x_grid, y_grid = np.meshgrid(xs, ys)

Z = w[0] * x_grid + w[1] * y_grid + b

im = plt.imshow(Z, origin='lower',
                cmap=plt.cm.gray, extent=(xmin,xmax,ymin,ymax) )

C = plt.contour(x_grid, y_grid, Z, 5,
                origin='lower',
                linewidths=(1,1,2,1,1),
                colors = 'black')

plt.clabel(C, inline=1, fmt='%1.1f')

plt.arrow(0,0,w[0],w[1], width=0.001,head_width=0.05, length_includes_head=True, alpha=1,linestyle='-',color='k');
plt.text(w[0], w[1]+0.05, r"$\mathbf{w}$");

```

### Using probabilities to quantify prediction confidence

As users of machine learning, we are interested in making *confident* predictions.  In the context of the linear discriminant function,
the magnitude of $f(\mathbf{x})$ can tell us something about our confidence in the prediction.  However, there is no easy way for us to interpret that value as a measure of confidence.  What would help us is a way to estimate $p(y | \mathbf{x})$.

In this notebook we will focus on binary classification problems.  In this case the only two options are $p(y=1 | \mathbf{x})$, and $p(y=0 | \mathbf{x})$, which satisfy:

$$
p(y=0 | \mathbf{x}) = 1 - p(y=1 | \mathbf{x})
$$

The following figure shows $p(y=1 | \mathbf{x})$ and $p(y=0 | \mathbf{x})$ for a hypothetical classification problem.

```python
from scipy.stats import norm
plt.style.use('bmh')
plt.xkcd(scale=0.3)
plt.figure(figsize=(6,4))
m1 = 2.5
std1 = 1.0
m2 = 5.0
std2 = 1.0
def solve(m1,m2,std1,std2):
  a = 1/(2*std1**2) - 1/(2*std2**2)
  b = m2/(std2**2) - m1/(std1**2)
  c = m1**2 /(2*std1**2) - m2**2 / (2*std2**2) - np.log(std2/std1)
  return np.roots([a,b,c])
result=solve(m1,m2,std1,std2)
x = np.linspace(-5,9,10000)
ax = plt.axes()
ax.plot(x,norm.pdf(x,m1,std1),label="p(y=0|x)")
ax.plot(x,norm.pdf(x,m2,std2),label="p(y=1|x)")
ax.plot(result,norm.pdf(result,m1,std1),'o')
plt.xlabel('x')
plt.ylabel('p(y|x)',rotation=0,labelpad=30)
ax.arrow(result[0],norm.pdf(result,m1,std1)[0],0,-2,head_width=0, head_length=0 ,fc='k', ec='k',color="yellow")
plt.text(result[0], -0.05, "x*")
plt.xticks([])
plt.yticks([])
plt.legend();
```

In the above figure, $\mathbf{x}^*$ is the point where
$p(y=1 | \mathbf{x}) = p(y=0 | \mathbf{x})$.  To the right of it, we would classify examples as belonging to the positive class, and negative to the left.

We would like to learn to predict a probability $ p(y | \mathbf{x})$  for a binary classification problem using dot products.
A dot product prduces arbitrary numbers, so cannot serve to produce probabilities because they need to be between 0 and 1.
However, we can convert a dot product to a probability by applying a "squashing function" to the dot product e.g. using the so-called **logistic function**:

$$
p(y=1|\mathbf{x})= \sigma(\mathbf{w}^\top \mathbf{x})
$$

where $\sigma(s)$ is the logistic function which is defined by:

$$
\sigma(s) = \frac{1}{1 + e^{-s}}.
$$

More explicitly, this can be expressed as:

$$
p(y=1|\mathbf{x})= \frac{1}{1 + e^{-\mathbf{w}^\top \mathbf{x}}}.
$$

Let's take a look at the graph of the logistic function (aka the sigmoid function), which demonstrates its ability to serve as a "squashing function":

```python
def sigmoid(s):
    return 1/(1+np.exp(-s))
```

```python
s = np.linspace(-10,10, 100)
plt.figure(figsize=(5,3))
plt.plot(s, sigmoid(s))
plt.xlabel("s", fontsize=20)
plt.ylabel(r"${\sigma}(s)$", rotation=0, fontsize=20, labelpad=20);

```

We can express the probability $p(y|\mathbf{x})$ as follows:

$$
p(y|\mathbf{x})= \begin{cases}    \sigma(\mathbf{w}^\top \mathbf{x}) & \textrm{for} \space {y}=1 \newline
1- \sigma(\mathbf{w}^\top \mathbf{x}) \space & \textrm{for} \space {y}=0
\end{cases}
$$

The choice of the logistic function seems arbitrary.  The following discussion will provide some motivation.

First, let's consider the ratio

$$\frac{p(y=1|\mathbf{x})}{p(y=0|\mathbf{x})} = \frac{p(y=1|\mathbf{x})}{1- p(y=1|\mathbf{x})}.$$

This is called the **odds**.

What did we gain?  Whereas $p(y=1|\mathbf{x})$ is a number between 0 and 1, the odds is a number between 0 and infinity.

That's an improvement, but again there's no natural way to model that using a dot product.  So instead, we will focus on the log odds:

$$\log \frac{p(y=1|\mathbf{x})}{p(y=0|\mathbf{x})} = \log \frac{p(y=1|\mathbf{x})}{1- p(y=1|\mathbf{x})} .$$

The log-odds is between negative infinity, and infinity and can therefore be modeled using a dot product.

For your reference, here's a nice [video](https://www.youtube.com/watch?v=ARfXDSkQf1Y) that explains odds and odds ratios.

Our choice to model the log-odds using a dot product gives the following:

$$\log \frac{p(y=1|\mathbf{x})}{1- p(y=1|\mathbf{x})} = \mathbf{w}^\top \mathbf{x}.$$

Solving for $p(y=1|\mathbf{x})$ we find that

$$
p(y=1|\mathbf{x})= \frac{1}{1 + e^{-\mathbf{w}^\top \mathbf{x}}}.
$$

In conclusion, the choice to model the log-odds ratio using a dot product gave us the motivation to use the logistic function as a "squashing function" to generate a probability from a dot product.

### Exercise

Check that solving for $p(y=1|\mathbf{x})$ in
$$\log \frac{p(y=1|\mathbf{x})}{1- p(y=1|\mathbf{x})} = \mathbf{w}^\top \mathbf{x}$$ indeed gives us
$$
p(y=1|\mathbf{x})= \frac{1}{1 + e^{-\mathbf{w}^\top \mathbf{x}}}.
$$

### Exercise: Properties of the logistic function

Verify that:

$$
\sigma(s) = \frac{1}{1+e^{-s}} = \frac{e^s}{1+e^s}
$$

$$
1- \sigma(s) = \sigma(-s) = \frac{1}{1+e^{s}}
$$

$$
\sigma'(s) = \sigma(s)(1-\sigma(s))
$$

The following figure plots these functions:

```python
plt.figure(figsize=(5,3))
s = np.linspace(-10,10, 100)
plt.plot(s, sigmoid(s), label = r'$\sigma(s)$')
plt.plot(s, 1-sigmoid(s), label = r'$1-\sigma(s)$')
plt.plot(s, sigmoid(s) *(1-sigmoid(s)), label = r"$\sigma'(s)$")

plt.xlabel("s")
plt.legend();
```

### Is logistic regression really linear?

Given that the dot product is squashed using a non-linear function, it's not clear that the resulting classifier is indeed linear.  To answer this question, we first observe that

$$
p(y=1|\mathbf{x})=\frac{e^{\mathbf{w}^\top \mathbf{x}}}
{e^{\mathbf{w}^\top \mathbf{x}}+1}
$$

$$
{p(y=0|\mathbf{x})}=1-{p(y=1|\mathbf{x})}=\frac{1}{1 + e^{\mathbf{w}^\top \mathbf{x}}}
$$

To  figure  out  how  the  decision  boundary  looks  like, we consider the following equation:

$$
p(y=1|\mathbf{x})=p(y=0|\mathbf{x})
$$

Solving for $\mathbf{x}$ we get that
$
e^{\mathbf{w}^\top \mathbf{x}}=1
$
i.e. the points on the decision boundary satisfy
$
\mathbf{w}^\top \mathbf{x}=0
$, which is th equation for a hyperplane.

Here's a plot that illustrates that:

```python
from mpl_toolkits.mplot3d import Axes3D
from matplotlib.ticker import LinearLocator, FormatStrFormatter
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
X = np.linspace(-10, 10, 1000)
X1, Y1 = np.meshgrid(X, X )

Z = sigmoid(X1)
surf=ax.plot_surface(X1,Y1,Z,linewidth=0, cmap=plt.get_cmap('bone'), antialiased=False)

ax.set_zlim(0, 1.01)
ax.set_xlim(-10, 10)
ax.set_ylim(-10, 10)

ax.set_zlabel('probability',rotation=0)
ax.view_init(azim = 180+60, elev = 40)
```

### Maximum likelihood

We have expressed our classification problem in the language of probabilities, and therefore, we will apply the principle of *maximum likelihood* in order to find the optimal value of the weight vector
$\mathbf{w}$.

The likelihood function is the probability of the labels $y_1,\ldots,y_N$ given the corresponding $\mathbf{x}_1,\ldots, \mathbf{x}_N$:

\begin{equation}
p(y_1,\ldots,y_N | \mathbf{x}_1,\ldots, \mathbf{x}_N) = \prod_{i=1}^{N} p({y_i| \mathbf{x}_i}).
\end{equation}

This is valid because we assume that the data points
$(\mathbf{x}_1,y_1),\ldots,(\mathbf{x}_N,y_N)$ are independent, a standard assumption in machine learning.

The likelihood function depends on the values of model parameters, and using maximum likelihood we seek to find the parameter values that maximize the likelihood function over all choices of those parameters.
Intuitively, this selects the parameter values that make the observed data most probable.

Let's define $p_i = p(y_i = 1 | \mathbf{x}_i)$.
Using this notation we can express $p({y_i| \mathbf{x}_i})$ as follows:

$$
p({y_i| \mathbf{x}_i}) = p_i^{y_i} (1- p_i)^{1-y_i}.
$$

Here we assumed that $y_i$ is 0 or 1 for our binary classification problem.
We are going to need the negative log of this probability:

$$
- \log p({y_i| \mathbf{x}_i}) = - y_i \log p_i - (1-y_i) \log(1- p_i)
$$

Recall that

$$
p(y=1|\mathbf{x}) = \sigma(\mathbf{x}^\top\mathbf{w}) = \frac{1}{1 + e^{-\mathbf{w}^\top \mathbf{x}}}.
$$

And inserting the form of $p({y_i| \mathbf{x}_i})$:

$$
- \log p({y_i| \mathbf{x}_i}) = - y_i \log \left( \sigma(\mathbf{w}^\top \mathbf{x}_i) \right) - (1-y_i) \log \left(1 - \sigma(\mathbf{w}^\top \mathbf{x}_i)\right)
$$

### Maximizing the likelihood

We would like to find the weight vector that maximizes the likelihood of the data, i.e. find
$$
\max_{\mathbf{w}} \prod_{i=1}^{N}p(y_i|\mathbf{x}_i)
$$
This is equivalent to maximizing the log-likelihood (since the logarithm is a monotonic function):
$$
\max_{\mathbf{w}} \log \prod_{i=1}^{N}p(y_i| \mathbf{x}_i)
$$
The logarithm of a product is a sum of the logs of the terms so our objective becomes

$$
\max_{\mathbf{w}} \sum_{i=1}^{N}\log p(y_i|\mathbf{x}_i)
$$

We'll replace maximization with minimization by considering

$$
\min_{\mathbf{w}} -\frac{1}{N}\sum_{i=1}^{N}\log(p(y_i|\mathbf{x}_i))
$$

Using the form we derived above for $p({y_i| \mathbf{x}_i})$

$$
- \log p({y_i| \mathbf{x}_i}) = - y_i \log \left( \sigma(\mathbf{w}^\top \mathbf{x}_i) \right) - (1-y_i) \log \left(1 - \sigma(\mathbf{w}^\top \mathbf{x}_i)\right)
$$

our objective now becomes:

$$
\min_{\mathbf{w}} \frac{1}{N}\sum_{i=1}^{N} \left[ - y_i \log \left( \sigma(\mathbf{w}^\top \mathbf{x}_i) \right) - (1-y_i) \log \left(1 - \sigma(\mathbf{w}^\top \mathbf{x}_i)\right) \right]
$$

Let's take a closer look at the expression we derived for $- \log p({y_i| \mathbf{x}_i})$.  And let's consider the case $y_i = 1$.  In that case, only the first term is nonzero.  When the predicted probability is equal to 1, which is what we would like, it is equal to 0, and the further away from the desired value it is, the larger the value.  A similar observation occurs for $y_i=0$:  In this case only the second term contributes, and this term equals to 0 when the predicted probability is equal to 0, which is the desired value for negative examples.  Thus, the function

$$
\mathcal{l}^{CE}(y,\mathbf{x}; \mathbf{w}) = - y \log \left( \sigma(\mathbf{w}^\top \mathbf{x}) \right) - (1-y) \log \left(1 - \sigma(\mathbf{w}^\top \mathbf{x})\right)
$$

which is known as the **cross entropy loss** or **log loss**.  It quantifies  the discrepancy of the predicted probability from the desired label.  To demonstrate that it indeed functions like a loss function,
let's plot this function for $y = 1$ and $y=0$:

```python
def cross_entropy(p, y):
    return -(y*np.log(p)+(1-y)*np.log(1-p))

plt.figure(figsize=(5,3))
p = np.linspace(0.01, 0.99, 100)
plt.plot(p, cross_entropy(p, 1), label='cross-entropy for y=1')
plt.plot(p, cross_entropy(p, 0), label='cross-entropy for y=0')

plt.xlabel('probability')
plt.ylabel('cross entropy')
plt.legend();
```

### Logistic regression loss

We have finally arrived at the final form of the loss function for logistic regression:

$$
J(\mathbf{w}) = \frac{1}{N}\sum_{i=1}^{N} \left[ - y_i \log \left( \sigma(\mathbf{w}^\top \mathbf{x}_i) \right) - (1-y_i) \log \left(1 - \sigma(\mathbf{w}^\top \mathbf{x}_i)\right) \right]
$$

Unlike the case of linear regression where there is a closed-form solution, there is no such solution for the logistic regression loss function.  It turns out that the loss function is convex, and therefore there is a global minimum.  Gradient descent is therefore a reasonable approach.

The gradient of our loss function is:
$$
\nabla J(\mathbf{w}) = \frac{1}{N}\sum_{i=1}^{N} \left(\sigma(\mathbf{w}^\top \mathbf{x}_i) - y_i\right) \mathbf{x}_i
$$

### Gradient descent

**Input:** A labeled dataset; learning rate $\eta$

1. initialize $\mathbf{w}(0)$
2. for t = 0, 1, 2,... do
3. $\;\;\;\;\;\;$       compute the gradient: $\mathbf{g}_t$ = $\nabla  J(\mathbf{w}(t))$
4. $\;\;\;\;\;\;$       update the weights: $\mathbf{w}(t + 1) = \mathbf{w}(t) - \eta \mathbf{g}_t$
5. $\;\;\;\;\;\;$       if it is time to stop, break from loop.
6. end for
7. return the final weights.

This is called **batch gradient descent**.
The halting condition can be a specific number of iterations, or that the loss function appears to have converged, i.e. shows little change across epochs.

### Implementation

Let's implement our gradient descent logistic regression algorithm as a Python class.

```python
class logistic_regression:
    def __init__(self, lr=0.01, epochs=350):
        self.lr = lr
        self.epochs = epochs

    def decision_function(self, X):
        return 1/(1+np.exp(-np.dot(X, self.w)))

    def gradient(self, X, y):
        return (self.decision_function(X) - y)@X/len(X)

    def loss(self, X, y):
        pred = self.decision_function(X)
        loss_values = - y * np.log(pred) - (1 - y) * np.log(1 - pred)
        return np.mean(loss_values)

    def fit(self, X, y):
        self.w = np.zeros(X.shape[1])
        # save a history of loss values
        self.loss_history = [self.loss(X, y)]
        for epoch in range(self.epochs):
            self.w = self.w - self.lr * self.gradient(X, y)
            self.loss_history.append(self.loss(X, y))

    def predict(self, X):
        pred = np.where(self.decision_function(X) >= .5, 1, 0)
        return np.squeeze(pred)

```

```python
N = 20
X = np.linspace(-10,10,20)
y=np.where(X>=0, 1, 0)
X=X.reshape(-1, 1)
print (X.shape, y.shape)

plt.style.use('default')
plt.figure(figsize=(5,3))
plt.scatter(X, y, alpha=0.8)
plt.xlabel("input")
plt.ylabel("class label");
```

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,
    test_size=0.2, shuffle=True)
```

```python
lr=logistic_regression()
lr.fit(X_train,y_train)
```

```python
plt.figure(figsize=(5,3))
plt.plot(lr.loss_history)
plt.ylabel('loss')
plt.xlabel('epoch');
```

```python
y_pred=lr.predict(X_test)
print('accuracy: ', np.sum(y_pred == y_test)/len(y_test))
```

    accuracy:  1.0

```python
from sklearn.datasets import make_classification
X,y = make_classification(n_samples=100, n_features=2, n_informative=2,
                          n_redundant=0, n_repeated=0, n_classes=2,
                          n_clusters_per_class=1, class_sep=0.3,
                          random_state=1)
lr=logistic_regression(0.01, 500)
lr.fit(X, y)
y_pred=lr.predict(X)
print('accuracy: ', np.sum(y_pred == y)/len(y))
```

    accuracy:  0.9

Let's plot the resulting decision boundary and weight vector:

```python

plt.style.use('default')
plt.scatter(X[:,0], X[:,1], c=1-y, alpha=0.5, s=20, cmap='magma');

w = lr.w/np.linalg.norm(lr.w)

delta = 0.01
xmin=-1.25;xmax=0.75;ymin=-0.75;ymax=1.5
xs = np.arange(xmin, xmax, delta)
ys = np.arange(ymin, ymax, delta)
x_grid, y_grid = np.meshgrid(xs, ys)

Z = sigmoid(lr.w[0]*x_grid + lr.w[1]*y_grid)

im = plt.imshow(Z, origin='lower',
                cmap=plt.cm.gray, extent=(xmin,xmax,ymin,ymax) )

C = plt.contour(x_grid, y_grid, Z, 7,
                origin='lower',
                linewidths=(1,1,1,1,2,1,1),
                colors = 'black')

plt.clabel(C, inline=1, fmt='%1.2f')

plt.arrow(0,0, w[0],w[1], width=0.001,head_width=0.05, length_includes_head=True, alpha=1,linestyle='-',color='k');
plt.text(w[0], w[1]+0.05, r"$\mathbf{w}$");

```

```python
plt.figure(figsize=(5,3))
plt.plot(lr.loss_history)
plt.ylabel('loss')
plt.xlabel('epoch')
```

Make sure to train the algorithm for a sufficiently large number of epochs so that the loss has a chance to converg.

### Logistic regression in scikit-learn

Let's compare results using our implementation with scikit-learn:

```python
from sklearn.datasets import load_breast_cancer
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler

X, y = load_breast_cancer(return_X_y=True)

# standardize
X = StandardScaler().fit_transform(X)

from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y,
    test_size=0.3, shuffle=True, random_state=2)

# first, our implementation
lr = logistic_regression(0.01, 1000)

# train the model using the training sets
%timeit lr.fit(X_train, y_train)

# making predictions on the testing set
y_pred = lr.predict(X_test)

# compute accuracy
print("accuracy using our implementation: {:.3f} ".format(
      np.mean(y_test==y_pred) ))

lr_sklearn = LogisticRegression()
%timeit lr_sklearn.fit(X_train, y_train)
y_pred = lr_sklearn.predict(X_test)
print("accuracy using scikit-learn implementation: {:.3f} ".format(
      np.mean(y_test==y_pred) ))
```

    229 ms ± 148 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
    accuracy using our implementation: 0.965
    24.2 ms ± 4.38 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
    accuracy using scikit-learn implementation: 0.971

### Stochastic gradient descent

In machine learning large training sets are necessary for good generalization.  But training models over large training sets is computationally expensive.
That has led to the development of alternatives to gradient descent.
Stochastic gradient descent is one such algorithm, and is the basis for many of the commonly used algorithms in neural networks and deep learning.

The cost functions used in machine learning typically decompose as a sum or average over the loss for individual training examples.
This is the case for logistic regression where the overall loss is the average of individual loss values:

$$
J(\mathbf{w})= \frac{1}{N}\sum_{i=1}^N \mathcal{l}^{CE}(y_i,\mathbf{x}_i; \mathbf{w}),
$$

where

$$
\mathcal{l}^{CE}(y,\mathbf{x}; \mathbf{w}) = - y \log \left( \sigma(\mathbf{w}^\top \mathbf{x}) \right) - (1-y) \log \left(1 - \sigma(\mathbf{w}^\top \mathbf{x})\right).
$$

The gradient of $J(\mathbf{w})$ is an average over the gradients of the contributions of individual training examples.
For large datasets, this average is expensive to compute.
The key insight, is that this mean can be estimated using a smaller number of examples.
The intuition why this is possible is that in practice, examples are often highly correlated, so you don't need to use all of them to get a reasonable estimate of the gradient.  This idea is implemented using the technique of **stochastic gradient descent (SGD)**.

The batch version of SGD uses a small  subset (or *batch*) of randomly chosen examples $\mathcal{B}$ which is a subset of the entire training set $\mathcal{D}$.  The gradient is estimated as the mean of the gradients over the examples that make up the batch:

$$
\mathbf{g} = \frac{1}{|\mathcal{B}|}\sum_{i\in \mathcal{B}} \nabla \mathcal{l}(y_i,\mathbf{x}_i; \mathbf{w}),
$$

where $\mathcal{l}(y_i,\mathbf{x}_i; \mathbf{w})$ is the loss function and $|\mathcal{B}|$ is the cardinality of the set $\mathcal{B}$.

#### Batch stochastic gradient descent

**Input:** Labeled dataset; learning rate, $\eta$.

1. initialize weights $\mathbf{w}$
2. for t = 0, 1, 2,... do
3. $\;\;\;\;\;\;$       partition the data into fixed-size batches $\mathcal{B}_1,\ldots,\mathcal{B}_b$
4. $\;\;\;\;\;\;$       for j = 1,...,b :  # loop over batches:
5. $\;\;\;\;\;\;\;\;\;$       compute the gradient over batch j: $\mathbf{g} = \frac{1}{|\mathcal{B_j}|}\sum_{i\in \mathcal{B}_j} \nabla \mathcal{l}(y_i,\mathbf{x}_i; \mathbf{w})$
6. $\;\;\;\;\;\;\;\;\;$       update the weights: $\mathbf{w} = \mathbf{w} - \eta \mathbf{g}$
7. $\;\;\;\;\;\;$       if it is time to stop, break
8. end for
7. return the final weights.

The extreme case where the batch size is equal to 1 will be called **stochastic gradient descent** (as opposed to batch SGD).  In this case the algorithm takes on the following form:

#### Stochastic gradient descent

Input:  learning rate, $\eta$.
1. initialize weights $\mathbf{w}$
2. for t = 0, 1, 2,... do
3. $\;\;\;\;\;\;$       randomly shuffle the examples in the training set
4. $\;\;\;\;\;\;$       for i = 1,...,N :  # loop over the shuffled data
5. $\;\;\;\;\;\;\;\;\;$       compute the gradient on example i: $\mathbf{g} = \nabla \mathcal{l}(y_i,\mathbf{x}_i; \mathbf{w})$
6. $\;\;\;\;\;\;\;\;\;$       update the weights: $\mathbf{w} = \mathbf{w} - \eta \mathbf{g}$
7. $\;\;\;\;\;\;$       if it is time to stop, break
8. end for
7. return the final weights.

In regular gradient descent, the loss function is guaranteed to decrease at each epoch (as long as the learning rate is sufficiently small).
This is no longer the case for SGD.  However, SGD will usually converge faster overall.

### Exercise:

Implement stochastic gradient descent and compare its convergence to that of batch gradient descent by plotting the loss function after each epoch.

```python
class lr_sgd :
    pass
```

### Comments:

1. In practice logistic regression is solved by faster methods than gradient descent

2. There is an extension to multi-class classification

### Pros and Cons of logistic regression

#### Pros

- Fast training and prediction
- Easy to use: no tuning required
- Produces well-calibrated probabilities as output
- Interpretable classifier (weights provide information about relative contribution of features)

#### Cons

- Linear classifier
- Performance is often not competitive with the best supervised learning methods

**Note:**  Use Multinomial logistic regression when you have data with multiple classes.