## [Problem 11](https://projecteuler.net/problem=11)
Largest product in a grid

```python
import numpy as np
import itertools
from tqdm import tqdm
import sympy
```

```python
a = np.loadtxt("./data/p011.txt",dtype=np.int64)
N = 20
```

```python
greatest_product=0
for i,j in itertools.product(range(N),range(N)):
    #down
    product = 1
    for k in range(4):
        product *= a[i][(j+k)%N]
    if product > greatest_product:
        greatest_product = product
    #left
    product = 1
    for k in range(4):
        product *= a[(i+k)%N][j]
    if product > greatest_product:
        greatest_product = product
    #diagonally_right
    product = 1
    for k in range(4):
        product *= a[(i+k)%N][(j+k)%N]
    if product > greatest_product:
        greatest_product = product
    #diagonally_left
    product = 1
    for k in range(4):
        product *= a[(i-k)%N][(j+k)%N]
    if product > greatest_product:
        greatest_product = product

```

```python
greatest_product
```

    70600674

## [Problem 12](https://projecteuler.net/problem=12)
Highly divisible triangular number

```python
def divisible_num(n):
    num=1
    hoge=sympy.factorint(n)
    for i in hoge:
        num *= (hoge[i]+1)
    return num
```

```python
N=0
for i in range(1,1000000):
    N+=i
    #print(N)
    if divisible_num(N)>500:
        break
print(N)
```

    76576500

## [Problem 13](https://projecteuler.net/problem=13)
Large sum

```python
a = np.loadtxt("./data/p013.txt",dtype=str)
```

```python
ans = 0
for i in range(100):
    ans += int(a[i])
int(str(ans)[:10])
```

    5537376230

## [Problem 14](https://projecteuler.net/problem=14)
Longest Collatz sequence

```python
def seq(n):
    if n%2==0:
        return n/2
    else:
        return 3*n+1
```

```python
def collatz(N):
    num = 1
    while True:
        N=seq(N)
        num+=1
        if N==1:
            break
    return num
```

```python
max_num=0
hoge=[]
for N in tqdm(range(1,10**6,2)):
    num=collatz(N)
    if num>max_num:
        max_num=num
        ans=N
if ans*2<10**6:
    ans*=2
print(ans)
```

    100%|██████████| 500000/500000 [00:21<00:00, 22802.29it/s]

    837799

## [Problem 16](https://projecteuler.net/problem=16)
Power digit sum

```python
hoge = str(2**1000)
ans = 0
for i in hoge:
    ans += int(i)
print(ans)
```

    1366

## [Problem 17](https://projecteuler.net/problem=17)
Number letter counts

```python
#one,two,three,four,five,six,seven,eight,nine
num_letter = {1:3,2:3,3:5,4:4,5:4,6:3,7:5,8:5,9:4}
#eleven,twelve,thirteen,fourteen,fifteen,sixteen,seventeen,eighteen,nineteen
num_letter.update({11:6,12:6,13:8,14:8,15:7,16:7,17:9,18:8,19:8})
#ten,twenty,thirty,forty,fifty,sixty,seventy,eighty,ninety
num_letter.update({10:3,20:6,30:6,40:5,50:5,60:5,70:7,80:6,90:6})
#hundred,thousand,and
num_letter.update({100:7,1000:8,"and":3})
```

```python
ans = 0
for i in range(1,20):
    ans += num_letter[i]
for i in range(20,100):
    if i%10==0:
        ans += num_letter[i]
    else:
        ans += (num_letter[i-i%10]+num_letter[i%10])
for i in range(100,1000):
    ans += (num_letter[i//100]+num_letter[100])
    mod = i%100
    if mod!=0:
        ans += num_letter["and"]
        if mod<20:
            ans += num_letter[mod]
        else:
            if mod%10==0:
                ans += num_letter[mod]
            else:
                ans += (num_letter[mod-mod%10]+num_letter[mod%10])
ans += (num_letter[1]+num_letter[1000])
ans
```

    21124

## [Problem 18](https://projecteuler.net/problem=18)
Maximum path sum I

```python
with open("./data/p018.txt") as f:
    l = f.readlines()
triangle = []
for line in l:
    line = line.split()
    a = [int(i) for i in line]
    triangle.append(a)
triangle
```

    [[75],
     [95, 64],
     [17, 47, 82],
     [18, 35, 87, 10],
     [20, 4, 82, 47, 65],
     [19, 1, 23, 75, 3, 34],
     [88, 2, 77, 73, 7, 63, 67],
     [99, 65, 4, 28, 6, 16, 70, 92],
     [41, 41, 26, 56, 83, 40, 80, 70, 33],
     [41, 48, 72, 33, 47, 32, 37, 16, 94, 29],
     [53, 71, 44, 65, 25, 43, 91, 52, 97, 51, 14],
     [70, 11, 33, 28, 77, 73, 17, 78, 39, 68, 17, 57],
     [91, 71, 52, 38, 17, 14, 91, 43, 58, 50, 27, 29, 48],
     [63, 66, 4, 68, 89, 53, 67, 30, 73, 16, 69, 87, 40, 31],
     [4, 62, 98, 27, 23, 9, 70, 98, 73, 93, 38, 53, 60, 4, 23]]

```python
dp = [[0 for _ in range(15)] for _ in range(15)]
```

```python
dp[0][0]=triangle[0][0]
for i in range(1,15):
    for j in range(i+1):
        dp[i][j]=dp[i][j]=dp[i-1][j]+triangle[i][j] if dp[i-1][j]>dp[i-1][j-1] else dp[i-1][j-1]+triangle[i][j]
#dp
max(max(dp))
```

    1074

## [Problem 19](https://projecteuler.net/problem=19)
Counting Sundays

```python
#1,2,3,4,5,6,7,8,9,10,11,12
months=[3,0,3,2,3,2,3,3,2,3,2,3]
```

```python
week=1
tgis=0
for _ in range(1901,2000):
    for month in months:
        if week==0:
            tgis+=1
        week+=month
        week=week%7
```

```python
months[2]=1
for month in months:
    if week==0:
        tgis+=1
    week+=month
    week=week%7
```

```python
print(tgis)
```

    171

## [Problem 20](https://projecteuler.net/problem=20)
Factorial digit sum

```python
import math
a=math.factorial(100)
```

```python
ans=0
for i in str(a):
    ans += int(i)
print(ans)
```

    648

```python

```


## Solving symmetrical Berman's equation (solution is gauranteed)
\begin{equation}
\Psi _{\eta \eta \tau }^{(n)}=\frac{1}{R_s}\Psi_{\eta \eta \eta \eta
}^{(n)}+ \sum\limits_{i = 0}^n{(2i+1)\left[ \Psi ^{(i)} \Psi _{\eta \eta
\eta }^{(n-i)} -\Psi_{\eta}^{(n-i)} \Psi _{\eta \eta}^{(i)}\right] } \\
\quad \Psi^{(n)}(\eta=\mp 1) = 0, \qquad \Psi^{(n)}_\eta(\eta=\mp 1) = \frac{(-1)^{n+1}}{(2n+1)!}, \qquad
-1 \le \eta \le1
\end{equation}
From the paper by Chunqing Lu "On the Existence of Steady Flow in a Channel
with one Porous Wall or Two Accelerating Walls" the solution for $\Psi^{(0)}$ is gauranteed at all Reynolds numbers.
The steady state equation above will be solved by using a transformation that will convert the problem to a IVP.
The ODE integrator will then use a callback function to check for when the integration has satisfied its termination criteria

```julia
using DifferentialEquations
using Plots
using Sundials
using BenchmarkTools
using Calculus
using DelimitedFiles
using Interpolations
using LaTeXStrings
using FFTW
```

## Solving the leading order equation $\Psi(\eta)^{(0)}$ steady state solution

### Low Reynolds number $R<17$
For small values of the Reynolds number ($R<17)$, the first order approximation $\Psi^{(0)}$ can be calculated fairly quickly using standard boundary value solvers (memory allocation is around 15GB at R=50 using MIRK4 two point boundary value solver). With a good initial guess value and correct values for ```abstol``` and ```reltol``` the Shooting method also works very well with all solvers e.g. Vern7()

### Intermediate Reynolds numbers $17<R<65$
Despite the equations being unstable to computation for $R>65$, they are still solvable for $17<R<65$ and the only reliable way to solve for $\Psi^{(0)}$ is to use the ```TwoPointBVProblem``` or Shooting with GeneralMirk4/Rosenbrock23. The latter solver requirement is due to the equations becoming stiff.

```julia
sol_psi = Array{ODESolution,1}(undef, 20) #Declaring an array for every ψ(η)^{(n)} solution
series_term = 0
Reynolds_number = 1.2345

p=(Reynolds_number, series_term)
const etaspan=(0.0,1.0)
ψ0= [0.0, 1.0, 0.0, -pi^2] #This contains ψ^{(0)}, ψ^{(0)}_η, ψ^{(0)}_ηη, ψ^{(0)}_ηηη

"""This is the differential equation for the ``\\psi(\\eta)^{(0)}`` term in the series.
``\\Psi_{\\eta \\eta \\eta \\eta
}^{(n)} = R_s( \\Psi_{\\eta}^{(0)} \\Psi _{\\eta \\eta}^{(0)} - \\Psi ^{(0)} \\Psi _{\\eta \\eta
\\eta }^{(0)})``
"""
function steady_diffeq_psi0!(dψ, ψ, p, η)
  dψ[1] = ψ[2]
  dψ[2] = ψ[3]
  dψ[3] = ψ[4]
  dψ[4] = (ψ[2]*ψ[3]-ψ[4]*ψ[1])*p[1]
end

"""This is the differential equation for the ``\\psi(\\eta)^{(n)}`` term in the series.
``\\Psi_{\\eta \\eta \\eta \\eta
}^{(n)} = R_s\\sum\\limits_{i = 0}^n{(2i+1)( \\Psi_{\\eta}^{(n-i)} \\Psi _{\\eta \\eta}^{(i)} - \\Psi ^{(i)} \\Psi _{\\eta \\eta
\\eta }^{(n-i)})}``
"""
function steady_diffeq_series!(dψ, ψ, p, η)
    solut_ψ0 = p[3]::ODESolution #This avoids the use of Global sol_psi0 and should speed up the code, this variable type checked
    solut_ψ  = p[4]::Vector{ODESolution}  #This avoids the use of Global sol_psi and should speed up the code,  this variable type checked
    sum_of_nonlinears = 0.0
    n = round(Int, p[2]) # This rounding off prevents errors are thrown when p vector is changed by Julia from Int to Floats
    for i= 1:(n-1)
        sum_of_nonlinears += (2*i+1)*(solut_ψ[n-i](η)[2]*solut_ψ[i](η)[3]- solut_ψ[i](η)[1]*solut_ψ[n-i](η)[4])
    end
    dψ[1] = ψ[2]::Float64
    dψ[2] = ψ[3]::Float64
    dψ[3] = ψ[4]::Float64
    dψ[4] = (ψ[2]*solut_ψ0(η)[3]-solut_ψ0(η)[1]*ψ[4])*p[1] + (2*p[2]+1)*(solut_ψ0(η)[2]*ψ[3]-ψ[1]*solut_ψ0(η)[4])*p[1] + sum_of_nonlinears*p[1]
end

"""
This is the Jacobian function for the steady ``\\psi(\\eta)^{(0)}`` differential equation
"""
function jacobian_steadypsi0(J,ψ,p,η)

    J[1,1] = J[1,3] = J[1,4] =  0.0;
    J[1,2] = 1.0

    J[2,1] = J[2,2] = J[2,4] =  0.0;
    J[2,3] = 1.0

    J[3,1] = J[3,2] = J[3,3] =  0.0;
    J[3,4] = 1.0

    J[3,1] = -ψ[4]*p[1]
    J[3,2] =  ψ[3]*p[1]
    J[3,3] =  ψ[2]*p[1]
    J[3,4] = -ψ[1]*p[1]
  nothing
end

"""
This is the boundary condition for the nth term  ``\\psi(\\eta)^{(n)}``. The p-vector supplies the correct value
of n in the 2nd position i.e. p[2] = n
"""
function bc!(residual, ψ, p, η)  # ψ[1] is the beginning of the etaspan 0< η < 1, and psi[end] is the ending
    residual[1] = ψ[1][1]         # The ψ[1] (i.e,. ψ^{n}(η=0)=0) solution at the beginning of the η span should be 0
    residual[2] = ψ[1][3]         #Second derivative ψ^{n}_ηη(η=0)=0 first η step
    residual[3] = ψ[end][1]       # the solution at the end of the time span should be 0
    residual[4] = ψ[end][2] - (-1)^(p[2]+1)/factorial(2*p[2]+1)      #First derivative should be -1 at end time step
end

function Ψ_smallr(θ::Float64, η::Float64 ,r::Float64)
    return(
        -η*(η^2-1)*sin(θ)/2.0 +
        r*η*(η^2-1)^2.0*(2.0+η^2)*sin(2*θ)/560.0 +
        r^2.0*η*(η^2-1)^2*((-591 -2294*η^2 + 161*η^4 + 1428*η^6)*sin(θ) +
            3.0*(423 + 166*η^2 - 553*η^4 - 84*η^6)sin(3*θ))/62092800.0#+
       # r^3.0*η*(η - 1)^2*(η + 1)^2*(2184*η^10*cos(θ)^2 + 1540*η^10 + 24213*η^8*cos(θ)^2 - 44205*η^8 + 18942*η^6*cos(θ)^2 + 33446*η^6 - 101444*η^4*cos(θ)^2 + 66052*η^4 - 23086*η^2*cos(θ)^2 - 6746*η^2 - 14889*cos(θ)^2 - 9383)*sin(θ)*cos(θ)/28252224000 +
       # r^4.0*η*(η - 1)^2*(η + 1)^2*(968647680*η^14*sin(θ)^5 + 14774464320*η^14*sin(θ) + 3510571680*η^14*sin(3*θ) - 393513120*η^14*sin(5*θ) - 39671478000*η^12*sin(θ)^5 - 75537556710*η^12*sin(θ) - 28276357725*η^12*sin(3*θ) - 3752563815*η^12*sin(5*θ) - 181021246560*η^10*sin(θ)^5 + 357286638324*η^10*sin(θ) + 75142631718*η^10*sin(3*θ) - 1180776366*η^10*sin(5*θ) + 677920708080*η^8*sin(θ)^5 - 645834258762*η^8*sin(θ) - 123813530379*η^8*sin(3*θ) - 1711397457*η^8*sin(5*θ) + 193167076032*η^6*sin(θ)^5 - 82870720296*η^6*sin(θ) + 215941919508*η^6*sin(3*θ) + 59820477948*η^6*sin(5*θ) + 153978403824*η^4*sin(θ)^5 - 123221433370*η^4*sin(θ) + 181687022685*η^4*sin(3*θ) - 60662329497*η^4*sin(5*θ) - 500785425504*η^2*sin(θ)^5 + 367229681332*η^2*sin(θ) - 134582502330*η^2*sin(3*θ) - 33288605262*η^2*sin(5*θ) - 1507832445168*sin(θ)^5 + 969877856394*sin(θ) - 482955016725*sin(3*θ) - 9092499471*sin(5*θ))/16189310893916160000
        )
end

function psi_smallr(i, η ,r)
    return((-1.0)^(i+1)*η*(η^2-1)/(2.0*factorial(2*i+1)) + r*(-1)^i*(2.0)^(2*i-3)*η*(η^2-1)^2.0*(2.0+η^2)/(35.0*factorial(2*i+1))+r^2.0*η*(η^2-1)^2*((-591 -2294*η^2 + 161*η^4 + 1428*η^6)*(-1.0)^i +
                        3.0*(423 + 166*η^2 - 553*η^4 - 84*η^6)*(-1)^i*(3)^(2*i+1))/(62092800*factorial(2*i+1)))
end

function psi_smallr_d1(i, η,  r )
    return((-1.0)^(i+1)*(3.0*η^2-1)/(2.0*factorial(2*i+1)) + r*(-1)^i*(2.0)^(2*i-3)*(7*η^6-9*η^2+2)/(35.0*factorial(2*i+1)))
end

function psi_smallr_d2(i, η, r)
    return((-1.0)^(i+1)*(6*η)/(2.0*factorial(2*i+1)) + r*(-1)^i*(2.0)^(2*i-3)*(42*η^5-18*η)/(35.0*factorial(2*i+1))+
    r^2.0*(3*(-197 - 1112*η^2 + 6930*η^4 - 2772*η^6 - 8085*η^8 + 5236*η^10)*(-1.0)^i +
            (1269 - 6120*η^2 - 6930*η^4 + 24948*η^6 - 10395*η^8 - 2772*η^10)*(-1)^i*(3)^(2*i+1)) /(62092800*factorial(2*i+1)))
end

function psi_smallr_d3(i, η,  r)
    return((-1.0)^(i+1)*(6)/(2.0*factorial(2*i+1)) + r*(-1)^i*(2.0)^(2*i-3)*(210*η^4-18)/(35.0*factorial(2*i+1)))
end

function psi_bigr(i, η ,r)
    return( (-1)^i*sin(pi*η)/(pi*factorial(2*i+1)))
end

function psi_bigr_d1(i, η ,r)
    return( (-1)^i *cos(pi*η)/factorial(2*i+1))
end

function psi_bigr_d3(i, η ,r)
    return( -(-1)^i*pi^2*cos(pi*η)/factorial(2*i+1))
end

psi0_d2_stored = [2.91, 2.57,  2.18, 1.82,  (1.5-0.3),       1.02,  0.847,   0.71, 0.599, 0.511, 0.442,  0.38,  0.33]
psi0_d3_stored = [-2.32,0.22,  2.98, 5.2,  (7.02+5.0), (9.29+2.7), 9.9598,  (10.42+1), 10.72, 10.93, 11.05, 11.14, 11.19]
Reynolds_values_stored = [1.0,5,10,  15,           20,         30,     35,     40,    45,    50,    55,    60,    65]
itp_d2 = Interpolations.interpolate((Reynolds_values_stored,), psi0_d2_stored , Gridded(Linear()))
itp_d3 = Interpolations.interpolate((Reynolds_values_stored,), psi0_d3_stored, Gridded(Linear()))

function psi0_bc_initialguess(Reynolds_number)
    if Reynolds_number < 50.0
        return[0.0, psi_smallr_d1(0, 0.0 , Reynolds_number), 0.0, psi_smallr_d3(0, 0.0 ,Reynolds_number)]
    else
        return[0.0, psi_bigr_d1(0, 0.0 ,Reynolds_number), 0.0, psi_bigr_d3(0, 0.0 ,Reynolds_number)] #This contains ψ, ψ_η, ψ_ηη, ψ_ηηη
    end
end

#The code below is for testing purposes only, it treats the psi problem as an IVP starting at eta=-1
#p=(Reynolds_number, 0)
#ψ0= [0.0, -1.0, 0.02, 2.9836060306009644]
#prob = ODEProblem(steady_diffeq_psi0!,ψ0,etaspan,p , jac = jacobian_steadypsi0)
#sol =  solve(prob,reltol=1e-6)
#@benchmark solve(prob,reltol=1e-6,save_everystep=true)
```

    psi0_bc_initialguess (generic function with 1 method)

```julia
Reynolds_number = 100.0
series_term = 0
p=(Reynolds_number, series_term)
ψ0 = psi0_bc_initialguess(Reynolds_number)
bvp_psi0_2point = TwoPointBVProblem(steady_diffeq_psi0!, bc!, ψ0, etaspan, p , jac = jacobian_steadypsi0)
sol_psi0 =  @time solve(bvp_psi0_2point, alg_hints = [:stiff], GeneralMIRK4(),dt=0.01) #Very accurate solver

```

      1.213639 seconds (14.38 M allocations: 1.350 GiB, 18.94% gc time, 3.44% compilation time)

    retcode: Success
    Interpolation: 1st order linear
    t: 101-element Vector{Float64}:
     0.0
     0.01
     0.02
     0.03
     0.04
     0.05
     0.06
     0.07
     0.08
     0.09
     0.1
     0.11
     0.12
     ⋮
     0.89
     0.9
     0.91
     0.92
     0.93
     0.94
     0.95
     0.96
     0.97
     0.98
     0.99
     1.0
    u: 101-element Vector{Vector{Float64}}:
     [3.834666615758363e-36, 0.9009447117537713, 0.0, -7.663934742673053]
     [0.009008169795013786, 0.9005615150166683, -0.07663933927508719, -7.663929855353097]
     [0.01800867565772932, 0.899411925782087, -0.1532783858968905, -7.66385680877828]
     [0.026993853694816324, 0.8974959528011967, -0.22991568649413924, -7.6635425365729555]
     [0.03595604020660508, 0.8948136298547821, -0.30654750762437627, -7.662705437992805]
     [0.044887572069110517, 0.8913650438306884, -0.38316679687935207, -7.660965248961628]
     [0.05378078744698747, 0.8871503726408891, -0.45976225917345626, -7.657856104713693]
     [0.06262802692956057, 0.8821699316776915, -0.5363175768012944, -7.652842105937127]
     [0.0714216351677245, 0.8764242272591171, -0.6128107943944138, -7.645334594388699]
     [0.08015396307312847, 0.8699140153429594, -0.6892138816695437, -7.6347102938820015]
     [0.08881737062332833, 0.8626403637024588, -0.7654924784058351, -7.620329479061467]
     [0.09740423029843583, 0.8546047157532146, -0.841605817967054, -7.601553393368539]
     [0.10590693115701463, 0.8458089542951968, -0.9175068183824171, -7.577760240865637]
     ⋮
     [0.10662400894332194, -0.9180088855344819, -1.2889447816730983, 9.18320546023379]
     [0.09738101185204243, -0.9304357926195989, -1.1961007205465262, 9.384754342850165]
     [0.0880184212718762, -0.9419242665376255, -1.1012665949921903, 9.581235182544678]
     [0.0785457201975831, -0.9524546581568348, -1.0044924969303612, 9.77277433385519]
     [0.06897258563392726, -0.9620078120495519, -0.9058269820900585, 9.959554047453839]
     [0.05930888374509837, -0.9705650481924701, -0.8053164674875252, 10.141821451054666]
     [0.0495646652188334, -0.9781081371643848, -0.7030045304430058, 10.3198993228493]
     [0.039750160915531685, -0.9846192677605756, -0.5989310892137031, 10.494199071675139]
     [0.029875777884383482, -0.9900810057214665, -0.4931314406785006, 10.665236443517669]
     [0.019952095843144748, -0.9944762419995633, -0.385635124666002, 10.83365060841718]
     [0.00998986423534185, -0.9977881286490407, -0.27646457715335393, 11.000227454100093]
     [1.1212799685637712e-37, -1.0, -0.16563352523866146, 11.16592813399404]

```julia
if(p[1] < 52.5)
    run(`cc -ansi steady_d4_julia.c -o steadyd4julia -lm `)
    io = open("Reynolds_number.txt", "w");
    write(io, string(p[1]));
    close(io);
    sleep(1)
    run(`./steadyd4julia $p\[1\]`)
    MatrixPsis=readdlm("terms_c.data",Float64,);
end
```

```julia
print(varinfo(r"sol_psi0"))
pyplot()
plot(sol_psi0,vars=(1), linecolor=:red, label = "Julia solution",xlabel=L"\eta", ylabel=L"\Psi^{(0)}")
if(p[1] < 52.5)
    plot!(MatrixPsis[:,1], MatrixPsis[:, 2], linestyle=:dot, linecolor=:green, label = "C code solution")
    plot!( x-> psi_smallr(0, x, p[1]), 0.0,1.0,linestyle=:dot, linecolor=:orange, label = "Small R analytical",title="R = "*string(p[1])*L",   \Psi^{(0)}", legend=:bottomright)
end

plot!( x-> psi_bigr(0, x, p[1]), 0.0,1.0,linestyle=:dash, linecolor=:blue, label = "Large R analytical",title="R = "*string(p[1])*L",   \Psi^{(0)}", legend=:bottomright, yguidefontrotation=-90)

```

```julia

psi_plot = Array{Plots.Plot{Plots.PyPlotBackend},1}()
u0 =[]
max_terms = 5
for i = 1:max_terms
    series_term = i
    p=(Reynolds_number, series_term, sol_psi0, sol_psi)
    if (Reynolds_number < 40.0 )
        push!(u0, [0.0, psi_smallr_d1(i, 0.0 ,Reynolds_number), 0.0, psi_smallr_d3(i, 0.0 ,Reynolds_number)])
    else
        push!(u0, [0.0, psi_bigr_d1(i, 0.0 ,Reynolds_number), 0.0, psi_bigr_d3(i, 0.0 ,Reynolds_number)])
    end

    bvp_sol= TwoPointBVProblem(steady_diffeq_series!, bc!, u0[i], etaspan, p)
    sol_psi[ i ] = @time solve(bvp_sol, alg_hints=[:stiff], GeneralMIRK4(), dt=0.01)
    plot(sol_psi[i],vars=(1), linecolor=:red, label = "Julia solution",xlabel=L"\eta", ylabel=L"\Psi^{(%$i)}", size=(800,400))
    if(p[1] < 35)
        plot!(MatrixPsis[:,1],MatrixPsis[:,i+2], linecolor=:green, label = "C code solution")
    end
    if(p[1] < 35  )
            plot!(x-> psi_smallr(i, x, p[1]),-1,1,linestyle=:dot, linecolor=:orange, label = "Small R analytical",title="R = "*string(p[1])*L",   \Psi^{(%$i)}", legend=:outertopright)
    end
    push!(psi_plot,plot!(x-> psi_bigr(i, x, p[1]),-1,1,linestyle=:dash, linecolor=:blue, label = "Large R analytical",title="R = "*string(p[1])*L",   \Psi^{(%$i)}", legend=:outertopright, yguidefontrotation=-90))
end
```

      1.170787 seconds (17.66 M allocations: 856.197 MiB, 13.52% gc time, 7.03% compilation time)
      2.125291 seconds (38.56 M allocations: 1.813 GiB, 13.40% gc time)
      6.470256 seconds (119.13 M allocations: 5.592 GiB, 14.75% gc time)
     15.899961 seconds (295.43 M allocations: 13.855 GiB, 15.48% gc time)
     28.847457 seconds (541.74 M allocations: 25.393 GiB, 15.48% gc time)

```julia
for i in 1:max_terms
          display(psi_plot[i])
          sleep(1)
 end

```

## Solving the for $\Psi^{0}$ using shooting method

```julia
Reynolds_number =200.0
series_term = 0
ψ0= psi0_bc_initialguess(Reynolds_number)
p=(Reynolds_number, series_term)
bvp_shooting = BVProblem(steady_diffeq_psi0!, bc!, ψ0, etaspan,p , jac=jacobian_steadypsi0)
sol_shooting = @time solve(bvp_shooting, alg_hints=[:stiff], reltol=1e-5, abstol=1e-5 , Shooting(Rosenbrock23()),dt=0.01) #Excellent solver
#sol_shooting = @time solve(bvp_shooting, alg_hints=[:stiff], reltol=1e-8, abstol=1e-6 , Shooting(CVODE_BDF())) #Ok but need to have tolerances adjusted some errors on boundary
#sol_shooting = @time solve(bvp_shooting, alg_hints=[:stiff], reltol=1e-8, abstol=1e-6 , Shooting(Rodas4P())) #Ok but need to have tolerances adjusted some errors on boundary
#sol_shooting = @time solve(bvp_shooting, alg_hints=[:stiff], reltol=1e-9, abstol=1e-9 , Shooting(RadauIIA5())) #Good, FAST, STABLE but some errors on boundary
#sol_shooting = @time solve(bvp_shooting, alg_hints=[:stiff], reltol=1e-8, abstol=1e-6 , Shooting(TRBDF2()))
#sol_shooting = @time solve(bvp_shooting, alg_hints=[:stiff], reltol=1e-6, abstol=1e-6 , Shooting(Rosenbrock23()))

```

      0.997417 seconds (1.63 M allocations: 94.171 MiB, 2.01% gc time, 99.01% compilation time)

    retcode: Success
    Interpolation: specialized 2nd order "free" stiffness-aware interpolation
    t: 106-element Vector{Float64}:
     0.0
     0.01
     0.013538511306414314
     0.026599447100528542
     0.032001100607511136
     0.045404104349177465
     0.0523556742560698
     0.06494631082669315
     0.07293882375769402
     0.08454842577221887
     0.09314438439630089
     0.10395776857966067
     0.11288032446163535
     ⋮
     0.8943665405635293
     0.9045710634338329
     0.9147755863041365
     0.9249801091744401
     0.9351846320447437
     0.9453891549150473
     0.9555936777853509
     0.9657982006556545
     0.9760027235259581
     0.9851267904234549
     0.9942508573209516
     1.0
    u: 106-element Vector{Vector{Float64}}:
     [-5.074926060678379e-22, 0.9504330727436989, 0.0, -8.568900846747393]
     [0.009502556049748314, 0.9500046276500344, -0.08568901833727831, -8.56890160313269]
     [0.012863543078418386, 0.949647770273577, -0.11601013162234369, -8.568875657145357]
     [0.025252982681622183, 0.9474017081918232, -0.22792508991179627, -8.568394201055547]
     [0.0303669135771105, 0.9460455356982318, -0.27420695493582964, -8.567759327917148]
     [0.043017864301823076, 0.9416008955461087, -0.3890190551166303, -8.564212950118199]
     [0.049553473458866626, 0.9386897069034277, -0.4485416433920138, -8.560592952413465]
     [0.06133308495240068, 0.932364082753612, -0.5562600081346668, -8.549703161191847]
     [0.0687663462780194, 0.927645225714964, -0.6245517840486476, -8.538943204412144]
     [0.07949108402548487, 0.9198195649891346, -0.7235613601198285, -8.516816448145605]
     [0.0873699635101399, 0.9132855363490431, -0.7966778273549502, -8.49454341085333]
     [0.09719687240789977, 0.90417496663582, -0.8883422986235184, -8.45853037712088]
     [0.10522782034191247, 0.8959125808091366, -0.9636510269717634, -8.421396877442495]
     ⋮
     [0.10327916904133468, -0.9366049380619135, -1.1249437061478904, 9.447296176771086]
     [0.09366508320000291, -0.9475891990821508, -1.0277429131235836, 9.602629549195072]
     [0.08394400268414184, -0.9575736366973023, -0.9289929314327601, 9.75093697160458]
     [0.07412620934259874, -0.9665428043368368, -0.8287645807137408, 9.892396200329461]
     [0.06422213900696004, -0.9744819672860493, -0.727126492722169, 10.027259515217748]
     [0.054242374358997945, -0.9813770761466644, -0.6241442713665095, 10.155869638396045]
     [0.04419763811653379, -0.987214730804014, -0.5198794680280224, 10.27868012804882]
     [0.03409878664531323, -0.991982132752436, -0.41438831921762775, 10.396281714409403]
     [0.023956804129533726, -0.9956670230026685, -0.3077201759726289, 10.509436586032173]
     [0.014861122091406343, -0.9980355359371386, -0.21138229119063104, 10.607680976169247]
     [0.005747854913650134, -0.999521000888278, -0.1141570302299134, 10.704036146317195]
     [1.6329927631522886e-9, -0.9999999947047237, -0.052444622523899455, 10.764339913702424]

```julia
plot(sol_shooting, vars=(1), label= "Steady "*L"\Psi^{(0)}"*" solution via shooting")
plot!(sol_psi0,vars=(1), linestyle=:dot, linewidth = 2,linecolor=:red, label = "Steady "*L" Ψ^{(0)}"*" solution via BVP",xlabel=L"\eta", ylabel=L"\Psi^{(0)}")
xlabel!(L"\eta")
ylabel!(L"\Psi^{(0)}")
```

## Comparing numerical results with asymptotic series.

```julia
function Ψ(θ::Float64,η::Float64,sol_psi0,sol_psi, number_terms::Int64)
    psi_value= sol_psi0(η)[1]*θ
    for i = 1:number_terms-1
        psi_value += sol_psi[i](η)[1]*θ^(2*i+1.0)
    end
    return psi_value
end

function Ψ_smallr(θ::Float64, η::Float64 ,r::Float64, number_of_terms::Int)
    terms = [
        -η*(η^2-1)*sin(θ)/2.0,

        r*η*(η^2-1)^2.0*(2.0+η^2)*sin(2*θ)/560.0,

        r^2.0*η*(η^2-1)^2*((-591 -2294*η^2 + 161*η^4 + 1428*η^6)*sin(θ) +
            +3.0*(423 + 166*η^2 - 553*η^4 - 84*η^6)sin(3*θ))/62092800.0,

        r^3.0*η*(η - 1)^2*(η + 1)^2*(2184*η^10*cos(θ)^2 + 1540*η^10 + 24213*η^8*cos(θ)^2 - 44205*η^8 +
            + 18942*η^6*cos(θ)^2 + 33446*η^6 - 101444*η^4*cos(θ)^2 + 66052*η^4 - 23086*η^2*cos(θ)^2 -
            6746*η^2 - 14889*cos(θ)^2 - 9383)*sin(θ)*cos(θ)/28252224000,

        r^4.0*η*(η - 1)^2*(η + 1)^2*(968647680*η^14*sin(θ)^5 + 14774464320*η^14*sin(θ) + 3510571680*η^14*sin(3*θ) -
            393513120*η^14*sin(5*θ) - 39671478000*η^12*sin(θ)^5 - 75537556710*η^12*sin(θ) -
            28276357725*η^12*sin(3*θ) - 3752563815*η^12*sin(5*θ) - 181021246560*η^10*sin(θ)^5 +
            357286638324*η^10*sin(θ) + 75142631718*η^10*sin(3*θ) - 1180776366*η^10*sin(5*θ) +
            677920708080*η^8*sin(θ)^5 - 645834258762*η^8*sin(θ) - 123813530379*η^8*sin(3*θ) -
            1711397457*η^8*sin(5*θ) + 193167076032*η^6*sin(θ)^5 - 82870720296*η^6*sin(θ) +
            215941919508*η^6*sin(3*θ) + 59820477948*η^6*sin(5*θ) + 153978403824*η^4*sin(θ)^5 -
            123221433370*η^4*sin(θ) + 181687022685*η^4*sin(3*θ) - 60662329497*η^4*sin(5*θ) -
            500785425504*η^2*sin(θ)^5 + 367229681332*η^2*sin(θ) - 134582502330*η^2*sin(3*θ) -
            33288605262*η^2*sin(5*θ) - 1507832445168*sin(θ)^5 + 969877856394*sin(θ) - 482955016725*sin(3*θ) -
            9092499471*sin(5*θ))/16189310893916160000
        ]
    if number_of_terms < 6
        return sum(terms[1:number_of_terms])
    else
        return term[1]
    end
end

function Ψ_bigr(θ::Float64, η::Float64)
    return(sin(pi*η)*sin(θ)/pi)
end
```

    Ψ_bigr (generic function with 1 method)

```julia
η_value = 0.6
plot(x->Ψ(x,η_value,sol_psi0,sol_psi, max_terms),0,pi, label = "Numerical series $max_terms terms")
xticks!(([0:π/6:π;], ["0", "\\pi/6", "\\pi/3", "\\pi/2", "2\\pi/3", "5\\pi/6","\\pi"]))
xlabel!(L"\theta")
ylabel!(L"\Psi(\theta,\eta= %$η_value)")
if Reynolds_number < 60
    plot!(x->Ψ_smallr(x,η_value,Reynolds_number, 3),0.0,pi, label = "Small R asymptotic")
else
    plot!(x->Ψ_bigr(x, η_value) ,0.0, pi, label = "Large R asymptotic")
end
```

```julia
ψ1=readdlm("psi_solution_R_200.0_1stOrder_sym.txt",Float64,); #check that N and M dimensions match
ψ2=readdlm("psi_solution_R_200.0_2ndOrder_sym.txt",Float64,); #check that N and M dimensions match
N = size(ψ1)[1] - 1 #Steps in η
M = size(ψ1)[2] - 1 #Steps in θ
Δη = 2.0/N
Δθ = pi/M
θ_value =0.6;
theta=range(0, stop=pi,length= M+1);
eta_vector = range(0, stop=1.0,length= round(Int,N+1));
m=2
plot( eta_vector, ψ1[1:end,m], label="1stOrder",linecolor=:orange, linestyle=:solid)
plot!(eta_vector,ψ2[1:end,m], label="2ndOrder",linecolor=:red, linestyle=:dot)
#plot!(x->Ψ_asym(Δθ*(m-1),x,R),-1,1,label="Asymmetric analytical", linestyle=:dot, linecolor=:grey)
plot!(title= L"θ=\frac{%$(m-1)}{%$M}π \quad R_s= %$Reynolds_number",size=(1100,500), legend=:outertopright, xlabel =L"η", ylabel=L"Ψ", yguidefontrotation=-90)
if Reynolds_number < 60
    plot!(x->Ψ_smallr(Δθ*(m-1),x,Reynolds_number),0,1, label="Small R analytical", linestyle=:dot, linecolor=:blue)
end
#plot!(-xinterval,-MatrixPsi[2:end-1,θ_scalefactor*(m-1)+1], label="Fortran FD", linestyle=:dash, linecolor=:black)
#plot!(x->Ψ_bigr((m-1)*Δθ,x),-1,1, label="Large R analytical",  linestyle=:dash, linecolor=:red,fmt = :svg)
plot!(x->Ψ(Δθ*(m-1),x,sol_psi0,sol_psi, max_terms),0,1.0, label="Series solution",linecolor=:green,linestyle=:dot, linewidth=3)
```

```julia

```


# Lab03: MLP

---
MSSV: 18120061

Họ và tên: Lê Nhựt Nam

## Import

```python
%matplotlib inline
#import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import _pickle as cPickle
import time

# ignore warning ...
import warnings
import math
warnings.filterwarnings('ignore')

!wget https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
!tar -xzf cifar-10-python.tar.gz
```

    --2020-12-18 08:13:27--  https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz
    Resolving www.cs.toronto.edu (www.cs.toronto.edu)... 128.100.3.30
    Connecting to www.cs.toronto.edu (www.cs.toronto.edu)|128.100.3.30|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 170498071 (163M) [application/x-gzip]
    Saving to: ‘cifar-10-python.tar.gz’

    cifar-10-python.tar 100%[===================>] 162.60M  16.8MB/s    in 11s

    2020-12-18 08:13:38 (15.2 MB/s) - ‘cifar-10-python.tar.gz’ saved [170498071/170498071]

## Định nghĩa các hàm
**Hàm đọc bộ dữ liệu CIFAR-10**

```python
def unpickle(file):
    fo = open(file, 'rb')
    dict = cPickle.load(fo, encoding='latin1')
    return dict
def read_data(containing_dir, num_train_batchs):
    # Read training data
    train_X_batchs = []
    train_Y_batchs = []
    for batch_idx in range(num_train_batchs):
        batch = unpickle(containing_dir + '/data_batch_' + str(batch_idx + 1))
        train_X_batchs.append(batch['data'])
        train_Y_batchs.append(np.array(batch['labels']).reshape(-1, 1))
    train_X = np.vstack(train_X_batchs)
    train_X = train_X / 255. # Normalize to [0, 1]
    train_X = np.hstack([np.ones((len(train_X), 1)), train_X])
    train_Y = np.vstack(train_Y_batchs)

    # Read test data
    batch = unpickle(containing_dir + '/test_batch')
    test_X = batch['data']
    test_X = test_X / 255. # Normalize to [0, 1]
    test_X = np.hstack([np.ones((len(test_X), 1)), test_X])
    test_Y = np.array(batch['labels']).reshape(-1, 1)

    return (train_X, train_Y, test_X, test_Y)
```

**Hàm tính output và gradient của Neural Net**

Ở đây, ta dùng hàm kích hoạt sigmoid ở các tầng ẩn, và hàm softmax ở tầng cuối.
Ta có:

$$ h=w^TX $$

$$\text{Sigmoid Activation: } z= \sigma \left(h\right)= \dfrac{1}{1+e^{-h}}$$

$$\text{Cross-entrophy loss: } J(w)=-\left({ylog(z)+(1-y)log(1-z)}\right)$$

$$\text{Chain rule: } \dfrac{\partial J(w)}{\partial w}=\dfrac{\partial J(w)}{\partial z} \dfrac{\partial z}{\partial h}\dfrac{\partial h}{\partial w}  $$

$$\dfrac{\partial J(w)}{\partial z}=-\left(\dfrac{y}{z}-\dfrac{1-y}{1-z}\right)=\dfrac{z-y}{z(1-z)}$$

$$\dfrac{\partial z}{\partial h}=z(1-z)$$

$$\dfrac{\partial h}{\partial w}=X$$

$$\dfrac{\partial J(w)}{\partial w}=X^T(z-y)$$

**Hàm cost và gradient tầng sigmoid:**
$$J(w)=\dfrac{1}{N} \sum_{n=1}^N-\left({ylog(z)+(1-y)log(1-z)}\right)  $$
$$\text{Gradient: } \dfrac{\partial J(w)}{\partial w}=\dfrac{1}{N}X^T(z-y)$$

Hàm softmax (C là số lớp):
$$h(z_i)=\dfrac{e^{z_i}}{\sum_{j=1}^{C}e^{z_j}}$$
**Hàm cost tầng softmax lúc này là:**
$$J(w)=\dfrac{1}{N} \sum_{n=1}^N \sum_{k=1}^K -(y_n)_k log \left(h(x_n)_k\right)  $$

Sử dụng kỹ thuật one-hot-encoding mã hóa tập label y, ta có:
 $$\text{với } (y_n)_k=    \begin{align}
        \begin{cases}
             0 \text { nếu } y_n\neq k\\
             1 \text { nếu } y_n= k\\
        \end{cases}
    \end{align} $$
lúc này số lớp sẽ là 2 hay C=2:

$$h(z_1)=\dfrac{e^{z_1}}{\sum_{j=1}^{2}e^{z_j}}$$

$$h(z_1)=\dfrac{e^{z_1}}{e^{z_1}+e^{z_2}}$$

$$h(z_1)=\dfrac{1}{1+e^{z_2-z_1}}$$

=> Đây chính là hàm sigmoid. Hơn nữa mặc dù có 2 output softmax vẫn có thể rút gọn thành 1 output vì tổng 2 outputs luôn bằng 1

**Do đó hàm cost của tầng softmax và tầng sigmoid là tương tự nhau**

**=> Gradient của tầng softmax cũng tương tự của tầng sigmoid**

```python
# Element x
def sigmoid(x):
    '''
    Computes sigmoid function for each element of array S.
    '''
    return 1 / (1 + np.exp(-x))

# As = ?
# Ws =
# md_X = mini batch X
# delta = ?
# i = layer i th
def gradient_sigmoid(As, Ws ,mb_X,delta, i):
    '''
    Computes gradient vector for sigmoid layer.
    '''
    delta = delta.dot(Ws[-i + 1].T[:, 1:]) * As[-i][:, 1:] * (1 - As[-i][:, 1:])
    if i == len(Ws):
      grad = np.dot(mb_X.T, delta) / len(mb_X)
    else:
      grad = np.dot(As[-i - 1].T, delta) / len(mb_X)
    return delta, grad

# Array S
def softmax(S):
    '''
    Computes softmax function for each row of array S.
    '''
    A = np.exp(S)
    A /=A.sum(axis=1, keepdims=True)
    return A

#
# minibach x = mb_x
# minibach y = mb_y
def gradient_softmax(As,mb_X,mb_Y):
    '''
    Computes gradient vector for softmaz layer
    '''
    delta = (As[-1] - mb_Y) / len(mb_Y)
    grad = np.dot(As[-2].T, delta)/ len(mb_X)
    return delta, grad

# weight matrix shape = Ws
def compute_nnet_outputs(Ws, X, need_all_layer_outputs):
    '''
    Computes the outputs of Neural Net by forward propagating X through the net.
    '''
    As = [X]
    for ws in Ws:
        As.append(np.zeros(shape = (len(X), ws.shape[1])))

    for i in range(1, len(As) - 1):
        As[i] = np.hstack([np.ones((len(X), 1)), sigmoid(np.dot(As[i - 1], Ws[i - 1]))])
    As[-1] = softmax(np.dot(As[-2], Ws[-1]))
    if need_all_layer_outputs:
      return As
    else:
      A = As[-1]
      return A
```

```python
nums = np.arange(-10, 10, step=1)
fig, ax = plt.subplots(figsize=(12,4))
ax.plot(nums, sigmoid(nums), 'r')
```

**Các hàm khởi tạo tham số**

```python

def compute_layer_sizes(X, Y, hid_layer_sizes):
    num_classes = len(np.unique(Y)) # Num classes
    layer_sizes = [X.shape[1] - 1] + hid_layer_sizes + [num_classes]
    print('Layer Sizes: ',layer_sizes)
    return layer_sizes

def one_hot_encoding(Y, num_classes):
    num_classes = len(np.unique(Y)) # Num classes
    one_hot_Y = np.zeros((len(Y), num_classes))
    one_hot_Y[np.arange(len(Y)), Y.reshape(-1)] = 1
    return one_hot_Y

def init_weight_matrix(X,Y,layer_sizes):
    #fix random for same result when init weight matrix
    np.random.seed(0)
    Ws = np.array([np.random.randn(layer_sizes[i]+1 , layer_sizes[i + 1]) / np.sqrt(layer_sizes[i]+1)
          for i in range(len(layer_sizes) - 1)]) # Init Ws
    print('Weight matrix shape: ',Ws[0].shape,Ws[1].shape)
    return Ws

```

**Hàm huấn luyện Neural net**

```python

def updateWeights(Ws,As,mb_X, mb_Y,  learning_rate):
    #update weights for softmax layer
    delta, grad = gradient_softmax(As, mb_X=mb_X, mb_Y=mb_Y)
    Ws[-1] -= learning_rate * grad

    #update weights for hidden layer
    for i in range(2, len(Ws) + 1):
        delta, grad = gradient_sigmoid(As,Ws, mb_X, delta, i)
        Ws[-i] -= learning_rate * grad
    return Ws

def train_nnet(X, Y, hid_layer_sizes, mb_size, learning_rate, max_epoch):

    #get layer sizes:
    layer_sizes = compute_layer_sizes(X, Y, hid_layer_sizes)

    # Prepare for training
    Ws=init_weight_matrix(X,Y,layer_sizes)
    one_hot_Y=one_hot_encoding(Y,layer_sizes[-1])

    costs = [] # To save costs during training
    errs = [] # To save mean binary errors during training
    N = len(X) # Num training examples
    rnd_idxs = list(range(N)) # Random indexes

    # Train
    for epoch in list(range(max_epoch)):
        start_time = time.time()
        #shuffle index
        np.random.shuffle(rnd_idxs)
        for start_idx in list(range(0, N, mb_size)):
            # Get minibach
            mb_X = X[rnd_idxs[start_idx:start_idx+mb_size]]
            mb_Y = one_hot_Y[rnd_idxs[start_idx:start_idx+mb_size]]

            # Forward-prop
            As = compute_nnet_outputs(Ws, mb_X, True)

            # Back-prop; on the way, compute each layer's gradient and update its W
            Ws=updateWeights(Ws,As,mb_X,mb_Y,learning_rate)

        # Compute training info, save it, and print it
        A = compute_nnet_outputs(Ws, X, False)
        cost = np.mean(-np.sum(one_hot_Y * np.log(A), axis=1))
        err = np.mean(np.argmax(A, axis=1) != Y.squeeze()) * 100
        costs.append(cost)
        errs.append(err)
        end_time = time.time()
        time_running = -(start_time - end_time)
        print ('Epoch %d, cost %.3f, err %.3f%%, , Time: %.3fs' %(epoch, cost, err, time_running))

    return Ws, costs, errs
```

## Chạy

**1.** Đọc dữ liệu (cần đặt thư mục chứa dữ liệu `cifar-10-batches-py` vào cùng thư mục chứa file notebook này)

```python
num_train_batchs = 3
train_X, train_Y, test_X, test_Y = read_data('/content/cifar-10-batches-py', num_train_batchs)
print ('train_X.shape = %s, train_Y.shape = %s' %(train_X.shape, train_Y.shape))
print ('test_X.shape  = %s, test_Y.shape  = %s' %(test_X.shape, test_Y.shape))
```

    train_X.shape = (30000, 3073), train_Y.shape = (30000, 1)
    test_X.shape  = (10000, 3073), test_Y.shape  = (10000, 1)

```python
f = open('cifar-10-batches-py/data_batch_1', 'rb')
datadict = cPickle.load(f,encoding='latin1')
f.close()
X = datadict["data"]
Y = datadict['labels']
X = X.reshape(10000, 3, 32, 32).transpose(0,2,3,1).astype("uint8")
Y = np.array(Y)

#Visualizing CIFAR 10
fig, axes1 = plt.subplots(5,5,figsize=(10,10))
for j in range(5):
    for k in range(5):
        i = np.random.choice(range(len(X)))
        axes1[j][k].set_axis_off()
        axes1[j][k].imshow(X[i:i+1][0])
```

**2.** Để thấy được sự ảnh hưởng của số lượng nơ-ron ẩn, ta sẽ lần lượt huấn luyện Neural Net với `hid_layer_sizes = [50]`, `[100]`, và `[200]` (cố định `mb_size = 32`, `learning_rate = 0.01`, `max_epoch = 200`)

```python
W_0, costs_0, errs_0 = train_nnet(train_X, train_Y, [50], 32, 0.01, 200)
```

    Layer Sizes:  [3072, 50, 10]
    Weight matrix shape:  (3073, 50) (51, 10)
    Epoch 0, cost 2.282, err 84.153%, , Time: 3.313s
    Epoch 1, cost 2.267, err 81.633%, , Time: 3.159s
    Epoch 2, cost 2.256, err 79.627%, , Time: 3.188s
    Epoch 3, cost 2.245, err 78.307%, , Time: 3.200s
    Epoch 4, cost 2.234, err 77.160%, , Time: 3.248s
    Epoch 5, cost 2.224, err 76.213%, , Time: 3.221s
    Epoch 6, cost 2.215, err 75.360%, , Time: 3.209s
    Epoch 7, cost 2.206, err 74.873%, , Time: 3.292s
    Epoch 8, cost 2.197, err 74.417%, , Time: 3.262s
    Epoch 9, cost 2.188, err 73.980%, , Time: 3.235s
    Epoch 10, cost 2.180, err 73.377%, , Time: 3.266s
    Epoch 11, cost 2.172, err 73.170%, , Time: 3.251s
    Epoch 12, cost 2.164, err 72.783%, , Time: 3.211s
    Epoch 13, cost 2.156, err 72.410%, , Time: 3.221s
    Epoch 14, cost 2.149, err 72.127%, , Time: 3.185s
    Epoch 15, cost 2.142, err 71.960%, , Time: 3.220s
    Epoch 16, cost 2.135, err 71.817%, , Time: 3.225s
    Epoch 17, cost 2.128, err 71.343%, , Time: 3.228s
    Epoch 18, cost 2.122, err 71.280%, , Time: 3.189s
    Epoch 19, cost 2.115, err 70.920%, , Time: 3.192s
    Epoch 20, cost 2.109, err 70.773%, , Time: 3.202s
    Epoch 21, cost 2.103, err 70.517%, , Time: 3.222s
    Epoch 22, cost 2.098, err 70.473%, , Time: 3.271s
    Epoch 23, cost 2.092, err 70.173%, , Time: 3.192s
    Epoch 24, cost 2.087, err 70.000%, , Time: 3.290s
    Epoch 25, cost 2.081, err 69.823%, , Time: 3.254s
    Epoch 26, cost 2.076, err 69.797%, , Time: 3.183s
    Epoch 27, cost 2.071, err 69.570%, , Time: 3.234s
    Epoch 28, cost 2.066, err 69.527%, , Time: 3.162s
    Epoch 29, cost 2.061, err 69.410%, , Time: 3.196s
    Epoch 30, cost 2.057, err 69.337%, , Time: 3.215s
    Epoch 31, cost 2.052, err 69.143%, , Time: 3.295s
    Epoch 32, cost 2.048, err 69.157%, , Time: 3.231s
    Epoch 33, cost 2.044, err 68.850%, , Time: 3.420s
    Epoch 34, cost 2.039, err 68.747%, , Time: 3.306s
    Epoch 35, cost 2.035, err 68.777%, , Time: 3.274s
    Epoch 36, cost 2.031, err 68.713%, , Time: 3.210s
    Epoch 37, cost 2.027, err 68.647%, , Time: 3.227s
    Epoch 38, cost 2.024, err 68.537%, , Time: 3.220s
    Epoch 39, cost 2.020, err 68.427%, , Time: 3.173s
    Epoch 40, cost 2.016, err 68.297%, , Time: 3.272s
    Epoch 41, cost 2.013, err 68.230%, , Time: 3.261s
    Epoch 42, cost 2.009, err 68.287%, , Time: 3.239s
    Epoch 43, cost 2.006, err 68.083%, , Time: 3.309s
    Epoch 44, cost 2.003, err 67.887%, , Time: 3.234s
    Epoch 45, cost 1.999, err 67.880%, , Time: 3.240s
    Epoch 46, cost 1.996, err 67.793%, , Time: 3.225s
    Epoch 47, cost 1.993, err 67.720%, , Time: 3.206s
    Epoch 48, cost 1.990, err 67.690%, , Time: 3.204s
    Epoch 49, cost 1.987, err 67.617%, , Time: 3.181s
    Epoch 50, cost 1.984, err 67.493%, , Time: 3.243s
    Epoch 51, cost 1.981, err 67.550%, , Time: 3.245s
    Epoch 52, cost 1.978, err 67.400%, , Time: 3.230s
    Epoch 53, cost 1.975, err 67.370%, , Time: 3.178s
    Epoch 54, cost 1.973, err 67.143%, , Time: 3.241s
    Epoch 55, cost 1.970, err 67.263%, , Time: 3.207s
    Epoch 56, cost 1.967, err 67.283%, , Time: 3.230s
    Epoch 57, cost 1.965, err 67.167%, , Time: 3.172s
    Epoch 58, cost 1.962, err 67.020%, , Time: 3.193s
    Epoch 59, cost 1.960, err 66.910%, , Time: 3.179s
    Epoch 60, cost 1.957, err 66.863%, , Time: 3.186s
    Epoch 61, cost 1.955, err 66.837%, , Time: 3.283s
    Epoch 62, cost 1.953, err 66.790%, , Time: 3.230s
    Epoch 63, cost 1.950, err 66.770%, , Time: 3.188s
    Epoch 64, cost 1.948, err 66.577%, , Time: 3.188s
    Epoch 65, cost 1.946, err 66.597%, , Time: 3.156s
    Epoch 66, cost 1.944, err 66.527%, , Time: 3.187s
    Epoch 67, cost 1.942, err 66.520%, , Time: 3.215s
    Epoch 68, cost 1.939, err 66.493%, , Time: 3.202s
    Epoch 69, cost 1.937, err 66.420%, , Time: 3.286s
    Epoch 70, cost 1.935, err 66.243%, , Time: 3.203s
    Epoch 71, cost 1.933, err 66.367%, , Time: 3.240s
    Epoch 72, cost 1.931, err 66.340%, , Time: 3.198s
    Epoch 73, cost 1.929, err 66.197%, , Time: 3.207s
    Epoch 74, cost 1.927, err 65.977%, , Time: 3.210s
    Epoch 75, cost 1.925, err 66.053%, , Time: 3.231s
    Epoch 76, cost 1.924, err 65.987%, , Time: 3.156s
    Epoch 77, cost 1.922, err 65.967%, , Time: 3.178s
    Epoch 78, cost 1.920, err 65.987%, , Time: 3.184s
    Epoch 79, cost 1.918, err 65.827%, , Time: 3.300s
    Epoch 80, cost 1.917, err 65.753%, , Time: 3.260s
    Epoch 81, cost 1.915, err 65.820%, , Time: 3.186s
    Epoch 82, cost 1.913, err 65.677%, , Time: 3.213s
    Epoch 83, cost 1.911, err 65.677%, , Time: 3.190s
    Epoch 84, cost 1.910, err 65.627%, , Time: 3.201s
    Epoch 85, cost 1.908, err 65.580%, , Time: 3.262s
    Epoch 86, cost 1.907, err 65.493%, , Time: 3.194s
    Epoch 87, cost 1.905, err 65.417%, , Time: 3.245s
    Epoch 88, cost 1.904, err 65.323%, , Time: 3.258s
    Epoch 89, cost 1.902, err 65.410%, , Time: 3.231s
    Epoch 90, cost 1.900, err 65.303%, , Time: 3.222s
    Epoch 91, cost 1.899, err 65.377%, , Time: 3.198s
    Epoch 92, cost 1.898, err 65.273%, , Time: 3.217s
    Epoch 93, cost 1.896, err 65.240%, , Time: 3.182s
    Epoch 94, cost 1.895, err 65.210%, , Time: 3.170s
    Epoch 95, cost 1.893, err 65.223%, , Time: 3.216s
    Epoch 96, cost 1.892, err 65.103%, , Time: 3.205s
    Epoch 97, cost 1.891, err 64.980%, , Time: 3.190s
    Epoch 98, cost 1.889, err 65.087%, , Time: 3.218s
    Epoch 99, cost 1.888, err 65.053%, , Time: 3.249s
    Epoch 100, cost 1.887, err 64.970%, , Time: 3.243s
    Epoch 101, cost 1.885, err 64.847%, , Time: 3.245s
    Epoch 102, cost 1.884, err 64.940%, , Time: 3.182s
    Epoch 103, cost 1.883, err 64.880%, , Time: 3.138s
    Epoch 104, cost 1.881, err 64.817%, , Time: 3.272s
    Epoch 105, cost 1.880, err 64.807%, , Time: 3.219s
    Epoch 106, cost 1.879, err 64.763%, , Time: 3.211s
    Epoch 107, cost 1.878, err 64.773%, , Time: 3.211s
    Epoch 108, cost 1.877, err 64.570%, , Time: 3.281s
    Epoch 109, cost 1.876, err 64.593%, , Time: 3.238s
    Epoch 110, cost 1.874, err 64.620%, , Time: 3.225s
    Epoch 111, cost 1.873, err 64.510%, , Time: 3.199s
    Epoch 112, cost 1.872, err 64.497%, , Time: 3.190s
    Epoch 113, cost 1.871, err 64.427%, , Time: 3.168s
    Epoch 114, cost 1.870, err 64.467%, , Time: 3.198s
    Epoch 115, cost 1.869, err 64.397%, , Time: 3.209s
    Epoch 116, cost 1.868, err 64.457%, , Time: 3.172s
    Epoch 117, cost 1.867, err 64.427%, , Time: 3.183s
    Epoch 118, cost 1.866, err 64.417%, , Time: 3.213s
    Epoch 119, cost 1.864, err 64.383%, , Time: 3.270s
    Epoch 120, cost 1.863, err 64.320%, , Time: 3.221s
    Epoch 121, cost 1.862, err 64.343%, , Time: 3.242s
    Epoch 122, cost 1.861, err 64.320%, , Time: 3.214s
    Epoch 123, cost 1.860, err 64.390%, , Time: 3.191s
    Epoch 124, cost 1.859, err 64.200%, , Time: 3.203s
    Epoch 125, cost 1.858, err 64.143%, , Time: 3.201s
    Epoch 126, cost 1.857, err 64.143%, , Time: 3.220s
    Epoch 127, cost 1.857, err 64.197%, , Time: 3.197s
    Epoch 128, cost 1.856, err 64.077%, , Time: 3.264s
    Epoch 129, cost 1.855, err 64.103%, , Time: 3.188s
    Epoch 130, cost 1.854, err 64.067%, , Time: 3.189s
    Epoch 131, cost 1.853, err 63.970%, , Time: 3.152s
    Epoch 132, cost 1.852, err 63.947%, , Time: 3.202s
    Epoch 133, cost 1.851, err 63.923%, , Time: 3.145s
    Epoch 134, cost 1.850, err 63.940%, , Time: 3.159s
    Epoch 135, cost 1.849, err 63.893%, , Time: 3.179s
    Epoch 136, cost 1.848, err 63.840%, , Time: 3.213s
    Epoch 137, cost 1.847, err 63.777%, , Time: 3.211s
    Epoch 138, cost 1.846, err 63.833%, , Time: 3.189s
    Epoch 139, cost 1.845, err 63.790%, , Time: 3.231s
    Epoch 140, cost 1.845, err 63.827%, , Time: 3.238s
    Epoch 141, cost 1.844, err 63.777%, , Time: 3.212s
    Epoch 142, cost 1.843, err 63.680%, , Time: 3.220s
    Epoch 143, cost 1.842, err 63.763%, , Time: 3.227s
    Epoch 144, cost 1.841, err 63.703%, , Time: 3.252s
    Epoch 145, cost 1.840, err 63.707%, , Time: 3.250s
    Epoch 146, cost 1.840, err 63.663%, , Time: 3.209s
    Epoch 147, cost 1.839, err 63.587%, , Time: 3.195s
    Epoch 148, cost 1.838, err 63.607%, , Time: 3.223s
    Epoch 149, cost 1.837, err 63.533%, , Time: 3.192s
    Epoch 150, cost 1.836, err 63.570%, , Time: 3.196s
    Epoch 151, cost 1.836, err 63.457%, , Time: 3.201s
    Epoch 152, cost 1.835, err 63.497%, , Time: 3.204s
    Epoch 153, cost 1.834, err 63.470%, , Time: 3.197s
    Epoch 154, cost 1.833, err 63.567%, , Time: 3.233s
    Epoch 155, cost 1.832, err 63.557%, , Time: 3.174s
    Epoch 156, cost 1.832, err 63.470%, , Time: 3.216s
    Epoch 157, cost 1.831, err 63.397%, , Time: 3.175s
    Epoch 158, cost 1.830, err 63.437%, , Time: 3.267s
    Epoch 159, cost 1.829, err 63.320%, , Time: 3.203s
    Epoch 160, cost 1.829, err 63.343%, , Time: 3.231s
    Epoch 161, cost 1.828, err 63.317%, , Time: 3.191s
    Epoch 162, cost 1.827, err 63.397%, , Time: 3.170s
    Epoch 163, cost 1.826, err 63.313%, , Time: 3.185s
    Epoch 164, cost 1.826, err 63.357%, , Time: 3.230s
    Epoch 165, cost 1.825, err 63.370%, , Time: 3.255s
    Epoch 166, cost 1.824, err 63.287%, , Time: 3.200s
    Epoch 167, cost 1.823, err 63.280%, , Time: 3.156s
    Epoch 168, cost 1.823, err 63.263%, , Time: 3.205s
    Epoch 169, cost 1.822, err 63.173%, , Time: 3.187s
    Epoch 170, cost 1.821, err 63.127%, , Time: 3.174s
    Epoch 171, cost 1.821, err 63.240%, , Time: 3.157s
    Epoch 172, cost 1.820, err 63.140%, , Time: 3.167s
    Epoch 173, cost 1.819, err 63.143%, , Time: 3.229s
    Epoch 174, cost 1.819, err 63.013%, , Time: 3.209s
    Epoch 175, cost 1.818, err 63.057%, , Time: 3.163s
    Epoch 176, cost 1.817, err 62.987%, , Time: 3.170s
    Epoch 177, cost 1.817, err 62.963%, , Time: 3.189s
    Epoch 178, cost 1.816, err 62.940%, , Time: 3.257s
    Epoch 179, cost 1.815, err 62.940%, , Time: 3.226s
    Epoch 180, cost 1.814, err 62.937%, , Time: 3.199s
    Epoch 181, cost 1.814, err 62.963%, , Time: 3.214s
    Epoch 182, cost 1.813, err 62.887%, , Time: 3.197s
    Epoch 183, cost 1.812, err 62.890%, , Time: 3.254s
    Epoch 184, cost 1.812, err 62.917%, , Time: 3.231s
    Epoch 185, cost 1.811, err 62.847%, , Time: 3.200s
    Epoch 186, cost 1.811, err 62.833%, , Time: 3.185s
    Epoch 187, cost 1.810, err 62.853%, , Time: 3.192s
    Epoch 188, cost 1.809, err 62.937%, , Time: 3.203s
    Epoch 189, cost 1.809, err 62.790%, , Time: 3.173s
    Epoch 190, cost 1.808, err 62.723%, , Time: 3.180s
    Epoch 191, cost 1.807, err 62.713%, , Time: 3.161s
    Epoch 192, cost 1.807, err 62.783%, , Time: 3.227s
    Epoch 193, cost 1.806, err 62.787%, , Time: 3.267s
    Epoch 194, cost 1.806, err 62.653%, , Time: 3.227s
    Epoch 195, cost 1.805, err 62.697%, , Time: 3.192s
    Epoch 196, cost 1.804, err 62.710%, , Time: 3.221s
    Epoch 197, cost 1.804, err 62.597%, , Time: 3.208s
    Epoch 198, cost 1.803, err 62.527%, , Time: 3.245s
    Epoch 199, cost 1.802, err 62.613%, , Time: 3.222s

```python
W_1, costs_1, errs_1 = train_nnet(train_X, train_Y, [100], 32, 0.01, 200)
```

    Layer Sizes:  [3072, 100, 10]
    Weight matrix shape:  (3073, 100) (101, 10)
    Epoch 0, cost 2.297, err 87.010%, , Time: 5.462s
    Epoch 1, cost 2.280, err 85.313%, , Time: 5.290s
    Epoch 2, cost 2.266, err 83.950%, , Time: 5.139s
    Epoch 3, cost 2.253, err 81.897%, , Time: 5.201s
    Epoch 4, cost 2.241, err 80.130%, , Time: 5.063s
    Epoch 5, cost 2.230, err 78.913%, , Time: 5.222s
    Epoch 6, cost 2.219, err 77.500%, , Time: 5.069s
    Epoch 7, cost 2.208, err 76.437%, , Time: 5.282s
    Epoch 8, cost 2.198, err 75.173%, , Time: 5.082s
    Epoch 9, cost 2.188, err 74.730%, , Time: 5.307s
    Epoch 10, cost 2.179, err 73.733%, , Time: 5.098s
    Epoch 11, cost 2.169, err 72.840%, , Time: 5.312s
    Epoch 12, cost 2.161, err 72.113%, , Time: 5.041s
    Epoch 13, cost 2.152, err 72.140%, , Time: 5.298s
    Epoch 14, cost 2.144, err 71.440%, , Time: 4.998s
    Epoch 15, cost 2.136, err 70.867%, , Time: 5.186s
    Epoch 16, cost 2.128, err 70.453%, , Time: 5.066s
    Epoch 17, cost 2.121, err 70.410%, , Time: 5.241s
    Epoch 18, cost 2.114, err 70.200%, , Time: 5.058s
    Epoch 19, cost 2.107, err 69.957%, , Time: 5.219s
    Epoch 20, cost 2.100, err 69.690%, , Time: 5.043s
    Epoch 21, cost 2.094, err 69.457%, , Time: 5.296s
    Epoch 22, cost 2.087, err 69.323%, , Time: 5.090s
    Epoch 23, cost 2.081, err 69.163%, , Time: 5.287s
    Epoch 24, cost 2.075, err 68.950%, , Time: 5.010s
    Epoch 25, cost 2.069, err 68.810%, , Time: 5.307s
    Epoch 26, cost 2.064, err 68.660%, , Time: 4.985s
    Epoch 27, cost 2.058, err 68.413%, , Time: 5.227s
    Epoch 28, cost 2.053, err 68.553%, , Time: 5.042s
    Epoch 29, cost 2.048, err 68.370%, , Time: 5.255s
    Epoch 30, cost 2.043, err 68.277%, , Time: 5.042s
    Epoch 31, cost 2.038, err 68.093%, , Time: 5.233s
    Epoch 32, cost 2.034, err 68.027%, , Time: 5.036s
    Epoch 33, cost 2.029, err 68.077%, , Time: 5.261s
    Epoch 34, cost 2.025, err 67.953%, , Time: 5.080s
    Epoch 35, cost 2.021, err 67.670%, , Time: 5.387s
    Epoch 36, cost 2.017, err 67.637%, , Time: 5.065s
    Epoch 37, cost 2.013, err 67.543%, , Time: 5.223s
    Epoch 38, cost 2.009, err 67.653%, , Time: 5.070s
    Epoch 39, cost 2.005, err 67.570%, , Time: 5.221s
    Epoch 40, cost 2.001, err 67.347%, , Time: 5.071s
    Epoch 41, cost 1.997, err 67.217%, , Time: 5.271s
    Epoch 42, cost 1.994, err 67.213%, , Time: 5.087s
    Epoch 43, cost 1.990, err 67.183%, , Time: 5.279s
    Epoch 44, cost 1.987, err 67.100%, , Time: 5.045s
    Epoch 45, cost 1.984, err 66.997%, , Time: 5.191s
    Epoch 46, cost 1.981, err 66.960%, , Time: 5.050s
    Epoch 47, cost 1.977, err 66.797%, , Time: 5.272s
    Epoch 48, cost 1.974, err 66.663%, , Time: 5.120s
    Epoch 49, cost 1.971, err 66.733%, , Time: 5.212s
    Epoch 50, cost 1.968, err 66.673%, , Time: 5.059s
    Epoch 51, cost 1.966, err 66.623%, , Time: 5.167s
    Epoch 52, cost 1.963, err 66.513%, , Time: 5.103s
    Epoch 53, cost 1.960, err 66.487%, , Time: 5.244s
    Epoch 54, cost 1.957, err 66.200%, , Time: 5.136s
    Epoch 55, cost 1.955, err 66.477%, , Time: 5.258s
    Epoch 56, cost 1.952, err 66.133%, , Time: 5.068s
    Epoch 57, cost 1.949, err 66.047%, , Time: 5.177s
    Epoch 58, cost 1.947, err 66.080%, , Time: 5.064s
    Epoch 59, cost 1.945, err 66.167%, , Time: 5.319s
    Epoch 60, cost 1.942, err 66.043%, , Time: 5.146s
    Epoch 61, cost 1.940, err 65.897%, , Time: 5.273s
    Epoch 62, cost 1.938, err 65.850%, , Time: 5.078s
    Epoch 63, cost 1.935, err 65.817%, , Time: 5.205s
    Epoch 64, cost 1.933, err 65.627%, , Time: 5.092s
    Epoch 65, cost 1.931, err 65.653%, , Time: 5.329s
    Epoch 66, cost 1.929, err 65.557%, , Time: 5.088s
    Epoch 67, cost 1.927, err 65.663%, , Time: 5.218s
    Epoch 68, cost 1.924, err 65.553%, , Time: 5.141s
    Epoch 69, cost 1.922, err 65.537%, , Time: 5.391s
    Epoch 70, cost 1.921, err 65.567%, , Time: 5.236s
    Epoch 71, cost 1.919, err 65.437%, , Time: 5.297s
    Epoch 72, cost 1.917, err 65.300%, , Time: 5.052s
    Epoch 73, cost 1.915, err 65.273%, , Time: 5.213s
    Epoch 74, cost 1.913, err 65.250%, , Time: 5.004s
    Epoch 75, cost 1.911, err 65.253%, , Time: 5.236s
    Epoch 76, cost 1.909, err 65.207%, , Time: 5.066s
    Epoch 77, cost 1.908, err 65.050%, , Time: 5.249s
    Epoch 78, cost 1.906, err 65.003%, , Time: 5.067s
    Epoch 79, cost 1.904, err 64.993%, , Time: 5.313s
    Epoch 80, cost 1.902, err 64.943%, , Time: 5.030s
    Epoch 81, cost 1.901, err 64.920%, , Time: 5.189s
    Epoch 82, cost 1.899, err 64.923%, , Time: 5.063s
    Epoch 83, cost 1.898, err 64.860%, , Time: 5.272s
    Epoch 84, cost 1.896, err 64.800%, , Time: 5.027s
    Epoch 85, cost 1.894, err 64.713%, , Time: 5.213s
    Epoch 86, cost 1.893, err 64.690%, , Time: 5.060s
    Epoch 87, cost 1.892, err 64.723%, , Time: 5.229s
    Epoch 88, cost 1.890, err 64.740%, , Time: 5.015s
    Epoch 89, cost 1.889, err 64.633%, , Time: 5.214s
    Epoch 90, cost 1.887, err 64.560%, , Time: 5.065s
    Epoch 91, cost 1.886, err 64.457%, , Time: 5.270s
    Epoch 92, cost 1.885, err 64.547%, , Time: 5.086s
    Epoch 93, cost 1.883, err 64.467%, , Time: 5.223s
    Epoch 94, cost 1.882, err 64.407%, , Time: 4.995s
    Epoch 95, cost 1.880, err 64.450%, , Time: 5.269s
    Epoch 96, cost 1.879, err 64.397%, , Time: 5.082s
    Epoch 97, cost 1.878, err 64.407%, , Time: 5.227s
    Epoch 98, cost 1.877, err 64.410%, , Time: 5.039s
    Epoch 99, cost 1.875, err 64.223%, , Time: 5.237s
    Epoch 100, cost 1.874, err 64.340%, , Time: 5.024s
    Epoch 101, cost 1.873, err 64.390%, , Time: 5.220s
    Epoch 102, cost 1.872, err 64.290%, , Time: 5.042s
    Epoch 103, cost 1.870, err 64.263%, , Time: 5.249s
    Epoch 104, cost 1.869, err 64.293%, , Time: 5.024s
    Epoch 105, cost 1.868, err 64.177%, , Time: 5.263s
    Epoch 106, cost 1.867, err 64.113%, , Time: 5.090s
    Epoch 107, cost 1.866, err 64.063%, , Time: 5.253s
    Epoch 108, cost 1.865, err 64.037%, , Time: 5.096s
    Epoch 109, cost 1.864, err 64.220%, , Time: 5.189s
    Epoch 110, cost 1.863, err 64.063%, , Time: 5.047s
    Epoch 111, cost 1.862, err 64.010%, , Time: 5.267s
    Epoch 112, cost 1.861, err 63.993%, , Time: 5.077s
    Epoch 113, cost 1.860, err 64.053%, , Time: 5.274s
    Epoch 114, cost 1.858, err 63.930%, , Time: 5.044s
    Epoch 115, cost 1.857, err 63.857%, , Time: 5.266s
    Epoch 116, cost 1.856, err 64.007%, , Time: 5.070s
    Epoch 117, cost 1.856, err 63.913%, , Time: 5.286s
    Epoch 118, cost 1.854, err 63.827%, , Time: 5.114s
    Epoch 119, cost 1.853, err 63.787%, , Time: 5.205s
    Epoch 120, cost 1.852, err 63.740%, , Time: 5.051s
    Epoch 121, cost 1.852, err 63.653%, , Time: 5.271s
    Epoch 122, cost 1.851, err 63.640%, , Time: 5.080s
    Epoch 123, cost 1.850, err 63.657%, , Time: 5.251s
    Epoch 124, cost 1.849, err 63.663%, , Time: 5.043s
    Epoch 125, cost 1.848, err 63.613%, , Time: 5.181s
    Epoch 126, cost 1.847, err 63.570%, , Time: 5.065s
    Epoch 127, cost 1.846, err 63.607%, , Time: 5.132s
    Epoch 128, cost 1.845, err 63.523%, , Time: 5.082s
    Epoch 129, cost 1.844, err 63.573%, , Time: 5.144s
    Epoch 130, cost 1.843, err 63.457%, , Time: 5.040s
    Epoch 131, cost 1.843, err 63.433%, , Time: 5.079s
    Epoch 132, cost 1.842, err 63.463%, , Time: 5.019s
    Epoch 133, cost 1.841, err 63.497%, , Time: 5.184s
    Epoch 134, cost 1.840, err 63.433%, , Time: 5.076s
    Epoch 135, cost 1.839, err 63.433%, , Time: 5.206s
    Epoch 136, cost 1.838, err 63.283%, , Time: 5.075s
    Epoch 137, cost 1.837, err 63.337%, , Time: 5.108s
    Epoch 138, cost 1.837, err 63.297%, , Time: 5.066s
    Epoch 139, cost 1.836, err 63.253%, , Time: 5.141s
    Epoch 140, cost 1.835, err 63.190%, , Time: 5.062s
    Epoch 141, cost 1.834, err 63.247%, , Time: 5.123s
    Epoch 142, cost 1.833, err 63.200%, , Time: 5.044s
    Epoch 143, cost 1.833, err 63.190%, , Time: 5.130s
    Epoch 144, cost 1.832, err 63.197%, , Time: 5.049s
    Epoch 145, cost 1.831, err 63.180%, , Time: 5.111s
    Epoch 146, cost 1.830, err 63.143%, , Time: 5.065s
    Epoch 147, cost 1.830, err 63.080%, , Time: 5.091s
    Epoch 148, cost 1.829, err 63.063%, , Time: 5.009s
    Epoch 149, cost 1.828, err 63.120%, , Time: 5.082s
    Epoch 150, cost 1.827, err 63.057%, , Time: 5.045s
    Epoch 151, cost 1.827, err 63.000%, , Time: 5.184s
    Epoch 152, cost 1.826, err 63.067%, , Time: 5.029s
    Epoch 153, cost 1.825, err 62.987%, , Time: 5.143s
    Epoch 154, cost 1.824, err 63.033%, , Time: 5.027s
    Epoch 155, cost 1.824, err 62.937%, , Time: 5.084s
    Epoch 156, cost 1.823, err 62.930%, , Time: 5.027s
    Epoch 157, cost 1.822, err 62.963%, , Time: 5.175s
    Epoch 158, cost 1.821, err 62.927%, , Time: 5.108s
    Epoch 159, cost 1.821, err 62.843%, , Time: 5.161s
    Epoch 160, cost 1.820, err 62.937%, , Time: 5.074s
    Epoch 161, cost 1.819, err 62.840%, , Time: 5.144s
    Epoch 162, cost 1.819, err 62.873%, , Time: 5.043s
    Epoch 163, cost 1.818, err 62.867%, , Time: 5.161s
    Epoch 164, cost 1.817, err 62.787%, , Time: 5.033s
    Epoch 165, cost 1.817, err 62.780%, , Time: 5.148s
    Epoch 166, cost 1.816, err 62.783%, , Time: 4.999s
    Epoch 167, cost 1.815, err 62.757%, , Time: 5.076s
    Epoch 168, cost 1.815, err 62.723%, , Time: 5.031s
    Epoch 169, cost 1.814, err 62.753%, , Time: 5.114s
    Epoch 170, cost 1.813, err 62.763%, , Time: 5.098s
    Epoch 171, cost 1.813, err 62.737%, , Time: 5.093s
    Epoch 172, cost 1.812, err 62.733%, , Time: 5.064s
    Epoch 173, cost 1.811, err 62.680%, , Time: 5.124s
    Epoch 174, cost 1.811, err 62.670%, , Time: 5.050s
    Epoch 175, cost 1.810, err 62.633%, , Time: 5.096s
    Epoch 176, cost 1.809, err 62.590%, , Time: 5.083s
    Epoch 177, cost 1.809, err 62.610%, , Time: 5.131s
    Epoch 178, cost 1.808, err 62.660%, , Time: 5.053s
    Epoch 179, cost 1.808, err 62.543%, , Time: 5.104s
    Epoch 180, cost 1.807, err 62.603%, , Time: 5.011s
    Epoch 181, cost 1.806, err 62.453%, , Time: 5.165s
    Epoch 182, cost 1.806, err 62.497%, , Time: 5.111s
    Epoch 183, cost 1.805, err 62.480%, , Time: 5.148s
    Epoch 184, cost 1.804, err 62.463%, , Time: 5.071s
    Epoch 185, cost 1.804, err 62.593%, , Time: 5.128s
    Epoch 186, cost 1.803, err 62.547%, , Time: 5.021s
    Epoch 187, cost 1.803, err 62.413%, , Time: 5.201s
    Epoch 188, cost 1.802, err 62.360%, , Time: 5.124s
    Epoch 189, cost 1.801, err 62.400%, , Time: 5.157s
    Epoch 190, cost 1.801, err 62.370%, , Time: 5.064s
    Epoch 191, cost 1.800, err 62.340%, , Time: 5.131s
    Epoch 192, cost 1.800, err 62.363%, , Time: 5.057s
    Epoch 193, cost 1.799, err 62.270%, , Time: 5.098s
    Epoch 194, cost 1.798, err 62.250%, , Time: 5.072s
    Epoch 195, cost 1.798, err 62.230%, , Time: 5.145s
    Epoch 196, cost 1.797, err 62.227%, , Time: 5.016s
    Epoch 197, cost 1.797, err 62.173%, , Time: 5.157s
    Epoch 198, cost 1.796, err 62.230%, , Time: 5.020s
    Epoch 199, cost 1.796, err 62.233%, , Time: 5.053s

```python
W_2, costs_2, errs_2 = train_nnet(train_X, train_Y, [200], 32, 0.01, 200)
```

    Layer Sizes:  [3072, 200, 10]
    Weight matrix shape:  (3073, 200) (201, 10)
    Epoch 0, cost 2.294, err 87.320%, , Time: 8.714s
    Epoch 1, cost 2.278, err 84.907%, , Time: 8.416s
    Epoch 2, cost 2.263, err 82.913%, , Time: 8.311s
    Epoch 3, cost 2.249, err 80.880%, , Time: 8.260s
    Epoch 4, cost 2.236, err 78.517%, , Time: 8.287s
    Epoch 5, cost 2.224, err 75.717%, , Time: 8.515s
    Epoch 6, cost 2.212, err 74.907%, , Time: 8.696s
    Epoch 7, cost 2.200, err 74.083%, , Time: 9.242s
    Epoch 8, cost 2.190, err 73.303%, , Time: 8.921s
    Epoch 9, cost 2.179, err 72.203%, , Time: 8.741s
    Epoch 10, cost 2.169, err 71.983%, , Time: 8.794s
    Epoch 11, cost 2.159, err 71.577%, , Time: 8.881s
    Epoch 12, cost 2.150, err 71.280%, , Time: 8.740s
    Epoch 13, cost 2.141, err 71.150%, , Time: 9.061s
    Epoch 14, cost 2.132, err 70.727%, , Time: 9.260s
    Epoch 15, cost 2.123, err 70.307%, , Time: 8.872s
    Epoch 16, cost 2.115, err 70.500%, , Time: 8.709s
    Epoch 17, cost 2.107, err 70.070%, , Time: 8.795s
    Epoch 18, cost 2.099, err 69.970%, , Time: 8.949s
    Epoch 19, cost 2.092, err 69.643%, , Time: 8.785s
    Epoch 20, cost 2.085, err 69.690%, , Time: 8.976s
    Epoch 21, cost 2.078, err 69.447%, , Time: 9.323s
    Epoch 22, cost 2.071, err 69.400%, , Time: 8.953s
    Epoch 23, cost 2.065, err 69.140%, , Time: 8.695s
    Epoch 24, cost 2.059, err 68.963%, , Time: 8.700s
    Epoch 25, cost 2.053, err 68.993%, , Time: 9.039s
    Epoch 26, cost 2.047, err 68.893%, , Time: 9.036s
    Epoch 27, cost 2.041, err 68.703%, , Time: 9.254s
    Epoch 28, cost 2.036, err 68.640%, , Time: 9.259s
    Epoch 29, cost 2.031, err 68.567%, , Time: 8.875s
    Epoch 30, cost 2.026, err 68.253%, , Time: 8.745s
    Epoch 31, cost 2.021, err 68.417%, , Time: 8.833s
    Epoch 32, cost 2.016, err 68.303%, , Time: 8.904s
    Epoch 33, cost 2.011, err 68.303%, , Time: 8.843s
    Epoch 34, cost 2.007, err 68.160%, , Time: 9.088s
    Epoch 35, cost 2.003, err 68.000%, , Time: 9.240s
    Epoch 36, cost 1.999, err 67.943%, , Time: 8.939s
    Epoch 37, cost 1.995, err 67.987%, , Time: 8.743s
    Epoch 38, cost 1.991, err 67.920%, , Time: 8.789s
    Epoch 39, cost 1.987, err 67.773%, , Time: 8.895s
    Epoch 40, cost 1.983, err 67.557%, , Time: 8.782s
    Epoch 41, cost 1.980, err 67.487%, , Time: 9.068s
    Epoch 42, cost 1.976, err 67.600%, , Time: 9.200s
    Epoch 43, cost 1.973, err 67.577%, , Time: 8.957s
    Epoch 44, cost 1.970, err 67.357%, , Time: 8.724s
    Epoch 45, cost 1.966, err 67.270%, , Time: 8.775s
    Epoch 46, cost 1.963, err 67.227%, , Time: 8.842s
    Epoch 47, cost 1.960, err 67.113%, , Time: 8.851s
    Epoch 48, cost 1.957, err 66.987%, , Time: 9.074s
    Epoch 49, cost 1.954, err 66.870%, , Time: 9.189s
    Epoch 50, cost 1.952, err 67.053%, , Time: 8.981s
    Epoch 51, cost 1.949, err 66.960%, , Time: 8.744s
    Epoch 52, cost 1.946, err 66.790%, , Time: 8.736s
    Epoch 53, cost 1.943, err 66.787%, , Time: 8.850s
    Epoch 54, cost 1.941, err 66.523%, , Time: 8.887s
    Epoch 55, cost 1.939, err 66.577%, , Time: 9.048s
    Epoch 56, cost 1.936, err 66.560%, , Time: 9.182s
    Epoch 57, cost 1.934, err 66.363%, , Time: 8.945s
    Epoch 58, cost 1.931, err 66.357%, , Time: 8.732s
    Epoch 59, cost 1.929, err 66.253%, , Time: 8.845s
    Epoch 60, cost 1.927, err 66.097%, , Time: 9.002s
    Epoch 61, cost 1.925, err 66.003%, , Time: 8.917s
    Epoch 62, cost 1.922, err 66.077%, , Time: 9.047s
    Epoch 63, cost 1.920, err 66.070%, , Time: 8.952s
    Epoch 64, cost 1.918, err 65.907%, , Time: 8.908s
    Epoch 65, cost 1.916, err 65.643%, , Time: 8.760s
    Epoch 66, cost 1.914, err 65.750%, , Time: 8.733s
    Epoch 67, cost 1.912, err 65.663%, , Time: 8.826s
    Epoch 68, cost 1.910, err 65.663%, , Time: 8.803s
    Epoch 69, cost 1.909, err 65.720%, , Time: 9.015s
    Epoch 70, cost 1.907, err 65.633%, , Time: 9.270s
    Epoch 71, cost 1.905, err 65.483%, , Time: 8.864s
    Epoch 72, cost 1.903, err 65.380%, , Time: 8.759s
    Epoch 73, cost 1.902, err 65.357%, , Time: 8.751s
    Epoch 74, cost 1.900, err 65.260%, , Time: 8.861s
    Epoch 75, cost 1.898, err 65.180%, , Time: 8.774s
    Epoch 76, cost 1.897, err 65.117%, , Time: 9.002s
    Epoch 77, cost 1.895, err 65.097%, , Time: 9.234s
    Epoch 78, cost 1.893, err 64.973%, , Time: 8.903s
    Epoch 79, cost 1.892, err 64.873%, , Time: 8.720s
    Epoch 80, cost 1.890, err 65.023%, , Time: 8.806s
    Epoch 81, cost 1.889, err 65.030%, , Time: 8.814s
    Epoch 82, cost 1.887, err 64.793%, , Time: 8.724s
    Epoch 83, cost 1.886, err 64.837%, , Time: 8.967s
    Epoch 84, cost 1.884, err 64.803%, , Time: 9.254s
    Epoch 85, cost 1.883, err 64.753%, , Time: 8.926s
    Epoch 86, cost 1.882, err 64.697%, , Time: 8.752s
    Epoch 87, cost 1.880, err 64.700%, , Time: 8.779s
    Epoch 88, cost 1.879, err 64.537%, , Time: 8.824s
    Epoch 89, cost 1.878, err 64.697%, , Time: 8.756s
    Epoch 90, cost 1.876, err 64.520%, , Time: 8.973s
    Epoch 91, cost 1.875, err 64.573%, , Time: 9.150s
    Epoch 92, cost 1.874, err 64.453%, , Time: 8.908s
    Epoch 93, cost 1.873, err 64.347%, , Time: 8.702s
    Epoch 94, cost 1.871, err 64.373%, , Time: 8.700s
    Epoch 95, cost 1.870, err 64.300%, , Time: 8.946s
    Epoch 96, cost 1.869, err 64.330%, , Time: 8.903s
    Epoch 97, cost 1.868, err 64.237%, , Time: 8.979s
    Epoch 98, cost 1.867, err 64.177%, , Time: 9.186s
    Epoch 99, cost 1.866, err 64.077%, , Time: 8.911s
    Epoch 100, cost 1.864, err 64.137%, , Time: 8.698s
    Epoch 101, cost 1.863, err 63.997%, , Time: 8.809s
    Epoch 102, cost 1.862, err 64.063%, , Time: 8.768s
    Epoch 103, cost 1.861, err 64.137%, , Time: 8.837s
    Epoch 104, cost 1.860, err 64.067%, , Time: 9.055s
    Epoch 105, cost 1.859, err 63.927%, , Time: 9.171s
    Epoch 106, cost 1.858, err 63.963%, , Time: 8.839s
    Epoch 107, cost 1.857, err 63.773%, , Time: 8.734s
    Epoch 108, cost 1.856, err 63.873%, , Time: 8.735s
    Epoch 109, cost 1.855, err 63.917%, , Time: 8.819s
    Epoch 110, cost 1.854, err 63.790%, , Time: 8.830s
    Epoch 111, cost 1.853, err 63.763%, , Time: 9.073s
    Epoch 112, cost 1.852, err 63.753%, , Time: 9.170s
    Epoch 113, cost 1.851, err 63.700%, , Time: 8.918s
    Epoch 114, cost 1.850, err 63.713%, , Time: 8.785s
    Epoch 115, cost 1.849, err 63.750%, , Time: 8.765s
    Epoch 116, cost 1.848, err 63.673%, , Time: 8.820s
    Epoch 117, cost 1.847, err 63.647%, , Time: 8.760s
    Epoch 118, cost 1.846, err 63.643%, , Time: 8.968s
    Epoch 119, cost 1.845, err 63.490%, , Time: 9.237s
    Epoch 120, cost 1.844, err 63.490%, , Time: 8.951s
    Epoch 121, cost 1.844, err 63.590%, , Time: 8.807s
    Epoch 122, cost 1.843, err 63.460%, , Time: 8.785s
    Epoch 123, cost 1.842, err 63.443%, , Time: 8.765s
    Epoch 124, cost 1.841, err 63.340%, , Time: 8.785s
    Epoch 125, cost 1.840, err 63.260%, , Time: 8.945s
    Epoch 126, cost 1.839, err 63.323%, , Time: 9.248s
    Epoch 127, cost 1.838, err 63.320%, , Time: 9.139s
    Epoch 128, cost 1.838, err 63.303%, , Time: 9.056s
    Epoch 129, cost 1.837, err 63.163%, , Time: 8.860s
    Epoch 130, cost 1.836, err 63.197%, , Time: 9.065s
    Epoch 131, cost 1.835, err 63.400%, , Time: 8.919s
    Epoch 132, cost 1.835, err 63.233%, , Time: 9.300s
    Epoch 133, cost 1.834, err 63.203%, , Time: 9.378s
    Epoch 134, cost 1.833, err 63.043%, , Time: 9.174s
    Epoch 135, cost 1.832, err 63.133%, , Time: 8.947s
    Epoch 136, cost 1.831, err 63.063%, , Time: 8.969s
    Epoch 137, cost 1.831, err 63.020%, , Time: 9.111s
    Epoch 138, cost 1.830, err 62.953%, , Time: 9.061s
    Epoch 139, cost 1.829, err 63.060%, , Time: 9.333s
    Epoch 140, cost 1.828, err 62.920%, , Time: 9.545s
    Epoch 141, cost 1.827, err 62.900%, , Time: 9.044s
    Epoch 142, cost 1.827, err 62.940%, , Time: 9.069s
    Epoch 143, cost 1.826, err 62.950%, , Time: 9.112s
    Epoch 144, cost 1.825, err 62.877%, , Time: 9.166s
    Epoch 145, cost 1.824, err 62.853%, , Time: 9.081s
    Epoch 146, cost 1.824, err 62.840%, , Time: 9.387s
    Epoch 147, cost 1.823, err 62.817%, , Time: 9.626s
    Epoch 148, cost 1.822, err 62.763%, , Time: 9.115s
    Epoch 149, cost 1.822, err 62.897%, , Time: 8.905s
    Epoch 150, cost 1.821, err 62.803%, , Time: 9.060s
    Epoch 151, cost 1.820, err 62.843%, , Time: 9.146s
    Epoch 152, cost 1.820, err 62.883%, , Time: 9.039s
    Epoch 153, cost 1.819, err 62.790%, , Time: 9.343s
    Epoch 154, cost 1.818, err 62.753%, , Time: 9.506s
    Epoch 155, cost 1.817, err 62.753%, , Time: 9.164s
    Epoch 156, cost 1.817, err 62.797%, , Time: 8.849s
    Epoch 157, cost 1.816, err 62.690%, , Time: 8.786s
    Epoch 158, cost 1.815, err 62.693%, , Time: 8.878s
    Epoch 159, cost 1.815, err 62.673%, , Time: 8.839s
    Epoch 160, cost 1.814, err 62.603%, , Time: 9.100s
    Epoch 161, cost 1.814, err 62.550%, , Time: 9.284s
    Epoch 162, cost 1.813, err 62.523%, , Time: 8.929s
    Epoch 163, cost 1.812, err 62.590%, , Time: 8.781s
    Epoch 164, cost 1.812, err 62.603%, , Time: 8.944s
    Epoch 165, cost 1.811, err 62.660%, , Time: 8.901s
    Epoch 166, cost 1.810, err 62.533%, , Time: 8.899s
    Epoch 167, cost 1.810, err 62.540%, , Time: 9.070s
    Epoch 168, cost 1.809, err 62.380%, , Time: 9.272s
    Epoch 169, cost 1.808, err 62.463%, , Time: 8.880s
    Epoch 170, cost 1.808, err 62.490%, , Time: 8.966s
    Epoch 171, cost 1.807, err 62.443%, , Time: 8.721s
    Epoch 172, cost 1.807, err 62.383%, , Time: 8.880s
    Epoch 173, cost 1.806, err 62.410%, , Time: 8.927s
    Epoch 174, cost 1.805, err 62.377%, , Time: 9.048s
    Epoch 175, cost 1.805, err 62.353%, , Time: 9.139s
    Epoch 176, cost 1.804, err 62.293%, , Time: 8.909s
    Epoch 177, cost 1.804, err 62.320%, , Time: 8.790s
    Epoch 178, cost 1.803, err 62.307%, , Time: 8.700s
    Epoch 179, cost 1.802, err 62.327%, , Time: 8.867s
    Epoch 180, cost 1.802, err 62.360%, , Time: 8.842s
    Epoch 181, cost 1.801, err 62.273%, , Time: 9.159s
    Epoch 182, cost 1.801, err 62.247%, , Time: 9.270s
    Epoch 183, cost 1.800, err 62.200%, , Time: 8.872s
    Epoch 184, cost 1.800, err 62.167%, , Time: 8.775s
    Epoch 185, cost 1.799, err 62.117%, , Time: 8.769s
    Epoch 186, cost 1.798, err 62.100%, , Time: 8.857s
    Epoch 187, cost 1.798, err 62.117%, , Time: 8.831s
    Epoch 188, cost 1.797, err 62.143%, , Time: 9.144s
    Epoch 189, cost 1.797, err 62.100%, , Time: 9.215s
    Epoch 190, cost 1.796, err 62.043%, , Time: 9.007s
    Epoch 191, cost 1.796, err 62.040%, , Time: 8.726s
    Epoch 192, cost 1.795, err 62.043%, , Time: 8.676s
    Epoch 193, cost 1.795, err 61.993%, , Time: 8.876s
    Epoch 194, cost 1.794, err 61.913%, , Time: 8.778s
    Epoch 195, cost 1.794, err 61.973%, , Time: 9.026s
    Epoch 196, cost 1.793, err 61.930%, , Time: 9.158s
    Epoch 197, cost 1.792, err 61.897%, , Time: 8.894s
    Epoch 198, cost 1.792, err 61.853%, , Time: 8.790s
    Epoch 199, cost 1.791, err 61.867%, , Time: 8.814s

```python
# Visualize results
epochs = np.arange(len(costs_0))
plt.figure(figsize=(10,8),dpi=400)
plt.plot(epochs, costs_0, label='50 hidden neurons')
plt.plot(epochs, costs_1, label='100 hidden neurons')
plt.plot(epochs, costs_2, label='200 hidden neurons')
plt.xlabel('epoch')
plt.ylabel('cost')
plt.legend(loc='best')
plt.savefig('figure_1.png')
```

Đánh giá mô  hình

```python

```


# Block 2: Linear programming: The min-cost flow problem
### Alfred Galichon (NYU)
## `math+econ+code' masterclass on matching models, optimal transport and applications
© 2018-2019 by Alfred Galichon. Support from NSF grant DMS-1716489 is acknowledged. Pauline Corblet, James Nesbit, and Lucas Vernet contributed.

### Learning Objectives

* Basic concepts of directed networks

* The min-cost flow problem

* Duality, optimality and equilibrium

### References

* [OTME], Ch.8

* Tolstoi (1930). Methods of finding the minimal total kilometrage in cargo transportation planning in space. *Transportation Planning* [in Russian]

* Koopmans (1949). Optimum utilization of the transportation system. *Econometrica*.

* Schrijver (2002). On the History of the Transportation and Maximum Flow Problems. *Mathematical programming*.

## Motivation

### Motivation: network flow problems

In 1930 Tolstoi, a Russian engineer, has to optimize the shipping of cement from factories to cities in the Soviet union through railway.

* each factory produces a fixed number of tons

* each city needs a fixed number of tons -- for now, we'll assume total production=total consumption

* each factory is connected by rail with a few cities, and the corresponding distance is given

* how to ship in order to minimize the total distance travelled?

This problem belongs to the class of *min-cost flow problems*, an important class of linear programming problems, which are the focus of today's lecture. A decade before the invention of linear programming and the work of Kantorovich, Koopmans and Dantzig, Tolstoi described a heuristic method for solving the problem, which led to the optimal solution. We'll solve the problem using modern tools (Gurobi), and will see that his solution was right.

The *shortest path problem*, or how to find the path of minimal distance from a point to another in a network, also belongs in this class.

```R
# Lets load up libraries now
library('gurobi')
library('Matrix')
```

## Data

We will be looking at 3 types of data:

### Sovet Planning Problem

Tolstoi's data, collected by Schrijver (2002). There are 10 factories and 68 Soviet cities, and 155 links connecting a factory to a city. This yields a sparse $68\times10$ matrix. Two vectors listing the demand of each city and the supply of each factory are also specified. This is stored in a $69\times11$ matrix, where we have appended the demand/supply vectors to the right and to the botton of the distance matrix.

```R
thePath = paste0(getwd(),"/../data_mec_optim/networks_sovietplanning")
spp_data = as.matrix(read.csv(paste0(thePath, "/distances.csv"), sep = ",", header = TRUE))  # loads the data
head(spp_data)
```

A matrix: 6 × 12 of type chr

X	Arkhangelsk	Yaroslavl.	Murom	Balakhonikha	Dzerzhinsk	Kishert.	Sverdlovsk	Artemovsk	Iledzhk	Dekonskaya	demand.

Agryz      	NA	NA	NA	 709	1064	 693	NA	NA  	NA	NA  	  2.0

Aleksandrov	NA	NA	NA	NA  	 397	NA  	NA	1180	NA	NA  	  4.0

Almaznaya  	NA	NA	NA	NA  	NA  	NA  	NA	  81	NA	  65	  1.5

Alchevskaya	NA	NA	NA	NA  	NA  	NA  	NA	 106	NA	 114	  4.0

Baku       	NA	NA	NA	NA  	NA  	NA  	NA	1554	NA	1563	 10.0

Barybino   	NA	NA	NA	NA  	NA  	NA  	NA	 985	NA	 968	  2.0

### Subway

Paris (NYC) subway network data: the network data are available on www.ratp.fr/opendata (http://www.mta.info). This data is made of two files. The file `nodes.csv` lists the stations: each station is indexed by the line number; each line has the name of the station, and its spatial coordinates. The file `arcs.csv` lists the links between stations: each link specifies the index of the origin, the index of the destination, and the length of the segment.

```R
city = "Paris"
head = FALSE
sep = ","
# city = 'NYC'
#if (city == "Paris") {
#    sep = ";"
#    head = FALSE
#} else {
#    sep = ","
#    head = TRUE
#}

thePath = paste0(getwd() ,"/../data_mec_optim/networks_subway")

sub_arcs = as.matrix(read.csv(paste0(thePath, "/", city, "/arcs.csv"), sep = sep,
    header = head))  # loads the arc data
sub_nodes = as.matrix(read.csv(paste0(thePath, "/", city, "/nodes.csv"), sep = sep,
    header = head))  # loads the nodes data
head(sub_arcs) # the colums are origin node index, destination node index and distance
head(sub_nodes) # name,
```

A matrix: 6 × 3 of type dbl

V1	V2	V3

1	  2	1027.280873

1	319	   9.062544

2	  1	1027.280873

2	  3	 785.939117

3	  2	 785.939117

3	  4	1010.642024

A matrix: 6 × 6 of type chr

V1	V2	V3	V4	V5	V6

La Défense (Grande Arche)           	14367265	2.237883	48.89193	 535	1282

Esplanade de la Défense             	 9992248	2.250442	48.88784	 603	1373

Pont de Neuilly                     	 6823045	2.259892	48.88451	 687	1456

Les Sablons (Jardin d'acclimatation)	 5500065	2.272539	48.88090	 767	1539

Porte Maillot                       	 9089205	2.281836	48.87796	 863	1633

Argentine                           	 2947071	2.289322	48.87553	 998	1766

```R
dim(sub_arcs)
dim(sub_nodes)
```

866

3

366

6

### NYC Roads

The NYC roads data is available,is available as a `shapefile`. The dataset is very similar in construction to the subway data-except that the network is much larger, as there are $67,316$ nodes and $169,399$ arcs. We won't load up this dataset yet, as it is quite large.

## Shortest path problems

### Topology of Networks

We start by defining the directed graph on which transportation takes place.

---
**Definition**

A (directed) *graph* $\left(\mathcal{X},\mathcal{A}\right)$ is a set of *nodes* (cities) $\mathcal{X}$, along with a set of *arcs* $\mathcal{A}\subseteq\mathcal{X}^{2}$ which are pairs $\left(x,y\right)$ where $x,y\in\mathcal{X}$.

---
Our definition allows an arc to have the same origin and destination. Note that for a dense network, $\mathcal{A}=\mathcal{X}^{2}$. For a line, $\left\vert \mathcal{A}\right\vert =\left\vert \mathcal{X}\right\vert -1$.

Next, we define the gradient matrix.

---
**Definition**

We define the *gradient matrix* (also called an `edge-node matrix') as the matrix with general term $\nabla_{ax}$, $a\in\mathcal{A}$, $x\in \mathcal{X}$, such that

\begin{align*}
\nabla_{ax}  &  =-1\text{ if }a\text{ is out of }x\text{,}\\
&  =+1\text{ if }a\text{ is into }x\text{,}\\
&  =0\text{ else.}
\end{align*}

---

Hence, if $f\in\mathbb{R}^{\mathcal{X}}$, then $\nabla f\in\mathbb{R}^{\mathcal{V}}$, and $\left(  \nabla f\right)  _{xy}=f_{y}-f_{x}$.

We shall denote $\nabla^{\intercal}$ the transpose of the gradient matrix. It is the network analog of the $-\operatorname{div}$ differential operator.

Next, we define paths and loops

---
**Definition**

Given two nodes $x$ and $y$, a *path* from $x$ to $y$ is a sequence $x_{1},x_{2},...,x_{K}$ in $\mathcal{X}$ where $x_{1}=x$, $x_{K}=y$, and for every $1\leq k\leq K-1$, $\left(  x_{k},x_{k+1}\right)  \in\mathcal{A}$.

A *loop* (also called `cycle') is a path from a node $x$ to itself.

---

A vector $c\in\mathbb{R}^{\mathcal{V}}$ defined transportation costs. That is, for $xy\in\mathcal{V}$, $c_{xy}$ is the transportation cost associated to arc $xy$. $c$ can also be thought of as the length of arc $xy$. The cost of moving the good from node $x$ to node $y$ along path $x_{1},x_{2},...,x_{K}$ is

\begin{align*}
\sum_{k=1}^{K-1}c_{x_{k}x_{k+1}}.
\end{align*}

No arbitrage conditions: there is no loop of negative cost:

---
**Assumption** [No profitable loop]

There is no profitable loop, which means that there is no sequence $x_{1},...,x_{K}$ in $\mathcal{X}$ such that $x_{K}=x_{1}$, $\left(  x_{k},x_{k+1}\right)  \in\mathcal{A}$, and $\sum_{k=1}^{K-1}c_{x_{k}x_{k+1}}<0$.

---

In particular, there is no profitable loop if $c\geq0$.

Let $n_{x}$ be the *net demand*, which is the flow of goods disappearing from the graph. The set of nodes defined by

\begin{align*}
\mathcal{X}_{0}=\left\{  x\in\mathcal{X}:n_{x}<0\right\}  \text{, and}\mathcal{X}_{1}=\left\{  x\in\mathcal{X}:n_{x}>0\right\}
\end{align*}

are called the *supply nodes* and *demand nodes* respectively. A node which is neither a supply node, neither a demand node is called a *transit node*.

Total supply is $-\sum_{x\in\mathcal{X}_{0}}n_{x}$, total demand is $\sum_{y\in\mathcal{X}_{1}}n_{y}$.

---
**Assumption** [Balancedness]

Assume that total supply equals total demand on the network, that is

\begin{align*}
\sum_{x\in\mathcal{X}_{0}}n_{x}+\sum_{y\in\mathcal{X}_{1}}n_{y}=0.
\end{align*}

---

---
**Assumption** [Connectedness]

Assume the set of supply nodes $\mathcal{X}_{0}$ is strongly connected to the set of demand nodes $\mathcal{X}_{1}$, i.e. for every $x\in\mathcal{X}_{0}$ and $y\in \mathcal{X}_{1}$, there is a path from $x$ to $y$.

---

The specification of the graph, the net demand vector, and the surplus vector
defines a network.

---
**Definition**

A directed graph $\left(\mathcal{X},\mathcal{A}\right)$ endowed with a net demand vector $\left(n_{z}\right)_{z \in \mathcal{X}}$ and a cost vector $\left(c_{a}\right)_{a\in\mathcal{A}}$ is called a *network* $\left(\mathcal{X}, \mathcal{A}, n, c\right)$. If Assumptions [no profitable loop](#noprofitloop), [balancedness](#balancedness) and [connectedness](#connectedness) all hold, the network is called *regular*.

---

Without mention of the contrary we shall assume that the network under consideration is regular.

The flow of mass disappearing at $x$ equals the flow arriving from other nodes minus the flow shipping to other nodes

\begin{align*}
n_{x}=\sum_{z:\left(  z,x\right)  \in\mathcal{A}}\pi_{zx}-\sum_{z:\left(x,z\right)  \in\mathcal{A}}\pi_{xz}
\end{align*}

and this equation can be rewritten as $\nabla^{\intercal}\pi=n$. This motivates the following definition:

---
**Definition**

The set of feasible flows, denoted $\mathcal{M}\left(  n\right)$, or $\mathcal{M}$ when there is no ambiguity, is defined as the set of flows $\pi\geq0$ that verify conservation equation

\begin{align*}
    \nabla^{\intercal}\pi=n.
\end{align*}

---

Let $\phi_{x}$ be the price of the commodity at $x$. Consider the strategy which consists in purchasing the good at $x$, shipping to $y$, and selling at $y$. The profit of this strategy is

\begin{align*}
\phi_{y}-\phi_{x}-c_{xy}=\left(  \nabla\phi-c\right)_{xy}
\end{align*}

and hence there is no arbitrage opportunity if $\phi_{y}-\phi_{x}-c_{xy}\leq0$ for every arc $xy$, that is

\begin{align*}
\nabla\phi\leq c.
\end{align*}

Consider the *minimum cost flow problem*

\begin{align*}
&  \min_{\pi\geq0}\sum_{\left(  x,y\right)  \in\mathcal{A}}\pi_{xy}%
c_{xy} \\
&  s.t.~\nabla^{\intercal}\pi=n
\end{align*}

which is a Linear Programming problem whose dual is

\begin{align*}
&  \max_{\phi\in\mathbb{R}^{\mathcal{X}}}\sum_{x\in\mathcal{X}}n_{x}\phi
_{x}\\
&  s.t.~\nabla\phi\leq c.
\end{align*}

---
**Proposition**
1. Under Assumption [No profitable loop](#noprofitloop), the [dual problem](#dual) is feasible, which means that there is a vector $\phi\in\mathbb{R}^{\mathcal{X}}$ such that $\nabla\phi\leq c$; and the value of [primal problem](#primal) is strictly less than $+\infty$.

2. Under Assumptions [balancedness](#balandedness) and [connectedness](#connectedness), the [primal problem](#primal) is feasible, which means that there is a flow $\pi\geq0$ such that $\nabla^{\intercal}\pi=n$; and the value of [dual problem](#dual) is strictly greater than $-\infty$.

---

Assume that $\left(\mathcal{X},\mathcal{A},n,c\right)$ is a regular network. Then the value of the [primal problem](#primal) coincides with the value of its [dual](#dual), and both problems have solutions. Further, if $\pi$ is a solution to the primal and $\phi$ is a solution to the dual, then $\pi_{xy}>0$ implies $\phi_{y}-\phi_{x}=c_{xy}$.

### Special cases

#### Shortest path problem

Assume there is only one source node $s \in \mathcal{X}$ and one target node $t\in\mathcal{X}$, each associated with unit flow. That is, $n_{x}=1\left\{  x=t\right\}  -1\left\{  x=s\right\}$. Then the problem boils down to how to push one unit of mass from $s$ to $t$. If we interpret $c_{xy}$ as the distance along arc $xy$, the solution of this problem corresponds to the shortest path from $s$ to $t$. This is why this problem is called shortest path problem. (More generally, this problem extends to the case when $c$ does not have a negative loop).

The dual problem is then

\begin{align*}
\max_{\phi}  &  ~\phi_{t}-\phi_{s}\\
s.t.  &  ~\phi_{y}-\phi_{x}\leq c_{xy}~\forall xy\in\mathcal{A}
\end{align*}

and we can impose normalization $\phi_{s}=0$, so that along the travelled path, $\phi_{x}$ interprets as the distance travelled thus far.

We have advocated for the use of Gurobi as a black box in this problem,but there exists a direct method to find out the minimal cost path by dynamic programming.

* The crucial remark here is that if there is a minimal cost path from $s$ to $t$, then there is one of length at most $\left\vert \mathcal{X}\right\vert$.

For $z\in\mathcal{X}$, and $t\in\mathbb{N}$, let $C_{sz}^{k}$ be the minimal cost of the path from $s$ to $z$ among paths of length at most $k$, with the convention that $C_{sz}^{k}=+\infty$ if there is no such path. One has:

* $C_{ss}^{0}=0$ and $C_{sz}^{0}=+\infty$ for all $z\neq s$, and

* for $t\geq1$, $C_{sz}^{k}$ satisfies the Bellman equation:

\begin{align*}
C_{sz}^{k}=\min\left\{  C_{sz}^{k},\min_{x\in\mathcal{Z}:sz\in\mathcal{A}%
}\left\{  c_{sx}+C_{xz}^{k-1}\right\}  \right\}  .
\end{align*}

It is easy to see that $C_{st}^{\left\vert \mathcal{X}\right\vert }$ is the minimal cost of any path from $s$ to $t$. Shortest-paths algorithms (Dijikstra and Bellman-Ford) implement this idea.

#### Transportation problem

Assume the problem is bipartite, that is $\mathcal{X}=\mathcal{S}\cup\mathcal{T}$ and $\mathcal{A}\subseteq\mathcal{S}\times\mathcal{T}$. That is, there are no intermediate nodes, and an arc can only go directly from a source to a target.

Note that any min-cost flow problem can be recast in this form, by taking the shortest distance between any source node and any target node.

The dual problem is then

\begin{align*}
\max_{\phi}  &  ~\sum_{x\in\mathcal{X}}n_{x}\phi_{x}\\
s.t.  &  ~\phi_{t}-\phi_{s}\leq c_{st}~\forall s\in\mathcal{S},t\in\mathcal{T}
\end{align*}

which is our first encounter with optimal transport (more on this tomorrow).

## Soviet Planning Problem

The problem that we are trying to solve is

\begin{align*}
\min_{\pi \geq 0} \sum_{(x,y) \in \mathcal{A}} \pi_{xy} c_{xy} \\
\text{s.t. } \nabla^T \pi = n
\end{align*}

Suppose that the network has `nbNodes` nodes and `nbArcs` arcs.
* $\pi$ is our object of interest,
* $c$ a vector of length `nbArcs` such that $c_a$ is the transportation cost at arc $a$,
* $\nabla$ is an matrix of size `nbArcs` $\times$ `nbNodes`. If the first arc has $i$ as the origin node and $j$ as the the destination node then $\nabla_{1i} = -1$ and $\nabla_{1j} = 1$,
* $n$, a vector of length `nbNodes` such that $n_x$ is the net demand at node $x$.

The first thing we will try and construct is the [gradient matrix](#grad_mat). It is a `nbArcs` $\times$ `nbNodes` matrix with $-1$ in the origin node of an arc and $1$ in the destination node. We will construct the gradient matrix as a sparse matrix.

```R
?sparseMatrix
```

```R
library("gurobi")
library("Matrix")

# Data cleaning
nsources = dim(spp_data)[1] - 1
ndests = dim(spp_data)[2] - 2
dists = matrix(as.numeric(spp_data[1:68, 2:11]), 68, 10)
demand = matrix(as.numeric(spp_data[1:68, 12]))
supply = matrix(as.numeric(spp_data[69, 2:11]))
nameNodes = c(spp_data[2:(nsources + 1)], dimnames(spp_data)[[2]][2:11])
nonzeros = which(!is.na(dists), arr.ind = TRUE)  # find nonzero entries

nbNodes = nsources+ndests
nbArcs = length(nonzeros[,1])

rows = nonzeros[, 1]
cols = nonzeros[, 2]
distances = dists[nonzeros]
arcs = cbind(rows, cols + nsources, distances)  # we add nsources to cols to generate a new `ID` for the destinations

head(arcs)
```

A matrix: 6 × 3 of type dbl

rows		distances

13	69	 634

16	69	 815

27	69	1237

42	69	1749

43	69	1394

 7	70	 135

We have constructed our matrix of arcs, now we can easily construct our $\nabla$ matrix, as the first column of `arcs` are the indices of the origin node (which will have $-1$ entry in $\nabla$) and the second column is are the indices of the destination nodes (which will have $+1$ entry).

```R
Nabla = sparseMatrix(i = 1:nbArcs, j = arcs[, 1], dims = c(nbArcs, nbNodes), x = -1) + sparseMatrix(i = 1:nbArcs, j = arcs[, 2], dims = c(nbArcs, nbNodes), x = 1)
```

$\nabla^\intercal$ is a `nbNodes` $\times$ `nbArcs` matrix, with the origins ordered first and then the destinations. So the net demand vector $n$ will just involve stacking the net demand of first the origins, then the destinations.

```R
n = c(-demand, supply)
```

Lastly, the cost vector is just the vector of distances.

```R
c = distances
```

Now let's solve using Gurobi:
* `A` = $\nabla^T$
* `obj` = $c$
* `sense` = '$=$'
* `rhs` = $n$
* `modelsense` = '$\min$'.

```R
result = gurobi(list(A = t(Nabla), obj = c, modelsense = "min", rhs = n, sense = "=",
    start = matrix(0, nbArcs, 1)), params = NULL)
pi = result$x
distance = result$objval

print(distance)
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 78 rows, 155 columns and 310 nonzeros
    Model fingerprint: 0xefeeca13
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [1e+01, 3e+03]
      Bounds range     [0e+00, 0e+00]
      RHS range        [5e-01, 3e+02]
    Presolve removed 48 rows and 48 columns
    Presolve time: 0.00s
    Presolved: 30 rows, 107 columns, 214 nonzeros

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    1.6658785e+05   4.252320e+02   0.000000e+00      0s
          37    3.9505200e+05   0.000000e+00   0.000000e+00      0s

    Solved in 37 iterations and 0.00 seconds
    Optimal objective  3.950520000e+05
    [1] 395052

## Subway Problem

We are trying to solve the exact same problem as the subway problem:
\begin{align}
\min_{\pi \geq 0} \sum_{(x,y) \in \mathcal{A}} \pi_{xy} c_{xy} \\
\text{s.t. } \nabla^T \pi = n
\end{align}

Once again let's first construct our $\nabla$ matrix. This will be pretty straightforward as the data is already in the correct format.

```R
sub_nbNodes = max(as.numeric(sub_arcs[, 1]))
sub_nbArcs = dim(sub_arcs)[1]
sub_namesNodes = sub_nodes[, 1]

sub_Nabla = sparseMatrix(i = 1:sub_nbArcs, j = as.numeric(sub_arcs[, 1]), dims = c(sub_nbArcs,
    sub_nbNodes), x = -1) + sparseMatrix(i = 1:sub_nbArcs, j = as.numeric(sub_arcs[,
    2]), dims = c(sub_nbArcs, sub_nbNodes), x = 1)
```

The net demand vector will just be $-1$ in our origin node and $1$ in our destination node.

```R
originNode = 84  #saint-germain des pres
destinationNode = 116  #trocadero
sub_n = rep(0, sub_nbNodes)  # construct vector of exiting flow, net demand is zero
sub_n[c(originNode, destinationNode)] = c(-1, 1)  # except for our origin and destination
```

And finally our cost vector

```R
sub_c = sub_arcs[, 3]
```

Now let's solve using Gurobi:
* `A` = $\nabla^T$
* `obj` = $c$
* `sense` = '$=$'
* `rhs` = $n$
* `modelsense` = '$\min$'.

```R
result = gurobi(list(A = t(sub_Nabla), obj = as.numeric(sub_c), sense = "=", rhs = sub_n,
    modelsense = "min", start = matrix(0, sub_nbArcs, 1)), params = NULL)
pi = result$x
distance = result$objval
```

    Gurobi Optimizer version 9.0.0 build v9.0.0rc2 (win64)
    Optimize a model with 366 rows, 866 columns and 1732 nonzeros
    Model fingerprint: 0xb4f97da7
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [6e+00, 6e+03]
      Bounds range     [0e+00, 0e+00]
      RHS range        [1e+00, 1e+00]
    Presolve removed 301 rows and 602 columns
    Presolve time: 0.00s
    Presolved: 65 rows, 264 columns, 528 nonzeros

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
           0    0.0000000e+00   2.000000e+00   0.000000e+00      0s
          25    5.3943935e+03   0.000000e+00   0.000000e+00      0s

    Solved in 25 iterations and 0.00 seconds
    Optimal objective  5.394393536e+03

Let's deduce what the minimal distance path is

```R
pi = as.logical(pi)
eqpath = which(pi > 0)
sub_arcs[pi, 1:2]
```

A matrix: 10 × 2 of type dbl

V1	V2

 81	238

 82	 81

 83	 82

 84	 83

117	116

118	117

119	118

120	119

237	120

238	237

Which isn't very illuminating...

```R
# Some plotting stuff
library("magick")
library("igraph")
geoCoordinates = sub_nodes[, 3:4]
class(geoCoordinates) = "numeric"

# Function to plot network
plotCurrentNetwork = function(network, curNode) {
    sizeNodes = rep(1, sub_nbNodes)
    sizeNodes[originNode] = 4
    sizeNodes[destinationNode] = 4
    sizeNodes[curNode] = 4
    labelNodes = rep(NA, sub_nbNodes)
    labelNodes[originNode] = sub_namesNodes[originNode]
    labelNodes[destinationNode] = sub_namesNodes[destinationNode]
    labelNodes[curNode] = sub_namesNodes[curNode]
    plot.igraph(network, vertex.label = labelNodes, vertex.label.cex = 1, vertex.size = sizeNodes,
        edge.arrow.size = 0, layout = geoCoordinates)
}

# Begin animation instance
frames = image_graph(width = 1600, height = 1600, res = 200)

thegraph = graph_from_edgelist(sub_arcs[, 1:2])

cont = TRUE
i = originNode
writeLines(paste("Starting from",paste0(sub_namesNodes[i], " (#", i, ")")))
rank = 0
while (cont) {
    rank = rank + 1
    leavingi = which(sub_Nabla[, i] == -1)
    a = intersect(eqpath, leavingi)[1]
    j = which(sub_Nabla[a, ] == 1)[1]
    plotCurrentNetwork(thegraph, j)
    writeLines(paste0(rank, ": ", sub_namesNodes[j], " (#", j, ")"))
    i = j
    if (j == destinationNode) {
        cont <- FALSE
    }
}
# done with plotting
dev.off()
```

    Starting from Saint-Germain des Prés (#84)
    1: Saint-Sulpice (#83)
    2: Saint-Placide (#82)
    3: Montparnasse-Bienvenue (#81)
    4: Duroc (#238)
    5: Ségur (#237)
    6: La Motte-Picquet-Grenelle (#120)
    7: Dupleix (#119)
    8: Bir-Hakeim (Grenelle) (#118)
    9: Passy (#117)
    10: Trocadéro (#116)

png: 2

```R
# animate
animation = image_animate(frames, 1)
image_write(animation, "subway.gif")
print(animation)
```

## NYC Roads

Again we are trying to solve the shortest path problem.

```R
library("rgdal")
library("rdist")

thePath = paste0(getwd() ,"/../data_mec_optim/networks_NYCroads")
load(paste0(thePath, "/DataNYC.Rdata"))
```

```R
shortestPath = function(arcs, nodes, originNode, destinationNode) {
    nbNodes = dim(nodes)[1]
    nbArcs = dim(arcs)[1]
    n = rep(0, nbNodes)
    n[c(originNode, destinationNode)] = c(-1, 1)
    Nabla = sparseMatrix(i = 1:nbArcs, j = arcs[, 1], dims = c(nbArcs, nbNodes),
        x = -1) + sparseMatrix(i = 1:nbArcs, j = arcs[, 2], dims = c(nbArcs, nbNodes),
        x = 1)
    Phi <- -arcs[, 3]

    result = gurobi(list(A = t(Nabla), obj = Phi, modelsense = "max", rhs = n, sense = "=",
        start = matrix(0, nbArcs, 1)), params = NULL)
    pi = result$x
    distance = -result$objval

    cont = TRUE
    i = originNode
    eqpath = which(pi > 0)
    rank = 0
    nodespath = c(0)
    nodespath[rank + 1] = i
    while (cont) {
        rank = rank + 1
        leavingi = which(Nabla[, i] == -1)
        a = intersect(eqpath, leavingi)[1]
        j = which(Nabla[a, ] == 1)[1]
        i = j
        nodespath[rank + 1] = i
        if (j == destinationNode) {
            cont <- FALSE
        }
    }

    return(nodespath)

    plot(shpNYC)
    points(nodes[nodespath, ], col = "red")
}
```

To get coordinates associated with an address:
https://www.gps-coordinates.net/

```R
startlat = 40.70102
startlong = -73.90414
finishlat = 40.7290094
finishlong = -73.9952367

StartGPS = matrix(c(startlong, startlat), ncol = 2)
FinishGPS = matrix(c(finishlong, finishlat), ncol = 2)
StartNAD83 = project(StartGPS, "+proj=lcc +lat_1=40.66666666666666 +lat_2=41.03333333333333 +lat_0=40.16666666666666 +lon_0=-74 +x_0=300000 +y_0=0 +datum=NAD83 +units=us-ft +no_defs")
FinishNAD83 = project(FinishGPS, "+proj=lcc +lat_1=40.66666666666666 +lat_2=41.03333333333333 +lat_0=40.16666666666666 +lon_0=-74 +x_0=300000 +y_0=0 +datum=NAD83 +units=us-ft +no_defs")
StartNode = which.min(cdist(StartNAD83, nodes, metric = "euclidean"))
FinishNode = which.min(cdist(FinishNAD83, nodes, metric = "euclidean"))
nodespath = shortestPath(arcs, nodes, StartNode, FinishNode)
```

    Optimize a model with 67316 rows, 169399 columns and 338796 nonzeros
    Coefficient statistics:
      Matrix range     [1e+00, 1e+00]
      Objective range  [5e+00, 1e+04]
      Bounds range     [0e+00, 0e+00]
      RHS range        [1e+00, 1e+00]

    Concurrent LP optimizer: dual simplex and barrier
    Showing barrier log only...

    Presolve removed 17773 rows and 28823 columns
    Presolve time: 0.54s
    Presolved: 49543 rows, 140576 columns, 281146 nonzeros

    Ordering time: 0.00s

    Barrier statistics:
     AA' NZ     : 8.984e+04
     Factor NZ  : 8.256e+05 (roughly 80 MBytes of memory)
     Factor Ops : 1.873e+07 (less than 1 second per iteration)
     Threads    : 3

                      Objective                Residual
    Iter       Primal          Dual         Primal    Dual     Compl     Time
       0  -4.22816139e+07  3.57483804e+02  5.30e+00 3.84e+03  8.76e+03     1s
       1  -3.80006823e+07 -2.24488020e+03  1.24e-01 2.51e+03  2.13e+03     1s
       2  -2.88697338e+07 -3.02933993e+03  6.99e-03 2.57e+02  3.64e+02     1s
       3  -6.13984474e+06 -5.15559077e+03  2.33e-04 1.35e+01  4.64e+01     1s
       4  -7.72223691e+05 -1.47773761e+04  2.35e-05 5.75e-01  5.43e+00     1s
       5  -3.32665358e+05 -2.65673557e+04  8.72e-06 1.54e-01  2.19e+00     1s
       6  -1.65534906e+05 -3.02040135e+04  3.64e-06 1.03e-01  9.65e-01     1s
       7  -9.77877154e+04 -3.18335966e+04  1.73e-06 2.04e-01  4.70e-01     1s
       8  -4.05263884e+04 -3.22636316e+04  1.98e-07 7.39e-02  5.89e-02     1s
       9  -3.24930927e+04 -3.24397597e+04  9.41e-10 2.86e-03  3.82e-04     1s
      10  -3.24453581e+04 -3.24453003e+04  2.36e-11 3.52e-12  4.11e-07     1s
      11  -3.24453076e+04 -3.24453075e+04  2.34e-13 3.47e-12  4.11e-10     2s

    Barrier solved model in 11 iterations and 1.56 seconds
    Optimal objective -3.24453076e+04

    Crossover log...

           0 DPushes remaining with DInf 6.9903905e-10                 2s

           0 PPushes remaining with PInf 0.0000000e+00                 2s

      Push phase complete: Pinf 0.0000000e+00, Dinf 6.9903905e-10      2s

    Iteration    Objective       Primal Inf.    Dual Inf.      Time
       49435   -3.2445308e+04   0.000000e+00   0.000000e+00      2s

    Solved with barrier
    Solved in 49435 iterations and 2.16 seconds
    Optimal objective -3.244530753e+04

Plot the result (this will take a while, so don't worry).

```R
plot(shpNYC)
points(nodes[nodespath, ], col = "red")
```