# ME-314 - Final Project - Yael Ben Shalom

*In this project, I simulated the dynamics of a planar jack-in-a-box system. I simulated and animated the dynamics of the multi-body system in time, with external forces applied on the box and impact accrues between the two rigid bodies (the jack and the box’s walls).*

*Both bodies in the planar systems are in gravity (in the -y direction of the world frame). In addition, two external forces are applied to the box – the first is an equal and opposite force to gravity (to ensure the box stays at the same position), and the second is a torque to rotate the box. The applied torque has a sinusoidal form to make the box rotate back and forth in a constant frequency.*

***

```python
import numpy as np
import sympy as sym
from sympy.abc import t
from sympy import sin, cos, symbols, Function, Inverse, Eq, Matrix, Rational, simplify, lambdify, solve
from math import pi
import matplotlib.pyplot as plt
```

```python
def mat_inverse(g):
    ''' A functions for ''invers'' operations.
    '''
    R = Matrix([[g[0,0], g[0,1], g[0,2]],
                [g[1,0], g[1,1], g[1,2]],
                [g[2,0], g[2,1], g[2,2]]])
    R_inv = R.T
    p = Matrix([g[0,3], g[1,3], g[2,3]])
    p_inv = -R_inv * p
    g_inv = Matrix([[R_inv[0,0], R_inv[0,1], R_inv[0,2], p_inv[0]],
                      [R_inv[1,0], R_inv[1,1], R_inv[1,2], p_inv[1]],
                      [R_inv[2,0], R_inv[2,1], R_inv[2,2], p_inv[2]],
                      [         0,          0,         0,        1]])
    return g_inv

def unhat(g):
    ''' A functions for ''unhat'' operations.
    '''
    V = Matrix([0, 0, 0, 0, 0, 0])
    V[0, 0] = g[0, 3]
    V[1, 0] = g[1, 3]
    V[2, 0] = g[2, 3]
    V[3, 0] = g[2, 1]
    V[4, 0] = g[0, 2]
    V[5, 0] = g[1, 0]

    return V

def get_se3_np(x, y, theta):
    """
    This function return SE(3) given x, y, theta
    """
    return np.array([[np.cos(theta), -np.sin(theta), 0, x],
                     [np.sin(theta),  np.cos(theta), 0, y],
                     [            0,              0, 1, 0],
                     [            0,              0, 0, 1]])

def integrate(f, xt, dt, time):
    """
    This function takes in an initial condition x(t) and a timestep dt,
    as well as a dynamical system f(x) that outputs a vector of the
    same dimension as x(t). It outputs a vector x(t+dt) at the future
    time step.

    Parameters
    ============
    dyn: Python function
        derivate of the system at a given step x(t),
        it can considered as \dot{x}(t) = func(x(t))
    x0: NumPy array
        current step x(t)
    dt:
        step size for integration
    time:
        step time

    Return
    ============
    new_x:
        value of x(t+dt) integrated from x(t)
    """
    k1 = dt * f(xt, time)
    k2 = dt * f(xt+k1/2., time)
    k3 = dt * f(xt+k2/2., time)
    k4 = dt * f(xt+k3, time)
    new_xt = xt + (1/6.) * (k1+2.0*k2+2.0*k3+k4)

    return new_xt

def simulate_impact(f, x0, tspan, dt, integrate):
    """
    This function takes in an initial condition x0, a timestep dt,
    a time span tspan consisting of a list [min_time, max_time],
    as well as a dynamical system f(x) that outputs a vector of the
    same dimension as x0. It outputs a full trajectory simulated
    over the time span of dimensions (xvec_size, time_vec_size).

    Parameters
    ============
    f: Python function
        derivate of the system at a given step x(t),
        it can considered as \dot{x}(t) = func(x(t))
    x0: NumPy array
        initial conditions
    tspan: Python list
        tspan = [min_time, max_time], it defines the start and end
        time of simulation
    dt:
        time step for numerical integration
    integrate: Python function
        numerical integration method used in this simulation

    Return
    ============
    x_traj:
        simulated trajectory of x(t) from t=0 to tf
    """
    N = int((max(tspan)-min(tspan))/dt)
    x = np.copy(x0)
    tvec = np.linspace(min(tspan),max(tspan), N)
    xtraj = np.zeros((len(x0), N))
    time = 0
    for i in range(N):
        time = time + dt
        (impact, impact_num) = impact_condition(x, phi_func, 1e-1)
        if impact is True:
            x = impact_update(x, impact_eqns_list[impact_num], dum_list)
            xtraj[:, i]=integrate(f, x, dt, time)
        else:
            xtraj[:, i]=integrate(f, x, dt, time)
        x = np.copy(xtraj[:,i])
    return xtraj
```

```python
# Box parameters:
l_box, M_box = 6, 100
j_box = 4*M_box*l_box**2

# Jack parameters:
l_jack, m_jack = 1, 1
j_jack = 4*m_jack*l_jack**2

# General parameters:
# g = 0 # Without gravity
g = 9.81 # With gravity

# Define box and jack's variable:
x_box = Function(r'x_box')(t)
y_box = Function(r'y_box')(t)
theta_box = Function(r'\theta_box')(t)
x_jack = Function(r'x_jack')(t)
y_jack = Function(r'y_jack')(t)
theta_jack = Function(r'\theta_jack')(t)

# Define the external force:
theta_d_box = sin(2*pi*t/5)
k = 30000
F_y_box = 4*M_box*g
F_theta_box = k*(theta_d_box)
F = Matrix([0, F_y_box, F_theta_box, 0, 0, 0])

# Define the configuration vector:
q = Matrix([x_box, y_box, theta_box, x_jack, y_jack, theta_jack])
qdot = q.diff(t)
qddot = qdot.diff(t)

# Finding the homogeneous representations of the rigid bodies:
g_w_b = Matrix([[cos(theta_box), -sin(theta_box), 0, x_box], [sin(theta_box), cos(theta_box), 0, y_box], [0, 0, 1, 0], [0, 0, 0, 1]])
g_b_b1 = Matrix([[1, 0, 0, l_box], [0, 1, 0, l_box], [0, 0, 1, 0], [0, 0, 0, 1]])
g_b_b2 = Matrix([[1, 0, 0, 0], [0, 1, 0, -l_box], [0, 0, 1, 0], [0, 0, 0, 1]])
g_b_b3 = Matrix([[1, 0, 0, -l_box], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
g_b_b4 = Matrix([[1, 0, 0, 0], [0, 1, 0, l_box], [0, 0, 1, 0], [0, 0, 0, 1]])
g_w_j = Matrix([[cos(theta_jack), -sin(theta_jack), 0, x_jack], [sin(theta_jack), cos(theta_jack), 0, y_jack], [0, 0, 1, 0], [0, 0, 0, 1]])
g_j_j1 = Matrix([[1, 0, 0, l_jack], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
g_j_j2 = Matrix([[1, 0, 0, 0], [0, 1, 0, -l_jack], [0, 0, 1, 0], [0, 0, 0, 1]])
g_j_j3 = Matrix([[1, 0, 0, -l_jack], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]])
g_j_j4 = Matrix([[1, 0, 0, 0], [0, 1, 0, l_jack], [0, 0, 1, 0], [0, 0, 0, 1]])

g_w_b1 = g_w_b * g_b_b1
g_w_b2 = g_w_b * g_b_b2
g_w_b3 = g_w_b * g_b_b3
g_w_b4 = g_w_b * g_b_b4

g_w_j1 = g_w_j * g_j_j1
g_w_j2 = g_w_j * g_j_j2
g_w_j3 = g_w_j * g_j_j3
g_w_j4 = g_w_j * g_j_j4

g_b1_j1 = mat_inverse(g_w_b1) * g_w_j1
g_b1_j2 = mat_inverse(g_w_b1) * g_w_j2
g_b1_j3 = mat_inverse(g_w_b1) * g_w_j3
g_b1_j4 = mat_inverse(g_w_b1) * g_w_j4

g_b2_j1 = mat_inverse(g_w_b2) * g_w_j1
g_b2_j2 = mat_inverse(g_w_b2) * g_w_j2
g_b2_j3 = mat_inverse(g_w_b2) * g_w_j3
g_b2_j4 = mat_inverse(g_w_b2) * g_w_j4

g_b3_j1 = mat_inverse(g_w_b3) * g_w_j1
g_b3_j2 = mat_inverse(g_w_b3) * g_w_j2
g_b3_j3 = mat_inverse(g_w_b3) * g_w_j3
g_b3_j4 = mat_inverse(g_w_b3) * g_w_j4

g_b4_j1 = mat_inverse(g_w_b4) * g_w_j1
g_b4_j2 = mat_inverse(g_w_b4) * g_w_j2
g_b4_j3 = mat_inverse(g_w_b4) * g_w_j3
g_b4_j4 = mat_inverse(g_w_b4) * g_w_j4

# Calculate the rigid body velocities for the box and the jack:
V_box = unhat(mat_inverse(g_w_b) * g_w_b.diff(t))
V_jack = unhat(mat_inverse(g_w_j) * g_w_j.diff(t))

# Calculate the body inertia matrix:
I_box = Matrix([[4*M_box, 0, 0, 0, 0, 0], [0, 4*M_box, 0, 0, 0, 0], [0 ,0 ,4*M_box ,0 ,0 ,0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, j_box]])
I_jack = Matrix([[4*m_jack, 0, 0, 0, 0, 0], [0, 4*m_jack, 0, 0, 0, 0], [0, 0, 4*m_jack, 0, 0, 0], [0, 0, 0, 0, 0, 0], [0, 0, 0 ,0, 0, 0], [0, 0, 0, 0, 0, j_jack]])

# Calculate the Lagrangian:
KE = simplify(0.5*(V_box.T)*I_box*V_box + 0.5*(V_jack.T)*I_jack*V_jack)[0]
PE = simplify(g*(4*M_box*y_box + 4*m_jack*y_jack))
L = simplify(KE - PE)

# Compute the Euler-Lagrange Equations:
dL_dq = simplify(Matrix([L]).jacobian(q).T)
dL_dqdot = simplify(Matrix([L]).jacobian(qdot).T)
ddL_dqdot_dt = simplify(dL_dqdot.diff(t))

lhs = simplify(ddL_dqdot_dt - dL_dq)
rhs = simplify(F)
EL_Eqs = simplify(Eq(lhs, rhs))
display(EL_Eqs)

# Solve the Euler-Lagrange Equations:
EL_solns = solve(EL_Eqs, qddot, dict=True)
```

```python
x_box_ddot_func = lambdify([q[0], q[1], q[2], q[3], q[4], q[5], qdot[0], qdot[1], qdot[2], qdot[3], qdot[4], qdot[5], t], EL_solns[0][qddot[0]])
y_box_ddot_func = lambdify([q[0], q[1], q[2], q[3], q[4], q[5], qdot[0], qdot[1], qdot[2], qdot[3], qdot[4], qdot[5], t], EL_solns[0][qddot[1]])
theta_box_ddot_func = lambdify([q[0], q[1], q[2], q[3], q[4], q[5], qdot[0], qdot[1], qdot[2], qdot[3], qdot[4], qdot[5], t], EL_solns[0][qddot[2]])
x_jack_ddot_func = lambdify([q[0], q[1], q[2], q[3], q[4], q[5], qdot[0], qdot[1], qdot[2], qdot[3], qdot[4], qdot[5], t], EL_solns[0][qddot[3]])
y_jack_ddot_func = lambdify([q[0], q[1], q[2], q[3], q[4], q[5], qdot[0], qdot[1], qdot[2], qdot[3], qdot[4], qdot[5], t], EL_solns[0][qddot[4]])
theta_jack_ddot_func = lambdify([q[0], q[1], q[2], q[3], q[4], q[5], qdot[0], qdot[1], qdot[2], qdot[3], qdot[4], qdot[5], t], EL_solns[0][qddot[5]])

def dynamics(s, t):
    sdot = np.array([
        s[6],
        s[7],
        s[8],
        s[9],
        s[10],
        s[11],
        x_box_ddot_func(*s, t),
        y_box_ddot_func(*s, t),
        theta_box_ddot_func(*s, t),
        x_jack_ddot_func(*s, t),
        y_jack_ddot_func(*s, t),
        theta_jack_ddot_func(*s, t)
    ])
    return sdot
```

```python
# Define acceleration matrix:
qddot_Matrix = Matrix([qdot[0], EL_solns[0][qddot[0]],
                       qdot[1], EL_solns[0][qddot[1]],
                       qdot[2], EL_solns[0][qddot[2]],
                       qdot[3], EL_solns[0][qddot[3]],
                       qdot[4], EL_solns[0][qddot[4]],
                       qdot[5], EL_solns[0][qddot[5]]])

# Define dummy symbols:
x_b_l, y_b_l, theta_b_l, x_j_l, y_j_l, theta_j_l, x_b_ldot, y_b_ldot, theta_b_ldot, x_j_ldot, y_j_ldot, theta_j_ldot = symbols('x_box_l, y_box_l, theta_box_l, x_jack_l, y_jack_l, theta_jack_l, x_box_ldot, y_box_ldot, theta_box_ldot, x_jack_ldot, y_jack_ldot, theta_jack_ldot')

dummy_dict = {q[0]:x_b_l, q[1]:y_b_l, q[2]:theta_b_l,
              q[3]:x_j_l, q[4]:y_j_l, q[5]:theta_j_l,
              qdot[0]:x_b_ldot, qdot[1]:y_b_ldot, qdot[2]:theta_b_ldot,
              qdot[3]:x_j_ldot, qdot[4]:y_j_ldot, qdot[5]:theta_j_ldot}
qddot_d = qddot_Matrix.subs(dummy_dict)
qddot_lambdify = lambdify([x_b_l, x_b_ldot ,y_b_l, y_b_ldot, theta_b_l, theta_b_ldot,
                           x_j_l, x_j_ldot ,y_j_l, y_j_ldot, theta_j_l, theta_j_ldot, t], qddot_d)
```

```python
r_jack_hat = Matrix([x_jack, y_jack, theta_jack, 1])

# Define impact constraint for wall 1:
phi_b1_j1 = (g_b1_j1[3]).subs(dummy_dict)
phi_b1_j2 = (g_b1_j2[3]).subs(dummy_dict)
phi_b1_j3 = (g_b1_j3[3]).subs(dummy_dict)
phi_b1_j4 = (g_b1_j4[3]).subs(dummy_dict)

# Define impact constraint for wall 2:
phi_b2_j1 = (g_b2_j1[7]).subs(dummy_dict)
phi_b2_j2 = (g_b2_j2[7]).subs(dummy_dict)
phi_b2_j3 = (g_b2_j3[7]).subs(dummy_dict)
phi_b2_j4 = (g_b2_j4[7]).subs(dummy_dict)

# Define impact constraint for wall 3:
phi_b3_j1 = (g_b3_j1[3]).subs(dummy_dict)
phi_b3_j2 = (g_b3_j2[3]).subs(dummy_dict)
phi_b3_j3 = (g_b3_j3[3]).subs(dummy_dict)
phi_b3_j4 = (g_b3_j4[3]).subs(dummy_dict)

# Define impact constraint for wall 4:
phi_b4_j1 = (g_b4_j1[7]).subs(dummy_dict)
phi_b4_j2 = (g_b4_j2[7]).subs(dummy_dict)
phi_b4_j3 = (g_b4_j3[7]).subs(dummy_dict)
phi_b4_j4 = (g_b4_j4[7]).subs(dummy_dict)

# Define impact constraint:
phi_dum = simplify(Matrix([[phi_b1_j1], [phi_b1_j2], [phi_b1_j3], [phi_b1_j4],  # all jack frames in the box1 frame
                           [phi_b2_j1], [phi_b2_j2], [phi_b2_j3], [phi_b2_j4],  # all jack frames in the box2 frame
                           [phi_b3_j1], [phi_b3_j2], [phi_b3_j3], [phi_b3_j4],  # all jack frames in the box3 frame
                           [phi_b4_j1], [phi_b4_j2], [phi_b4_j3], [phi_b4_j4]])) # all jack frames in the box4 frame

# Compute the Hamiltonian:
H = simplify((dL_dqdot.T * qdot)[0] - L)

# Compute expressions:
H_dum = H.subs(dummy_dict)
dL_dqdot_dum = dL_dqdot.subs(dummy_dict)
dPhidq_dum = phi_dum.jacobian([x_b_l, y_b_l, theta_b_l, x_j_l, y_j_l, theta_j_l])

# Define dummy symbols for tau+:
lamb = symbols(r'lambda')
x_b_dot_Plus, y_b_dot_Plus, theta_b_dot_Plus, x_j_dot_Plus, y_j_dot_Plus, theta_j_dot_Plus = symbols(r'x_box_dot_+, y_box_dot_+, theta_box_dot_+, x_jack_dot_+, y_jack_dot_+, theta_jack_dot_+')

impact_dict = {x_b_ldot:x_b_dot_Plus, y_b_ldot:y_b_dot_Plus, theta_b_ldot:theta_b_dot_Plus,
               x_j_ldot:x_j_dot_Plus, y_j_ldot:y_j_dot_Plus, theta_j_ldot:theta_j_dot_Plus}

# Evaluate expressions at tau+:
dL_dqdot_dumPlus = simplify(dL_dqdot_dum.subs(impact_dict))
dPhidq_dumPlus = simplify(dPhidq_dum.subs(impact_dict))
H_dumPlus = simplify(H_dum.subs(impact_dict))

impact_eqns_list = []

# Define equations
lhs = Matrix([dL_dqdot_dumPlus[0] - dL_dqdot_dum[0],
              dL_dqdot_dumPlus[1] - dL_dqdot_dum[1],
              dL_dqdot_dumPlus[2] - dL_dqdot_dum[2],
              dL_dqdot_dumPlus[3] - dL_dqdot_dum[3],
              dL_dqdot_dumPlus[4] - dL_dqdot_dum[4],
              dL_dqdot_dumPlus[5] - dL_dqdot_dum[5],
              H_dumPlus - H_dum])

for i in range(phi_dum.shape[0]):
    rhs = Matrix([lamb*dPhidq_dum[i,0],
                  lamb*dPhidq_dum[i,1],
                  lamb*dPhidq_dum[i,2],
                  lamb*dPhidq_dum[i,3],
                  lamb*dPhidq_dum[i,4],
                  lamb*dPhidq_dum[i,5],
                  0])
    impact_eqns_list.append(simplify(Eq(lhs, rhs)))
```

```python
dum_list = [x_b_dot_Plus, y_b_dot_Plus, theta_b_dot_Plus,
            x_j_dot_Plus, y_j_dot_Plus, theta_j_dot_Plus]

def impact_update(s, impact_eqns, dum_list):
    """ This function updated the system after impact.
    It returns the uptadet s array after impact.
    """
    subs_dict = {x_b_l:s[0], y_b_l:s[1], theta_b_l:s[2],
                 x_j_l:s[3], y_j_l:s[4], theta_j_l:s[5],
                 x_b_ldot:s[6], y_b_ldot:s[7], theta_b_ldot:s[8],
                 x_j_ldot:s[9], y_j_ldot:s[10], theta_j_ldot:s[11]}
    new_impact_eqns = impact_eqns.subs(subs_dict)
    impact_solns = solve(new_impact_eqns, [x_b_dot_Plus, y_b_dot_Plus, theta_b_dot_Plus,
                                           x_j_dot_Plus, y_j_dot_Plus, theta_j_dot_Plus,
                                           lamb], dict=True)
    if len(impact_solns) == 1:
        print("Damn, only one solution")

    else:
        for sol in impact_solns:
            lamb_sol = sol[lamb]
            if abs(lamb_sol) < 1e-06:
                pass
            else:
                return np.array([
                    s[0], #q will be the same after impact
                    s[1],
                    s[2],
                    s[3],
                    s[4],
                    s[5],
                    float(sym.N(sol[dum_list[0]])), #q_dot will change after impact
                    float(sym.N(sol[dum_list[1]])),
                    float(sym.N(sol[dum_list[2]])),
                    float(sym.N(sol[dum_list[3]])),
                    float(sym.N(sol[dum_list[4]])),
                    float(sym.N(sol[dum_list[5]])),
                ])
```

```python
phi_func = lambdify([x_b_l, y_b_l, theta_b_l,
                     x_j_l, y_j_l, theta_j_l,
                     x_b_ldot, y_b_ldot, theta_b_ldot,
                     x_j_ldot, y_j_ldot, theta_j_ldot],
                    phi_dum)

def impact_condition(s, phi_func, threshold = 1e-1):
    """ This function checks the systems for impact.
    In the case of an impact (abs(phi_val) < threshold),
    the function returns True and the row number of the
    phi matrix in which the impact accured.
    """
    phi_val = phi_func(*s)
    for i in range(phi_val.shape[0]):
        if (phi_val[i] > -threshold) and (phi_val[i] < threshold):
            return (True, i)
    return (False, None)

# print('term impact condition function:', impact_condition(s_test, phi_func, 0.01))
```

```python
# Simulate the motion:
tspan = [0, 10]
dt = 0.01
s0 = np.array([0, 0, 0, 0, 0, 0, 0, 0, -1.6, 0, 0, 0])

N = int((max(tspan) - min(tspan))/dt)
tvec = np.linspace(min(tspan), max(tspan), N)
traj = simulate_impact(dynamics, s0, tspan, dt, integrate)

plt.figure()
plt.plot(tvec, traj[0], label='x_box')
plt.plot(tvec, traj[1], label='y_box')
plt.plot(tvec, traj[2], label='theta_box')
plt.title('Box Motion Simulation')
plt.xlabel('t')
plt.ylabel('state')
plt.legend(loc="best")
plt.show()

plt.figure()
plt.plot(tvec, traj[3], label='x_jack')
plt.plot(tvec, traj[4], label='y_jack')
plt.plot(tvec, traj[5], label='theta_jack')
plt.title('Jack Motion Simulation')
plt.xlabel('t')
plt.ylabel('state')
plt.legend(loc="best")
plt.show()

plt.figure()
plt.plot(tvec, traj[6], label='x_box_dot')
plt.plot(tvec, traj[7], label='y_box_dot')
plt.plot(tvec, traj[8], label='theta_box_dot')
plt.title('Box Velocity Simulation')
plt.xlabel('t')
plt.ylabel('state')
plt.legend(loc="best")
plt.show()

plt.figure()
plt.plot(tvec, traj[9], label='x_jack_dot')
plt.plot(tvec, traj[10], label='y_jack_dot')
plt.plot(tvec, traj[11], label='theta_jack_dot')
plt.title('Jack Velocity Simulation')
plt.xlabel('t')
plt.ylabel('state')
plt.legend(loc="best")
plt.show()
```

```python
def animate_jack_in_a_box(config_array,l=1,w=0.2,T=10):
    """
    Function to generate web-based animation of the system

    Parameters:
    ================================================
    config_array:
        trajectory of theta1 and theta2, should be a NumPy array with
        shape of (2,N)
    L1:
        length of the first pendulum
    L2:
        length of the second pendulum
    T:
        length/seconds of animation duration

    Returns: None
    """
    ################################
    # Imports required for animation. (leave this part)
    from plotly.offline import init_notebook_mode, iplot
    from IPython.display import display, HTML
    import plotly.graph_objects as go

    #######################
    # Browser configuration. (leave this part)
    def configure_plotly_browser_state():
        import IPython
        display(IPython.core.display.HTML('''

            '''))
    configure_plotly_browser_state()
    init_notebook_mode(connected=False)

    ###############################################
    # Getting data from pendulum angle trajectories.
    N = len(config_array[0])

    x_box_array = config_array[0]
    y_box_array = config_array[1]
    theta_box_array = config_array[2]
    x_jack_array = config_array[3]
    y_jack_array = config_array[4]
    theta_jack_array = config_array[5]

    b1_x_array = np.zeros(N, dtype=np.float32)
    b1_y_array = np.zeros(N, dtype=np.float32)
    b2_x_array = np.zeros(N, dtype=np.float32)
    b2_y_array = np.zeros(N, dtype=np.float32)
    b3_x_array = np.zeros(N, dtype=np.float32)
    b3_y_array = np.zeros(N, dtype=np.float32)
    b4_x_array = np.zeros(N, dtype=np.float32)
    b4_y_array = np.zeros(N, dtype=np.float32)

    j_x_array = np.zeros(N, dtype=np.float32)
    j_y_array = np.zeros(N, dtype=np.float32)
    j1_x_array = np.zeros(N, dtype=np.float32)
    j1_y_array = np.zeros(N, dtype=np.float32)
    j2_x_array = np.zeros(N, dtype=np.float32)
    j2_y_array = np.zeros(N, dtype=np.float32)
    j3_x_array = np.zeros(N, dtype=np.float32)
    j3_y_array = np.zeros(N, dtype=np.float32)
    j4_x_array = np.zeros(N, dtype=np.float32)
    j4_y_array = np.zeros(N, dtype=np.float32)

    for t in range(N):
        g_w_b = get_se3_np(x_box_array[t], y_box_array[t], theta_box_array[t])
        g_w_j = get_se3_np(x_jack_array[t], y_jack_array[t], theta_jack_array[t])

        b1 = g_w_b.dot(np.array([l_box, l_box, 0, 1]))
        b1_x_array[t] = b1[0]
        b1_y_array[t] = b1[1]
        b2 = g_w_b.dot(np.array([l_box, -l_box, 0, 1]))
        b2_x_array[t] = b2[0]
        b2_y_array[t] = b2[1]
        b3 = g_w_b.dot(np.array([-l_box, -l_box, 0, 1]))
        b3_x_array[t] = b3[0]
        b3_y_array[t] = b3[1]
        b4 = g_w_b.dot(np.array([-l_box, l_box, 0, 1]))
        b4_x_array[t] = b4[0]
        b4_y_array[t] = b4[1]

        j = g_w_j.dot(np.array([0, 0, 0, 1]))
        j_x_array[t] = j[0]
        j_y_array[t] = j[1]
        j1 = g_w_j.dot(np.array([l_jack, 0, 0, 1]))
        j1_x_array[t] = j1[0]
        j1_y_array[t] = j1[1]
        j2 = g_w_j.dot(np.array([0, -l_jack, 0, 1]))
        j2_x_array[t] = j2[0]
        j2_y_array[t] = j2[1]
        j3 = g_w_j.dot(np.array([-l_jack, 0, 0, 1]))
        j3_x_array[t] = j3[0]
        j3_y_array[t] = j3[1]
        j4 = g_w_j.dot(np.array([0, l_jack, 0, 1]))
        j4_x_array[t] = j4[0]
        j4_y_array[t] = j4[1]

    ####################################
    # Axis limits.
    xm = -13
    xM = 13
    ym = -13
    yM = 13

    ###########################
    # Defining data dictionary.
    data=[dict(name = 'Box'),
          dict(name = 'Jack'),
          dict(name = 'Mass1_Jack'),
        ]

    ################################
    # Preparing simulation layout.
    layout=dict(xaxis=dict(range=[xm, xM], autorange=False, zeroline=False,dtick=1),
                yaxis=dict(range=[ym, yM], autorange=False, zeroline=False,scaleanchor = "x",dtick=1),
                title='Jack in a Box Simulation',
                hovermode='closest',
                updatemenus= [{'type': 'buttons',
                               'buttons': [{'label': 'Play','method': 'animate',
                                            'args': [None, {'frame': {'duration': T, 'redraw': False}}]},
                                           {'args': [[None], {'frame': {'duration': T, 'redraw': False}, 'mode': 'immediate',
                                            'transition': {'duration': 0}}],'label': 'Pause','method': 'animate'}
                                          ]
                              }]
               )

    ########################################
    # Defining the frames of the simulation.
    frames=[dict(data=[
            dict(x=[b1_x_array[k],b2_x_array[k],b3_x_array[k],b4_x_array[k],b1_x_array[k]],
                 y=[b1_y_array[k],b2_y_array[k],b3_y_array[k],b4_y_array[k],b1_y_array[k]],
                 mode='lines',
                 line=dict(color='blue', width=3)
                 ),
            dict(x=[j1_x_array[k],j3_x_array[k],j_x_array[k],j2_x_array[k],j4_x_array[k]],
                 y=[j1_y_array[k],j3_y_array[k],j_y_array[k],j2_y_array[k],j4_y_array[k]],
                 mode='lines',
                 line=dict(color='green', width=3)
                 ),
            go.Scatter(
                x=[j1_x_array[k],j2_x_array[k],j3_x_array[k],j4_x_array[k]],
                y=[j1_y_array[k],j2_y_array[k],j3_y_array[k],j4_y_array[k]],
                mode="markers",
                marker=dict(color='darkgreen', size=6)),
                 ]) for k in range(N)]

    #######################################
    # Putting it all together and plotting.
    figure1=dict(data=data, layout=layout, frames=frames)
    iplot(figure1)

##############
# The animation:
animate_jack_in_a_box(traj)
```

```python

```


# Tutorial 2. Flow Visualization

### Description: Flow visualization is important when assessing a flow field and is helpful in better understanding the flow characteristics. Streamlines, pathlines and streaklines are introduced in this exercise. The first part deals with defining, setting up and visualizing a vector field. Here it represents a steady flow, more precisely a pure rotational field. In-built streamline plot has also underlying numerical approximation due to which the streamline is instead of a circle (analytical solution) a spiral. Isocontours are used to show the velocity magnitude. Numerical integration methods are implemented to show the differences for pathline calculation. This follows the Lagrangian view, i.e. particle tracking. While the flow domain is defined by a vector field independent of time, i.e. steady, the particles in the flow have a changing velocity as this depends by definition on the position. Some exercises are proposed.

#### Students are advised to complete the exercises.

Project: Structural Wind Engineering WS19-20
        Chair of Structural Analysis @ TUM - R. Wüchner, M. Péntek

Author: kodakkal.anoop@tum.de mate.pentek@tum.de

Created on:  08.11.2015

Last update: 27.09.2019

##### Contents:

1. Calculation of divergence and curl
2. Vector field visualization
3. Streamlines
4. Pathlines with different numerical integration schemes

```python
# import
import time
import matplotlib.pyplot as plt
import numpy as np
import sympy
from matplotlib import pyplot as plt
```

#### Symbolic vector calculus

using the symbolic toolbox (sympy) to define and display the velocity field u.

```python
# declare symbolic variables
x = sympy.Symbol('x')
y = sympy.Symbol('y')
z = sympy.Symbol('z')
t = sympy.Symbol('t')
```

Let us define each component of the velocity field.

```python
##
## modify here for a different velocity field
## use here the format: symbolic_vel_component = f(t,x,y) = a + b*x + c*y + d*t
##
symbolic_velocity_x = 0 + 0 * x + (-2) * y + 0 * t
symbolic_velocity_y = 0 + 2 * x + 0 * y + 0 * t
symbolic_velocity_z = 0 + 0 * x + 0 * y + 0 * t
print('u = [', symbolic_velocity_x,' ', symbolic_velocity_y,' ', symbolic_velocity_z,']')

# creating function from the symbolic expression
# which are later used for numerical evaluations
velocity_x = sympy.lambdify((t,x,y), symbolic_velocity_x)
velocity_y = sympy.lambdify((t,x,y), symbolic_velocity_y)
velocity_z = sympy.lambdify((t,x,y), symbolic_velocity_z)
```

    u = [ -2*y   2*x   0 ]

###### for exercise 1
Try a different velocity field.

```python
# symbolic_velocity_x = 2
# symbolic_velocity_y = 2
# symbolic_velocity_z = 0
# print('u = [', symbolic_velocity_x,' ', symbolic_velocity_y,' ', symbolic_velocity_z,']')
```

### Calculation of  divergence and curl

#### Two key operators in vector calculus are divergence and curl. Recall from lecture the definitions of these two quantities. What do these two quantities signify?

Divergence signifies how a vector field changes its magnitude in the neighborhood of a point and curl is an indicator of how its direction changes.

Divergence of a velocity field $u(x,y,z)$ is

$$ div (u) = \nabla \cdot u =  \frac{\partial u_x}{\partial x} + \frac{\partial u_y}{\partial y} + \frac{\partial u_z}{\partial z}  $$

```python
symbolic_div = sympy.diff(symbolic_velocity_x, x) + sympy.diff(symbolic_velocity_y, y) + sympy.diff(symbolic_velocity_z, z);
print('div(u)= ', symbolic_div)
```

    div(u)=  0

Curl of a velocity field $u(x,y,z)$ is

$$ curl (u) = \nabla \times u =
\begin{vmatrix}
\hat{i} & \hat{j} & \hat{k}  \\
\frac{\partial }{\partial x } & \frac{\partial}{\partial y} & \frac{\partial }{\partial z} \\
u_x & u_y & u_z
\end{vmatrix} $$

```python
symbolic_curl_x = sympy.diff(symbolic_velocity_z, y) - sympy.diff(symbolic_velocity_y, z)
symbolic_curl_y = sympy.diff(symbolic_velocity_x, z) - sympy.diff(symbolic_velocity_z, x)
symbolic_curl_z = sympy.diff(symbolic_velocity_y, x) - sympy.diff(symbolic_velocity_x, y)
print('curl(u) = [', symbolic_curl_x,' ', symbolic_curl_y,' ', symbolic_curl_z,']')
```

    curl(u) = [ 0   0   4 ]

### Vector (velocity) field  visualization

We need to generate a grid to visualize.

```python
# give grid size parameters
grid_spacing_start = -2
grid_spacing_end = 2
grid_spacing_size = 0.4

# generate grid size vectors 1D
grid_spacing_x = np.arange(grid_spacing_start, grid_spacing_end + grid_spacing_size, grid_spacing_size)
grid_spacing_y = np.arange(grid_spacing_start, grid_spacing_end + grid_spacing_size, grid_spacing_size)

meshgrid_x, meshgrid_y = np.meshgrid(grid_spacing_x, grid_spacing_y)
```

Refer to [meshgrid](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.meshgrid.html) for more details on mesh grid function on numpy

##### Let us plot the vector field to visualize

```python
plt.figure(num=1, figsize=(8, 8))
plt.title('Vector field for Vx = ' + str(symbolic_velocity_x) + ' and Vy = ' + str(symbolic_velocity_y))
plt.xlabel('x')
plt.ylabel('y')
plt.xlim(grid_spacing_start-1, grid_spacing_end+1)
plt.ylim(grid_spacing_start-1, grid_spacing_end+1)

# 0.0 stands for t=0 -> considering one certain time step
t = 0.0
plt.quiver(meshgrid_x, meshgrid_y,
           velocity_x(t, meshgrid_x, meshgrid_y), velocity_y(t, meshgrid_x, meshgrid_y),
           angles='xy',scale_units='xy',scale=10)

#plt.axis('equal')
plt.grid()
```

### Exercise 1: Visualizing different fields

Try to visualize following vector fields and compute their divergence and curl.
Is the relationship between these quantities and the vector fields?

1. constant u = [ 2, 2, 0 ]
2. constant u = [ 2, -2, 0 ]
3. u = [ x, y, 0 ]
4. u = [ 2y, -2x, 0 ]
5. u = [ x-y, x+y, 0 ]

### Streamline visualization

using the inbuilt function of [streamline](https://matplotlib.org/api/_as_gen/matplotlib.pyplot.streamplot.html) to plot the streamlines

```python
plt.figure(num=2, figsize=(8, 8))

# 0.0 stands for t=0 -> considering one certain time step
t = 0.0
plt.streamplot(meshgrid_x, meshgrid_y,
               velocity_x(t, meshgrid_x, meshgrid_y), velocity_y(t, meshgrid_x, meshgrid_y),
               density= 0.5)

plt.title('Streamlines for vector field')
plt.xlabel('x')
plt.ylabel('y')
plt.xlim([grid_spacing_start-1, grid_spacing_end+1])
plt.ylim([grid_spacing_start-1, grid_spacing_end+1])
plt.grid(True)
```

### Contour plot for velocity magnitude

Velocity magnitude $ \mid U\mid= \sqrt{U_x^2 + U_y^2}$

```python
# 0.0 stands for t=0 -> considering one certain time step
t = 0.0
meshgrid_velocity_magnitude = np.sqrt((np.square(velocity_x(t, meshgrid_x, meshgrid_y)) +
                                       np.square(velocity_y(t, meshgrid_x, meshgrid_y))))

# plot separately a contour line for the magnitude of velocity for an initial point
# x0 = 1, y0 = 1 -> looking for magnitude Ux(x0=1)= -2 and Uy(y0=1) = 2
# magnitude is 2*sqrt(2) = 2.8284
value_looked_for = np.sqrt((-2*1)**2 + (2*1)**2)

# plot first plot line of interest
# after plot contours lines dashed
plt.figure(num=3, figsize=(8, 8))
# plot a selected countour line
CS = plt.contour(meshgrid_x, meshgrid_y, meshgrid_velocity_magnitude, levels = [value_looked_for], linewidths= 2.5)
plt.clabel(CS, inline=1, fontsize=12)
# plot multiple countour lines
plt.contour(meshgrid_x, meshgrid_y, meshgrid_velocity_magnitude, linestyles='dashed',linewidths=1.5)

plt.title('Vector magnitude contour plot')
plt.xlabel('x')
plt.ylabel('y')
plt.xlim([grid_spacing_start-1, grid_spacing_end+1])
plt.ylim([grid_spacing_start-1, grid_spacing_end+1])
plt.grid(True)
```

## Check Point 1: Discussion

#### Discuss in groups the insights into vector (velocity) field visualization and other flow related quantities. Can you make some predictions of the flow fields from the the divergence and curl ?

### Pathline

Pathlines are actual paths traveled by each particle.
$$ x = x_{start} + \int_{t_{start}} ^{t}{u dt }$$

###### Here, the particle is introduced into flow at time $ t = 0$ at position $x(t=0) = 1 $ and $ y(t=0) = 1$

We follow this particle in time and evaluate the pathline. The determination of the pathline for this case is done numerically. This implies an approximation of position (distance) by numerically integrating the given (known) velocity.
$$ U_x = \frac {dx} {dt} \rightarrow dx = U_x dt$$

This can be solved using various (time) integration schemes.

##### 1. A forward Euler integration

check out the (forward) [Euler integration](https://en.wikipedia.org/wiki/Euler_method) for details

#### Creating the time instances as an array

The start time, end time and the number of time steps are specified here for generating the time series.

```python
# start time
start_time = 0.0
# end time
end_time = np.pi
#delta time
delta_time = 0.1
# time step
n_steps = round((end_time - start_time)/delta_time)
# time series
time_series = np.linspace(start_time, end_time, n_steps)
```

end_time is chosen as $\pi$ so that the particle will come back to its initial position. Recall the fact the pathlines and streamlines are same for a steady flow.

The pathline is evaluated using the forward Euler scheme. The pathline length, x and y coordinates are stored for each time step.

```python
# defining the velocity as a function
# one can define with the lambda placeholder for variables
# velocities defined generically as functions depending on x, y, t
# so positions x and y (maybe z) and time t
# sympy.lambdify creates a function from the symbolic definition of
# symbolic_velocity_component with the variables x, y, t which can be
# evaluated numerically

# initialize with zero values
coord_x = np.zeros(len(time_series))
coord_y = np.zeros(len(time_series))
# initial position
coord_x[0] = 1
coord_y[0] = 1

# length set to zero
pathline_length = 0

# calculation loop for forward Euler
# x(i) = x(i-1) + Vx(i-1) * delta_t
# y(i) = y(i-1) + Vy(i-1) * delta_t
for i in range(0, len(time_series)-1):
    # calculate new position
    coord_x[i+1] = coord_x[i] + velocity_x(time_series[i] + delta_time,
                                           coord_x[i],
                                           coord_y[i]) * delta_time
    coord_y[i+1] = coord_y[i] + velocity_y(time_series[i] + delta_time,
                                           coord_x[i],
                                           coord_y[i]) * delta_time

    # increment pathline length
    pathline_length += np.sqrt((coord_x[i+1] - coord_x[i])**2 + (coord_y[i+1] - coord_y[i])**2)

# store results under new name to be more specific
euler_coord_x = coord_x
euler_coord_y = coord_y
euler_pathline_length = pathline_length
```

##### 2. 4th order Runga-Kutta (RK-4) integration

check out the [ Runga-Kutta (RK-4)](https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods#Examples) for details

```python
# initialize with zero values
coord_x = np.zeros(len(time_series))
coord_y = np.zeros(len(time_series))
# initial position
coord_x[0] = 1
coord_y[0] = 1

# length set to zero
pathline_length = 0

# calculation loop for RK4
for i in range(0, len(time_series)-1):
    # calculate new position
    k_1 = velocity_x(time_series[i],
                    coord_x[i],
                    coord_y[i])
    k_2 = velocity_x(time_series[i] + 0.5 * delta_time,
                    coord_x[i] + 0.5 * delta_time * k_1,
                    coord_y[i] + 0.5 * delta_time * k_1)
    k_3 = velocity_x(time_series[i] + 0.5 * delta_time,
                    coord_x[i] + 0.5 * delta_time * k_2,
                    coord_y[i] + 0.5 * delta_time * k_2)
    k_4 = velocity_x(time_series[i] + delta_time,
                    coord_x[i] + k_3 * delta_time,
                    coord_y[i] + k_3 * delta_time)

    coord_x[i+1] = coord_x[i] + (1.0/6.0) * (k_1 + 2 * k_2 + 2 * k_3 + k_4) * delta_time

    k_1 = velocity_y(time_series[i],
                     coord_x[i],
                     coord_y[i])
    k_2 = velocity_y(time_series[i] + 0.5 * delta_time,
                     coord_x[i] + 0.5 * delta_time * k_1,
                     coord_y[i] + 0.5 * delta_time * k_1)
    k_3 = velocity_y(time_series[i] + 0.5 * delta_time,
                     coord_x[i] + 0.5 * delta_time * k_2,
                     coord_y[i] + 0.5 * delta_time * k_2)
    k_4 = velocity_y(time_series[i] + delta_time,
                     coord_x[i] + k_3 * delta_time,
                     coord_y[i] + k_3 * delta_time)

    coord_y[i+1] = coord_y[i] + (1.0/6.0) * (k_1 + 2 * k_2 + 2 * k_3 + k_4) * delta_time

    # increment pathline length
    pathline_length += np.sqrt((coord_x[i+1] - coord_x[i])**2 + (coord_y[i+1] - coord_y[i])**2)

# store results under new name to be more specific
rk4_coord_x = coord_x
rk4_coord_y = coord_y
rk4_pathline_length = pathline_length
```

##### 3. Adams-Bashforth

check out the [ Adams-Bashforth](https://en.wikipedia.org/wiki/Linear_multistep_method#Two-step_Adams%E2%80%93Bashforth) for details

```python
# initialize with zero values
coord_x = np.zeros(len(time_series))
coord_y = np.zeros(len(time_series))
# initial position
coord_x[0] = 1
coord_y[0] = 1

# length set to zero
pathline_length = 0

# predict first step with Euler as Adams-Bashforth needs more information to
# start because the method depends on 2 previous values (positions) of x
# and y
# note that it introduces an inital error, one could substitute it with RK4
# to predict instead and reduce the error
coord_x[1] = coord_x[0] + velocity_x(time_series[0] + delta_time,
                                    coord_x[0],
                                    coord_y[0]) * delta_time
coord_y[1] = coord_y[0] + velocity_y(time_series[0] + delta_time,
                                    coord_x[0],
                                    coord_y[0]) * delta_time

pathline_length += (np.sqrt((coord_x[1] - coord_x[0])**2 + (coord_y[1] - coord_y[0])**2))

# calculation loop for Adams-Bashforth
for i in range(0 + 1, len(time_series)-1):
    # calculate new position
    coord_x[i+1] = coord_x[i] + 3.0/2.0 * velocity_x(time_series[i] + delta_time,
                                                    coord_x[i],
                                                    coord_y[i]) * delta_time \
                              - 1.0/2.0 * velocity_x(time_series[i-1] + delta_time,
                                                    coord_x[i-1],
                                                    coord_y[i-1]) * delta_time
    coord_y[i+1] = coord_y[i] + 3.0/2.0 * velocity_y(time_series[i] + delta_time,
                                                    coord_x[i],
                                                    coord_y[i]) * delta_time \
                              - 1.0/2.0 * velocity_y(time_series[i-1] + delta_time,
                                                    coord_x[i-1],
                                                    coord_y[i-1]) * delta_time

    # increment pathline length
    pathline_length += np.sqrt((coord_x[i+1] - coord_x[i])**2 + (coord_y[i+1] - coord_y[i])**2)

# store results under new name to be more specific
ab_coord_x = coord_x
ab_coord_y = coord_y
ab_pathline_length = pathline_length
```

##### 4. Analytic solution

The (exact) analytic value for particles (for this specific case) is a circle as the corresponding pathline/streamline are the same (for a steady flow field).

```python
# radius
r = np.sqrt((1)**2 + (1)**2)
# parametric time steps
# taking a center angle theta
theta = np.linspace(0, 2*np.pi, 1000)
# center of circle a,b
a = 0
b = 0
# point on the circle
coord_x = r * np.cos(theta) + a
coord_y = r * np.sin(theta) + b

# store results under new name to be more specific
exact_coord_x = coord_x
exact_coord_y = coord_y
exact_pathline_length = 2 * np.pi * r
```

##### Let us plot the pathline

```python
plt.figure(num=4, figsize=(8, 8))
plt.title('Pathline for point x(t(0)) = y(t(0)) = 1, delta_time = ' + str(round(delta_time, 3)))
# Euler
plt.plot(euler_coord_x, euler_coord_y,'-r', label='Euler')
# RK4
plt.plot(rk4_coord_x, rk4_coord_y,'-.b', label='RK4')
# AB
plt.plot(ab_coord_x, ab_coord_y,'--k', label='AB')
# exact analytic
plt.plot(exact_coord_x, exact_coord_y,'-b', label='Exact')

plt.xlabel('x')
plt.ylabel('y')
plt.xlim([grid_spacing_start-1, grid_spacing_end+1])
plt.ylim([grid_spacing_start-1, grid_spacing_end+1])

plt.legend(loc='upper right', shadow=True)
plt.grid(True)
```

Let's zoom to point [1,1]

```python
plt.figure(num=5, figsize=(15, 6))

plt.subplot(1,2,1)
plt.title('Pathline for point x(t(0)) = y(t(0)) = 1, delta_time = ' + str(round(delta_time, 3)))
# Euler
plt.plot(euler_coord_x, euler_coord_y,'-r', label='Euler')
# RK4
plt.plot(rk4_coord_x, rk4_coord_y,'-.b', label='RK4')
# AB
plt.plot(ab_coord_x, ab_coord_y,'--k', label='AB')
# exact analytic
plt.plot(exact_coord_x, exact_coord_y,'-b', label='Exact')

plt.xlabel('x')
plt.ylabel('y')
plt.xlim([0, 2])
plt.ylim([0, 2])
plt.legend(loc='upper right', shadow=True)
plt.grid(True)

plt.subplot(1,2,2)
plt.title('Pathline for point x(t(0)) = y(t(0)) = 1, delta_time = ' + str(round(delta_time, 3)))
# Euler
plt.plot(euler_coord_x, euler_coord_y,'-r', label='Euler')
# RK4
plt.plot(rk4_coord_x, rk4_coord_y,'-.b', label='RK4')
# AB
plt.plot(ab_coord_x, ab_coord_y,'--k', label='AB')
# exact analytic
plt.plot(exact_coord_x, exact_coord_y,'-b', label='Exact')

plt.xlabel('x')
plt.ylabel('y')
plt.xlim([0.95, 1.05])
plt.ylim([0.95, 1.05])
plt.legend(loc='upper right', shadow=True)
plt.grid(True)

plt.show()
```

Let's compare the pathline length for different integration schemes

```python
print('Pathline length comparison for dt = ', str(round(delta_time,3)))
print('Euler: ', str(round(euler_pathline_length, 3)))
print('RK4: ', str(round(rk4_pathline_length, 3)))
print('AB: ', str(round(ab_pathline_length, 3)))
print('Exact: ', str(round(exact_pathline_length, 3)))
```

    Pathline length comparison for dt =  0.1
    Euler:  11.439
    RK4:  11.523
    AB:  8.842
    Exact:  8.886

## Check Point 2: Discussion

#### What do you observe with different integration schemes? Discuss in groups

### Exercise 2: Effect of time step

Reduce the time step to delta_time to 0.01. Observe the differences and comment. Modify to 0.001

### Exercise 3: Modify the velocity field

Redefine the velocity field as u = [1 x 0] and observe differences.

## Check Point 3: Discussion

#### Discuss in groups results and impressions of exercise 2-3.

### Assignment: Modify the velocity field to time varying one & Streaklines

1. Redefine the velocity field as u = [-2yt 2xt 0] and observe differences.You should have an updating vector plot with time, also the streamlines will be time-dependent. For each time instance there will be new streamlines for the whole field.
2. For the initial velocity field u = [-2y 2x 0] implement on your own the functionality for calculating streaklines.This is the family of pathlines defined before (connecting all the particles i which are released into the field at position x(t=i)=1 and y(t=i)=1) for a succsession of initial times ti0.


# Model analysis project

For the model analysis project, we have chosen to focus on the utility maximization problem from basic consumer theory. It is stated that a rational consumer will maximize their utility based on their disposable income for a given commodity. This means that the consumer have a chosen amount of money to spend on a specific group of interchangable commodities, and therefore, must choose a quantity of each commodity within their budget. This is done based on prices of each commodity and individual preferences of the consumer.

## Describing and solving utility-maximization subject to a budgetconstraint

Utility-maximization is based on a consumers individual preferences, illustrated by an equation explaining utility given by interchangable goods. These preferences are subject to a budget-constraint, illustrating the chosen amount of money, the consumer has chosen to spend on the group of goods in question. The two equations together is an example of a consumerproblem.

We have chosen to explain the optimization of utility based on the simple equations underneath.

```python
# we start out by importing the necessary packages to both manipulate and illustrate the equations and data we have chosen.
import numpy as np
import scipy as sp
import math
from scipy import linalg
from scipy import optimize
from scipy import interpolate
import sympy as sm
import matplotlib.pyplot as plt
```

```python
# we also toggle on nice printing to illustrate our equations
sm.init_printing(use_unicode=True)
```

```python
# we go on to defining our variables as sympy symbols for equation handling
q1 = sm.symbols('q_1')
q2 = sm.symbols('q_2')
p1 = sm.symbols('p_1')
p2 = sm.symbols('p_2')
I = sm.symbols('I')
U = sm.symbols('U')
```

```python
# Next thing is to state the utility function
utility=q1*q2/(q1+q2)
utility
```

```python
# The consumers preferences are subject to the following budgetconstraint
budgcons=sm.Eq(p1*q1+p2*q2,I)
budgcons
```

```python
# The indifference curve with respect to q2 is defined for later use
indiff=sm.solve(sm.Eq(q1*q2/(q1+q2),U),q2)
indiff
```

With the model set, we can begin solving the optimization problem theoretically.

```python
# Isolating q2 in the budgetconstraint for later substitutaion
q2_iso=sm.solve(budgcons,q2)
q2_iso
```

```python
q1_iso=sm.solve(budgcons,q1)
q1_iso
```

```python
# Substituting q2 into the utility function
util_subsq2=utility.subs(q2,q2_iso[0])
util_subsq2
```

```python
util_subsq1=utility.subs(q1,q1_iso[0])
util_subsq1
```

```python
# Deriving the first order condition
foc1=sm.diff(util_subsq2,q1)
foc1
```

```python
foc2=sm.diff(util_subsq1,q2)
foc2
```

```python
# Deriving the solution for the optimal quantity of commodity 1 based on prices of two commodities and budget.
sol1 = sm.solve(sm.Eq(foc1,0),q1)
sol1
```

```python
sol2 = sm.solve(sm.Eq(foc2,0),q2)
sol2
```

## Optimization with values

As an example of the utility optimization problem, we start out by defining the utility function, the budgetconstraint and the indifference curve which we found in the algebraic solutions above.

```python
# We start out by defining the functions found above
# The utility function is defined by
def util(q1,q2):
    return (q1*q2)/(q1+q2)
# The budget constraint with respect to q2 is defined by
def budgcons(q1,p1,p2,I):
    return (I/p2)-(p1/p2)*q1
# The indifference function with respect to q2 is defined by
def indiff(q1,u):
    return (-u*q1)/(u-q1)
```

As we have found the solutions for the quantities with the chosen utility function and budgetconstraint above, we can define q1 and q2 as part of the the function to find the optimal values. Furthermore, as we got two optimal solutions for each of the quantities of the two commodities, we know that the solution must be positive for each of the commoditites. Therefore we choose the solution functions that gives positive amounts.

```python
# We then go on to defining the funtion for finding the optimal quantities.
def optimum(p1,p2,I):
    q1= (I*p1-I*math.sqrt(p1*p2))/(p1**2-p1*p2)
    q2= -(I*p2-I*math.sqrt(p1*p2))/(p1*p2-p2**2)
    u=util(q1,q2)
    return q1, q2, u
```

As we need prices and a budget value to solve for the optimal quantities, we have chosen the price of q1 as 0.5 and the price of q2 as 1. Furthermore, the budget amount is set to 100.

```python
# Defining the prices and the budget amount. Furthermore, defining boundaries for the visualization later on.
p1, p2 = 0.5, 1
I = 100

pmin, pmax = 0.5, 4
Imin, Imax = 10, 200
qmax = (3/4)*Imax/pmin
```

```python
# We then find the optimal quantities for the commodities, given our set prices and budget amount.
optimum(0.5,1,100)
```

```python
# Finally we define a function to plot the budget constraint and indifference curve for the model to show the optimal point.

def consume_plot(p1=p1, p2=p2, I=I):

    q1 = np.linspace(0.1,qmax,num=100)
    q1e, q2e, u = optimum(p1, p2 ,I)
    idfc = indiff(q1,u)
    budg = budgcons(q1, p1, p2, I)

    fig, ax = plt.subplots(figsize=(8,8))
    ax.plot(q1, budg, lw=2.5)
    ax.plot(q1, idfc, lw=2.5)
    ax.vlines(q1e,0,q2e, linestyles="dashed")
    ax.hlines(q2e,0,q1e, linestyles="dashed")
    ax.plot(q1e,q2e,'ob')
    ax.set_xlim(0, qmax)
    ax.set_ylim(0, qmax)
    ax.set_xlabel(r'$c_1$', fontsize=16)
    ax.set_ylabel('$c_2$', fontsize=16)
    ax.spines['right'].set_visible(False)
    ax.spines['top'].set_visible(False)
    ax.grid()
    plt.show()
```

```python
# Calling the function for the graph gives us a visualization of the optimal quantities of the
# commodities, also given by our optimum function.
consume_plot()
```

```python

```


<h1> Logaritmo Discreto

El logaritmo discreto, es la base de la seguridad de los metodos anteriores. Consiste en encontrar las llaves privadas de Bob y Alicia, a y b. Imaginemos que existe un tercer sujeto llamado Pancho. Pancho intercepta la comunicación entre Alicia y Bob y lo único que ve en el camino son A y B, es decir $g^a$ y $g^b$. Si Pancho pudiera saber a o b, entonces solo tomo ya sea A o B y lo eleva a la llave que encontro. Para encontrarlo necesita del logaritmo discreto.

El logaritmo discreto se define como:

$\ g^a \equiv \ A \ mod\ p$

Donde A, g y p son conocidos y buscamos a, es decir:

$ a \equiv \ log_g(A) \ mod\ p$

Hay varias formas de encontrar el logaritmo discreto. Tomemos del ejemplo anterior en el encriptado de Diffie Helman, los mismos números.

Digamos que Pancho consigue A = 656334248, sabemos que g = 13 y p = 3367900313

```python
A = 656334248
g = 13
p = 3367900313

```

<h3> Algoritmo Pohlig-Helman

```python
from sympy.ntheory import factorint
factorint(3367900312)
```

    {2: 3, 7: 1, 109: 1, 551753: 1}

Tomamos primero el factor $2^3$

```python
int1=int((3367900313-1)/2)

alpha1=pow(13,int1,p)
print("Alpha1 =",alpha1)
beta1=pow(A,int1,p)
print("Beta1 =",beta1)

```

    Alpha1 = 3367900312
    Beta1 = 3367900312

Por lo tanto $x_0=1$, y ahora incrementamos $2^2$

```python
s=modinv(g,p)
A1=A*s%p
print(A1)
int2=int((3367900312)/4)
beta2=pow(A1,int2,p)
print("Beta2 =",beta2)
prueba=pow(alpha1,5,p)
print("prueba =",prueba)
```

    3159318308
    Beta2 = 3367900312
    prueba = 3367900312

Por lo tanto $x_1=1$

```python
s2=modinv(g,p)
A2=A1*s2*s2%p
print(A2)
int3=int((3367900312)/8)
beta3=pow(A2,int3,p)
print("Beta3 =",beta3)

```

    1732536954
    Beta3 = 3367900312

Por lo tanto $x_2=1$, y la primera ecuación es: $x_1 = x_0+2*x_1+4*x_2 \equiv 7\ mod \ 8$

Tomamos segundo el factor $7$

```python
int11=int((3367900313-1)/7)

alpha11=pow(13,int11,p)
print("Alpha1 =",alpha11)
beta11=pow(A,int11,p)
print("Beta1 =",beta11)
```

    Alpha1 = 2974477553
    Beta1 = 1

Por lo tanto  $x_1=0$ y la segunda ecuación es: $x_2 \equiv 0\ mod \ 7$

Tomamos tercero el factor $109$

```python
int111=int((3367900313-1)/109)

alpha111=pow(13,int111,p)
print("Alpha1 =",alpha111)
beta111=pow(A,int111,p)
print("Beta1 =",beta111)
```

    Alpha1 = 2920505388
    Beta1 = 2497218448

```python
prueba=pow(alpha111,3,p)
print (prueba)
```

    2497218448

Por lo tanto  $x_1=3$ y la segunda ecuación es: $x_3 \equiv 3\ mod \ 109$

Tomamos por último el factor $551753$

```python
int1111=int((3367900313-1)/551753)

alpha1111=pow(13,int1111,p)
print("Alpha1 =",alpha1111)
beta1111=pow(A,int1111,p)
print("Beta1 =",beta1111)
```

    Alpha1 = 600054388
    Beta1 = 2491247764

```python
i=0
result=0
while result != beta1111:
    result=pow(alpha1111,i,p)
    i=i+1
print(i-1)
print (result)
```

    46552
    2491247764

```python
print(pow(alpha1111,46552,p))
```

    2491247764

Por lo tanto  $x_1=46552$ y la segunda ecuación es: $x_4 \equiv 46552\ mod \ 551753$

```python
def extended_gcd2(a, b):
    """Return (r, s, d) where a*r + b*s = d and d = gcd(a,b)"""
    x,y = 0, 1
    lastx, lasty = 1, 0

    while b:
        a, (q, b) = b, divmod(a,b)
        x, lastx = lastx-q*x, x
        y, lasty = lasty-q*y, y

    return (lastx, lasty, a)
def chinese_remainder_theorem(items):
  """Solve the chinese remainder theorem
  Given a list of items (a_i, n_i) solve for x such that x = a_i (mod n_i)
  such that 0 <= x < product(n_i)
  Assumes that n_i are pairwise co-prime.
  """

  # Determine N, the product of all n_i
  N = 1
  for a, n in items:
    N *= n

  # Find the solution (mod N)
  result = 0
  for a, n in items:
    m = N//n
    r, s, d = extended_gcd2(n, m)
    if d != 1:
      raise "Input not pairwise co-prime"
    result += a*s*m

  # Make sure we return the canonical solution.
  return result % N
result_final=chinese_remainder_theorem([(7,8),(0,7),(3,109),(46552,551753)])
print("a =",result_final)
```

    a = 12736871

# Fuerza Bruta

Con este algoritmo vamos a intentar descifrar un mensaje encriptado con ROTn. El modo más natural de proceder es intentar con las 25 llaves disponibles.

I. Insertamos el mensaje a descifrar.

```python
mensaje = 'FSCJCQ Y RGLEY.'
```

```python
alfabeto = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ'
```

II. Probaremos cada rotación para ver si tiene sentido el mensaje.

```python
for rot in range(len(alfabeto)):

    resultado = ''

    for letra in mensaje:
        if letra in alfabeto:
            num = alfabeto.find(letra)
            num = num - rot

            if num < 0:
                num = num + len(alfabeto)

            resultado = resultado + alfabeto[num]

        else:

            resultado = resultado + letra

    print('Rot #%s: %s' % (rot, resultado))
```

    Rot #0: FSCJCQ Y RGLEY.
    Rot #1: ERBIBP X QFKDX.
    Rot #2: DQAHAO W PEJCW.
    Rot #3: CPZGZN V ODIBV.
    Rot #4: BOYFYM U NCHAU.
    Rot #5: ANXEXL T MBGZT.
    Rot #6: ZMWDWK S LAFYS.
    Rot #7: YLVCVJ R KZEXR.
    Rot #8: XKUBUI Q JYDWQ.
    Rot #9: WJTATH P IXCVP.
    Rot #10: VISZSG O HWBUO.
    Rot #11: UHRYRF N GVATN.
    Rot #12: TGQXQE M FUZSM.
    Rot #13: SFPWPD L ETYRL.
    Rot #14: REOVOC K DSXQK.
    Rot #15: QDNUNB J CRWPJ.
    Rot #16: PCMTMA I BQVOI.
    Rot #17: OBLSLZ H APUNH.
    Rot #18: NAKRKY G ZOTMG.
    Rot #19: MZJQJX F YNSLF.
    Rot #20: LYIPIW E XMRKE.
    Rot #21: KXHOHV D WLQJD.
    Rot #22: JWGNGU C VKPIC.
    Rot #23: IVFMFT B UJOHB.
    Rot #24: HUELES A TINGA.
    Rot #25: GTDKDR Z SHMFZ.

```python

```


```python
import matplotlib.pyplot as plt
import numpy as np
from scipy.stats import expon
from typing import List
```

(a) Let the random variable $x$ be the lifetime of a light bulb in days. We assume that the lifetime is exponential with density:
$$p(x) = \lambda^{-1} e^{-x/\lambda}$$

Then given $\lambda$ (the mean time to failure):
$$ P (x \ge 30) = 1 - cdf(p(x)) = e^{-x/\lambda}$$

(b) Let $x_1,...,x_n \sim Exp(\lambda)$. The contribution of $x_i$ to the likelihood is:
$$L_i(\lambda) = \lambda^{-1} e^{-x/\lambda}$$

and the total log-likelihood is:
\begin{align}
\log L(\lambda) & = \sum_{i=1}^n \log L_i(\lambda) \\
& = -n \log \lambda - \frac{1}{\lambda} \sum_{i=1}^n x_i \\
& = -n (\log \lambda + \bar{x}/\lambda)
\end{align}

```python
data = np.array([6, 28]) # only the first two order statistics are reported
n = data.shape[0]
x_bar = np.mean(data)
lambda_ = np.linspace(1, 30, 100)
```

```python
log_like = -n * (np.log(lambda_) + x_bar / lambda_)
likelihood = np.exp(log_like)
likelihood /= np.max(likelihood)
```

```python
plt.plot(lambda_, likelihood)
plt.title('Light bulb failure likelihood')
plt.xlabel(r'$\lambda$')
plt.ylabel('Likelihood');
```

Not at all sure about parts (d) and (e).


# Python's Next Top Model
By The Good, The Bad and the Ugly aka. The Three Musketeers.

```python
# Importing packages
import matplotlib.pyplot as plt
import pandas as pd
import ipywidgets as widgets
import numpy as np
import itertools
from ipywidgets import Layout
import math

import time
from scipy import linalg
import scipy.optimize as optimize
import sympy as sm
#from data_gen import gen_df
from IPython.display import clear_output
```

Welcome back to the wonderful Python universe of Credible Threats. We will continue the exploration of movies, and we're sure that our data project notebook will look at this notebook and think, *you complete me*. You might want to ask, *Magic mirror on the wall, who's the fairest one of all*, and the answer will be this notebook. This notebook will try to estimate the IMDb ratings of movies. And as a bonus, we even have a live-updating figure, which will leave you thinking, *it's alive, it's alive*.

We will calculate the utility of each movie, given genres, awards, duration and so forth, and we will try to calculate the IMDb ratings for each movie, based on this utility. Finally, we will minimize the distance between our estimated ratings and the actual IMDb ratings. Now, we're sure you're thinking, *show me the money*, so this is it.

Our utility function is given by:
$$ U_i = \sum_{k=1}^{23}(\alpha_k G_{ik}) + \sum_{k=1920}^{2010} (\beta_k D_{ik}) + \gamma N_i + \delta W_i + \rho L_i $$.

$$\text{Where, } G_{ik} \text{ is genre } k \text{, } D_{ik} \text{ is decade } k \text{, } N_i \text{ is award nominations, } W_i \text{ is award wins, and } L_i \text{ is duration. } $$

I think we can agree, it is *beauty kills the beast*.
Now, based on this utility function, we will estimate the ratings of each movies,
$$ R_i^{model} = \frac{\exp(\omega x_i')}{1 + \exp(\omega x_i')} $$

where,
$$ x_i = \big[G_1, G_2, ..., G_{23}, D_{1920}, D_{1930}, ..., D_{2010}, N_i, W_i, L_i \big] $$
$$ \omega = \big[\alpha_1, \alpha_2, ..., \alpha_{23}, \beta_{1920}, \beta_{1930}, ..., \beta_{2010}, \gamma, \delta, \rho_1 \big] $$

We then use optimize methods to solve the following:
$$ \min_{\omega} \Big\{ \sum_{i=1}^{n} \left( R_i^{model} - R_i^{data} \right)^2 \Big\} $$

$$ \text{where, } R_i^{data} \text{ are the true IMDb ratings from the dataset, and} R_i^{model} \text{are the ratings calculated from the utility function.}$$

Throughout our notebook, you might *feel the need - the need for speed*. But, *patience you must have, my young Padawan*, because optimizing takes time.

*Of all the python notebooks in all the towns in all the world, you walk into ours*. How lucky you are, you'll soon see why.

So, *say hello to my little friend*, Python's next top model.
Let's *get busy coding, or get busy dying*.

```python
# List for later use
variables = ['Action','Adventure','Animation','Biography','Comedy','Crime','Documentary',
            'Drama','Family','Fantasy','FilmNoir','History','Horror','Music','Musical',
            'Mystery','Romance','SciFi','Short','Sport','Thriller','War','Western',
            'decade_1920','decade_1930','decade_1940','decade_1950','decade_1960',
            'decade_1970','decade_1980','decade_1990','decade_2000','decade_2010',
            'nrOfNominations','nrOfWins','duration']

vars_dec = ['Action','Adventure','Animation','Biography','Comedy','Crime','Documentary',
        'Drama','Family','Fantasy','FilmNoir','History','Horror','Music','Musical',
        'Mystery','Romance','SciFi','Short','Sport','Thriller','War','Western',
        'nrOfNominations','nrOfWins','duration']

decade_list = [1920, 1930, 1940, 1950, 1960, 1970, 1980, 1990, 2000, 2010]

# Function to calculate squared mean differences
def sqr_diff_sum(df_X,pars):
    """ Generates a float of the sum of squared differences between the ratings from the data and the model.

    Args:
        df_X (DataFrame): DataFrame containing the variable in x for all observations.
        pars (List): List of parameters in omega

    Returns:
        A float.
    """
    # Calculate the matrix product between omega and X
    util = df_X@pars
    # Scale the product so it's between 0 and 10. This is the R_model
    df_Y['rat_model'] = 10*np.exp(util)/(1+np.exp(util))
    # Calculate the squared difference between R_data and R_model
    df_Y['sqr_diff'] = (df_Y['rat_model']-df_Y['rat_data'])**2

    return df_Y['sqr_diff'].sum() # Returns the sum of the squared differences
```

```python
def gen_df(filename):

    # Get .csv.file
    data = pd.read_csv(filename, sep=';', encoding='latin-1', escapechar='\\')

    # Read file into pandas dataframe
    df = pd.DataFrame(data)

    # Drop unwanted columns
    for i in range(44,48):
        df.drop(columns=[f'Unnamed: {i}'], inplace=True)

    # Filters out movies with less than 5000 ratings and drops some genres and other columns
    df = df.loc[(df['ratingCount']>=5000)]
    df.drop(columns=['fn','wordsInTitle','url','Adult','GameShow','News','RealityTV','TalkShow'], inplace=True)

    # Keep only observations of movie-type
    I = df['type'] == 'video.movie'
    df = df.loc[I]
    df.drop(columns=['type'], inplace=True)

    # Drop observations with missing data
    df.dropna(inplace=True)

    # Replace 0's in imdb-ratings
    df['imdbRating'] = df['imdbRating'].astype(str)
    df['imdbRating'].replace(regex=True, inplace=True,to_replace='0',value='')
    df['imdbRating'] = df['imdbRating'].astype(float)

    # Transform duration from seconds to hours
    df['duration'] = df['duration']/60**2

    # Drop years before 1920 and 2014 because of few obervations
    I = (df['year']>=1920) & (df['year']<=2013)
    df = df.loc[I]

    # Change the type of 'year' to integer
    df['year'] = df['year'].astype(int)

    # Sort observations and reset index
    df.sort_values('year', inplace=True)
    df.reset_index(inplace=True)

    # Generating variable that shows the decade (as a string)
    year_list = [1920, 1930, 1940, 1950, 1960, 1970, 1980, 1990, 2000, 2010]

    df['decade'] = ''
    for i,start in enumerate(year_list):
        end = start+10
        df.loc[(df['year'] >= start) & (df['year'] < end), 'decade'] = f'{year_list[i]}s'

    return df
```

```python
def df_s(keep_top=None, decade=None):
    """ Prepares data for optimization by creating dummmy-variables for decades, dropping movies with less than 5000 ratings,
        dropping some variables that we don't wish to use in the model, and splitting the dataset, so we have a data set
        consisting only of the variables used in the model.

    Args:
        keep_top (optional, type: int): When a number (n) is put in this option, only the top n movies a kept in the dataset. This is used to remove movies for a better fit of the model

    Returns:
        df_X (type: Pandas dataframe): A dataframe consisting only of the varaibles that are used to calculate the rating.
        df_Y (type: Pandas dataframe): A dataframe consisting of the true ratings from the dataset and an index-variable, used for merging with original dataset, later in the process.

    Notice:
        The function includes global for 'df', which means that the dataframe 'df' generated, can be called outside the function. This is used so that we won't have to filter the original dataframe everytime we use it.
    """
    global df

    # Calls the file 'imdb.csv' which is located in the repository, and contains the dataset used
    filename = 'imdb.csv'

    # A function generated earlier, which cleans the dataset [Should we include this, and do some of the filtering below in that funciton?]
    df = gen_df(filename)

    # Keeps only top n movies, if this is specified when calling the function
    if keep_top != None:
        df = df.sort_values('imdbRating', ascending=False)
        df = df.iloc[:keep_top]

    if decade == None:
        # Decade dummies
        for i in decade_list:
            df[f'decade_{i}'] = 0
            df.loc[(df['decade'] == f'{i}s'),f'decade_{i}'] = 1

        # Splits the dataset into two datasets
        df_X = df.copy()
        df_Y = pd.DataFrame(df[['imdbRating', 'index']].copy())
        df_Y = df_Y.rename(columns = {'imdbRating':'rat_data'})

        # Rearrange and keep given columns
        df_X = df_X.reindex(['Action','Adventure','Animation','Biography','Comedy','Crime','Documentary',
                              'Drama','Family','Fantasy','FilmNoir','History','Horror','Music','Musical',
                              'Mystery','Romance','SciFi','Short','Sport','Thriller','War','Western',
                              'decade_1920','decade_1930','decade_1940','decade_1950','decade_1960',
                              'decade_1970','decade_1980','decade_1990','decade_2000','decade_2010',
                              'nrOfNominations','nrOfWins','duration'], axis=1)

    if decade != None:
        # Keeps movies from the specified decade
        df = df.loc[df['decade'] == f'{decade}s']

        # Splits the dataset into two datasets
        df_X = df.copy()
        df_Y = pd.DataFrame(df[['imdbRating', 'index']].copy())
        df_Y = df_Y.rename(columns = {'imdbRating':'rat_data'})

        # Rearrange columns and keep the specified variables
        df_X = df_X.reindex(['Action','Adventure','Animation','Biography','Comedy','Crime','Documentary',
                                  'Drama','Family','Fantasy','FilmNoir','History','Horror','Music','Musical',
                                  'Mystery','Romance','SciFi','Short','Sport','Thriller','War','Western',
                                  'nrOfNominations','nrOfWins','duration'], axis=1)

    return df_X, df_Y
```

## Optimization
We analyze preferences for movies by minimizing the sum of the squared differences from our model prediction and the true rating, i.e.:
$$
\min_{\omega} \Big\{ \sum_{i=1}^{n} \left( R_i^{model} - R_i^{data} \right)^2 \Big\}
$$
By doing so, we'll find parameters that best describe movie preferences given our model. As we will discuss later, our model is not necessarily the best model to describe preferences, which might lead to these preferences not being accurate.

If the model could predict all movie ratings correctly the parameters would perfectly describe movie preferences. For example if the parameter for action-movies were negative, the prediction would be precise, if action-movies generally had a negative impact on ratings. Of course people have much different preferences, and an interesting analysis would be, to predict preferences for different people, and by this look at a distribution of these parameters. This is not included in this project, though, mainly because of we don't have the data for it.

To shorten the length of the optimizing process, and to sort out movies that makes it harder for the model to predict preferences, the optimization can be based on the top n movies, based on IMDb rating. When all movies are included in the optimizer the lowest ranking movies will have ratings around 2. Our rating predictor will in this case not rank any movies under 5. Thereby it is clear that the predictor model is not able to predict movies with very low rating. But it is able to predict movies with higher ratings.

```python
def optimizer(keep_top=None, live_graph=True):
    """ Creates and optimizes the function which calculates rating based on variables in the dataset.

    Args:
        keep_top (optional, type: int): When a number (n) is put in this option, only the top n movies a kept in the dataset. This is used to sort out movies for a better fit of the model
        live_graph(optional, type: boolean): Produces a live graph of the optimization proces, plotting the function values against the iterations. Makes the optimizer slower, but is a good visualization of the process.

    Returns:
        result (type: scipy optimizer object): An object containing results from the optimizer among other information such as status etc.
        timer (type: int): Time of running optimizer in seconds.

    Notice:
        Uses the package scipy.optimize.minimize which optimizes functions based on a given method. Here 'Nelder-Mead' is used.
        This function defines a set of functions used in the optimizer which are not documented by docstring but comments in code.
    """
    # "Initializes" a timer for printing time of optimization
    start = time.time()

    # Set of globals, so variables can be called inside the functions defined in this function
    global fs
    global evals
    global x0
    global df
    global df_Y

    # Plots a live graph of optimization process, if chosen.
    if live_graph:
        # The live plot is created by taking the function value for each iteration and saving in a list
        def live_plot(evals, fs, ymax=10000, figsize=(7,5)):
            # Clears plot every time a new plot is created
            clear_output(wait=True)
            plt.figure(figsize=figsize)

            # Plots values and sets title, grid, labels, etc.
            plt.plot(evals, fs)
            plt.title('Figure 1: Optimizing path')
            plt.grid(True)
            plt.xlabel('Iterations')
            plt.ylabel('Function value')
            plt.xlim(0,17500)
            plt.ylim(0,ymax)

            plt.show();

    # Function to be called by optimizer each iteration. This returns function values for each iteration and plots live graph
    def collect(x):
        # Set of globals to be called in live graph-function
        global fs
        global evals
        global x0
        global fig
        global ax
        global ymax
        global x_1, x_2, x_3, x_4, x_5, x_6, x_7, x_8, x_9, x_10, x_11, x_12, x_13, x_14, x_15, x_16, x_17, x_18, x_19, x_20
        global x_21, x_22, x_23, x_24, x_25, x_26, x_27, x_28, x_29, x_30, x_31, x_32, x_33, x_34, x_35, x_36

        # Calculates function value for initial guess, for first iteration
        if evals == 0:
            fs = []
            fs.append(obj_fun(x0))
            # Generates vector of parameter estimates
            for i in range(1,37): # 1 to 36
                globals()[f'x_{i}'] = [x0[i-1]]

        # Calculates function values for current parameter values
        if evals != 0:
            fs.append(obj_fun(x))

        # Plots live graph if chosen
        if live_graph:
            # Calculates max y-value for axis for first iteration
            if evals == 1:
                ymax = math.ceil(obj_fun(x)/2000)*2000

            # Updates plot every 100th iteration for the first 1000 iterations
            if evals < 1000:
                if evals > 0 and evals%100 == 0:
                    live_plot(range(evals+1), fs, ymax)
            # Updates plot every 500th iteration for the rest of the process
            if evals >=1000:
                if evals%500 == 0:
                    live_plot(range(evals+1), fs, ymax)

        # Appends esimtates to x_vectors
        for i in range(1,37):
            globals()[f'x_{i}'].append(x[i-1])

        # Adds one to the number of iterations each iteration
        evals += 1

    # Define datasets to be used using function defined earlier
    df_X, df_Y = df_s(keep_top=keep_top, decade=None)

    # Intiial guess
    x0 = np.zeros(len(variables))

    # Start iteration number
    evals = 0

    # Defines function to be minimized
    obj_fun = lambda x: sqr_diff_sum(df_X,x)

    # Run optimizer
    result = optimize.minimize(obj_fun,x0,
                               method="Nelder-Mead",
                               options={"disp":True, "maxiter":50000}, # display the results
                               callback=collect
                               )

    # End timer
    end = time.time()-start

    # Returns
    return result, end
```

By runing the code below you can see a live-updating graph that shows the value of the sum of the squared difference for each iteration. As you will see, the sqaured difference reaches close to its minimum very quickly so the last 10,000 iterations almost don't change the value of the squard difference. The live-graph slows the optimizer down, so choose option live_graph=False, if you are impacient and don't care about an awesome live-updating graph.

```python
# Run optimizer function and print results and time
result, timer = optimizer()
print(f'{" ":9s}Time: {timer:.4f} seconds')

# Keep copy of df and df_Y for later purpose
df_Y_all = df_Y.copy()
df_all = df.copy()
```

The following cell will give a list of the estimated parameters/preferences, and the cell after that produce a figure showing the path of the estimated parameters, i.e. the parameter estimate for each iteration.

```python
# List of Parameter Estimates

# Constructs dictionary with parameters
preben = dict()
for j,i in enumerate(variables):
    preben[i] = f'{result.x[j]:.2f}'

# Prints list of parameterestimes
for i in preben:
    print (i, ":", preben[i])

```

    Action : -0.02
    Adventure : -0.03
    Animation : 0.17
    Biography : -0.17
    Comedy : -0.02
    Crime : -0.03
    Documentary : 0.11
    Drama : 0.13
    Family : -0.06
    Fantasy : -0.33
    FilmNoir : 0.02
    History : -0.08
    Horror : -0.02
    Music : -0.03
    Musical : -0.08
    Mystery : -0.09
    Romance : -0.04
    SciFi : -0.10
    Short : 0.12
    Sport : -0.06
    Thriller : -0.06
    War : 0.19
    Western : 0.03
    decade_1920 : -0.04
    decade_1930 : 0.12
    decade_1940 : 0.18
    decade_1950 : -0.03
    decade_1960 : 0.31
    decade_1970 : 0.10
    decade_1980 : 0.01
    decade_1990 : -0.22
    decade_2000 : -0.20
    decade_2010 : -0.21
    nrOfNominations : 0.00
    nrOfWins : 0.02
    duration : 0.45

```python
## Convergence of Parameter Estimates

# Constructing dataframe which contains all parameter vectors as variables
par_dict = {"Action": x_1,
             "Adventure": x_2,
             "Animation": x_3,
             "Biography": x_4,
             "Comedy": x_5,
             "Crime": x_6,
             "Documentary": x_7,
             "Drama": x_8,
             "Family": x_9,
             "Fantasy": x_10,
             "Film Noir": x_11,
             "History": x_12,
             "Horror": x_13,
             "Music": x_14,
             "Musical": x_15,
             "Mystery": x_16,
             "Romance": x_17,
             "SciFi": x_18,
             "Short": x_19,
             "Sport": x_20,
             "Thriller": x_21,
             "War": x_22,
             "Western": x_23,
             "1920's": x_24,
             "1930's": x_25,
             "1940's": x_26,
             "1950's": x_27,
             "1960's": x_28,
             "1970's": x_29,
             "1980's": x_30,
             "1990's": x_31,
             "2000's": x_32,
             "2010's": x_33,
             "Nr. of Nominations": x_34,
             "Nr. of Wins": x_35,
             "Duration": x_36
            }

df_par = pd.DataFrame(par_dict)

##  Interactive graph of parametervalues for each iteration in optimizer
# Chose which parameter to show

# Interactive graph of parameter estimates
def graph(par):
    """
    Constructs a graph which shows the progress in parameter estimates foreach iteration in
    the optimizing process.

    Args:
        par (type: str): Variable name, for which the graphs shows the parameter estimates.

    Returns:
        Graph of progress in parameter estimates.

    Notice:
        The function is meant to be called within the graph_int function.
        The graph_int function turns this graph into an interactive function, where the
        user can choose between variables.
        Thus, the graph is not meant to be run on its own.
    """
    # Generating figure
    plt.plot(df_par[par])

    # Grid and axes
    plt.grid(True)
    plt.axhline(df_par[par].iloc[-1], linestyle="dashed", color="orange", label="Optimum")

    # Labels and Titles
    plt.title("Figure 2: Progress in Parameter Estimation")
    plt.ylabel("Parameter Estimate")
    plt.xlabel("Nr. of Iterations")
    plt.legend()

# Interactive part: Choose between parameters
def graph_int():
    """
    Constructs interactive graph from the graph function.
    """
    widgets.interact(graph,
                     par = widgets.Dropdown(
                     description="Variable",
                     options=df_par.columns.values,
                     value="Action"));
graph_int()

```

    interactive(children=(Dropdown(description='Variable', options=('Action', 'Adventure', 'Animation', 'Biography…

From the above figure, we notice that most parameter estimates are very volatile in the first 10,000-15,000 iterations, following which they converge towards the optimum value.

However, this does not appear to be the case for the variables, Nr. of nominations and Nr. of awards, which get quite close to the optimum value at around the 7500th iteration.
Additionally, we notice that the variable Duration appears to have the largest effect on ratings, which the optimzer seems to realize after approximately 2500 iterations.

In the next part of the project we will plot the optimal estimates in a figure. Furthermore, we show how the estimates changes a lot when the sample size is restricted to only the best movies.

```python
## This piece of code solves the model for three different sample size

#### OBS: The code will take a couple of minutes to run because the model is solved three times!! ####

# List to store result for the optimization with the different sample sizes
results_list = [result.x]

# This loop solve the model for the top 500, 1000, and 2000 movies
for i in [500, 1000, 2000]:
    res_temp, timer_temp = optimizer(keep_top=i, live_graph=False)
    temp = res_temp.x
    print(f'{" ":9s}Time: {timer_temp:.4f} seconds')
    results_list.append(temp) # Store the results in the 'results_list'
```

    Optimization terminated successfully.
             Current function value: 197.838633
             Iterations: 13628
             Function evaluations: 15779
             Time: 35.7865 seconds
    Optimization terminated successfully.
             Current function value: 439.055453
             Iterations: 21224
             Function evaluations: 24448
             Time: 55.7610 seconds
    Optimization terminated successfully.
             Current function value: 903.321786
             Iterations: 16176
             Function evaluations: 18626
             Time: 47.0429 seconds

```python
## The code creates an interactive plot of the estimated parameters for all variables.
## In the interactive part you can choose between estimates when the model is solved
## for all movies or just top top 500, 1000, or 2000 rated movies.

def fig_2(val):
    """ Generates a figure which plots estimated parameters for all variables.

    Args:
        val (string or int): Should be one of the elements in the options-list

    Returns:
        One interactive plot.

    Notice:
        The function is generated so that it can be called using widgets.interact.
        Thus, it is not intended to be used on its own.
    """
    # Initiates figure
    fig = plt.figure(figsize=(12,6))
    ax1 = fig.add_subplot(1,1,1)

    # Plots the estimated parameters for the chosen sample of movies
    ax1.bar(variables, results_list[options.index(val)], label=f'Estimates with {options[options.index(val)]} movies')

    # Scatter plot with the estimated paramters for the entire sample
    ax1.scatter(variables, results_list[0], marker='D', s=15, zorder=2, label='Estimates with all movies')

    # Legends and labels
    ax1.legend(loc="lower right")
    ax1.set_ylabel('Parameter estimates')
    ax1.set_title(f'Figure 3: Parameter estimates for {options[options.index(val)]} movies')
    ax1.set_ylim([-0.7,0.7])
    ax1.axhline(y=0,color='black',linewidth=1)
    for tick in ax1.get_xticklabels():
        tick.set_rotation(90)

options = ['All', 500, 1000, 2000] # Option list

# Making the figure interactive so the estimates are shown for the chosen sample size
widgets.interact(fig_2,
    val = widgets.Dropdown(description='Nr. of movies', value='All', options=options,
                ),
);
```

    interactive(children=(Dropdown(description='Nr. of movies', options=('All', 500, 1000, 2000), value='All'), Ou…

As also showed in the list above, duration is the varaible that effects the ratings the most. This is also the case when the model is solved for top 500, 1000, and 2000 rated movies, cf. figure 3. However, it seems like duration is the only estimate which is fairly stable. The sign of the decade dummies is an example. When the model is solved for the entire sample, the estimates are positive for the older decade-dummies, i.e. the 1920s and 1930s, and negative for the newer decade-dummies, the 2000s and 2010s. However, when the sample only contains the top 2000 rated movies we see the opposite picture. We know from figure 10 in our dataproject that the ratings of the younger movies have a larger variance with more high-rated movies but also a lot of poorly rated movies. By restricting the sample to only the best movies we drop all the bad new movies, and this is probably the reason to the positive estimates of the newer movies.

## Robustness checks

### Prediction precision
Ideally our model would predict movie ratings perfectly, and thereby our parameters would perfectly describe the general preferences for movies. Whether this is possible, even with an advanced model, is not sure. People have different preferences, and two movies that would seem identical in our dataset, based on genres, duration, time of release, etc. could have very different ratings. Therefore some devitaion in our prediction from the true ratings are expected. For that reason, this section will explore how well our model describe the actual ratings, and why some estimates are more reliable than others.

```python
def merge_df(df_org):
    """ Merges the original dataset with the optimal solution from the optimizer, i.e. the last dataset of calculated raitngs.

    Args:
        df_org (type: Pandas dataframe): The orignal dataset of movies, containing true ratings.

    Returns:
        df_merge (type: Pandas dataframe): Original dataset combined with calculated ratings from optimal parameters.
    """

    # Merges original dataset with optimal ratings based on model
    df_merge = df_org.merge(df_Y_all, how='right', left_on='index', right_on='index')

    # Calculates deviations both in normal and absolute form
    df_merge['abs_diff'] = abs(df_merge['rat_model']-df_merge['rat_data'])
    df_merge['diff'] = df_merge['rat_model']-df_merge['rat_data']

    return df_merge

def _mean_genre(df,group, diff):
    """ Calculates the mean of deviations from true ratings and ratings based on optimal parameters from optimizer.

    Args:
        df (type: Pandas dataframe): Dataframe consisting information on groups and deivations
        group (type: string): Defines which group mean of deviations are presented for. Chosen by fixed list in widget.

    Returns:
        None

    Notice:
        Only to be run through widget.interact()
    """
    # List of values used for each group
    genre_list = ['Action','Adventure','Animation','Biography','Comedy','Crime','Documentary',
                    'Drama','Family','Fantasy','FilmNoir','History','Horror','Music','Musical',
                    'Mystery','Romance','SciFi','Short','Sport','Thriller','War','Western']

    decade_list = ['1920s', '1930s', '1940s', '1950s', '1960s', '1970s', '1980s', '1990s', '2000s', '2010s']

    dur_list = [0, .5, 1, 1.5, 2, 2.5, 3, 3.5]

    rat_list = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

    # Dictionaries for plotting
    mean_dict = dict()
    count_dict = dict()

    # Choose which type of difference to plot
    if diff == 'Absolute':
        diff_col = 'abs_diff'
    else:
        diff_col = 'diff'

    # Creates dictionaries based on chosen group
    if group == 'Genres':
        for i in genre_list:
            I = df[i] == 1
            mean_dict[i] = df.loc[I][diff_col].mean()
            count_dict[i] = df.loc[I]['index'].count()

    if group == 'Decades':
        for i in decade_list:
            I = df['decade'] == i
            mean_dict[i] = df.loc[I][diff_col].mean()
            count_dict[i] = df.loc[I]['index'].count()

    if group == 'Duration':
        for j,i in enumerate(dur_list):
            if j != len(dur_list)-1:
                # Uses values from dict, to find movies between to items in the lists.
                I = ((df['duration'] >= dur_list[j]) & (df['duration'] < dur_list[j+1]))
                mean_dict[f'{dur_list[j]:2.1f} - {dur_list[j+1]:2.1f}'] = df.loc[I][diff_col].mean()
                count_dict[f'{dur_list[j]:2.1f} - {dur_list[j+1]:2.1f}'] = df.loc[I]['index'].count()
            else:
                I = df['duration'] >= dur_list[j]
                mean_dict[f'{dur_list[j]:2.1f} {"+":5s}'] = df.loc[I][diff_col].mean()
                count_dict[f'{dur_list[j]:2.1f} {"+":5s}'] = df.loc[I]['index'].count()

    if group == 'True rating':
        for j,i in enumerate(rat_list):
            if j != len(rat_list)-1:
                # Uses values from dict, to find movies between to items in the lists.
                I = ((df['rat_data'] >= rat_list[j]) & (df['rat_data'] < rat_list[j+1]))
                mean_dict[f'{rat_list[j]:4.1f} - {rat_list[j+1]:3.1f}'] = df.loc[I][diff_col].mean()
                count_dict[f'{rat_list[j]:4.1f} - {rat_list[j+1]:3.1f}'] = df.loc[I]['index'].count()
            else:
                pass

    # Creates figure to hold two subplots
    fig1, (ax1,ax2) = plt.subplots(nrows=1,ncols=2,sharey=True,figsize=(12,5))
    #plt.title("Figure 4")

    # Plots means of deviations
    ax1.barh(*zip(*mean_dict.items()))
    ax1.set_ylabel(group)
    ax1.set_xlabel('Model deviation')
    ax1.grid(axis='x')
    ax1.set_title('Figure 4: Deviation between the model and the actual ratings')

    # Plots count of movies
    ax2.barh(*zip(*count_dict.items()))
    ax2.set_xlabel('Number of observations')
    ax2.grid(axis='x')
    ax2.set_title('Figure 5: Number of observation')

# Calls merge-function
df_merge = merge_df(df_all)

# Runs interactive figure
mean_genre = widgets.interact(_mean_genre,
                             df = widgets.fixed(df_merge),
                             group = widgets.Dropdown(
                                 options = ['Genres','Decades','Duration','True rating'],
                                 description = 'Group',
                                 value = 'Genres'),
                             diff = widgets.RadioButtons(
                                 options = ['Average', 'Absolute'],
                                 description = 'Difference',
                                 value = 'Average'))

```

    interactive(children=(Dropdown(description='Group', options=('Genres', 'Decades', 'Duration', 'True rating'), …

Figure 4 plot the mean of deviations based on genres, duration, decades, and true ratings. Figure 5 plot the number of movies in the dataset based on the same groups. From these graphs we see a clear correlation between precision and number of movies in the dataset. The more movies of a given genre, duration, etc. the more precise the prediction of movie-ratings in these groups. Take drama-movies as an example. The mean of the deviation is close to zero, and movies of this genre is also quite overrepresentated in the dataset. The complete opposite case is short-movies. But we also see that sci-fi-movies are quite well predicted, even though there aren't relatively many of these movies in the dataset. The same is seen in other different genres. This general image is seen when we group movies on decades, duration, and true rating as well.

When looking at absolute differences instead, the correlation is not as strong, especially not for the genres.

The next section will look at the changes in the estimates when the model is solved for one decade at a time.

```python
## The following code produces the parameter estimates where the sample is restricted to all the decades,
## one at the time. The function return a list which contains a list of the estimates for every decade.

#### OBS: The code will take about 3-4 minutes to run, since the model is estimated 10 times!! ####

def optimizer_dec():
    """ Generates a list containing 10 list with estimates of the model for every decade.

    Args:
        No arguments are needed.

    Returns:
        A list.

    Notice:
        It will raise an error if the function is given an argument!
    """

    result = [] # Empty list to store the estimated parameters

    # Make a loop so the model is estimated for all ten decades
    for j,decade in enumerate(decade_list,1):
        df_X, df_Y = df_s(keep_top=None, decade=decade) # Call the function to generate the two dataframes
        x0 = np.zeros(len(vars_dec)) # Starting values

        obj_fun = lambda x: sqr_diff_sum(df_X,x) # The objective function -> sum of squared differences
        start = time.time()

        # Use Scipy optimizer to solve the model
        print(f'Opmization {j} out of {len(decade_list)}: {decade}')

        result_i = optimize.minimize(obj_fun,x0,
                               method='Nelder-Mead',
                               options={"disp":True, "maxiter":50000}, # display the results
                               );

        end_time = time.time()-start
        print(f'{" ":9s}Time: {end_time:.4f} seconds')

        # Add the result for each deacde to the result-list
        result.append(list(result_i.x))

    return result # Returns the result-list

# Call the optimize_dec function
result_dec = optimizer_dec()
```

    Opmization 1 out of 10: 1920
    Optimization terminated successfully.
             Current function value: 2.582968
             Iterations: 11001
             Function evaluations: 13220
             Time: 17.1451 seconds
    Opmization 2 out of 10: 1930
    Optimization terminated successfully.
             Current function value: 15.556532
             Iterations: 11136
             Function evaluations: 13352
             Time: 17.3901 seconds
    Opmization 3 out of 10: 1940
    Optimization terminated successfully.
             Current function value: 26.531688
             Iterations: 8876
             Function evaluations: 10731
             Time: 14.3220 seconds
    Opmization 4 out of 10: 1950
    Optimization terminated successfully.
             Current function value: 51.634688
             Iterations: 9578
             Function evaluations: 11517
             Time: 15.2239 seconds
    Opmization 5 out of 10: 1960
    Optimization terminated successfully.
             Current function value: 87.282326
             Iterations: 13058
             Function evaluations: 15676
             Time: 20.5838 seconds
    Opmization 6 out of 10: 1970
    Optimization terminated successfully.
             Current function value: 43.500430
             Iterations: 9229
             Function evaluations: 11116
             Time: 14.7109 seconds
    Opmization 7 out of 10: 1980
    Optimization terminated successfully.
             Current function value: 77.604277
             Iterations: 8448
             Function evaluations: 10155
             Time: 14.2556 seconds
    Opmization 8 out of 10: 1990
    Optimization terminated successfully.
             Current function value: 88.532631
             Iterations: 7482
             Function evaluations: 9030
             Time: 12.3654 seconds
    Opmization 9 out of 10: 2000
    Optimization terminated successfully.
             Current function value: 183.284443
             Iterations: 9095
             Function evaluations: 10933
             Time: 15.7564 seconds
    Opmization 10 out of 10: 2010
    Optimization terminated successfully.
             Current function value: 44.839180
             Iterations: 8593
             Function evaluations: 10363
             Time: 14.0481 seconds

```python
## The following code produces a figure with the estimated parameters for each deacde for a chosen variable

result_dec_mod = [] # Empty list to storage

# The loop changes the order of the result-list so the
# estimates are ordered by the variables and subordered by decade
# insted of being ordered by decade and subordered by variables
for j,var in enumerate(vars_dec):
    temp = []
    for i,dec in enumerate(decade_list):
        temp.append(result_dec[i][j])

    result_dec_mod.append(temp)

# Defining a figure to plot the estimates
def fig(var):
    """ Generates a figure which plots estimated parameters for each decade for one variable

    Args:
        var (type: string): Should be one of the variables in the X-vector

    Returns:
        One interactive plot.

    Notice:
        The function is generated so that it can be called using widgets.interact()
        Thus, it is not intended to be used on its own.
        """
    # Initalize figure
    fig = plt.figure(figsize=(8,6))
    ax1 = fig.add_subplot(1,1,1)

    # Plotting variables for chosen decade
    ax1.bar(decade_list, result_dec_mod[vars_dec.index(var)],width=6)

    # Setting labels, ticks etc.
    ax1.set_ylabel('Parameter estimates')
    ax1.set_title(f'Figure 6: Parameter estimates for {var} per decade')
    ax1.set_xticks(decade_list)
    ax1.axhline(y=0,color='black',linewidth=1)

# Making the figure interactive so the estimates are shown for the chosen variable
widgets.interact(fig,
    var = widgets.Dropdown(description='Variable', value='Action', options=vars_dec,
                ),
);
```

    interactive(children=(Dropdown(description='Variable', options=('Action', 'Adventure', 'Animation', 'Biography…

Figure 6 shows the final robustness check, how the estimates vary when the sample only contains movies from one deacde. Going through the different variables, it is hard to find a generel trend. E.g. the estimates of action movies are negative in some decades and positive in other decades. On the other hand, the estimates of war movies are negative for all but two decades, and duration has positive estimates in all decades.

The messy picture could be caused by too small samples whereby the estimates will be very volatile. Another reason could be that the preferences change. By looking at the estimates of drama movies you could make the arguement that the the genre of drama affect the ratings positively for old movies, but the opposite after the 1950s. This could be an indication that people have less preferences for newer drama movies compared to older drama movies.

## Conclusion

Based on the above results, it appears that duration increases ratings by quite a lot, while Nr. of nominations and Nr. of wins doesn't appear to play a role. Meanwhile, the effect of different genres and decades are quite different. We see that genres like Action and Horror have very little effect on ratings, while the Fantasy movies appear to have a signifiant positive marginal effect on ratings.

Our estimates are not precise though, but this is to be expected. It is very difficult to create a model that predicts ratings like these 100 pct. right (if it is even possible with the given variables). Our robustness checks shows, that estimates are volatile to whether only top movies are included and how many of these are included, as shown in figure 4. This volatility is probably a combination of model misspecification and changing preferences. Additionally, genres could have very different distributions (e.g. the best movies are often action-movies, but there are A LOT of bad action movies as well).

A way to deal with this volatility could be to consider another functional form of the model or maybe find other variables that could improve how well the model predicts ratings.

We hope you enjoyed our movie model project, and don't think we are completely idiots. Because as our model we conclude that *much to learn, we still have*.

Credible Threats/modelproject/Python's Next Top Model.ipynb