#**Chapter 4 – Training Models**

     Open the original notebook

Let's begin like in the last notebook: importing a few common modules, ensuring MatplotLib plots figures inline and preparing a function to save the figures. We also check that Python 3.5 or later is installed (although Python 2.x may work, it is deprecated so once again we strongly recommend you use Python 3 instead), as well as Scikit-Learn ≥0.20.

You don't need to worry about understanding everything that is written in this section.

```python
# Python ≥3.5 is required
import sys
assert sys.version_info >= (3, 5)

# Is this notebook running on Colab or Kaggle?
IS_COLAB = "google.colab" in sys.modules

# Scikit-Learn ≥0.20 is required
import sklearn
assert sklearn.__version__ >= "0.20"

# Common imports
import numpy as np
import os

# To make this notebook's output stable across runs
rnd_seed = 42
rnd_gen = np.random.default_rng(rnd_seed)

# To plot pretty figures
%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
mpl.rc('axes', labelsize=14)
mpl.rc('xtick', labelsize=12)
mpl.rc('ytick', labelsize=12)

# Where to save the figures
PROJECT_ROOT_DIR = "."
CHAPTER_ID = "classification"
IMAGES_PATH = os.path.join(PROJECT_ROOT_DIR, "images", CHAPTER_ID)
os.makedirs(IMAGES_PATH, exist_ok=True)

def save_fig(fig_id, tight_layout=True, fig_extension="png", resolution=300):
    path = os.path.join(IMAGES_PATH, fig_id + "." + fig_extension)
    print("Saving figure", fig_id)
    if tight_layout:
        plt.tight_layout()
    plt.savefig(path, format=fig_extension, dpi=resolution)
```

In this notebook we will be working with the [*Iris Flower Dataset*](https://en.wikipedia.org/wiki/Iris_flower_data_set), in which the length and width of both the sepals and petals of three types of Iris flowes were recorded. For reference, these are pictures of the three flowers:

 In order: Iris Setosa,  Iris Versicolor, and Iris Virginica

Photo Credits:[Kosaciec szczecinkowaty Iris setosa](https://en.wikipedia.org/wiki/File:Kosaciec_szczecinkowaty_Iris_setosa.jpg) by [Radomil Binek](https://commons.wikimedia.org/wiki/User:Radomil) licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.en); [Blue flag flower close-up (Iris versicolor)](https://en.wikipedia.org/wiki/File:Iris_versicolor_3.jpg)by Danielle Langlois licensed under [CC BY-SA 3.0](https://creativecommons.org/licenses/by-sa/3.0/deed.en); [image of Iris virginica shrevei](https://en.wikipedia.org/wiki/File:Iris_virginica.jpg) by [Frank Mayfield](https://www.flickr.com/photos/33397993@N05) licensed under [CC BY-SA 2.0](https://creativecommons.org/licenses/by-sa/2.0/deed.en).

As you can imagine, this dataset is normally used to train *multiclass*/*multinomial* classification algorithms and not *binary* classification algorithms, since there *are* more than 2 classes.

"*Three classes, even!*" - an observant TA

For this exercise, however, we will be implemented the binary classification algorithm referred to as the *logistic regression* algorithm (also called logit regression).

```python
# Let's load the Iris Dataset
from sklearn import datasets
iris = datasets.load_iris()

# Print out some information about the data
print(f'Keys in Iris dictionary: \n{list(iris.keys())}\n\n')
print(iris.DESCR)

# And load the petal lengths and widths as our input data
X = iris["data"][:, (2, 3)]  # petal length, petal width
y = iris["target"]

print(iris['data_module'])

# The target data labels Setosa as 0, Versicolor as 1, and Virginica as 2. For
# this exercise we will be using only the Versicolor and Virgina sets.
bin_indices = np.logical_or(y==1,y==2)
bin_X = X[bin_indices]
bin_y = (y[bin_indices]==2).astype(np.uint8) # convert to binary
```

    Keys in Iris dictionary:
    ['data', 'target', 'frame', 'target_names', 'DESCR', 'feature_names', 'filename', 'data_module']

    .. _iris_dataset:

    Iris plants dataset
    --------------------

    **Data Set Characteristics:**

        :Number of Instances: 150 (50 in each of three classes)
        :Number of Attributes: 4 numeric, predictive attributes and the class
        :Attribute Information:
            - sepal length in cm
            - sepal width in cm
            - petal length in cm
            - petal width in cm
            - class:
                    - Iris-Setosa
                    - Iris-Versicolour
                    - Iris-Virginica

        :Summary Statistics:

        ============== ==== ==== ======= ===== ====================
                        Min  Max   Mean    SD   Class Correlation
        ============== ==== ==== ======= ===== ====================
        sepal length:   4.3  7.9   5.84   0.83    0.7826
        sepal width:    2.0  4.4   3.05   0.43   -0.4194
        petal length:   1.0  6.9   3.76   1.76    0.9490  (high!)
        petal width:    0.1  2.5   1.20   0.76    0.9565  (high!)
        ============== ==== ==== ======= ===== ====================

        :Missing Attribute Values: None
        :Class Distribution: 33.3% for each of 3 classes.
        :Creator: R.A. Fisher
        :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
        :Date: July, 1988

    The famous Iris database, first used by Sir R.A. Fisher. The dataset is taken
    from Fisher's paper. Note that it's the same as in R, but not as in the UCI
    Machine Learning Repository, which has two wrong data points.

    This is perhaps the best known database to be found in the
    pattern recognition literature.  Fisher's paper is a classic in the field and
    is referenced frequently to this day.  (See Duda & Hart, for example.)  The
    data set contains 3 classes of 50 instances each, where each class refers to a
    type of iris plant.  One class is linearly separable from the other 2; the
    latter are NOT linearly separable from each other.

    .. topic:: References

       - Fisher, R.A. "The use of multiple measurements in taxonomic problems"
         Annual Eugenics, 7, Part II, 179-188 (1936); also in "Contributions to
         Mathematical Statistics" (John Wiley, NY, 1950).
       - Duda, R.O., & Hart, P.E. (1973) Pattern Classification and Scene Analysis.
         (Q327.D83) John Wiley & Sons.  ISBN 0-471-22361-1.  See page 218.
       - Dasarathy, B.V. (1980) "Nosing Around the Neighborhood: A New System
         Structure and Classification Rule for Recognition in Partially Exposed
         Environments".  IEEE Transactions on Pattern Analysis and Machine
         Intelligence, Vol. PAMI-2, No. 1, 67-71.
       - Gates, G.W. (1972) "The Reduced Nearest Neighbor Rule".  IEEE Transactions
         on Information Theory, May 1972, 431-433.
       - See also: 1988 MLC Proceedings, 54-64.  Cheeseman et al"s AUTOCLASS II
         conceptual clustering system finds 3 classes in the data.
       - Many, many more ...
    sklearn.datasets.data

```python
bin_X.shape
bin_y
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], dtype=uint8)

We now have a set of binary classification data we can use to train an algorithm.

As we saw during our reading, we need to define three things in order to train our algorithm: the type of algorithm we will train, the cost function (which will tell us how close our prediction is to the truth), and a method for updating the parameters in our model according to the value of the cost function (e.g., the gradient descent method).

Let's begin by defining the type of algorithm we will use. We will train a logistic regression model to differentiate between two classes. A reminder of how the logistic regression algorithm works is given below.

The logistic regression algorithm will thus take an input $t$ that is a linear combination of the features:

 $t_{\small{n}} = \beta_{\small{0}} + \beta_{\small{1}} \cdot X_{1,n} + \beta_{\small{2}} \cdot X_{2,n}$

where
* $n$ is the ID of the sample
* $X_{\small{0}}$ represents the petal length
* $X_{\small{1}}$ represents the petal width

This input is then fed into the logistic function, $\sigma$:
\begin{align}
\sigma: t\mapsto \dfrac{1}{1+e^ {-t}}
\end{align}

Let's plot it below to remember the shape of the function

```python
t = np.arange(-4,4,.1)
def logistic(in_val):
    # Return the value of the logistic function
    return 1/(1 + np.exp(- in_val))

fig, ax = plt.subplots()

ax.axvline(0, c='black', alpha=1)
ax.axhline(0, c='black', alpha=1)

[ax.axhline(y_val, c='black', alpha=0.5, linestyle='dotted') for y_val in (0.5,1)]

plt.autoscale(axis='x', tight=True)

ax.plot(t, logistic(t));
ax.set_xlabel('$t$')
ax.set_ylabel('$\\sigma\\  \\left(t\\right)$')
fig.tight_layout()
```

With the logistic function, we define inputs resulting in $\sigma\geq.5$ as belonging to the ***one*** class, and any value below that is considered to belong to the ***zero*** class.

We now have a function which lets us map the value of the petal length and width to the class to which the observation belongs (i.e., whether the length and width correspond to Iris Versicolor or Iris Virginica). However, there is a parameter vector **$\theta$** with a number of parameters that we do not have a value for:
 $\theta = [ \beta_{\small{0}}, \beta_{\small{1}}$, $\beta_{\small{2}} ]$

**Q1) Set up an array of random numbers between 0 and 1 representing the $\theta$ vector.**

Hint:  Use `rnd_gen`! If you're not sure how to use it, consult the `default_rng` documentation [at this link](https://numpy.org/doc/stable/reference/random/generator.html). For instance, you may use the `random` method of `rnd_gen`.

```python
# Write your code here
rng = np.random.default_rng()
theta_init = rng.random( size=3)
print(theta_init)
```

    [0.20693996 0.43319825 0.30702669]

In order to determine whether a set of $\beta$ values is better than the other, we need to quantify well the values are able to predict the class. This is where the cost function comes in.

The cost function, $c$, will return a value close to zero when the prediction, $\hat{p}$, is correct and a large value when it is wrong. In a binary classification problem, we can use the log loss function. For a single prediction and truth value, it is given by:
\begin{align}
        \text{c}(\hat{p},y) = \left\{
        \begin{array}{cl}
        -\log(\hat{p})& \text{if}\; y=1\\
        -\log(1-\hat{p}) & \text{if}\; y=0
        \end{array}
        \right.
    \end{align}

However, we want to apply the cost function to an n-dimensional set of predictions and truth values. Thankfully, we can find the average value of the log loss function $J$ for an an-dimensional set of $\hat{y}$ & $y$ as follows:

\begin{align}
        \text{J}(\mathbf{\hat{p}},y) = - \dfrac{1}{n} \sum_{i=1}^{n}
        \left[ y_i\cdot \log\left( \hat{p}_i \right) \right] +
        \left[ \left( 1 - y_i \right) \cdot \log\left( 1-\hat{p}_i \right) \right]
    \end{align}

We now have a formula that can be used to calculate the average cost over the training set of data.

**Q2) Define a log_loss function that takes in an arbitrarily large set of prediction and truths**

Hint 1: You need to encode the function $J$ above, for which Numpy's functions may be quite convenient (e.g., [`log`](https://numpy.org/doc/stable/reference/generated/numpy.log.html), [`mean`](https://numpy.org/doc/stable/reference/generated/numpy.mean.html), etc.)

Hint 2: Asserting the dimensions of the vector is a good way to check that your function is working correctly. [Here's a tutorial on how to use `assert`](https://swcarpentry.github.io/python-novice-inflammation/10-defensive/index.html#assertions). For instance, to assert that two vectors `X` and `Y` have the same dimension, you may use:
```
assert X.shape==Y.shape
```

Now let's code 💻

```python
def log_loss(p_hat, y, epsilon=1e-7):
  # Write your code here.
  # We can also run into problems if p_hat = 0, so add an _epsilon_ term
  # when evaluating log(p_hat).
  log_p_hat =  np.log10(p_hat) +epsilon
  n=y.shape[0]
  Jarray=y*log_p_hat + (1-y)*np.log10(1-p_hat)
  J = -np.sum(Jarray)*1/n

  # After calculating J, assert that J has the same shape as p_hat and y
  assert Jarray.shape==p_hat.shape
  assert Jarray.shape==y.shape

  return J
```

We now have a way of quantifying how good our predictions are. The final thing needed for us to train our algorithm is figuring out a way to update the parameters in a way that improves the average quality of our predictions.

**Warning**: we'll go into a bit of math below

Let's look at the change in a single parameter within $\theta$: $\beta_1$ (given $X_{1,i} = X_1$, $\;\hat{p}_{i} = \hat{p}$, $\;y_{i} = y$). If we want to know what the effect of changing the value of $\beta_1$ will have on the log loss function we can find this with the partial derivative:
$
        \dfrac{\partial J}{\partial \beta_1}
$

This may not seem very helpful by itself - after all, $\beta_1$ isn't even in the expression of $J$. But if we use the chain rule, we can rewrite the expression as:

        $\dfrac{\partial J}{\partial \hat{p}} \cdot
        \dfrac{\partial \hat{p}}{\partial \theta} \cdot
        \dfrac{\partial \theta}{\partial \beta_1}$

We'll spare you the math (feel free to verify it youself, however!):

$\dfrac{\partial J}{\partial \hat{p}} =  \dfrac{\hat{p} - y}{\hat{p}(1-\hat{p})}, \quad
        \dfrac{\partial \hat{p}}{\partial \theta} = \hat{p} (1-\hat{p}), \quad
        \dfrac{\partial \theta}{\partial \beta_1} = X_1 $

and thus
$
        \dfrac{\partial J}{\partial \beta_1} = (\hat{p} - y) \cdot X_1
$

We can calculate the partial derivative for each parameter in $\theta$ which, as you may have realized, is simply the $\theta$ gradient of $J$: $\nabla_{\theta}(J)$

With all of this information, we can now write $\nabla_{\theta} J$ in terms of the error, the feature vector, and the number of samples we're training on!

$\nabla_{\mathbf{\theta}^{(k)}} \, J(\mathbf{\theta^{(k)}}) = \dfrac{1}{n} \sum\limits_{i=1}^{n}{ \left ( \hat{p}^{(k)}_{i} - y_{i} \right ) \mathbf{X}_{i}}$

Note that here $k$ represents the iteration of the parameters we are currently on.

We now have a gradient we can calculate and use in the batch gradient descent method! The updated parameters will thus be:

\begin{align}
{\mathbf{\theta}^{(k+1)}} = {\mathbf{\theta}^{(k)}} - \eta\,\nabla_{\theta^{(k)}}J(\theta^{(k)})
\end{align}

Where $\eta$ is the learning rate parameter. It's also worth pointing out that $\;\hat{p}^{(k)}_i = \sigma\left(\theta^{(k)}, X_i\right) $

In order to easily calculate the input to the logistic regression, we'll multiply the $\theta$ vector with the X data, and as we have a non-zero bias  $\beta_0$ we'd like to have an X matrix whose first column is filled with ones.

\begin{align}
    X_{\small{with\ bias}} = \begin{pmatrix}
        1 & X_{1,0} & X_{2,0}\\
        1 & X_{1,1} & X_{2,1}\\
        &...&\\
        1 & X_{1,n} & X_{2,n}
        \end{pmatrix}
\end{align}

**Q3) Prepare the `X_with_bias` matrix (remember to use the `bin_X` data and not just `X`). Write a function called `predict` that takes in the parameter vector $\theta$ and the `X_with_bias` matrix and evaluates the logistic function for each of the samples.**

Hint 1: You recently learned how to initialize arrays in the `Numpy` notebook [at this link](https://nbviewer.org/github/tbeucler/2022_ML_Earth_Env_Sci/blob/main/Lab_Notebooks/S1_2_Numpy.ipynb). There are many ways to add a columns of 1 to `bin_X`, for instance using [`np.concatenate`](https://numpy.org/doc/stable/reference/generated/numpy.concatenate.html) or [`np.append`](https://numpy.org/doc/stable/reference/generated/numpy.append.html).

Hint 2:  To clarify, the function `predict` calculates $\hat{p}$ from $\beta$ and $\boldsymbol{X}$.

Hint 3: In practice, to calculate the logistic function for each sample, you may follow the equations [higher up in the notebook](#logit) and (1) calculate $t$ from $\beta$ and $\boldsymbol{X_{\mathrm{with\ bias}}}$ before (2) applying the logistic function $\sigma$ to $t$.

```python
# Prepare the X_with_bias matrix
ones=np.ones(bin_X.shape[0])
X_with_bias= np.concatenate(( np.transpose([ones]),bin_X), axis=1)
```

```python
# Write your function predict here
def predict(X,theta_model):
  t=np.sum(theta_model*X,axis=1)
  p_hat= logistic(t)
  y_hat= p_hat>=0.5
  return y_hat.astype(int)

```

**Q4) Now that you have a `predict` function, write a `gradient_calc` function that calculates the gradient for the logistic function.**

Hint 1: You'll have to feed `theta`, `X`, and `y` to the `gradient_calc` function.

Hint 2: You can use [this equation](#grad_eq) to calculate the gradient of the cost function.

```python
# Write your code here
def gradient_calc(p_hat,y,X):
  n=y.size
  grad_matrix=(p_hat-y)*np.transpose(X)
  gradJ=1/n*np.sum(np.transpose(grad_matrix), axis=0)
  return gradJ

```

We can now write a function that will train a logistic regression algorithm!

Your `logistic_regression` function needs to:
* Take in a set of training input/output data, validation input/output data, a number of iterations to train for, a set of initial parameters $\theta$, and a learning rate $\eta$
* At each iteration:
 * Generate a set of predictions on the training data. Hint: You may use your function `predict` on inputs `X_train` from the training set.
 * Calculate and store the loss function for the training data at each iteration. Hint: You may use your function `log_loss` on inputs `X_train` and outputs `y_train` from the training set.
 * Calculate the gradient. Hint: You may use your function `grad_calc`.
 * Update the $\theta$ parameters. Hint: You need to implement [this equation](#grad_descent).
 * Generate a set of predictions on the validation data using the updated parameters. Hint: You may use your function `predict` on inputs `X_valid` from the validation set.
 * Calculate and store the loss function for the validation data. Hint: You may use your function `log_loss` on inputs `X_valid` and outputs `y_valid` from the validation set.
 * Bonus: Calculate and store the accuracy of the model on the training and validation data as a metric!
* Return the final set of parameters $\theta$ & the stored training/validation loss function values (and the accuracy, if you did the bonus)

**Q5) Write the `logistic_regression` function**

```python
# Write your code here
def logistic_regression(eta,theta,max_iter,X_train,y_train,X_test,y_test):

  #intializing variables
  tol=1e-8
  n_iter=1

  #training set
  t=np.sum(theta*X_train,axis=1)
  p_hat= logistic(t)
  loss=log_loss(p_hat, y_train, epsilon=1e-7)
  lossar0=np.array(loss)

  #test_set
  t_test=np.sum(theta*X_test,axis=1)
  p_hat_test= logistic(t_test)
  loss_test=log_loss(p_hat_test, y_test, epsilon=1e-7)
  lossar1=np.array(loss_test)

#train
  while ( n_iter < max_iter and loss>tol ):
    n_iter =n_iter+1
    print('iteration',n_iter)
    #calculating the gradient
    gradJ=gradient_calc(p_hat,y_train,X_train)
    #new theta values
    theta=theta -eta*gradJ
    #evaluating p_hat
    t=np.sum(theta*X_train,axis=1)
    p_hat= logistic(t)
    #new loss function
    loss=log_loss(p_hat, y_train, epsilon=1e-7)
    print('loss_train',loss)
    lossar0 =np.append(lossar0,loss)

    #Evaluate the test set
    t_test=np.sum(theta*X_test,axis=1)
    p_hat_test=logistic(t_test)
    loss_test=log_loss(p_hat_test, y_test, epsilon=1e-7)
    lossar1 =np.append(lossar1,loss_test)

  return np.concatenate(([lossar0],[lossar1]), axis=0), theta

```

**¡¡¡Important Note!!!**

The notebook assumes that you will return
1. a Losses list, where Losses[0] is the training loss and Losses[1] is the validation loss
2. a tuple with the 3 final coefficients ($\beta_0$, $\beta_1$, $\beta_2$)

The code for visualizing the bonus accuracy is not included - but it should be simple enough to do in a way similar to that which is done with the losses.

---------------------

Now that we have our logistic regression function, we're all set to train our algorithm! Or are we?

There's an important data step that we've neglected up to this point - we need to split the data into the train, validation, and test datasets.

```python
test_ratio = 0.2
validation_ratio = 0.2
total_size = len(X_with_bias)

test_size = int(total_size * test_ratio)
validation_size = int(total_size * validation_ratio)
train_size = total_size - test_size - validation_size

rnd_indices = rnd_gen.permutation(total_size)

X_train = X_with_bias[rnd_indices[:train_size]]
y_train = bin_y[rnd_indices[:train_size]]
X_valid = X_with_bias[rnd_indices[train_size:-test_size]]
y_valid = bin_y[rnd_indices[train_size:-test_size]]
X_test = X_with_bias[rnd_indices[-test_size:]]
y_test = bin_y[rnd_indices[-test_size:]]
```

Now we're ready!

**Q6) Train your logistic regression algorithm. Use 5000 iterations, $\eta$=0.1**

Hint: It's time to use the `logistic_regression` function you defined in Q5.

```python
# Complete the code
max_iter=5000
eta=0.1
theta=theta_init

losses, coeffs =  logistic_regression(eta,theta,max_iter,X_train,y_train,X_test,y_test)
```

    Streaming output truncated to the last 5000 lines.
    iteration 2501
    loss_train 0.1320256000226267
    iteration 2502
    loss_train 0.13200902990066943
    iteration 2503
    loss_train 0.1319924684054566
    iteration 2504
    loss_train 0.13197591552956867
    iteration 2505
    loss_train 0.13195937126559518
    iteration 2506
    loss_train 0.13194283560613465
    iteration 2507
    loss_train 0.1319263085437948
    iteration 2508
    loss_train 0.1319097900711924
    iteration 2509
    loss_train 0.1318932801809532
    iteration 2510
    loss_train 0.13187677886571206
    iteration 2511
    loss_train 0.13186028611811274
    iteration 2512
    loss_train 0.13184380193080816
    iteration 2513
    loss_train 0.13182732629646007
    iteration 2514
    loss_train 0.1318108592077393
    iteration 2515
    loss_train 0.13179440065732567
    iteration 2516
    loss_train 0.13177795063790776
    iteration 2517
    loss_train 0.13176150914218332
    iteration 2518
    loss_train 0.13174507616285883
    iteration 2519
    loss_train 0.13172865169264986
    iteration 2520
    loss_train 0.1317122357242806
    iteration 2521
    loss_train 0.1316958282504844
    iteration 2522
    loss_train 0.13167942926400325
    iteration 2523
    loss_train 0.13166303875758814
    iteration 2524
    loss_train 0.13164665672399878
    iteration 2525
    loss_train 0.1316302831560038
    iteration 2526
    loss_train 0.13161391804638056
    iteration 2527
    loss_train 0.13159756138791526
    iteration 2528
    loss_train 0.13158121317340277
    iteration 2529
    loss_train 0.13156487339564688
    iteration 2530
    loss_train 0.13154854204746003
    iteration 2531
    loss_train 0.13153221912166338
    iteration 2532
    loss_train 0.1315159046110869
    iteration 2533
    loss_train 0.1314995985085691
    iteration 2534
    loss_train 0.13148330080695744
    iteration 2535
    loss_train 0.13146701149910783
    iteration 2536
    loss_train 0.13145073057788498
    iteration 2537
    loss_train 0.1314344580361621
    iteration 2538
    loss_train 0.13141819386682124
    iteration 2539
    loss_train 0.131401938062753
    iteration 2540
    loss_train 0.1313856906168564
    iteration 2541
    loss_train 0.13136945152203938
    iteration 2542
    loss_train 0.13135322077121828
    iteration 2543
    loss_train 0.13133699835731794
    iteration 2544
    loss_train 0.13132078427327196
    iteration 2545
    loss_train 0.13130457851202232
    iteration 2546
    loss_train 0.1312883810665196
    iteration 2547
    loss_train 0.13127219192972284
    iteration 2548
    loss_train 0.1312560110945997
    iteration 2549
    loss_train 0.13123983855412616
    iteration 2550
    loss_train 0.1312236743012869
    iteration 2551
    loss_train 0.1312075183290748
    iteration 2552
    loss_train 0.13119137063049133
    iteration 2553
    loss_train 0.13117523119854646
    iteration 2554
    loss_train 0.13115910002625847
    iteration 2555
    loss_train 0.13114297710665407
    iteration 2556
    loss_train 0.13112686243276836
    iteration 2557
    loss_train 0.13111075599764493
    iteration 2558
    loss_train 0.1310946577943356
    iteration 2559
    loss_train 0.1310785678159006
    iteration 2560
    loss_train 0.13106248605540852
    iteration 2561
    loss_train 0.1310464125059362
    iteration 2562
    loss_train 0.13103034716056902
    iteration 2563
    loss_train 0.13101429001240036
    iteration 2564
    loss_train 0.1309982410545321
    iteration 2565
    loss_train 0.1309822002800743
    iteration 2566
    loss_train 0.13096616768214533
    iteration 2567
    loss_train 0.13095014325387183
    iteration 2568
    loss_train 0.13093412698838863
    iteration 2569
    loss_train 0.13091811887883886
    iteration 2570
    loss_train 0.13090211891837372
    iteration 2571
    loss_train 0.13088612710015277
    iteration 2572
    loss_train 0.13087014341734368
    iteration 2573
    loss_train 0.13085416786312234
    iteration 2574
    loss_train 0.13083820043067268
    iteration 2575
    loss_train 0.130822241113187
    iteration 2576
    loss_train 0.13080628990386542
    iteration 2577
    loss_train 0.13079034679591653
    iteration 2578
    loss_train 0.13077441178255683
    iteration 2579
    loss_train 0.13075848485701094
    iteration 2580
    loss_train 0.13074256601251158
    iteration 2581
    loss_train 0.13072665524229957
    iteration 2582
    loss_train 0.13071075253962375
    iteration 2583
    loss_train 0.13069485789774105
    iteration 2584
    loss_train 0.13067897130991643
    iteration 2585
    loss_train 0.13066309276942278
    iteration 2586
    loss_train 0.1306472222695412
    iteration 2587
    loss_train 0.13063135980356055
    iteration 2588
    loss_train 0.1306155053647779
    iteration 2589
    loss_train 0.13059965894649814
    iteration 2590
    loss_train 0.13058382054203413
    iteration 2591
    loss_train 0.13056799014470674
    iteration 2592
    loss_train 0.13055216774784476
    iteration 2593
    loss_train 0.13053635334478494
    iteration 2594
    loss_train 0.13052054692887188
    iteration 2595
    loss_train 0.13050474849345808
    iteration 2596
    loss_train 0.1304889580319039
    iteration 2597
    loss_train 0.13047317553757776
    iteration 2598
    loss_train 0.1304574010038557
    iteration 2599
    loss_train 0.13044163442412174
    iteration 2600
    loss_train 0.13042587579176768
    iteration 2601
    loss_train 0.1304101251001932
    iteration 2602
    loss_train 0.13039438234280587
    iteration 2603
    loss_train 0.1303786475130208
    iteration 2604
    loss_train 0.13036292060426125
    iteration 2605
    loss_train 0.1303472016099579
    iteration 2606
    loss_train 0.13033149052354945
    iteration 2607
    loss_train 0.1303157873384823
    iteration 2608
    loss_train 0.13030009204821044
    iteration 2609
    loss_train 0.13028440464619578
    iteration 2610
    loss_train 0.1302687251259079
    iteration 2611
    loss_train 0.13025305348082405
    iteration 2612
    loss_train 0.1302373897044292
    iteration 2613
    loss_train 0.13022173379021598
    iteration 2614
    loss_train 0.13020608573168466
    iteration 2615
    loss_train 0.1301904455223433
    iteration 2616
    loss_train 0.13017481315570745
    iteration 2617
    loss_train 0.13015918862530038
    iteration 2618
    loss_train 0.13014357192465298
    iteration 2619
    loss_train 0.13012796304730376
    iteration 2620
    loss_train 0.13011236198679882
    iteration 2621
    loss_train 0.13009676873669176
    iteration 2622
    loss_train 0.13008118329054394
    iteration 2623
    loss_train 0.13006560564192413
    iteration 2624
    loss_train 0.1300500357844087
    iteration 2625
    loss_train 0.13003447371158158
    iteration 2626
    loss_train 0.13001891941703422
    iteration 2627
    loss_train 0.13000337289436562
    iteration 2628
    loss_train 0.12998783413718223
    iteration 2629
    loss_train 0.129972303139098
    iteration 2630
    loss_train 0.12995677989373447
    iteration 2631
    loss_train 0.12994126439472048
    iteration 2632
    loss_train 0.12992575663569245
    iteration 2633
    loss_train 0.12991025661029418
    iteration 2634
    loss_train 0.12989476431217709
    iteration 2635
    loss_train 0.12987927973499974
    iteration 2636
    loss_train 0.12986380287242832
    iteration 2637
    loss_train 0.1298483337181364
    iteration 2638
    loss_train 0.1298328722658048
    iteration 2639
    loss_train 0.12981741850912196
    iteration 2640
    loss_train 0.12980197244178343
    iteration 2641
    loss_train 0.12978653405749233
    iteration 2642
    loss_train 0.12977110334995903
    iteration 2643
    loss_train 0.1297556803129012
    iteration 2644
    loss_train 0.12974026494004398
    iteration 2645
    loss_train 0.12972485722511962
    iteration 2646
    loss_train 0.12970945716186794
    iteration 2647
    loss_train 0.12969406474403575
    iteration 2648
    loss_train 0.12967867996537738
    iteration 2649
    loss_train 0.1296633028196543
    iteration 2650
    loss_train 0.12964793330063534
    iteration 2651
    loss_train 0.12963257140209647
    iteration 2652
    loss_train 0.12961721711782095
    iteration 2653
    loss_train 0.1296018704415993
    iteration 2654
    loss_train 0.1295865313672292
    iteration 2655
    loss_train 0.12957119988851556
    iteration 2656
    loss_train 0.12955587599927046
    iteration 2657
    loss_train 0.1295405596933132
    iteration 2658
    loss_train 0.1295252509644702
    iteration 2659
    loss_train 0.12950994980657513
    iteration 2660
    loss_train 0.12949465621346873
    iteration 2661
    loss_train 0.12947937017899888
    iteration 2662
    loss_train 0.12946409169702064
    iteration 2663
    loss_train 0.12944882076139616
    iteration 2664
    loss_train 0.12943355736599468
    iteration 2665
    loss_train 0.12941830150469255
    iteration 2666
    loss_train 0.1294030531713733
    iteration 2667
    loss_train 0.12938781235992727
    iteration 2668
    loss_train 0.12937257906425212
    iteration 2669
    loss_train 0.12935735327825257
    iteration 2670
    loss_train 0.1293421349958401
    iteration 2671
    loss_train 0.12932692421093364
    iteration 2672
    loss_train 0.12931172091745874
    iteration 2673
    loss_train 0.12929652510934822
    iteration 2674
    loss_train 0.1292813367805418
    iteration 2675
    loss_train 0.1292661559249862
    iteration 2676
    loss_train 0.1292509825366351
    iteration 2677
    loss_train 0.12923581660944924
    iteration 2678
    loss_train 0.12922065813739622
    iteration 2679
    loss_train 0.12920550711445059
    iteration 2680
    loss_train 0.12919036353459393
    iteration 2681
    loss_train 0.12917522739181467
    iteration 2682
    loss_train 0.1291600986801081
    iteration 2683
    loss_train 0.12914497739347658
    iteration 2684
    loss_train 0.1291298635259292
    iteration 2685
    loss_train 0.12911475707148204
    iteration 2686
    loss_train 0.12909965802415807
    iteration 2687
    loss_train 0.12908456637798701
    iteration 2688
    loss_train 0.12906948212700547
    iteration 2689
    loss_train 0.12905440526525708
    iteration 2690
    loss_train 0.12903933578679203
    iteration 2691
    loss_train 0.12902427368566752
    iteration 2692
    loss_train 0.12900921895594744
    iteration 2693
    loss_train 0.12899417159170265
    iteration 2694
    loss_train 0.12897913158701063
    iteration 2695
    loss_train 0.12896409893595578
    iteration 2696
    loss_train 0.12894907363262909
    iteration 2697
    loss_train 0.12893405567112856
    iteration 2698
    loss_train 0.12891904504555873
    iteration 2699
    loss_train 0.12890404175003103
    iteration 2700
    loss_train 0.12888904577866347
    iteration 2701
    loss_train 0.12887405712558095
    iteration 2702
    loss_train 0.12885907578491498
    iteration 2703
    loss_train 0.12884410175080374
    iteration 2704
    loss_train 0.12882913501739227
    iteration 2705
    loss_train 0.12881417557883207
    iteration 2706
    loss_train 0.12879922342928143
    iteration 2707
    loss_train 0.12878427856290534
    iteration 2708
    loss_train 0.12876934097387538
    iteration 2709
    loss_train 0.12875441065636975
    iteration 2710
    loss_train 0.1287394876045734
    iteration 2711
    loss_train 0.12872457181267774
    iteration 2712
    loss_train 0.1287096632748809
    iteration 2713
    loss_train 0.1286947619853876
    iteration 2714
    loss_train 0.12867986793840908
    iteration 2715
    loss_train 0.12866498112816332
    iteration 2716
    loss_train 0.1286501015488747
    iteration 2717
    loss_train 0.12863522919477427
    iteration 2718
    loss_train 0.12862036406009966
    iteration 2719
    loss_train 0.12860550613909488
    iteration 2720
    loss_train 0.12859065542601067
    iteration 2721
    loss_train 0.12857581191510412
    iteration 2722
    loss_train 0.128560975600639
    iteration 2723
    loss_train 0.1285461464768855
    iteration 2724
    loss_train 0.12853132453812027
    iteration 2725
    loss_train 0.12851650977862655
    iteration 2726
    loss_train 0.12850170219269397
    iteration 2727
    loss_train 0.12848690177461863
    iteration 2728
    loss_train 0.1284721085187031
    iteration 2729
    loss_train 0.1284573224192565
    iteration 2730
    loss_train 0.1284425434705942
    iteration 2731
    loss_train 0.12842777166703817
    iteration 2732
    loss_train 0.12841300700291666
    iteration 2733
    loss_train 0.12839824947256442
    iteration 2734
    loss_train 0.1283834990703226
    iteration 2735
    loss_train 0.1283687557905387
    iteration 2736
    loss_train 0.12835401962756662
    iteration 2737
    loss_train 0.12833929057576665
    iteration 2738
    loss_train 0.1283245686295054
    iteration 2739
    loss_train 0.1283098537831558
    iteration 2740
    loss_train 0.1282951460310973
    iteration 2741
    loss_train 0.12828044536771546
    iteration 2742
    loss_train 0.12826575178740238
    iteration 2743
    loss_train 0.12825106528455624
    iteration 2744
    loss_train 0.1282363858535818
    iteration 2745
    loss_train 0.12822171348888983
    iteration 2746
    loss_train 0.12820704818489756
    iteration 2747
    loss_train 0.12819238993602855
    iteration 2748
    loss_train 0.12817773873671245
    iteration 2749
    loss_train 0.12816309458138536
    iteration 2750
    loss_train 0.12814845746448947
    iteration 2751
    loss_train 0.1281338273804733
    iteration 2752
    loss_train 0.12811920432379156
    iteration 2753
    loss_train 0.12810458828890528
    iteration 2754
    loss_train 0.12808997927028157
    iteration 2755
    loss_train 0.12807537726239382
    iteration 2756
    loss_train 0.1280607822597216
    iteration 2757
    loss_train 0.12804619425675065
    iteration 2758
    loss_train 0.12803161324797296
    iteration 2759
    loss_train 0.12801703922788665
    iteration 2760
    loss_train 0.12800247219099592
    iteration 2761
    loss_train 0.12798791213181124
    iteration 2762
    loss_train 0.1279733590448492
    iteration 2763
    loss_train 0.1279588129246324
    iteration 2764
    loss_train 0.12794427376568976
    iteration 2765
    loss_train 0.12792974156255613
    iteration 2766
    loss_train 0.12791521630977265
    iteration 2767
    loss_train 0.12790069800188636
    iteration 2768
    loss_train 0.12788618663345055
    iteration 2769
    loss_train 0.1278716821990245
    iteration 2770
    loss_train 0.12785718469317364
    iteration 2771
    loss_train 0.12784269411046936
    iteration 2772
    loss_train 0.12782821044548912
    iteration 2773
    loss_train 0.12781373369281657
    iteration 2774
    loss_train 0.12779926384704118
    iteration 2775
    loss_train 0.12778480090275857
    iteration 2776
    loss_train 0.12777034485457037
    iteration 2777
    loss_train 0.12775589569708423
    iteration 2778
    loss_train 0.12774145342491378
    iteration 2779
    loss_train 0.1277270180326786
    iteration 2780
    loss_train 0.1277125895150043
    iteration 2781
    loss_train 0.1276981678665225
    iteration 2782
    loss_train 0.1276837530818707
    iteration 2783
    loss_train 0.12766934515569234
    iteration 2784
    loss_train 0.127654944082637
    iteration 2785
    loss_train 0.12764054985735998
    iteration 2786
    loss_train 0.12762616247452263
    iteration 2787
    loss_train 0.1276117819287922
    iteration 2788
    loss_train 0.12759740821484178
    iteration 2789
    loss_train 0.1275830413273505
    iteration 2790
    loss_train 0.12756868126100332
    iteration 2791
    loss_train 0.12755432801049102
    iteration 2792
    loss_train 0.12753998157051036
    iteration 2793
    loss_train 0.1275256419357639
    iteration 2794
    loss_train 0.1275113091009602
    iteration 2795
    loss_train 0.1274969830608135
    iteration 2796
    loss_train 0.12748266381004392
    iteration 2797
    loss_train 0.1274683513433775
    iteration 2798
    loss_train 0.12745404565554608
    iteration 2799
    loss_train 0.1274397467412873
    iteration 2800
    loss_train 0.12742545459534457
    iteration 2801
    loss_train 0.12741116921246723
    iteration 2802
    loss_train 0.12739689058741024
    iteration 2803
    loss_train 0.12738261871493456
    iteration 2804
    loss_train 0.12736835358980667
    iteration 2805
    loss_train 0.1273540952067991
    iteration 2806
    loss_train 0.12733984356068986
    iteration 2807
    loss_train 0.12732559864626297
    iteration 2808
    loss_train 0.12731136045830801
    iteration 2809
    loss_train 0.12729712899162046
    iteration 2810
    loss_train 0.12728290424100133
    iteration 2811
    loss_train 0.12726868620125759
    iteration 2812
    loss_train 0.12725447486720157
    iteration 2813
    loss_train 0.12724027023365175
    iteration 2814
    loss_train 0.127226072295432
    iteration 2815
    loss_train 0.1272118810473719
    iteration 2816
    loss_train 0.12719769648430687
    iteration 2817
    loss_train 0.12718351860107782
    iteration 2818
    loss_train 0.12716934739253144
    iteration 2819
    loss_train 0.12715518285351998
    iteration 2820
    loss_train 0.12714102497890153
    iteration 2821
    loss_train 0.12712687376353954
    iteration 2822
    loss_train 0.1271127292023033
    iteration 2823
    loss_train 0.12709859129006762
    iteration 2824
    loss_train 0.12708446002171297
    iteration 2825
    loss_train 0.12707033539212548
    iteration 2826
    loss_train 0.12705621739619674
    iteration 2827
    loss_train 0.12704210602882407
    iteration 2828
    loss_train 0.12702800128491024
    iteration 2829
    loss_train 0.12701390315936373
    iteration 2830
    loss_train 0.12699981164709848
    iteration 2831
    loss_train 0.12698572674303404
    iteration 2832
    loss_train 0.12697164844209552
    iteration 2833
    loss_train 0.12695757673921354
    iteration 2834
    loss_train 0.1269435116293243
    iteration 2835
    loss_train 0.1269294531073694
    iteration 2836
    loss_train 0.12691540116829614
    iteration 2837
    loss_train 0.12690135580705722
    iteration 2838
    loss_train 0.12688731701861086
    iteration 2839
    loss_train 0.12687328479792084
    iteration 2840
    loss_train 0.12685925913995633
    iteration 2841
    loss_train 0.126845240039692
    iteration 2842
    loss_train 0.12683122749210807
    iteration 2843
    loss_train 0.12681722149219007
    iteration 2844
    loss_train 0.12680322203492922
    iteration 2845
    loss_train 0.126789229115322
    iteration 2846
    loss_train 0.12677524272837035
    iteration 2847
    loss_train 0.12676126286908168
    iteration 2848
    loss_train 0.1267472895324689
    iteration 2849
    loss_train 0.1267333227135502
    iteration 2850
    loss_train 0.12671936240734924
    iteration 2851
    loss_train 0.1267054086088951
    iteration 2852
    loss_train 0.1266914613132223
    iteration 2853
    loss_train 0.12667752051537057
    iteration 2854
    loss_train 0.1266635862103852
    iteration 2855
    loss_train 0.1266496583933168
    iteration 2856
    loss_train 0.12663573705922132
    iteration 2857
    loss_train 0.12662182220316007
    iteration 2858
    loss_train 0.12660791382019976
    iteration 2859
    loss_train 0.12659401190541236
    iteration 2860
    loss_train 0.1265801164538752
    iteration 2861
    loss_train 0.126566227460671
    iteration 2862
    loss_train 0.1265523449208877
    iteration 2863
    loss_train 0.12653846882961867
    iteration 2864
    loss_train 0.12652459918196243
    iteration 2865
    loss_train 0.12651073597302295
    iteration 2866
    loss_train 0.12649687919790942
    iteration 2867
    loss_train 0.1264830288517362
    iteration 2868
    loss_train 0.12646918492962322
    iteration 2869
    loss_train 0.12645534742669534
    iteration 2870
    loss_train 0.12644151633808295
    iteration 2871
    loss_train 0.12642769165892148
    iteration 2872
    loss_train 0.12641387338435175
    iteration 2873
    loss_train 0.12640006150951974
    iteration 2874
    loss_train 0.1263862560295767
    iteration 2875
    loss_train 0.12637245693967908
    iteration 2876
    loss_train 0.12635866423498854
    iteration 2877
    loss_train 0.12634487791067198
    iteration 2878
    loss_train 0.12633109796190142
    iteration 2879
    loss_train 0.12631732438385426
    iteration 2880
    loss_train 0.1263035571717128
    iteration 2881
    loss_train 0.12628979632066487
    iteration 2882
    loss_train 0.1262760418259031
    iteration 2883
    loss_train 0.1262622936826255
    iteration 2884
    loss_train 0.12624855188603526
    iteration 2885
    loss_train 0.1262348164313406
    iteration 2886
    loss_train 0.12622108731375498
    iteration 2887
    loss_train 0.1262073645284969
    iteration 2888
    loss_train 0.1261936480707901
    iteration 2889
    loss_train 0.12617993793586335
    iteration 2890
    loss_train 0.12616623411895053
    iteration 2891
    loss_train 0.12615253661529074
    iteration 2892
    loss_train 0.12613884542012807
    iteration 2893
    loss_train 0.12612516052871164
    iteration 2894
    loss_train 0.1261114819362959
    iteration 2895
    loss_train 0.12609780963814013
    iteration 2896
    loss_train 0.12608414362950876
    iteration 2897
    loss_train 0.12607048390567138
    iteration 2898
    loss_train 0.12605683046190247
    iteration 2899
    loss_train 0.12604318329348174
    iteration 2900
    loss_train 0.1260295423956937
    iteration 2901
    loss_train 0.12601590776382818
    iteration 2902
    loss_train 0.1260022793931798
    iteration 2903
    loss_train 0.12598865727904834
    iteration 2904
    loss_train 0.12597504141673857
    iteration 2905
    loss_train 0.12596143180156022
    iteration 2906
    loss_train 0.12594782842882807
    iteration 2907
    loss_train 0.1259342312938619
    iteration 2908
    loss_train 0.1259206403919864
    iteration 2909
    loss_train 0.12590705571853128
    iteration 2910
    loss_train 0.12589347726883127
    iteration 2911
    loss_train 0.12587990503822602
    iteration 2912
    loss_train 0.1258663390220601
    iteration 2913
    loss_train 0.1258527792156832
    iteration 2914
    loss_train 0.12583922561444968
    iteration 2915
    loss_train 0.12582567821371907
    iteration 2916
    loss_train 0.12581213700885577
    iteration 2917
    loss_train 0.12579860199522905
    iteration 2918
    loss_train 0.1257850731682131
    iteration 2919
    loss_train 0.12577155052318714
    iteration 2920
    loss_train 0.12575803405553512
    iteration 2921
    loss_train 0.12574452376064596
    iteration 2922
    loss_train 0.12573101963391362
    iteration 2923
    loss_train 0.1257175216707367
    iteration 2924
    loss_train 0.1257040298665187
    iteration 2925
    loss_train 0.12569054421666825
    iteration 2926
    loss_train 0.12567706471659862
    iteration 2927
    loss_train 0.12566359136172786
    iteration 2928
    loss_train 0.12565012414747911
    iteration 2929
    loss_train 0.1256366630692802
    iteration 2930
    loss_train 0.12562320812256383
    iteration 2931
    loss_train 0.12560975930276758
    iteration 2932
    loss_train 0.12559631660533369
    iteration 2933
    loss_train 0.12558288002570944
    iteration 2934
    loss_train 0.12556944955934674
    iteration 2935
    loss_train 0.12555602520170236
    iteration 2936
    loss_train 0.12554260694823793
    iteration 2937
    loss_train 0.1255291947944198
    iteration 2938
    loss_train 0.1255157887357191
    iteration 2939
    loss_train 0.12550238876761177
    iteration 2940
    loss_train 0.12548899488557852
    iteration 2941
    loss_train 0.12547560708510475
    iteration 2942
    loss_train 0.12546222536168078
    iteration 2943
    loss_train 0.12544884971080145
    iteration 2944
    loss_train 0.12543548012796657
    iteration 2945
    loss_train 0.12542211660868058
    iteration 2946
    loss_train 0.12540875914845254
    iteration 2947
    loss_train 0.12539540774279648
    iteration 2948
    loss_train 0.12538206238723096
    iteration 2949
    loss_train 0.12536872307727934
    iteration 2950
    loss_train 0.12535538980846964
    iteration 2951
    loss_train 0.1253420625763345
    iteration 2952
    loss_train 0.12532874137641153
    iteration 2953
    loss_train 0.12531542620424269
    iteration 2954
    loss_train 0.1253021170553748
    iteration 2955
    loss_train 0.12528881392535943
    iteration 2956
    loss_train 0.12527551680975252
    iteration 2957
    loss_train 0.125262225704115
    iteration 2958
    loss_train 0.1252489406040122
    iteration 2959
    loss_train 0.12523566150501428
    iteration 2960
    loss_train 0.125222388402696
    iteration 2961
    loss_train 0.1252091212926366
    iteration 2962
    loss_train 0.12519586017042014
    iteration 2963
    loss_train 0.12518260503163522
    iteration 2964
    loss_train 0.12516935587187505
    iteration 2965
    loss_train 0.1251561126867375
    iteration 2966
    loss_train 0.12514287547182495
    iteration 2967
    loss_train 0.1251296442227445
    iteration 2968
    loss_train 0.12511641893510772
    iteration 2969
    loss_train 0.12510319960453078
    iteration 2970
    loss_train 0.1250899862266346
    iteration 2971
    loss_train 0.12507677879704437
    iteration 2972
    loss_train 0.1250635773113901
    iteration 2973
    loss_train 0.12505038176530622
    iteration 2974
    loss_train 0.12503719215443182
    iteration 2975
    loss_train 0.12502400847441042
    iteration 2976
    loss_train 0.1250108307208902
    iteration 2977
    loss_train 0.12499765888952365
    iteration 2978
    loss_train 0.12498449297596813
    iteration 2979
    loss_train 0.12497133297588527
    iteration 2980
    loss_train 0.12495817888494125
    iteration 2981
    loss_train 0.12494503069880684
    iteration 2982
    loss_train 0.12493188841315725
    iteration 2983
    loss_train 0.12491875202367221
    iteration 2984
    loss_train 0.12490562152603589
    iteration 2985
    loss_train 0.12489249691593703
    iteration 2986
    loss_train 0.12487937818906886
    iteration 2987
    loss_train 0.12486626534112891
    iteration 2988
    loss_train 0.12485315836781935
    iteration 2989
    loss_train 0.12484005726484675
    iteration 2990
    loss_train 0.12482696202792215
    iteration 2991
    loss_train 0.12481387265276106
    iteration 2992
    loss_train 0.12480078913508334
    iteration 2993
    loss_train 0.12478771147061338
    iteration 2994
    loss_train 0.12477463965508
    iteration 2995
    loss_train 0.12476157368421634
    iteration 2996
    loss_train 0.12474851355376008
    iteration 2997
    loss_train 0.12473545925945324
    iteration 2998
    loss_train 0.12472241079704231
    iteration 2999
    loss_train 0.12470936816227804
    iteration 3000
    loss_train 0.12469633135091582
    iteration 3001
    loss_train 0.1246833003587152
    iteration 3002
    loss_train 0.12467027518144015
    iteration 3003
    loss_train 0.12465725581485912
    iteration 3004
    loss_train 0.12464424225474488
    iteration 3005
    loss_train 0.12463123449687451
    iteration 3006
    loss_train 0.12461823253702957
    iteration 3007
    loss_train 0.12460523637099583
    iteration 3008
    loss_train 0.12459224599456349
    iteration 3009
    loss_train 0.1245792614035271
    iteration 3010
    loss_train 0.1245662825936855
    iteration 3011
    loss_train 0.12455330956084189
    iteration 3012
    loss_train 0.1245403423008038
    iteration 3013
    loss_train 0.12452738080938307
    iteration 3014
    loss_train 0.1245144250823958
    iteration 3015
    loss_train 0.12450147511566247
    iteration 3016
    loss_train 0.12448853090500786
    iteration 3017
    loss_train 0.12447559244626095
    iteration 3018
    loss_train 0.12446265973525518
    iteration 3019
    loss_train 0.12444973276782802
    iteration 3020
    loss_train 0.12443681153982152
    iteration 3021
    loss_train 0.12442389604708179
    iteration 3022
    loss_train 0.12441098628545923
    iteration 3023
    loss_train 0.12439808225080859
    iteration 3024
    loss_train 0.12438518393898881
    iteration 3025
    loss_train 0.12437229134586304
    iteration 3026
    loss_train 0.12435940446729875
    iteration 3027
    loss_train 0.12434652329916769
    iteration 3028
    loss_train 0.12433364783734564
    iteration 3029
    loss_train 0.12432077807771286
    iteration 3030
    loss_train 0.12430791401615364
    iteration 3031
    loss_train 0.1242950556485565
    iteration 3032
    loss_train 0.12428220297081438
    iteration 3033
    loss_train 0.12426935597882413
    iteration 3034
    loss_train 0.124256514668487
    iteration 3035
    loss_train 0.12424367903570828
    iteration 3036
    loss_train 0.12423084907639767
    iteration 3037
    loss_train 0.12421802478646879
    iteration 3038
    loss_train 0.12420520616183965
    iteration 3039
    loss_train 0.12419239319843228
    iteration 3040
    loss_train 0.12417958589217291
    iteration 3041
    loss_train 0.12416678423899201
    iteration 3042
    loss_train 0.12415398823482417
    iteration 3043
    loss_train 0.12414119787560798
    iteration 3044
    loss_train 0.12412841315728641
    iteration 3045
    loss_train 0.1241156340758064
    iteration 3046
    loss_train 0.12410286062711913
    iteration 3047
    loss_train 0.12409009280717978
    iteration 3048
    loss_train 0.12407733061194773
    iteration 3049
    loss_train 0.12406457403738651
    iteration 3050
    loss_train 0.12405182307946364
    iteration 3051
    loss_train 0.12403907773415085
    iteration 3052
    loss_train 0.12402633799742398
    iteration 3053
    loss_train 0.12401360386526282
    iteration 3054
    loss_train 0.12400087533365142
    iteration 3055
    loss_train 0.1239881523985778
    iteration 3056
    loss_train 0.12397543505603412
    iteration 3057
    loss_train 0.12396272330201659
    iteration 3058
    loss_train 0.12395001713252543
    iteration 3059
    loss_train 0.12393731654356503
    iteration 3060
    loss_train 0.12392462153114374
    iteration 3061
    loss_train 0.12391193209127398
    iteration 3062
    loss_train 0.12389924821997225
    iteration 3063
    loss_train 0.12388656991325907
    iteration 3064
    loss_train 0.12387389716715899
    iteration 3065
    loss_train 0.12386122997770058
    iteration 3066
    loss_train 0.12384856834091643
    iteration 3067
    loss_train 0.12383591225284318
    iteration 3068
    loss_train 0.12382326170952146
    iteration 3069
    loss_train 0.12381061670699589
    iteration 3070
    loss_train 0.12379797724131514
    iteration 3071
    loss_train 0.12378534330853182
    iteration 3072
    loss_train 0.12377271490470253
    iteration 3073
    loss_train 0.12376009202588795
    iteration 3074
    loss_train 0.12374747466815263
    iteration 3075
    loss_train 0.12373486282756516
    iteration 3076
    loss_train 0.12372225650019801
    iteration 3077
    loss_train 0.12370965568212777
    iteration 3078
    loss_train 0.12369706036943486
    iteration 3079
    loss_train 0.1236844705582037
    iteration 3080
    loss_train 0.12367188624452265
    iteration 3081
    loss_train 0.12365930742448403
    iteration 3082
    loss_train 0.12364673409418406
    iteration 3083
    loss_train 0.12363416624972294
    iteration 3084
    loss_train 0.12362160388720479
    iteration 3085
    loss_train 0.12360904700273763
    iteration 3086
    loss_train 0.12359649559243342
    iteration 3087
    loss_train 0.12358394965240799
    iteration 3088
    loss_train 0.12357140917878116
    iteration 3089
    loss_train 0.12355887416767655
    iteration 3090
    loss_train 0.12354634461522176
    iteration 3091
    loss_train 0.12353382051754824
    iteration 3092
    loss_train 0.12352130187079133
    iteration 3093
    loss_train 0.1235087886710903
    iteration 3094
    loss_train 0.12349628091458821
    iteration 3095
    loss_train 0.1234837785974321
    iteration 3096
    loss_train 0.12347128171577275
    iteration 3097
    loss_train 0.12345879026576485
    iteration 3098
    loss_train 0.1234463042435671
    iteration 3099
    loss_train 0.12343382364534175
    iteration 3100
    loss_train 0.12342134846725519
    iteration 3101
    loss_train 0.12340887870547744
    iteration 3102
    loss_train 0.12339641435618244
    iteration 3103
    loss_train 0.12338395541554802
    iteration 3104
    loss_train 0.12337150187975572
    iteration 3105
    loss_train 0.12335905374499105
    iteration 3106
    loss_train 0.12334661100744314
    iteration 3107
    loss_train 0.1233341736633051
    iteration 3108
    loss_train 0.12332174170877373
    iteration 3109
    loss_train 0.12330931514004975
    iteration 3110
    loss_train 0.12329689395333758
    iteration 3111
    loss_train 0.12328447814484543
    iteration 3112
    loss_train 0.12327206771078543
    iteration 3113
    loss_train 0.1232596626473732
    iteration 3114
    loss_train 0.12324726295082851
    iteration 3115
    loss_train 0.12323486861737469
    iteration 3116
    loss_train 0.12322247964323879
    iteration 3117
    loss_train 0.12321009602465174
    iteration 3118
    loss_train 0.12319771775784821
    iteration 3119
    loss_train 0.12318534483906655
    iteration 3120
    loss_train 0.1231729772645489
    iteration 3121
    loss_train 0.12316061503054122
    iteration 3122
    loss_train 0.12314825813329307
    iteration 3123
    loss_train 0.1231359065690578
    iteration 3124
    loss_train 0.12312356033409254
    iteration 3125
    loss_train 0.12311121942465807
    iteration 3126
    loss_train 0.12309888383701892
    iteration 3127
    loss_train 0.12308655356744332
    iteration 3128
    loss_train 0.1230742286122032
    iteration 3129
    loss_train 0.12306190896757425
    iteration 3130
    loss_train 0.1230495946298358
    iteration 3131
    loss_train 0.1230372855952709
    iteration 3132
    loss_train 0.1230249818601663
    iteration 3133
    loss_train 0.12301268342081235
    iteration 3134
    loss_train 0.12300039027350324
    iteration 3135
    loss_train 0.12298810241453668
    iteration 3136
    loss_train 0.12297581984021409
    iteration 3137
    loss_train 0.12296354254684062
    iteration 3138
    loss_train 0.12295127053072502
    iteration 3139
    loss_train 0.12293900378817973
    iteration 3140
    loss_train 0.12292674231552088
    iteration 3141
    loss_train 0.12291448610906804
    iteration 3142
    loss_train 0.12290223516514467
    iteration 3143
    loss_train 0.12288998948007773
    iteration 3144
    loss_train 0.12287774905019791
    iteration 3145
    loss_train 0.12286551387183944
    iteration 3146
    loss_train 0.12285328394134022
    iteration 3147
    loss_train 0.12284105925504167
    iteration 3148
    loss_train 0.12282883980928896
    iteration 3149
    loss_train 0.12281662560043088
    iteration 3150
    loss_train 0.12280441662481964
    iteration 3151
    loss_train 0.12279221287881126
    iteration 3152
    loss_train 0.12278001435876519
    iteration 3153
    loss_train 0.12276782106104456
    iteration 3154
    loss_train 0.12275563298201608
    iteration 3155
    loss_train 0.12274345011805005
    iteration 3156
    loss_train 0.12273127246552028
    iteration 3157
    loss_train 0.12271910002080423
    iteration 3158
    loss_train 0.12270693278028284
    iteration 3159
    loss_train 0.12269477074034073
    iteration 3160
    loss_train 0.12268261389736601
    iteration 3161
    loss_train 0.12267046224775031
    iteration 3162
    loss_train 0.12265831578788886
    iteration 3163
    loss_train 0.12264617451418047
    iteration 3164
    loss_train 0.12263403842302735
    iteration 3165
    loss_train 0.12262190751083536
    iteration 3166
    loss_train 0.12260978177401394
    iteration 3167
    loss_train 0.12259766120897592
    iteration 3168
    loss_train 0.1225855458121377
    iteration 3169
    loss_train 0.12257343557991929
    iteration 3170
    loss_train 0.12256133050874404
    iteration 3171
    loss_train 0.12254923059503896
    iteration 3172
    loss_train 0.12253713583523451
    iteration 3173
    loss_train 0.1225250462257646
    iteration 3174
    loss_train 0.12251296176306675
    iteration 3175
    loss_train 0.12250088244358182
    iteration 3176
    loss_train 0.12248880826375429
    iteration 3177
    loss_train 0.12247673922003205
    iteration 3178
    loss_train 0.12246467530886647
    iteration 3179
    loss_train 0.12245261652671244
    iteration 3180
    loss_train 0.12244056287002823
    iteration 3181
    loss_train 0.12242851433527568
    iteration 3182
    loss_train 0.12241647091892
    iteration 3183
    loss_train 0.12240443261742996
    iteration 3184
    loss_train 0.12239239942727767
    iteration 3185
    loss_train 0.12238037134493863
    iteration 3186
    loss_train 0.12236834836689203
    iteration 3187
    loss_train 0.1223563304896203
    iteration 3188
    loss_train 0.1223443177096093
    iteration 3189
    loss_train 0.12233231002334838
    iteration 3190
    loss_train 0.12232030742733034
    iteration 3191
    loss_train 0.12230830991805136
    iteration 3192
    loss_train 0.12229631749201096
    iteration 3193
    loss_train 0.12228433014571223
    iteration 3194
    loss_train 0.12227234787566159
    iteration 3195
    loss_train 0.12226037067836876
    iteration 3196
    loss_train 0.12224839855034701
    iteration 3197
    loss_train 0.12223643148811297
    iteration 3198
    loss_train 0.12222446948818659
    iteration 3199
    loss_train 0.1222125125470913
    iteration 3200
    loss_train 0.12220056066135379
    iteration 3201
    loss_train 0.12218861382750422
    iteration 3202
    loss_train 0.12217667204207619
    iteration 3203
    loss_train 0.12216473530160646
    iteration 3204
    loss_train 0.12215280360263531
    iteration 3205
    loss_train 0.12214087694170632
    iteration 3206
    loss_train 0.12212895531536652
    iteration 3207
    loss_train 0.12211703872016609
    iteration 3208
    loss_train 0.12210512715265874
    iteration 3209
    loss_train 0.12209322060940155
    iteration 3210
    loss_train 0.12208131908695473
    iteration 3211
    loss_train 0.12206942258188196
    iteration 3212
    loss_train 0.1220575310907503
    iteration 3213
    loss_train 0.12204564461013004
    iteration 3214
    loss_train 0.12203376313659478
    iteration 3215
    loss_train 0.12202188666672152
    iteration 3216
    loss_train 0.12201001519709051
    iteration 3217
    loss_train 0.12199814872428535
    iteration 3218
    loss_train 0.12198628724489288
    iteration 3219
    loss_train 0.12197443075550335
    iteration 3220
    loss_train 0.12196257925271016
    iteration 3221
    loss_train 0.12195073273311016
    iteration 3222
    loss_train 0.12193889119330335
    iteration 3223
    loss_train 0.1219270546298931
    iteration 3224
    loss_train 0.12191522303948601
    iteration 3225
    loss_train 0.12190339641869197
    iteration 3226
    loss_train 0.1218915747641242
    iteration 3227
    loss_train 0.12187975807239905
    iteration 3228
    loss_train 0.12186794634013629
    iteration 3229
    loss_train 0.12185613956395883
    iteration 3230
    loss_train 0.1218443377404929
    iteration 3231
    loss_train 0.121832540866368
    iteration 3232
    loss_train 0.12182074893821673
    iteration 3233
    loss_train 0.12180896195267514
    iteration 3234
    loss_train 0.12179717990638238
    iteration 3235
    loss_train 0.12178540279598087
    iteration 3236
    loss_train 0.12177363061811625
    iteration 3237
    loss_train 0.12176186336943744
    iteration 3238
    loss_train 0.12175010104659649
    iteration 3239
    loss_train 0.12173834364624875
    iteration 3240
    loss_train 0.12172659116505273
    iteration 3241
    loss_train 0.12171484359967019
    iteration 3242
    loss_train 0.12170310094676608
    iteration 3243
    loss_train 0.12169136320300851
    iteration 3244
    loss_train 0.12167963036506886
    iteration 3245
    loss_train 0.12166790242962167
    iteration 3246
    loss_train 0.12165617939334467
    iteration 3247
    loss_train 0.12164446125291878
    iteration 3248
    loss_train 0.12163274800502807
    iteration 3249
    loss_train 0.12162103964635983
    iteration 3250
    loss_train 0.12160933617360456
    iteration 3251
    loss_train 0.12159763758345576
    iteration 3252
    loss_train 0.12158594387261033
    iteration 3253
    loss_train 0.12157425503776816
    iteration 3254
    loss_train 0.12156257107563244
    iteration 3255
    loss_train 0.1215508919829093
    iteration 3256
    loss_train 0.1215392177563082
    iteration 3257
    loss_train 0.12152754839254172
    iteration 3258
    loss_train 0.12151588388832554
    iteration 3259
    loss_train 0.12150422424037852
    iteration 3260
    loss_train 0.1214925694454226
    iteration 3261
    loss_train 0.12148091950018289
    iteration 3262
    loss_train 0.1214692744013876
    iteration 3263
    loss_train 0.12145763414576811
    iteration 3264
    loss_train 0.12144599873005885
    iteration 3265
    loss_train 0.12143436815099745
    iteration 3266
    loss_train 0.12142274240532457
    iteration 3267
    loss_train 0.12141112148978404
    iteration 3268
    loss_train 0.12139950540112275
    iteration 3269
    loss_train 0.12138789413609068
    iteration 3270
    loss_train 0.121376287691441
    iteration 3271
    loss_train 0.12136468606392982
    iteration 3272
    loss_train 0.12135308925031647
    iteration 3273
    loss_train 0.12134149724736328
    iteration 3274
    loss_train 0.12132991005183572
    iteration 3275
    loss_train 0.1213183276605023
    iteration 3276
    loss_train 0.12130675007013464
    iteration 3277
    loss_train 0.12129517727750733
    iteration 3278
    loss_train 0.1212836092793982
    iteration 3279
    loss_train 0.12127204607258793
    iteration 3280
    loss_train 0.12126048765386045
    iteration 3281
    loss_train 0.12124893402000261
    iteration 3282
    loss_train 0.12123738516780432
    iteration 3283
    loss_train 0.12122584109405869
    iteration 3284
    loss_train 0.12121430179556164
    iteration 3285
    loss_train 0.12120276726911232
    iteration 3286
    loss_train 0.12119123751151276
    iteration 3287
    loss_train 0.12117971251956822
    iteration 3288
    loss_train 0.12116819229008673
    iteration 3289
    loss_train 0.12115667681987957
    iteration 3290
    loss_train 0.12114516610576095
    iteration 3291
    loss_train 0.12113366014454806
    iteration 3292
    loss_train 0.12112215893306112
    iteration 3293
    loss_train 0.1211106624681234
    iteration 3294
    loss_train 0.12109917074656115
    iteration 3295
    loss_train 0.12108768376520367
    iteration 3296
    loss_train 0.12107620152088314
    iteration 3297
    loss_train 0.12106472401043483
    iteration 3298
    loss_train 0.12105325123069696
    iteration 3299
    loss_train 0.12104178317851075
    iteration 3300
    loss_train 0.12103031985072041
    iteration 3301
    loss_train 0.12101886124417316
    iteration 3302
    loss_train 0.12100740735571906
    iteration 3303
    loss_train 0.12099595818221133
    iteration 3304
    loss_train 0.12098451372050599
    iteration 3305
    loss_train 0.12097307396746217
    iteration 3306
    loss_train 0.12096163891994181
    iteration 3307
    loss_train 0.12095020857480995
    iteration 3308
    loss_train 0.1209387829289345
    iteration 3309
    loss_train 0.12092736197918628
    iteration 3310
    loss_train 0.12091594572243926
    iteration 3311
    loss_train 0.12090453415557006
    iteration 3312
    loss_train 0.12089312727545848
    iteration 3313
    loss_train 0.1208817250789871
    iteration 3314
    loss_train 0.12087032756304154
    iteration 3315
    loss_train 0.12085893472451023
    iteration 3316
    loss_train 0.12084754656028467
    iteration 3317
    loss_train 0.12083616306725917
    iteration 3318
    loss_train 0.12082478424233102
    iteration 3319
    loss_train 0.12081341008240035
    iteration 3320
    loss_train 0.1208020405843703
    iteration 3321
    loss_train 0.1207906757451468
    iteration 3322
    loss_train 0.12077931556163882
    iteration 3323
    loss_train 0.12076796003075808
    iteration 3324
    loss_train 0.12075660914941937
    iteration 3325
    loss_train 0.12074526291454016
    iteration 3326
    loss_train 0.12073392132304095
    iteration 3327
    loss_train 0.12072258437184516
    iteration 3328
    loss_train 0.12071125205787903
    iteration 3329
    loss_train 0.12069992437807157
    iteration 3330
    loss_train 0.12068860132935483
    iteration 3331
    loss_train 0.12067728290866368
    iteration 3332
    loss_train 0.12066596911293585
    iteration 3333
    loss_train 0.12065465993911194
    iteration 3334
    loss_train 0.12064335538413537
    iteration 3335
    loss_train 0.12063205544495247
    iteration 3336
    loss_train 0.12062076011851244
    iteration 3337
    loss_train 0.12060946940176726
    iteration 3338
    loss_train 0.12059818329167177
    iteration 3339
    loss_train 0.12058690178518369
    iteration 3340
    loss_train 0.12057562487926361
    iteration 3341
    loss_train 0.12056435257087487
    iteration 3342
    loss_train 0.12055308485698364
    iteration 3343
    loss_train 0.12054182173455906
    iteration 3344
    loss_train 0.12053056320057293
    iteration 3345
    loss_train 0.12051930925199995
    iteration 3346
    loss_train 0.12050805988581767
    iteration 3347
    loss_train 0.12049681509900635
    iteration 3348
    loss_train 0.12048557488854918
    iteration 3349
    loss_train 0.12047433925143206
    iteration 3350
    loss_train 0.12046310818464374
    iteration 3351
    loss_train 0.12045188168517584
    iteration 3352
    loss_train 0.1204406597500227
    iteration 3353
    loss_train 0.12042944237618138
    iteration 3354
    loss_train 0.12041822956065189
    iteration 3355
    loss_train 0.12040702130043696
    iteration 3356
    loss_train 0.1203958175925421
    iteration 3357
    loss_train 0.12038461843397551
    iteration 3358
    loss_train 0.12037342382174841
    iteration 3359
    loss_train 0.12036223375287453
    iteration 3360
    loss_train 0.12035104822437055
    iteration 3361
    loss_train 0.12033986723325582
    iteration 3362
    loss_train 0.12032869077655252
    iteration 3363
    loss_train 0.12031751885128554
    iteration 3364
    loss_train 0.12030635145448258
    iteration 3365
    loss_train 0.12029518858317399
    iteration 3366
    loss_train 0.12028403023439302
    iteration 3367
    loss_train 0.12027287640517556
    iteration 3368
    loss_train 0.12026172709256032
    iteration 3369
    loss_train 0.12025058229358862
    iteration 3370
    loss_train 0.12023944200530469
    iteration 3371
    loss_train 0.12022830622475539
    iteration 3372
    loss_train 0.12021717494899035
    iteration 3373
    loss_train 0.12020604817506182
    iteration 3374
    loss_train 0.12019492590002502
    iteration 3375
    loss_train 0.12018380812093761
    iteration 3376
    loss_train 0.12017269483486012
    iteration 3377
    loss_train 0.12016158603885582
    iteration 3378
    loss_train 0.12015048172999061
    iteration 3379
    loss_train 0.12013938190533316
    iteration 3380
    loss_train 0.12012828656195476
    iteration 3381
    loss_train 0.1201171956969295
    iteration 3382
    loss_train 0.12010610930733413
    iteration 3383
    loss_train 0.12009502739024805
    iteration 3384
    loss_train 0.12008394994275345
    iteration 3385
    loss_train 0.12007287696193515
    iteration 3386
    loss_train 0.12006180844488057
    iteration 3387
    loss_train 0.12005074438868006
    iteration 3388
    loss_train 0.12003968479042634
    iteration 3389
    loss_train 0.120028629647215
    iteration 3390
    loss_train 0.12001757895614433
    iteration 3391
    loss_train 0.12000653271431515
    iteration 3392
    loss_train 0.11999549091883102
    iteration 3393
    loss_train 0.1199844535667982
    iteration 3394
    loss_train 0.11997342065532551
    iteration 3395
    loss_train 0.11996239218152452
    iteration 3396
    loss_train 0.11995136814250944
    iteration 3397
    loss_train 0.11994034853539708
    iteration 3398
    loss_train 0.11992933335730692
    iteration 3399
    loss_train 0.11991832260536112
    iteration 3400
    loss_train 0.11990731627668437
    iteration 3401
    loss_train 0.11989631436840423
    iteration 3402
    loss_train 0.1198853168776506
    iteration 3403
    loss_train 0.11987432380155624
    iteration 3404
    loss_train 0.11986333513725644
    iteration 3405
    loss_train 0.1198523508818891
    iteration 3406
    loss_train 0.11984137103259479
    iteration 3407
    loss_train 0.11983039558651667
    iteration 3408
    loss_train 0.11981942454080054
    iteration 3409
    loss_train 0.1198084578925948
    iteration 3410
    loss_train 0.11979749563905041
    iteration 3411
    loss_train 0.11978653777732107
    iteration 3412
    loss_train 0.11977558430456296
    iteration 3413
    loss_train 0.11976463521793486
    iteration 3414
    loss_train 0.11975369051459822
    iteration 3415
    loss_train 0.11974275019171704
    iteration 3416
    loss_train 0.11973181424645789
    iteration 3417
    loss_train 0.11972088267599004
    iteration 3418
    loss_train 0.11970995547748514
    iteration 3419
    loss_train 0.11969903264811764
    iteration 3420
    loss_train 0.11968811418506446
    iteration 3421
    loss_train 0.11967720008550503
    iteration 3422
    loss_train 0.11966629034662155
    iteration 3423
    loss_train 0.11965538496559858
    iteration 3424
    loss_train 0.11964448393962333
    iteration 3425
    loss_train 0.11963358726588565
    iteration 3426
    loss_train 0.11962269494157786
    iteration 3427
    loss_train 0.11961180696389476
    iteration 3428
    loss_train 0.11960092333003391
    iteration 3429
    loss_train 0.11959004403719523
    iteration 3430
    loss_train 0.11957916908258134
    iteration 3431
    loss_train 0.11956829846339731
    iteration 3432
    loss_train 0.11955743217685076
    iteration 3433
    loss_train 0.11954657022015187
    iteration 3434
    loss_train 0.1195357125905133
    iteration 3435
    loss_train 0.11952485928515039
    iteration 3436
    loss_train 0.11951401030128088
    iteration 3437
    loss_train 0.11950316563612501
    iteration 3438
    loss_train 0.11949232528690566
    iteration 3439
    loss_train 0.11948148925084819
    iteration 3440
    loss_train 0.11947065752518042
    iteration 3441
    loss_train 0.1194598301071327
    iteration 3442
    loss_train 0.119449006993938
    iteration 3443
    loss_train 0.11943818818283167
    iteration 3444
    loss_train 0.11942737367105159
    iteration 3445
    loss_train 0.11941656345583819
    iteration 3446
    loss_train 0.11940575753443444
    iteration 3447
    loss_train 0.11939495590408565
    iteration 3448
    loss_train 0.11938415856203975
    iteration 3449
    loss_train 0.11937336550554715
    iteration 3450
    loss_train 0.11936257673186072
    iteration 3451
    loss_train 0.1193517922382358
    iteration 3452
    loss_train 0.11934101202193026
    iteration 3453
    loss_train 0.11933023608020439
    iteration 3454
    loss_train 0.11931946441032105
    iteration 3455
    loss_train 0.11930869700954551
    iteration 3456
    loss_train 0.11929793387514538
    iteration 3457
    loss_train 0.11928717500439107
    iteration 3458
    loss_train 0.11927642039455513
    iteration 3459
    loss_train 0.11926567004291276
    iteration 3460
    loss_train 0.11925492394674155
    iteration 3461
    loss_train 0.11924418210332154
    iteration 3462
    loss_train 0.11923344450993523
    iteration 3463
    loss_train 0.11922271116386758
    iteration 3464
    loss_train 0.119211982062406
    iteration 3465
    loss_train 0.11920125720284043
    iteration 3466
    loss_train 0.11919053658246298
    iteration 3467
    loss_train 0.11917982019856857
    iteration 3468
    loss_train 0.11916910804845424
    iteration 3469
    loss_train 0.11915840012941964
    iteration 3470
    loss_train 0.11914769643876684
    iteration 3471
    loss_train 0.11913699697380022
    iteration 3472
    loss_train 0.11912630173182667
    iteration 3473
    loss_train 0.11911561071015556
    iteration 3474
    loss_train 0.11910492390609857
    iteration 3475
    loss_train 0.11909424131696983
    iteration 3476
    loss_train 0.11908356294008593
    iteration 3477
    loss_train 0.11907288877276583
    iteration 3478
    loss_train 0.11906221881233085
    iteration 3479
    loss_train 0.1190515530561048
    iteration 3480
    loss_train 0.1190408915014139
    iteration 3481
    loss_train 0.11903023414558667
    iteration 3482
    loss_train 0.11901958098595412
    iteration 3483
    loss_train 0.11900893201984956
    iteration 3484
    loss_train 0.1189982872446088
    iteration 3485
    loss_train 0.11898764665756999
    iteration 3486
    loss_train 0.11897701025607363
    iteration 3487
    loss_train 0.11896637803746261
    iteration 3488
    loss_train 0.1189557499990823
    iteration 3489
    loss_train 0.11894512613828029
    iteration 3490
    loss_train 0.11893450645240669
    iteration 3491
    loss_train 0.11892389093881389
    iteration 3492
    loss_train 0.11891327959485665
    iteration 3493
    loss_train 0.11890267241789211
    iteration 3494
    loss_train 0.11889206940527984
    iteration 3495
    loss_train 0.11888147055438167
    iteration 3496
    loss_train 0.11887087586256183
    iteration 3497
    loss_train 0.1188602853271869
    iteration 3498
    loss_train 0.11884969894562585
    iteration 3499
    loss_train 0.11883911671524992
    iteration 3500
    loss_train 0.11882853863343278
    iteration 3501
    loss_train 0.11881796469755034
    iteration 3502
    loss_train 0.118807394904981
    iteration 3503
    loss_train 0.11879682925310539
    iteration 3504
    loss_train 0.11878626773930646
    iteration 3505
    loss_train 0.11877571036096952
    iteration 3506
    loss_train 0.11876515711548229
    iteration 3507
    loss_train 0.11875460800023477
    iteration 3508
    loss_train 0.11874406301261917
    iteration 3509
    loss_train 0.11873352215003014
    iteration 3510
    loss_train 0.11872298540986465
    iteration 3511
    loss_train 0.11871245278952197
    iteration 3512
    loss_train 0.11870192428640362
    iteration 3513
    loss_train 0.11869139989791362
    iteration 3514
    loss_train 0.11868087962145801
    iteration 3515
    loss_train 0.11867036345444536
    iteration 3516
    loss_train 0.1186598513942865
    iteration 3517
    loss_train 0.11864934343839448
    iteration 3518
    loss_train 0.11863883958418471
    iteration 3519
    loss_train 0.11862833982907496
    iteration 3520
    loss_train 0.11861784417048511
    iteration 3521
    loss_train 0.11860735260583753
    iteration 3522
    loss_train 0.11859686513255677
    iteration 3523
    loss_train 0.11858638174806965
    iteration 3524
    loss_train 0.11857590244980533
    iteration 3525
    loss_train 0.1185654272351952
    iteration 3526
    loss_train 0.11855495610167296
    iteration 3527
    loss_train 0.1185444890466746
    iteration 3528
    loss_train 0.11853402606763828
    iteration 3529
    loss_train 0.11852356716200463
    iteration 3530
    loss_train 0.11851311232721633
    iteration 3531
    loss_train 0.11850266156071841
    iteration 3532
    loss_train 0.11849221485995823
    iteration 3533
    loss_train 0.11848177222238526
    iteration 3534
    loss_train 0.11847133364545134
    iteration 3535
    loss_train 0.11846089912661055
    iteration 3536
    loss_train 0.11845046866331922
    iteration 3537
    loss_train 0.11844004225303587
    iteration 3538
    loss_train 0.1184296198932213
    iteration 3539
    loss_train 0.11841920158133858
    iteration 3540
    loss_train 0.11840878731485296
    iteration 3541
    loss_train 0.11839837709123202
    iteration 3542
    loss_train 0.11838797090794552
    iteration 3543
    loss_train 0.11837756876246538
    iteration 3544
    loss_train 0.11836717065226589
    iteration 3545
    loss_train 0.11835677657482349
    iteration 3546
    loss_train 0.11834638652761684
    iteration 3547
    loss_train 0.11833600050812686
    iteration 3548
    loss_train 0.11832561851383658
    iteration 3549
    loss_train 0.11831524054223148
    iteration 3550
    loss_train 0.11830486659079902
    iteration 3551
    loss_train 0.11829449665702894
    iteration 3552
    loss_train 0.11828413073841322
    iteration 3553
    loss_train 0.11827376883244607
    iteration 3554
    loss_train 0.11826341093662385
    iteration 3555
    loss_train 0.11825305704844515
    iteration 3556
    loss_train 0.11824270716541072
    iteration 3557
    loss_train 0.1182323612850236
    iteration 3558
    loss_train 0.11822201940478888
    iteration 3559
    loss_train 0.11821168152221401
    iteration 3560
    loss_train 0.11820134763480845
    iteration 3561
    loss_train 0.118191017740084
    iteration 3562
    loss_train 0.11818069183555456
    iteration 3563
    loss_train 0.11817036991873628
    iteration 3564
    loss_train 0.1181600519871474
    iteration 3565
    loss_train 0.11814973803830837
    iteration 3566
    loss_train 0.11813942806974184
    iteration 3567
    loss_train 0.11812912207897265
    iteration 3568
    loss_train 0.11811882006352774
    iteration 3569
    loss_train 0.11810852202093626
    iteration 3570
    loss_train 0.11809822794872954
    iteration 3571
    loss_train 0.118087937844441
    iteration 3572
    loss_train 0.11807765170560627
    iteration 3573
    loss_train 0.11806736952976318
    iteration 3574
    loss_train 0.11805709131445166
    iteration 3575
    loss_train 0.11804681705721379
    iteration 3576
    loss_train 0.11803654675559377
    iteration 3577
    loss_train 0.11802628040713806
    iteration 3578
    loss_train 0.11801601800939517
    iteration 3579
    loss_train 0.11800575955991574
    iteration 3580
    loss_train 0.11799550505625264
    iteration 3581
    loss_train 0.11798525449596076
    iteration 3582
    loss_train 0.11797500787659719
    iteration 3583
    loss_train 0.11796476519572119
    iteration 3584
    loss_train 0.11795452645089412
    iteration 3585
    loss_train 0.11794429163967944
    iteration 3586
    loss_train 0.11793406075964269
    iteration 3587
    loss_train 0.11792383380835167
    iteration 3588
    loss_train 0.11791361078337617
    iteration 3589
    loss_train 0.11790339168228817
    iteration 3590
    loss_train 0.11789317650266179
    iteration 3591
    loss_train 0.11788296524207316
    iteration 3592
    loss_train 0.11787275789810063
    iteration 3593
    loss_train 0.11786255446832454
    iteration 3594
    loss_train 0.11785235495032745
    iteration 3595
    loss_train 0.11784215934169402
    iteration 3596
    loss_train 0.1178319676400109
    iteration 3597
    loss_train 0.11782177984286692
    iteration 3598
    loss_train 0.11781159594785305
    iteration 3599
    loss_train 0.11780141595256222
    iteration 3600
    loss_train 0.11779123985458957
    iteration 3601
    loss_train 0.11778106765153228
    iteration 3602
    loss_train 0.11777089934098962
    iteration 3603
    loss_train 0.117760734920563
    iteration 3604
    loss_train 0.11775057438785581
    iteration 3605
    loss_train 0.11774041774047356
    iteration 3606
    loss_train 0.1177302649760239
    iteration 3607
    loss_train 0.11772011609211645
    iteration 3608
    loss_train 0.11770997108636301
    iteration 3609
    loss_train 0.11769982995637739
    iteration 3610
    loss_train 0.11768969269977543
    iteration 3611
    loss_train 0.11767955931417512
    iteration 3612
    loss_train 0.11766942979719645
    iteration 3613
    loss_train 0.11765930414646154
    iteration 3614
    loss_train 0.11764918235959447
    iteration 3615
    loss_train 0.11763906443422148
    iteration 3616
    loss_train 0.11762895036797077
    iteration 3617
    loss_train 0.11761884015847265
    iteration 3618
    loss_train 0.1176087338033595
    iteration 3619
    loss_train 0.11759863130026567
    iteration 3620
    loss_train 0.1175885326468276
    iteration 3621
    loss_train 0.11757843784068382
    iteration 3622
    loss_train 0.11756834687947476
    iteration 3623
    loss_train 0.11755825976084305
    iteration 3624
    loss_train 0.11754817648243322
    iteration 3625
    loss_train 0.11753809704189198
    iteration 3626
    loss_train 0.11752802143686795
    iteration 3627
    loss_train 0.1175179496650118
    iteration 3628
    loss_train 0.11750788172397629
    iteration 3629
    loss_train 0.11749781761141607
    iteration 3630
    loss_train 0.117487757324988
    iteration 3631
    loss_train 0.11747770086235078
    iteration 3632
    loss_train 0.11746764822116523
    iteration 3633
    loss_train 0.1174575993990942
    iteration 3634
    loss_train 0.11744755439380247
    iteration 3635
    loss_train 0.11743751320295688
    iteration 3636
    loss_train 0.11742747582422627
    iteration 3637
    loss_train 0.11741744225528153
    iteration 3638
    loss_train 0.11740741249379544
    iteration 3639
    loss_train 0.11739738653744289
    iteration 3640
    loss_train 0.11738736438390074
    iteration 3641
    loss_train 0.11737734603084785
    iteration 3642
    loss_train 0.11736733147596505
    iteration 3643
    loss_train 0.11735732071693514
    iteration 3644
    loss_train 0.11734731375144299
    iteration 3645
    loss_train 0.11733731057717547
    iteration 3646
    loss_train 0.11732731119182128
    iteration 3647
    loss_train 0.11731731559307122
    iteration 3648
    loss_train 0.1173073237786181
    iteration 3649
    loss_train 0.11729733574615667
    iteration 3650
    loss_train 0.11728735149338364
    iteration 3651
    loss_train 0.11727737101799764
    iteration 3652
    loss_train 0.11726739431769946
    iteration 3653
    loss_train 0.11725742139019164
    iteration 3654
    loss_train 0.11724745223317887
    iteration 3655
    loss_train 0.11723748684436766
    iteration 3656
    loss_train 0.11722752522146655
    iteration 3657
    loss_train 0.11721756736218608
    iteration 3658
    loss_train 0.11720761326423869
    iteration 3659
    loss_train 0.11719766292533881
    iteration 3660
    loss_train 0.11718771634320277
    iteration 3661
    loss_train 0.11717777351554896
    iteration 3662
    loss_train 0.11716783444009761
    iteration 3663
    loss_train 0.11715789911457092
    iteration 3664
    loss_train 0.11714796753669314
    iteration 3665
    loss_train 0.1171380397041903
    iteration 3666
    loss_train 0.11712811561479053
    iteration 3667
    loss_train 0.11711819526622373
    iteration 3668
    loss_train 0.11710827865622195
    iteration 3669
    loss_train 0.11709836578251899
    iteration 3670
    loss_train 0.11708845664285066
    iteration 3671
    loss_train 0.11707855123495468
    iteration 3672
    loss_train 0.11706864955657076
    iteration 3673
    loss_train 0.11705875160544044
    iteration 3674
    loss_train 0.11704885737930727
    iteration 3675
    loss_train 0.11703896687591667
    iteration 3676
    loss_train 0.11702908009301599
    iteration 3677
    loss_train 0.1170191970283545
    iteration 3678
    loss_train 0.11700931767968342
    iteration 3679
    loss_train 0.11699944204475583
    iteration 3680
    loss_train 0.1169895701213268
    iteration 3681
    loss_train 0.11697970190715318
    iteration 3682
    loss_train 0.11696983739999384
    iteration 3683
    loss_train 0.11695997659760955
    iteration 3684
    loss_train 0.11695011949776293
    iteration 3685
    loss_train 0.11694026609821849
    iteration 3686
    loss_train 0.1169304163967428
    iteration 3687
    loss_train 0.11692057039110404
    iteration 3688
    loss_train 0.11691072807907253
    iteration 3689
    loss_train 0.11690088945842043
    iteration 3690
    loss_train 0.11689105452692168
    iteration 3691
    loss_train 0.11688122328235226
    iteration 3692
    loss_train 0.11687139572248996
    iteration 3693
    loss_train 0.11686157184511442
    iteration 3694
    loss_train 0.11685175164800726
    iteration 3695
    loss_train 0.11684193512895187
    iteration 3696
    loss_train 0.1168321222857336
    iteration 3697
    loss_train 0.11682231311613965
    iteration 3698
    loss_train 0.11681250761795911
    iteration 3699
    loss_train 0.11680270578898289
    iteration 3700
    loss_train 0.1167929076270038
    iteration 3701
    loss_train 0.11678311312981661
    iteration 3702
    loss_train 0.11677332229521778
    iteration 3703
    loss_train 0.11676353512100571
    iteration 3704
    loss_train 0.11675375160498078
    iteration 3705
    loss_train 0.11674397174494501
    iteration 3706
    loss_train 0.11673419553870247
    iteration 3707
    loss_train 0.116724422984059
    iteration 3708
    loss_train 0.11671465407882224
    iteration 3709
    loss_train 0.11670488882080185
    iteration 3710
    loss_train 0.11669512720780915
    iteration 3711
    loss_train 0.11668536923765743
    iteration 3712
    loss_train 0.11667561490816178
    iteration 3713
    loss_train 0.11666586421713913
    iteration 3714
    loss_train 0.11665611716240827
    iteration 3715
    loss_train 0.1166463737417898
    iteration 3716
    loss_train 0.11663663395310622
    iteration 3717
    loss_train 0.1166268977941818
    iteration 3718
    loss_train 0.11661716526284266
    iteration 3719
    loss_train 0.11660743635691678
    iteration 3720
    loss_train 0.11659771107423397
    iteration 3721
    loss_train 0.11658798941262577
    iteration 3722
    loss_train 0.11657827136992567
    iteration 3723
    loss_train 0.11656855694396895
    iteration 3724
    loss_train 0.11655884613259267
    iteration 3725
    loss_train 0.11654913893363573
    iteration 3726
    loss_train 0.11653943534493891
    iteration 3727
    loss_train 0.11652973536434467
    iteration 3728
    loss_train 0.11652003898969737
    iteration 3729
    loss_train 0.11651034621884324
    iteration 3730
    loss_train 0.11650065704963015
    iteration 3731
    loss_train 0.11649097147990801
    iteration 3732
    loss_train 0.1164812895075283
    iteration 3733
    loss_train 0.1164716111303444
    iteration 3734
    loss_train 0.11646193634621159
    iteration 3735
    loss_train 0.1164522651529868
    iteration 3736
    loss_train 0.11644259754852883
    iteration 3737
    loss_train 0.11643293353069827
    iteration 3738
    loss_train 0.11642327309735742
    iteration 3739
    loss_train 0.11641361624637057
    iteration 3740
    loss_train 0.1164039629756036
    iteration 3741
    loss_train 0.1163943132829243
    iteration 3742
    loss_train 0.11638466716620216
    iteration 3743
    loss_train 0.11637502462330854
    iteration 3744
    loss_train 0.11636538565211646
    iteration 3745
    loss_train 0.11635575025050086
    iteration 3746
    loss_train 0.11634611841633842
    iteration 3747
    loss_train 0.1163364901475075
    iteration 3748
    loss_train 0.11632686544188833
    iteration 3749
    loss_train 0.11631724429736293
    iteration 3750
    loss_train 0.11630762671181497
    iteration 3751
    loss_train 0.11629801268313002
    iteration 3752
    loss_train 0.11628840220919531
    iteration 3753
    loss_train 0.11627879528789999
    iteration 3754
    loss_train 0.11626919191713478
    iteration 3755
    loss_train 0.11625959209479224
    iteration 3756
    loss_train 0.11624999581876676
    iteration 3757
    loss_train 0.11624040308695442
    iteration 3758
    loss_train 0.11623081389725302
    iteration 3759
    loss_train 0.11622122824756219
    iteration 3760
    loss_train 0.11621164613578323
    iteration 3761
    loss_train 0.11620206755981928
    iteration 3762
    loss_train 0.11619249251757514
    iteration 3763
    loss_train 0.11618292100695744
    iteration 3764
    loss_train 0.11617335302587452
    iteration 3765
    loss_train 0.11616378857223636
    iteration 3766
    loss_train 0.11615422764395489
    iteration 3767
    loss_train 0.11614467023894355
    iteration 3768
    loss_train 0.11613511635511772
    iteration 3769
    loss_train 0.11612556599039434
    iteration 3770
    loss_train 0.11611601914269222
    iteration 3771
    loss_train 0.11610647580993182
    iteration 3772
    loss_train 0.11609693599003536
    iteration 3773
    loss_train 0.11608739968092675
    iteration 3774
    loss_train 0.11607786688053164
    iteration 3775
    loss_train 0.11606833758677751
    iteration 3776
    loss_train 0.11605881179759334
    iteration 3777
    loss_train 0.11604928951091005
    iteration 3778
    loss_train 0.11603977072466014
    iteration 3779
    loss_train 0.11603025543677786
    iteration 3780
    loss_train 0.11602074364519921
    iteration 3781
    loss_train 0.11601123534786188
    iteration 3782
    loss_train 0.1160017305427052
    iteration 3783
    loss_train 0.1159922292276703
    iteration 3784
    loss_train 0.11598273140070002
    iteration 3785
    loss_train 0.1159732370597388
    iteration 3786
    loss_train 0.11596374620273299
    iteration 3787
    loss_train 0.11595425882763036
    iteration 3788
    loss_train 0.11594477493238056
    iteration 3789
    loss_train 0.11593529451493492
    iteration 3790
    loss_train 0.11592581757324649
    iteration 3791
    loss_train 0.11591634410526987
    iteration 3792
    loss_train 0.11590687410896151
    iteration 3793
    loss_train 0.11589740758227947
    iteration 3794
    loss_train 0.11588794452318356
    iteration 3795
    loss_train 0.11587848492963516
    iteration 3796
    loss_train 0.11586902879959743
    iteration 3797
    loss_train 0.11585957613103529
    iteration 3798
    loss_train 0.11585012692191504
    iteration 3799
    loss_train 0.11584068117020509
    iteration 3800
    loss_train 0.11583123887387511
    iteration 3801
    loss_train 0.11582180003089669
    iteration 3802
    loss_train 0.11581236463924308
    iteration 3803
    loss_train 0.11580293269688911
    iteration 3804
    loss_train 0.11579350420181135
    iteration 3805
    loss_train 0.11578407915198799
    iteration 3806
    loss_train 0.11577465754539892
    iteration 3807
    loss_train 0.11576523938002571
    iteration 3808
    loss_train 0.11575582465385155
    iteration 3809
    loss_train 0.11574641336486127
    iteration 3810
    loss_train 0.11573700551104148
    iteration 3811
    loss_train 0.11572760109038024
    iteration 3812
    loss_train 0.11571820010086754
    iteration 3813
    loss_train 0.11570880254049475
    iteration 3814
    loss_train 0.11569940840725507
    iteration 3815
    loss_train 0.1156900176991433
    iteration 3816
    loss_train 0.11568063041415587
    iteration 3817
    loss_train 0.11567124655029087
    iteration 3818
    loss_train 0.11566186610554804
    iteration 3819
    loss_train 0.11565248907792877
    iteration 3820
    loss_train 0.11564311546543606
    iteration 3821
    loss_train 0.11563374526607459
    iteration 3822
    loss_train 0.11562437847785065
    iteration 3823
    loss_train 0.11561501509877212
    iteration 3824
    loss_train 0.11560565512684867
    iteration 3825
    loss_train 0.11559629856009142
    iteration 3826
    loss_train 0.11558694539651324
    iteration 3827
    loss_train 0.11557759563412859
    iteration 3828
    loss_train 0.11556824927095352
    iteration 3829
    loss_train 0.11555890630500579
    iteration 3830
    loss_train 0.11554956673430473
    iteration 3831
    loss_train 0.11554023055687128
    iteration 3832
    loss_train 0.11553089777072803
    iteration 3833
    loss_train 0.1155215683738992
    iteration 3834
    loss_train 0.1155122423644105
    iteration 3835
    loss_train 0.11550291974028952
    iteration 3836
    loss_train 0.11549360049956521
    iteration 3837
    loss_train 0.11548428464026823
    iteration 3838
    loss_train 0.11547497216043093
    iteration 3839
    loss_train 0.11546566305808705
    iteration 3840
    loss_train 0.11545635733127214
    iteration 3841
    loss_train 0.1154470549780233
    iteration 3842
    loss_train 0.11543775599637922
    iteration 3843
    loss_train 0.1154284603843802
    iteration 3844
    loss_train 0.11541916814006807
    iteration 3845
    loss_train 0.11540987926148638
    iteration 3846
    loss_train 0.11540059374668016
    iteration 3847
    loss_train 0.11539131159369613
    iteration 3848
    loss_train 0.11538203280058258
    iteration 3849
    loss_train 0.11537275736538936
    iteration 3850
    loss_train 0.11536348528616788
    iteration 3851
    loss_train 0.1153542165609713
    iteration 3852
    loss_train 0.11534495118785412
    iteration 3853
    loss_train 0.11533568916487257
    iteration 3854
    loss_train 0.11532643049008454
    iteration 3855
    loss_train 0.11531717516154931
    iteration 3856
    loss_train 0.11530792317732794
    iteration 3857
    loss_train 0.11529867453548284
    iteration 3858
    loss_train 0.11528942923407819
    iteration 3859
    loss_train 0.11528018727117967
    iteration 3860
    loss_train 0.1152709486448546
    iteration 3861
    loss_train 0.11526171335317167
    iteration 3862
    loss_train 0.11525248139420137
    iteration 3863
    loss_train 0.1152432527660157
    iteration 3864
    loss_train 0.11523402746668816
    iteration 3865
    loss_train 0.1152248054942938
    iteration 3866
    loss_train 0.11521558684690934
    iteration 3867
    loss_train 0.11520637152261298
    iteration 3868
    loss_train 0.11519715951948453
    iteration 3869
    loss_train 0.11518795083560528
    iteration 3870
    loss_train 0.1151787454690582
    iteration 3871
    loss_train 0.11516954341792765
    iteration 3872
    loss_train 0.1151603446802997
    iteration 3873
    loss_train 0.1151511492542619
    iteration 3874
    loss_train 0.11514195713790332
    iteration 3875
    loss_train 0.11513276832931467
    iteration 3876
    loss_train 0.1151235828265881
    iteration 3877
    loss_train 0.11511440062781739
    iteration 3878
    loss_train 0.11510522173109781
    iteration 3879
    loss_train 0.11509604613452622
    iteration 3880
    loss_train 0.11508687383620093
    iteration 3881
    loss_train 0.11507770483422192
    iteration 3882
    loss_train 0.11506853912669059
    iteration 3883
    loss_train 0.11505937671170989
    iteration 3884
    loss_train 0.11505021758738443
    iteration 3885
    loss_train 0.11504106175182013
    iteration 3886
    loss_train 0.11503190920312471
    iteration 3887
    loss_train 0.11502275993940715
    iteration 3888
    loss_train 0.11501361395877822
    iteration 3889
    loss_train 0.11500447125934991
    iteration 3890
    loss_train 0.11499533183923602
    iteration 3891
    loss_train 0.1149861956965517
    iteration 3892
    loss_train 0.11497706282941374
    iteration 3893
    loss_train 0.11496793323594029
    iteration 3894
    loss_train 0.11495880691425117
    iteration 3895
    loss_train 0.11494968386246764
    iteration 3896
    loss_train 0.11494056407871248
    iteration 3897
    loss_train 0.11493144756111007
    iteration 3898
    loss_train 0.11492233430778612
    iteration 3899
    loss_train 0.11491322431686803
    iteration 3900
    loss_train 0.11490411758648462
    iteration 3901
    loss_train 0.11489501411476617
    iteration 3902
    loss_train 0.11488591389984457
    iteration 3903
    loss_train 0.11487681693985315
    iteration 3904
    loss_train 0.11486772323292681
    iteration 3905
    loss_train 0.11485863277720185
    iteration 3906
    loss_train 0.11484954557081609
    iteration 3907
    loss_train 0.11484046161190896
    iteration 3908
    loss_train 0.11483138089862123
    iteration 3909
    loss_train 0.11482230342909523
    iteration 3910
    loss_train 0.11481322920147487
    iteration 3911
    loss_train 0.11480415821390536
    iteration 3912
    loss_train 0.11479509046453357
    iteration 3913
    loss_train 0.11478602595150773
    iteration 3914
    loss_train 0.11477696467297772
    iteration 3915
    loss_train 0.11476790662709477
    iteration 3916
    loss_train 0.11475885181201155
    iteration 3917
    loss_train 0.11474980022588238
    iteration 3918
    loss_train 0.11474075186686288
    iteration 3919
    loss_train 0.1147317067331103
    iteration 3920
    loss_train 0.1147226648227833
    iteration 3921
    loss_train 0.11471362613404201
    iteration 3922
    loss_train 0.11470459066504805
    iteration 3923
    loss_train 0.11469555841396445
    iteration 3924
    loss_train 0.11468652937895583
    iteration 3925
    loss_train 0.11467750355818815
    iteration 3926
    loss_train 0.114668480949829
    iteration 3927
    loss_train 0.11465946155204722
    iteration 3928
    loss_train 0.11465044536301329
    iteration 3929
    loss_train 0.11464143238089909
    iteration 3930
    loss_train 0.11463242260387797
    iteration 3931
    loss_train 0.11462341603012471
    iteration 3932
    loss_train 0.11461441265781562
    iteration 3933
    loss_train 0.11460541248512837
    iteration 3934
    loss_train 0.11459641551024216
    iteration 3935
    loss_train 0.11458742173133762
    iteration 3936
    loss_train 0.11457843114659685
    iteration 3937
    loss_train 0.11456944375420335
    iteration 3938
    loss_train 0.11456045955234208
    iteration 3939
    loss_train 0.11455147853919952
    iteration 3940
    loss_train 0.11454250071296354
    iteration 3941
    loss_train 0.11453352607182345
    iteration 3942
    loss_train 0.11452455461397003
    iteration 3943
    loss_train 0.11451558633759544
    iteration 3944
    loss_train 0.11450662124089336
    iteration 3945
    loss_train 0.11449765932205885
    iteration 3946
    loss_train 0.11448870057928848
    iteration 3947
    loss_train 0.11447974501078019
    iteration 3948
    loss_train 0.11447079261473332
    iteration 3949
    loss_train 0.11446184338934881
    iteration 3950
    loss_train 0.1144528973328288
    iteration 3951
    loss_train 0.11444395444337703
    iteration 3952
    loss_train 0.1144350147191986
    iteration 3953
    loss_train 0.11442607815850006
    iteration 3954
    loss_train 0.11441714475948936
    iteration 3955
    loss_train 0.11440821452037596
    iteration 3956
    loss_train 0.11439928743937058
    iteration 3957
    loss_train 0.11439036351468548
    iteration 3958
    loss_train 0.11438144274453436
    iteration 3959
    loss_train 0.11437252512713224
    iteration 3960
    loss_train 0.11436361066069566
    iteration 3961
    loss_train 0.11435469934344247
    iteration 3962
    loss_train 0.11434579117359205
    iteration 3963
    loss_train 0.11433688614936502
    iteration 3964
    loss_train 0.11432798426898365
    iteration 3965
    loss_train 0.11431908553067141
    iteration 3966
    loss_train 0.1143101899326533
    iteration 3967
    loss_train 0.11430129747315568
    iteration 3968
    loss_train 0.11429240815040628
    iteration 3969
    loss_train 0.11428352196263433
    iteration 3970
    loss_train 0.11427463890807034
    iteration 3971
    loss_train 0.11426575898494634
    iteration 3972
    loss_train 0.1142568821914957
    iteration 3973
    loss_train 0.11424800852595315
    iteration 3974
    loss_train 0.1142391379865549
    iteration 3975
    loss_train 0.11423027057153849
    iteration 3976
    loss_train 0.11422140627914291
    iteration 3977
    loss_train 0.11421254510760846
    iteration 3978
    loss_train 0.11420368705517694
    iteration 3979
    loss_train 0.11419483212009143
    iteration 3980
    loss_train 0.11418598030059647
    iteration 3981
    loss_train 0.11417713159493798
    iteration 3982
    loss_train 0.11416828600136318
    iteration 3983
    loss_train 0.11415944351812081
    iteration 3984
    loss_train 0.11415060414346086
    iteration 3985
    loss_train 0.11414176787563486
    iteration 3986
    loss_train 0.11413293471289553
    iteration 3987
    loss_train 0.11412410465349712
    iteration 3988
    loss_train 0.1141152776956952
    iteration 3989
    loss_train 0.1141064538377466
    iteration 3990
    loss_train 0.11409763307790982
    iteration 3991
    loss_train 0.11408881541444438
    iteration 3992
    loss_train 0.11408000084561147
    iteration 3993
    loss_train 0.11407118936967345
    iteration 3994
    loss_train 0.11406238098489405
    iteration 3995
    loss_train 0.11405357568953858
    iteration 3996
    loss_train 0.11404477348187346
    iteration 3997
    loss_train 0.11403597436016662
    iteration 3998
    loss_train 0.11402717832268731
    iteration 3999
    loss_train 0.11401838536770613
    iteration 4000
    loss_train 0.11400959549349508
    iteration 4001
    loss_train 0.11400080869832746
    iteration 4002
    loss_train 0.11399202498047802
    iteration 4003
    loss_train 0.11398324433822271
    iteration 4004
    loss_train 0.11397446676983904
    iteration 4005
    loss_train 0.11396569227360566
    iteration 4006
    loss_train 0.1139569208478027
    iteration 4007
    loss_train 0.11394815249071168
    iteration 4008
    loss_train 0.1139393872006153
    iteration 4009
    loss_train 0.11393062497579777
    iteration 4010
    loss_train 0.11392186581454454
    iteration 4011
    loss_train 0.11391310971514246
    iteration 4012
    loss_train 0.11390435667587977
    iteration 4013
    loss_train 0.11389560669504593
    iteration 4014
    loss_train 0.1138868597709318
    iteration 4015
    loss_train 0.11387811590182963
    iteration 4016
    loss_train 0.1138693750860329
    iteration 4017
    loss_train 0.11386063732183652
    iteration 4018
    loss_train 0.11385190260753669
    iteration 4019
    loss_train 0.11384317094143095
    iteration 4020
    loss_train 0.1138344423218182
    iteration 4021
    loss_train 0.11382571674699858
    iteration 4022
    loss_train 0.11381699421527372
    iteration 4023
    loss_train 0.11380827472494641
    iteration 4024
    loss_train 0.11379955827432084
    iteration 4025
    loss_train 0.11379084486170257
    iteration 4026
    loss_train 0.11378213448539842
    iteration 4027
    loss_train 0.11377342714371654
    iteration 4028
    loss_train 0.11376472283496639
    iteration 4029
    loss_train 0.11375602155745886
    iteration 4030
    loss_train 0.113747323309506
    iteration 4031
    loss_train 0.1137386280894212
    iteration 4032
    loss_train 0.11372993589551934
    iteration 4033
    loss_train 0.1137212467261164
    iteration 4034
    loss_train 0.11371256057952982
    iteration 4035
    loss_train 0.11370387745407823
    iteration 4036
    loss_train 0.1136951973480817
    iteration 4037
    loss_train 0.11368652025986144
    iteration 4038
    loss_train 0.11367784618774021
    iteration 4039
    loss_train 0.11366917513004182
    iteration 4040
    loss_train 0.11366050708509158
    iteration 4041
    loss_train 0.113651842051216
    iteration 4042
    loss_train 0.11364318002674292
    iteration 4043
    loss_train 0.11363452101000145
    iteration 4044
    loss_train 0.11362586499932209
    iteration 4045
    loss_train 0.11361721199303658
    iteration 4046
    loss_train 0.11360856198947791
    iteration 4047
    loss_train 0.11359991498698041
    iteration 4048
    loss_train 0.11359127098387978
    iteration 4049
    loss_train 0.11358262997851283
    iteration 4050
    loss_train 0.11357399196921787
    iteration 4051
    loss_train 0.1135653569543344
    iteration 4052
    loss_train 0.11355672493220313
    iteration 4053
    loss_train 0.11354809590116623
    iteration 4054
    loss_train 0.11353946985956703
    iteration 4055
    loss_train 0.11353084680575018
    iteration 4056
    loss_train 0.11352222673806169
    iteration 4057
    loss_train 0.11351360965484865
    iteration 4058
    loss_train 0.11350499555445963
    iteration 4059
    loss_train 0.11349638443524447
    iteration 4060
    loss_train 0.11348777629555419
    iteration 4061
    loss_train 0.11347917113374108
    iteration 4062
    loss_train 0.11347056894815882
    iteration 4063
    loss_train 0.1134619697371623
    iteration 4064
    loss_train 0.11345337349910765
    iteration 4065
    loss_train 0.11344478023235234
    iteration 4066
    loss_train 0.1134361899352551
    iteration 4067
    loss_train 0.11342760260617582
    iteration 4068
    loss_train 0.11341901824347585
    iteration 4069
    loss_train 0.11341043684551769
    iteration 4070
    loss_train 0.1134018584106651
    iteration 4071
    loss_train 0.11339328293728311
    iteration 4072
    loss_train 0.11338471042373803
    iteration 4073
    loss_train 0.11337614086839753
    iteration 4074
    loss_train 0.11336757426963036
    iteration 4075
    loss_train 0.11335901062580662
    iteration 4076
    loss_train 0.11335044993529769
    iteration 4077
    loss_train 0.11334189219647618
    iteration 4078
    loss_train 0.11333333740771594
    iteration 4079
    loss_train 0.11332478556739206
    iteration 4080
    loss_train 0.11331623667388101
    iteration 4081
    loss_train 0.11330769072556036
    iteration 4082
    loss_train 0.11329914772080901
    iteration 4083
    loss_train 0.11329060765800711
    iteration 4084
    loss_train 0.11328207053553598
    iteration 4085
    loss_train 0.11327353635177827
    iteration 4086
    loss_train 0.11326500510511785
    iteration 4087
    loss_train 0.11325647679393987
    iteration 4088
    loss_train 0.11324795141663066
    iteration 4089
    loss_train 0.11323942897157782
    iteration 4090
    loss_train 0.11323090945717024
    iteration 4091
    loss_train 0.11322239287179796
    iteration 4092
    loss_train 0.11321387921385233
    iteration 4093
    loss_train 0.11320536848172585
    iteration 4094
    loss_train 0.11319686067381243
    iteration 4095
    loss_train 0.113188355788507
    iteration 4096
    loss_train 0.11317985382420587
    iteration 4097
    loss_train 0.11317135477930652
    iteration 4098
    loss_train 0.11316285865220771
    iteration 4099
    loss_train 0.11315436544130933
    iteration 4100
    loss_train 0.11314587514501263
    iteration 4101
    loss_train 0.11313738776172004
    iteration 4102
    loss_train 0.11312890328983513
    iteration 4103
    loss_train 0.11312042172776282
    iteration 4104
    loss_train 0.11311194307390923
    iteration 4105
    loss_train 0.1131034673266816
    iteration 4106
    loss_train 0.1130949944844885
    iteration 4107
    loss_train 0.11308652454573967
    iteration 4108
    loss_train 0.11307805750884611
    iteration 4109
    loss_train 0.11306959337222001
    iteration 4110
    loss_train 0.11306113213427474
    iteration 4111
    loss_train 0.11305267379342498
    iteration 4112
    loss_train 0.11304421834808655
    iteration 4113
    loss_train 0.11303576579667646
    iteration 4114
    loss_train 0.11302731613761303
    iteration 4115
    loss_train 0.11301886936931571
    iteration 4116
    loss_train 0.1130104254902052
    iteration 4117
    loss_train 0.11300198449870331
    iteration 4118
    loss_train 0.11299354639323322
    iteration 4119
    loss_train 0.11298511117221925
    iteration 4120
    loss_train 0.11297667883408685
    iteration 4121
    loss_train 0.11296824937726276
    iteration 4122
    loss_train 0.11295982280017486
    iteration 4123
    loss_train 0.11295139910125232
    iteration 4124
    loss_train 0.11294297827892538
    iteration 4125
    loss_train 0.11293456033162565
    iteration 4126
    loss_train 0.11292614525778578
    iteration 4127
    loss_train 0.11291773305583969
    iteration 4128
    loss_train 0.11290932372422244
    iteration 4129
    loss_train 0.11290091726137037
    iteration 4130
    loss_train 0.11289251366572098
    iteration 4131
    loss_train 0.1128841129357129
    iteration 4132
    loss_train 0.11287571506978607
    iteration 4133
    loss_train 0.11286732006638148
    iteration 4134
    loss_train 0.11285892792394138
    iteration 4135
    loss_train 0.11285053864090924
    iteration 4136
    loss_train 0.11284215221572967
    iteration 4137
    loss_train 0.11283376864684845
    iteration 4138
    loss_train 0.11282538793271263
    iteration 4139
    loss_train 0.11281701007177028
    iteration 4140
    loss_train 0.11280863506247082
    iteration 4141
    loss_train 0.11280026290326478
    iteration 4142
    loss_train 0.11279189359260382
    iteration 4143
    loss_train 0.11278352712894084
    iteration 4144
    loss_train 0.11277516351072994
    iteration 4145
    loss_train 0.11276680273642632
    iteration 4146
    loss_train 0.11275844480448641
    iteration 4147
    loss_train 0.11275008971336778
    iteration 4148
    loss_train 0.11274173746152917
    iteration 4149
    loss_train 0.11273338804743056
    iteration 4150
    loss_train 0.11272504146953301
    iteration 4151
    loss_train 0.11271669772629875
    iteration 4152
    loss_train 0.11270835681619122
    iteration 4153
    loss_train 0.11270001873767503
    iteration 4154
    loss_train 0.11269168348921596
    iteration 4155
    loss_train 0.11268335106928093
    iteration 4156
    loss_train 0.11267502147633797
    iteration 4157
    loss_train 0.1126666947088564
    iteration 4158
    loss_train 0.11265837076530656
    iteration 4159
    loss_train 0.1126500496441601
    iteration 4160
    loss_train 0.11264173134388965
    iteration 4161
    loss_train 0.11263341586296914
    iteration 4162
    loss_train 0.1126251031998736
    iteration 4163
    loss_train 0.11261679335307921
    iteration 4164
    loss_train 0.11260848632106331
    iteration 4165
    loss_train 0.11260018210230442
    iteration 4166
    loss_train 0.11259188069528216
    iteration 4167
    loss_train 0.11258358209847734
    iteration 4168
    loss_train 0.11257528631037192
    iteration 4169
    loss_train 0.11256699332944894
    iteration 4170
    loss_train 0.11255870315419271
    iteration 4171
    loss_train 0.11255041578308855
    iteration 4172
    loss_train 0.11254213121462303
    iteration 4173
    loss_train 0.11253384944728378
    iteration 4174
    loss_train 0.11252557047955966
    iteration 4175
    loss_train 0.11251729430994059
    iteration 4176
    loss_train 0.11250902093691768
    iteration 4177
    loss_train 0.11250075035898316
    iteration 4178
    loss_train 0.11249248257463039
    iteration 4179
    loss_train 0.1124842175823539
    iteration 4180
    loss_train 0.11247595538064933
    iteration 4181
    loss_train 0.11246769596801338
    iteration 4182
    loss_train 0.1124594393429441
    iteration 4183
    loss_train 0.11245118550394038
    iteration 4184
    loss_train 0.11244293444950247
    iteration 4185
    loss_train 0.11243468617813171
    iteration 4186
    loss_train 0.11242644068833046
    iteration 4187
    loss_train 0.11241819797860231
    iteration 4188
    loss_train 0.11240995804745188
    iteration 4189
    loss_train 0.11240172089338511
    iteration 4190
    loss_train 0.11239348651490878
    iteration 4191
    loss_train 0.11238525491053106
    iteration 4192
    loss_train 0.11237702607876109
    iteration 4193
    loss_train 0.11236880001810916
    iteration 4194
    loss_train 0.1123605767270867
    iteration 4195
    loss_train 0.11235235620420621
    iteration 4196
    loss_train 0.11234413844798145
    iteration 4197
    loss_train 0.11233592345692707
    iteration 4198
    loss_train 0.11232771122955901
    iteration 4199
    loss_train 0.11231950176439426
    iteration 4200
    loss_train 0.11231129505995094
    iteration 4201
    loss_train 0.1123030911147483
    iteration 4202
    loss_train 0.11229488992730663
    iteration 4203
    loss_train 0.11228669149614742
    iteration 4204
    loss_train 0.11227849581979324
    iteration 4205
    loss_train 0.1122703028967677
    iteration 4206
    loss_train 0.1122621127255956
    iteration 4207
    loss_train 0.11225392530480277
    iteration 4208
    loss_train 0.11224574063291635
    iteration 4209
    loss_train 0.1122375587084642
    iteration 4210
    loss_train 0.11222937952997566
    iteration 4211
    loss_train 0.11222120309598099
    iteration 4212
    loss_train 0.11221302940501153
    iteration 4213
    loss_train 0.11220485845559984
    iteration 4214
    loss_train 0.11219669024627946
    iteration 4215
    loss_train 0.11218852477558512
    iteration 4216
    loss_train 0.11218036204205253
    iteration 4217
    loss_train 0.1121722020442186
    iteration 4218
    loss_train 0.11216404478062131
    iteration 4219
    loss_train 0.1121558902497997
    iteration 4220
    loss_train 0.11214773845029395
    iteration 4221
    loss_train 0.11213958938064524
    iteration 4222
    loss_train 0.11213144303939603
    iteration 4223
    loss_train 0.11212329942508958
    iteration 4224
    loss_train 0.11211515853627056
    iteration 4225
    loss_train 0.11210702037148446
    iteration 4226
    loss_train 0.11209888492927797
    iteration 4227
    loss_train 0.11209075220819889
    iteration 4228
    loss_train 0.11208262220679607
    iteration 4229
    loss_train 0.11207449492361943
    iteration 4230
    loss_train 0.11206637035722
    iteration 4231
    loss_train 0.11205824850614986
    iteration 4232
    loss_train 0.11205012936896215
    iteration 4233
    loss_train 0.11204201294421115
    iteration 4234
    loss_train 0.11203389923045219
    iteration 4235
    loss_train 0.11202578822624167
    iteration 4236
    loss_train 0.11201767993013707
    iteration 4237
    loss_train 0.11200957434069697
    iteration 4238
    loss_train 0.11200147145648096
    iteration 4239
    loss_train 0.11199337127604972
    iteration 4240
    loss_train 0.11198527379796505
    iteration 4241
    loss_train 0.11197717902078977
    iteration 4242
    loss_train 0.11196908694308785
    iteration 4243
    loss_train 0.11196099756342416
    iteration 4244
    loss_train 0.11195291088036481
    iteration 4245
    loss_train 0.11194482689247692
    iteration 4246
    loss_train 0.11193674559832861
    iteration 4247
    loss_train 0.11192866699648914
    iteration 4248
    loss_train 0.11192059108552878
    iteration 4249
    loss_train 0.11191251786401897
    iteration 4250
    loss_train 0.11190444733053198
    iteration 4251
    loss_train 0.11189637948364148
    iteration 4252
    loss_train 0.11188831432192187
    iteration 4253
    loss_train 0.1118802518439488
    iteration 4254
    loss_train 0.11187219204829892
    iteration 4255
    loss_train 0.1118641349335499
    iteration 4256
    loss_train 0.11185608049828052
    iteration 4257
    loss_train 0.1118480287410706
    iteration 4258
    loss_train 0.11183997966050105
    iteration 4259
    loss_train 0.11183193325515371
    iteration 4260
    loss_train 0.11182388952361161
    iteration 4261
    loss_train 0.11181584846445874
    iteration 4262
    loss_train 0.11180781007628017
    iteration 4263
    loss_train 0.11179977435766199
    iteration 4264
    loss_train 0.11179174130719143
    iteration 4265
    loss_train 0.11178371092345665
    iteration 4266
    loss_train 0.1117756832050469
    iteration 4267
    loss_train 0.11176765815055247
    iteration 4268
    loss_train 0.11175963575856468
    iteration 4269
    loss_train 0.11175161602767596
    iteration 4270
    loss_train 0.1117435989564797
    iteration 4271
    loss_train 0.11173558454357035
    iteration 4272
    loss_train 0.11172757278754337
    iteration 4273
    loss_train 0.11171956368699539
    iteration 4274
    loss_train 0.11171155724052387
    iteration 4275
    loss_train 0.11170355344672747
    iteration 4276
    loss_train 0.11169555230420587
    iteration 4277
    loss_train 0.11168755381155968
    iteration 4278
    loss_train 0.11167955796739061
    iteration 4279
    loss_train 0.11167156477030143
    iteration 4280
    loss_train 0.11166357421889582
    iteration 4281
    loss_train 0.11165558631177866
    iteration 4282
    loss_train 0.11164760104755576
    iteration 4283
    loss_train 0.11163961842483397
    iteration 4284
    loss_train 0.11163163844222114
    iteration 4285
    loss_train 0.1116236610983262
    iteration 4286
    loss_train 0.11161568639175905
    iteration 4287
    loss_train 0.11160771432113067
    iteration 4288
    loss_train 0.11159974488505303
    iteration 4289
    loss_train 0.11159177808213912
    iteration 4290
    loss_train 0.11158381391100297
    iteration 4291
    loss_train 0.11157585237025956
    iteration 4292
    loss_train 0.11156789345852498
    iteration 4293
    loss_train 0.11155993717441635
    iteration 4294
    loss_train 0.1115519835165517
    iteration 4295
    loss_train 0.11154403248355015
    iteration 4296
    loss_train 0.11153608407403177
    iteration 4297
    loss_train 0.11152813828661777
    iteration 4298
    loss_train 0.11152019511993026
    iteration 4299
    loss_train 0.11151225457259241
    iteration 4300
    loss_train 0.1115043166432284
    iteration 4301
    loss_train 0.11149638133046333
    iteration 4302
    loss_train 0.11148844863292351
    iteration 4303
    loss_train 0.11148051854923602
    iteration 4304
    loss_train 0.11147259107802913
    iteration 4305
    loss_train 0.11146466621793198
    iteration 4306
    loss_train 0.11145674396757488
    iteration 4307
    loss_train 0.11144882432558902
    iteration 4308
    loss_train 0.11144090729060657
    iteration 4309
    loss_train 0.11143299286126077
    iteration 4310
    loss_train 0.11142508103618587
    iteration 4311
    loss_train 0.11141717181401707
    iteration 4312
    loss_train 0.11140926519339063
    iteration 4313
    loss_train 0.11140136117294369
    iteration 4314
    loss_train 0.11139345975131458
    iteration 4315
    loss_train 0.11138556092714243
    iteration 4316
    loss_train 0.1113776646990675
    iteration 4317
    loss_train 0.11136977106573097
    iteration 4318
    loss_train 0.11136188002577505
    iteration 4319
    loss_train 0.11135399157784294
    iteration 4320
    loss_train 0.1113461057205788
    iteration 4321
    loss_train 0.11133822245262782
    iteration 4322
    loss_train 0.11133034177263618
    iteration 4323
    loss_train 0.11132246367925104
    iteration 4324
    loss_train 0.11131458817112048
    iteration 4325
    loss_train 0.11130671524689369
    iteration 4326
    loss_train 0.1112988449052208
    iteration 4327
    loss_train 0.11129097714475286
    iteration 4328
    loss_train 0.11128311196414196
    iteration 4329
    loss_train 0.11127524936204122
    iteration 4330
    loss_train 0.11126738933710463
    iteration 4331
    loss_train 0.11125953188798725
    iteration 4332
    loss_train 0.11125167701334512
    iteration 4333
    loss_train 0.11124382471183522
    iteration 4334
    loss_train 0.11123597498211546
    iteration 4335
    loss_train 0.11122812782284486
    iteration 4336
    loss_train 0.11122028323268332
    iteration 4337
    loss_train 0.11121244121029172
    iteration 4338
    loss_train 0.11120460175433201
    iteration 4339
    loss_train 0.11119676486346695
    iteration 4340
    loss_train 0.11118893053636049
    iteration 4341
    loss_train 0.11118109877167726
    iteration 4342
    loss_train 0.11117326956808316
    iteration 4343
    loss_train 0.11116544292424486
    iteration 4344
    loss_train 0.1111576188388301
    iteration 4345
    loss_train 0.11114979731050756
    iteration 4346
    loss_train 0.11114197833794687
    iteration 4347
    loss_train 0.11113416191981865
    iteration 4348
    loss_train 0.11112634805479446
    iteration 4349
    loss_train 0.1111185367415469
    iteration 4350
    loss_train 0.1111107279787494
    iteration 4351
    loss_train 0.11110292176507645
    iteration 4352
    loss_train 0.1110951180992035
    iteration 4353
    loss_train 0.11108731697980698
    iteration 4354
    loss_train 0.11107951840556414
    iteration 4355
    loss_train 0.11107172237515338
    iteration 4356
    loss_train 0.11106392888725396
    iteration 4357
    loss_train 0.11105613794054608
    iteration 4358
    loss_train 0.11104834953371095
    iteration 4359
    loss_train 0.11104056366543068
    iteration 4360
    loss_train 0.11103278033438845
    iteration 4361
    loss_train 0.1110249995392682
    iteration 4362
    loss_train 0.11101722127875503
    iteration 4363
    loss_train 0.11100944555153484
    iteration 4364
    loss_train 0.11100167235629455
    iteration 4365
    loss_train 0.11099390169172203
    iteration 4366
    loss_train 0.11098613355650606
    iteration 4367
    loss_train 0.11097836794933644
    iteration 4368
    loss_train 0.11097060486890382
    iteration 4369
    loss_train 0.11096284431389992
    iteration 4370
    loss_train 0.11095508628301731
    iteration 4371
    loss_train 0.1109473307749495
    iteration 4372
    loss_train 0.11093957778839102
    iteration 4373
    loss_train 0.11093182732203728
    iteration 4374
    loss_train 0.11092407937458469
    iteration 4375
    loss_train 0.1109163339447305
    iteration 4376
    loss_train 0.110908591031173
    iteration 4377
    loss_train 0.11090085063261142
    iteration 4378
    loss_train 0.1108931127477458
    iteration 4379
    loss_train 0.11088537737527733
    iteration 4380
    loss_train 0.11087764451390796
    iteration 4381
    loss_train 0.11086991416234063
    iteration 4382
    loss_train 0.11086218631927926
    iteration 4383
    loss_train 0.11085446098342858
    iteration 4384
    loss_train 0.11084673815349445
    iteration 4385
    loss_train 0.11083901782818352
    iteration 4386
    loss_train 0.11083130000620339
    iteration 4387
    loss_train 0.1108235846862626
    iteration 4388
    loss_train 0.11081587186707063
    iteration 4389
    loss_train 0.11080816154733793
    iteration 4390
    loss_train 0.11080045372577575
    iteration 4391
    loss_train 0.11079274840109642
    iteration 4392
    loss_train 0.11078504557201309
    iteration 4393
    loss_train 0.11077734523723991
    iteration 4394
    loss_train 0.11076964739549186
    iteration 4395
    loss_train 0.11076195204548502
    iteration 4396
    loss_train 0.1107542591859362
    iteration 4397
    loss_train 0.11074656881556319
    iteration 4398
    loss_train 0.11073888093308476
    iteration 4399
    loss_train 0.11073119553722058
    iteration 4400
    loss_train 0.11072351262669117
    iteration 4401
    loss_train 0.11071583220021802
    iteration 4402
    loss_train 0.1107081542565236
    iteration 4403
    loss_train 0.11070047879433119
    iteration 4404
    loss_train 0.11069280581236506
    iteration 4405
    loss_train 0.11068513530935034
    iteration 4406
    loss_train 0.11067746728401316
    iteration 4407
    loss_train 0.11066980173508045
    iteration 4408
    loss_train 0.11066213866128011
    iteration 4409
    loss_train 0.11065447806134097
    iteration 4410
    loss_train 0.11064681993399281
    iteration 4411
    loss_train 0.1106391642779662
    iteration 4412
    loss_train 0.11063151109199268
    iteration 4413
    loss_train 0.1106238603748047
    iteration 4414
    loss_train 0.11061621212513566
    iteration 4415
    loss_train 0.11060856634171982
    iteration 4416
    loss_train 0.11060092302329233
    iteration 4417
    loss_train 0.11059328216858931
    iteration 4418
    loss_train 0.11058564377634772
    iteration 4419
    loss_train 0.11057800784530541
    iteration 4420
    loss_train 0.11057037437420124
    iteration 4421
    loss_train 0.11056274336177488
    iteration 4422
    loss_train 0.11055511480676694
    iteration 4423
    loss_train 0.11054748870791889
    iteration 4424
    loss_train 0.11053986506397309
    iteration 4425
    loss_train 0.11053224387367294
    iteration 4426
    loss_train 0.11052462513576256
    iteration 4427
    loss_train 0.110517008848987
    iteration 4428
    loss_train 0.11050939501209231
    iteration 4429
    loss_train 0.11050178362382539
    iteration 4430
    loss_train 0.11049417468293395
    iteration 4431
    loss_train 0.11048656818816667
    iteration 4432
    loss_train 0.11047896413827314
    iteration 4433
    loss_train 0.11047136253200382
    iteration 4434
    loss_train 0.11046376336811005
    iteration 4435
    loss_train 0.11045616664534402
    iteration 4436
    loss_train 0.11044857236245892
    iteration 4437
    loss_train 0.11044098051820872
    iteration 4438
    loss_train 0.11043339111134833
    iteration 4439
    loss_train 0.11042580414063358
    iteration 4440
    loss_train 0.11041821960482108
    iteration 4441
    loss_train 0.11041063750266843
    iteration 4442
    loss_train 0.11040305783293407
    iteration 4443
    loss_train 0.11039548059437733
    iteration 4444
    loss_train 0.11038790578575841
    iteration 4445
    loss_train 0.11038033340583842
    iteration 4446
    loss_train 0.11037276345337935
    iteration 4447
    loss_train 0.11036519592714404
    iteration 4448
    loss_train 0.11035763082589621
    iteration 4449
    loss_train 0.11035006814840052
    iteration 4450
    loss_train 0.11034250789342241
    iteration 4451
    loss_train 0.11033495005972832
    iteration 4452
    loss_train 0.11032739464608543
    iteration 4453
    loss_train 0.1103198416512619
    iteration 4454
    loss_train 0.11031229107402675
    iteration 4455
    loss_train 0.11030474291314979
    iteration 4456
    loss_train 0.11029719716740183
    iteration 4457
    loss_train 0.11028965383555449
    iteration 4458
    loss_train 0.11028211291638017
    iteration 4459
    loss_train 0.11027457440865235
    iteration 4460
    loss_train 0.11026703831114516
    iteration 4461
    loss_train 0.1102595046226338
    iteration 4462
    loss_train 0.11025197334189413
    iteration 4463
    loss_train 0.1102444444677031
    iteration 4464
    loss_train 0.11023691799883834
    iteration 4465
    loss_train 0.11022939393407846
    iteration 4466
    loss_train 0.11022187227220286
    iteration 4467
    loss_train 0.11021435301199184
    iteration 4468
    loss_train 0.11020683615222664
    iteration 4469
    loss_train 0.11019932169168922
    iteration 4470
    loss_train 0.1101918096291625
    iteration 4471
    loss_train 0.11018429996343015
    iteration 4472
    loss_train 0.11017679269327692
    iteration 4473
    loss_train 0.11016928781748819
    iteration 4474
    loss_train 0.11016178533485027
    iteration 4475
    loss_train 0.11015428524415041
    iteration 4476
    loss_train 0.11014678754417663
    iteration 4477
    loss_train 0.11013929223371786
    iteration 4478
    loss_train 0.1101317993115638
    iteration 4479
    loss_train 0.11012430877650513
    iteration 4480
    loss_train 0.11011682062733329
    iteration 4481
    loss_train 0.11010933486284057
    iteration 4482
    loss_train 0.11010185148182022
    iteration 4483
    loss_train 0.1100943704830662
    iteration 4484
    loss_train 0.11008689186537338
    iteration 4485
    loss_train 0.11007941562753755
    iteration 4486
    loss_train 0.11007194176835523
    iteration 4487
    loss_train 0.11006447028662389
    iteration 4488
    loss_train 0.11005700118114178
    iteration 4489
    loss_train 0.11004953445070807
    iteration 4490
    loss_train 0.11004207009412266
    iteration 4491
    loss_train 0.11003460811018638
    iteration 4492
    loss_train 0.11002714849770093
    iteration 4493
    loss_train 0.11001969125546876
    iteration 4494
    loss_train 0.11001223638229324
    iteration 4495
    loss_train 0.11000478387697861
    iteration 4496
    loss_train 0.1099973337383298
    iteration 4497
    loss_train 0.10998988596515274
    iteration 4498
    loss_train 0.10998244055625418
    iteration 4499
    loss_train 0.10997499751044162
    iteration 4500
    loss_train 0.10996755682652348
    iteration 4501
    loss_train 0.10996011850330892
    iteration 4502
    loss_train 0.10995268253960812
    iteration 4503
    loss_train 0.10994524893423185
    iteration 4504
    loss_train 0.109937817685992
    iteration 4505
    loss_train 0.10993038879370107
    iteration 4506
    loss_train 0.10992296225617242
    iteration 4507
    loss_train 0.10991553807222038
    iteration 4508
    loss_train 0.10990811624065995
    iteration 4509
    loss_train 0.10990069676030709
    iteration 4510
    loss_train 0.10989327962997852
    iteration 4511
    loss_train 0.10988586484849179
    iteration 4512
    loss_train 0.10987845241466536
    iteration 4513
    loss_train 0.10987104232731837
    iteration 4514
    loss_train 0.10986363458527096
    iteration 4515
    loss_train 0.1098562291873439
    iteration 4516
    loss_train 0.10984882613235898
    iteration 4517
    loss_train 0.10984142541913879
    iteration 4518
    loss_train 0.10983402704650658
    iteration 4519
    loss_train 0.10982663101328659
    iteration 4520
    loss_train 0.10981923731830383
    iteration 4521
    loss_train 0.10981184596038411
    iteration 4522
    loss_train 0.10980445693835411
    iteration 4523
    loss_train 0.1097970702510413
    iteration 4524
    loss_train 0.10978968589727395
    iteration 4525
    loss_train 0.10978230387588117
    iteration 4526
    loss_train 0.10977492418569294
    iteration 4527
    loss_train 0.10976754682554
    iteration 4528
    loss_train 0.10976017179425389
    iteration 4529
    loss_train 0.10975279909066707
    iteration 4530
    loss_train 0.10974542871361265
    iteration 4531
    loss_train 0.1097380606619247
    iteration 4532
    loss_train 0.10973069493443809
    iteration 4533
    loss_train 0.10972333152998841
    iteration 4534
    loss_train 0.10971597044741216
    iteration 4535
    loss_train 0.10970861168554655
    iteration 4536
    loss_train 0.10970125524322977
    iteration 4537
    loss_train 0.10969390111930064
    iteration 4538
    loss_train 0.1096865493125989
    iteration 4539
    loss_train 0.10967919982196506
    iteration 4540
    loss_train 0.10967185264624042
    iteration 4541
    loss_train 0.10966450778426719
    iteration 4542
    loss_train 0.10965716523488823
    iteration 4543
    loss_train 0.10964982499694732
    iteration 4544
    loss_train 0.10964248706928904
    iteration 4545
    loss_train 0.10963515145075872
    iteration 4546
    loss_train 0.10962781814020255
    iteration 4547
    loss_train 0.10962048713646745
    iteration 4548
    loss_train 0.10961315843840122
    iteration 4549
    loss_train 0.10960583204485244
    iteration 4550
    loss_train 0.10959850795467053
    iteration 4551
    loss_train 0.10959118616670556
    iteration 4552
    loss_train 0.10958386667980859
    iteration 4553
    loss_train 0.1095765494928314
    iteration 4554
    loss_train 0.10956923460462648
    iteration 4555
    loss_train 0.10956192201404728
    iteration 4556
    loss_train 0.10955461171994799
    iteration 4557
    loss_train 0.10954730372118353
    iteration 4558
    loss_train 0.10953999801660964
    iteration 4559
    loss_train 0.1095326946050829
    iteration 4560
    loss_train 0.1095253934854607
    iteration 4561
    loss_train 0.10951809465660114
    iteration 4562
    loss_train 0.10951079811736318
    iteration 4563
    loss_train 0.10950350386660652
    iteration 4564
    loss_train 0.10949621190319175
    iteration 4565
    loss_train 0.10948892222598013
    iteration 4566
    loss_train 0.10948163483383379
    iteration 4567
    loss_train 0.10947434972561557
    iteration 4568
    loss_train 0.10946706690018922
    iteration 4569
    loss_train 0.10945978635641916
    iteration 4570
    loss_train 0.10945250809317071
    iteration 4571
    loss_train 0.10944523210930984
    iteration 4572
    loss_train 0.10943795840370339
    iteration 4573
    loss_train 0.10943068697521906
    iteration 4574
    loss_train 0.10942341782272513
    iteration 4575
    loss_train 0.10941615094509088
    iteration 4576
    loss_train 0.10940888634118622
    iteration 4577
    loss_train 0.10940162400988192
    iteration 4578
    loss_train 0.10939436395004948
    iteration 4579
    loss_train 0.10938710616056126
    iteration 4580
    loss_train 0.10937985064029031
    iteration 4581
    loss_train 0.10937259738811053
    iteration 4582
    loss_train 0.10936534640289658
    iteration 4583
    loss_train 0.10935809768352382
    iteration 4584
    loss_train 0.10935085122886852
    iteration 4585
    loss_train 0.10934360703780768
    iteration 4586
    loss_train 0.109336365109219
    iteration 4587
    loss_train 0.10932912544198099
    iteration 4588
    loss_train 0.10932188803497309
    iteration 4589
    loss_train 0.10931465288707524
    iteration 4590
    loss_train 0.10930741999716836
    iteration 4591
    loss_train 0.1093001893641341
    iteration 4592
    loss_train 0.10929296098685484
    iteration 4593
    loss_train 0.10928573486421371
    iteration 4594
    loss_train 0.1092785109950947
    iteration 4595
    loss_train 0.10927128937838251
    iteration 4596
    loss_train 0.10926407001296261
    iteration 4597
    loss_train 0.10925685289772129
    iteration 4598
    loss_train 0.1092496380315455
    iteration 4599
    loss_train 0.10924242541332307
    iteration 4600
    loss_train 0.10923521504194257
    iteration 4601
    loss_train 0.10922800691629325
    iteration 4602
    loss_train 0.10922080103526524
    iteration 4603
    loss_train 0.10921359739774941
    iteration 4604
    loss_train 0.10920639600263733
    iteration 4605
    loss_train 0.10919919684882136
    iteration 4606
    loss_train 0.10919199993519468
    iteration 4607
    loss_train 0.10918480526065115
    iteration 4608
    loss_train 0.10917761282408546
    iteration 4609
    loss_train 0.10917042262439296
    iteration 4610
    loss_train 0.10916323466046993
    iteration 4611
    loss_train 0.10915604893121318
    iteration 4612
    loss_train 0.10914886543552058
    iteration 4613
    loss_train 0.1091416841722904
    iteration 4614
    loss_train 0.10913450514042199
    iteration 4615
    loss_train 0.10912732833881522
    iteration 4616
    loss_train 0.10912015376637085
    iteration 4617
    loss_train 0.10911298142199037
    iteration 4618
    loss_train 0.10910581130457599
    iteration 4619
    loss_train 0.10909864341303069
    iteration 4620
    loss_train 0.10909147774625823
    iteration 4621
    loss_train 0.10908431430316308
    iteration 4622
    loss_train 0.10907715308265047
    iteration 4623
    loss_train 0.10906999408362643
    iteration 4624
    loss_train 0.10906283730499765
    iteration 4625
    loss_train 0.10905568274567169
    iteration 4626
    loss_train 0.10904853040455671
    iteration 4627
    loss_train 0.10904138028056178
    iteration 4628
    loss_train 0.10903423237259656
    iteration 4629
    loss_train 0.10902708667957156
    iteration 4630
    loss_train 0.10901994320039804
    iteration 4631
    loss_train 0.10901280193398794
    iteration 4632
    loss_train 0.10900566287925398
    iteration 4633
    loss_train 0.10899852603510966
    iteration 4634
    loss_train 0.10899139140046914
    iteration 4635
    loss_train 0.1089842589742474
    iteration 4636
    loss_train 0.10897712875536013
    iteration 4637
    loss_train 0.10897000074272373
    iteration 4638
    loss_train 0.10896287493525542
    iteration 4639
    loss_train 0.10895575133187314
    iteration 4640
    loss_train 0.10894862993149547
    iteration 4641
    loss_train 0.10894151073304181
    iteration 4642
    loss_train 0.10893439373543237
    iteration 4643
    loss_train 0.10892727893758791
    iteration 4644
    loss_train 0.10892016633843012
    iteration 4645
    loss_train 0.10891305593688135
    iteration 4646
    loss_train 0.10890594773186461
    iteration 4647
    loss_train 0.1088988417223038
    iteration 4648
    loss_train 0.1088917379071234
    iteration 4649
    loss_train 0.10888463628524869
    iteration 4650
    loss_train 0.1088775368556057
    iteration 4651
    loss_train 0.10887043961712124
    iteration 4652
    loss_train 0.10886334456872274
    iteration 4653
    loss_train 0.10885625170933835
    iteration 4654
    loss_train 0.10884916103789709
    iteration 4655
    loss_train 0.10884207255332864
    iteration 4656
    loss_train 0.10883498625456335
    iteration 4657
    loss_train 0.10882790214053238
    iteration 4658
    loss_train 0.10882082021016758
    iteration 4659
    loss_train 0.10881374046240151
    iteration 4660
    loss_train 0.10880666289616753
    iteration 4661
    loss_train 0.10879958751039961
    iteration 4662
    loss_train 0.1087925143040326
    iteration 4663
    loss_train 0.10878544327600195
    iteration 4664
    loss_train 0.10877837442524381
    iteration 4665
    loss_train 0.10877130775069518
    iteration 4666
    loss_train 0.10876424325129373
    iteration 4667
    loss_train 0.10875718092597782
    iteration 4668
    loss_train 0.10875012077368654
    iteration 4669
    loss_train 0.10874306279335968
    iteration 4670
    loss_train 0.10873600698393786
    iteration 4671
    loss_train 0.10872895334436228
    iteration 4672
    loss_train 0.10872190187357499
    iteration 4673
    loss_train 0.1087148525705186
    iteration 4674
    loss_train 0.1087078054341366
    iteration 4675
    loss_train 0.10870076046337308
    iteration 4676
    loss_train 0.10869371765717291
    iteration 4677
    loss_train 0.1086866770144817
    iteration 4678
    loss_train 0.10867963853424564
    iteration 4679
    loss_train 0.1086726022154118
    iteration 4680
    loss_train 0.10866556805692788
    iteration 4681
    loss_train 0.10865853605774227
    iteration 4682
    loss_train 0.10865150621680413
    iteration 4683
    loss_train 0.1086444785330633
    iteration 4684
    loss_train 0.10863745300547037
    iteration 4685
    loss_train 0.1086304296329766
    iteration 4686
    loss_train 0.10862340841453394
    iteration 4687
    loss_train 0.10861638934909507
    iteration 4688
    loss_train 0.1086093724356135
    iteration 4689
    loss_train 0.10860235767304326
    iteration 4690
    loss_train 0.10859534506033912
    iteration 4691
    loss_train 0.10858833459645668
    iteration 4692
    loss_train 0.10858132628035214
    iteration 4693
    loss_train 0.10857432011098243
    iteration 4694
    loss_train 0.10856731608730523
    iteration 4695
    loss_train 0.10856031420827884
    iteration 4696
    loss_train 0.10855331447286236
    iteration 4697
    loss_train 0.10854631688001547
    iteration 4698
    loss_train 0.1085393214286987
    iteration 4699
    loss_train 0.10853232811787317
    iteration 4700
    loss_train 0.10852533694650073
    iteration 4701
    loss_train 0.10851834791354399
    iteration 4702
    loss_train 0.10851136101796616
    iteration 4703
    loss_train 0.10850437625873122
    iteration 4704
    loss_train 0.10849739363480387
    iteration 4705
    loss_train 0.1084904131451494
    iteration 4706
    loss_train 0.10848343478873389
    iteration 4707
    loss_train 0.10847645856452413
    iteration 4708
    loss_train 0.10846948447148753
    iteration 4709
    loss_train 0.10846251250859226
    iteration 4710
    loss_train 0.10845554267480714
    iteration 4711
    loss_train 0.10844857496910174
    iteration 4712
    loss_train 0.10844160939044628
    iteration 4713
    loss_train 0.10843464593781166
    iteration 4714
    loss_train 0.10842768461016951
    iteration 4715
    loss_train 0.10842072540649217
    iteration 4716
    loss_train 0.10841376832575263
    iteration 4717
    loss_train 0.10840681336692455
    iteration 4718
    loss_train 0.10839986052898234
    iteration 4719
    loss_train 0.10839290981090108
    iteration 4720
    loss_train 0.10838596121165653
    iteration 4721
    loss_train 0.10837901473022513
    iteration 4722
    loss_train 0.10837207036558404
    iteration 4723
    loss_train 0.10836512811671109
    iteration 4724
    loss_train 0.1083581879825848
    iteration 4725
    loss_train 0.10835124996218436
    iteration 4726
    loss_train 0.1083443140544896
    iteration 4727
    loss_train 0.10833738025848123
    iteration 4728
    loss_train 0.10833044857314039
    iteration 4729
    loss_train 0.10832351899744906
    iteration 4730
    loss_train 0.10831659153038986
    iteration 4731
    loss_train 0.10830966617094613
    iteration 4732
    loss_train 0.10830274291810181
    iteration 4733
    loss_train 0.10829582177084161
    iteration 4734
    loss_train 0.10828890272815088
    iteration 4735
    loss_train 0.10828198578901567
    iteration 4736
    loss_train 0.10827507095242264
    iteration 4737
    loss_train 0.10826815821735922
    iteration 4738
    loss_train 0.10826124758281347
    iteration 4739
    loss_train 0.10825433904777418
    iteration 4740
    loss_train 0.10824743261123072
    iteration 4741
    loss_train 0.10824052827217322
    iteration 4742
    loss_train 0.10823362602959244
    iteration 4743
    loss_train 0.10822672588247988
    iteration 4744
    loss_train 0.10821982782982761
    iteration 4745
    loss_train 0.10821293187062851
    iteration 4746
    loss_train 0.10820603800387607
    iteration 4747
    loss_train 0.10819914622856434
    iteration 4748
    loss_train 0.10819225654368828
    iteration 4749
    loss_train 0.10818536894824325
    iteration 4750
    loss_train 0.10817848344122548
    iteration 4751
    loss_train 0.10817160002163186
    iteration 4752
    loss_train 0.10816471868845987
    iteration 4753
    loss_train 0.10815783944070766
    iteration 4754
    loss_train 0.10815096227737413
    iteration 4755
    loss_train 0.10814408719745877
    iteration 4756
    loss_train 0.10813721419996178
    iteration 4757
    loss_train 0.10813034328388406
    iteration 4758
    loss_train 0.10812347444822705
    iteration 4759
    loss_train 0.10811660769199298
    iteration 4760
    loss_train 0.10810974301418469
    iteration 4761
    loss_train 0.10810288041380575
    iteration 4762
    loss_train 0.10809601988986033
    iteration 4763
    loss_train 0.10808916144135328
    iteration 4764
    loss_train 0.10808230506729007
    iteration 4765
    loss_train 0.10807545076667689
    iteration 4766
    loss_train 0.10806859853852063
    iteration 4767
    loss_train 0.10806174838182878
    iteration 4768
    loss_train 0.10805490029560945
    iteration 4769
    loss_train 0.10804805427887151
    iteration 4770
    loss_train 0.10804121033062444
    iteration 4771
    loss_train 0.10803436844987838
    iteration 4772
    loss_train 0.10802752863564412
    iteration 4773
    loss_train 0.10802069088693314
    iteration 4774
    loss_train 0.10801385520275754
    iteration 4775
    loss_train 0.10800702158213013
    iteration 4776
    loss_train 0.10800019002406432
    iteration 4777
    loss_train 0.10799336052757415
    iteration 4778
    loss_train 0.10798653309167448
    iteration 4779
    loss_train 0.10797970771538065
    iteration 4780
    loss_train 0.10797288439770865
    iteration 4781
    loss_train 0.10796606313767529
    iteration 4782
    loss_train 0.10795924393429791
    iteration 4783
    loss_train 0.10795242678659447
    iteration 4784
    loss_train 0.1079456116935837
    iteration 4785
    loss_train 0.10793879865428488
    iteration 4786
    loss_train 0.10793198766771804
    iteration 4787
    loss_train 0.10792517873290378
    iteration 4788
    loss_train 0.10791837184886326
    iteration 4789
    loss_train 0.10791156701461854
    iteration 4790
    loss_train 0.10790476422919215
    iteration 4791
    loss_train 0.10789796349160727
    iteration 4792
    loss_train 0.1078911648008878
    iteration 4793
    loss_train 0.10788436815605827
    iteration 4794
    loss_train 0.10787757355614379
    iteration 4795
    loss_train 0.10787078100017024
    iteration 4796
    loss_train 0.10786399048716397
    iteration 4797
    loss_train 0.10785720201615215
    iteration 4798
    loss_train 0.1078504155861625
    iteration 4799
    loss_train 0.10784363119622344
    iteration 4800
    loss_train 0.1078368488453639
    iteration 4801
    loss_train 0.10783006853261368
    iteration 4802
    loss_train 0.10782329025700298
    iteration 4803
    loss_train 0.10781651401756286
    iteration 4804
    loss_train 0.10780973981332483
    iteration 4805
    loss_train 0.10780296764332119
    iteration 4806
    loss_train 0.10779619750658474
    iteration 4807
    loss_train 0.1077894294021491
    iteration 4808
    loss_train 0.1077826633290483
    iteration 4809
    loss_train 0.10777589928631727
    iteration 4810
    loss_train 0.10776913727299134
    iteration 4811
    loss_train 0.10776237728810666
    iteration 4812
    loss_train 0.10775561933069985
    iteration 4813
    loss_train 0.10774886339980833
    iteration 4814
    loss_train 0.10774210949447002
    iteration 4815
    loss_train 0.10773535761372358
    iteration 4816
    loss_train 0.1077286077566082
    iteration 4817
    loss_train 0.10772185992216385
    iteration 4818
    loss_train 0.10771511410943092
    iteration 4819
    loss_train 0.1077083703174507
    iteration 4820
    loss_train 0.1077016285452649
    iteration 4821
    loss_train 0.1076948887919159
    iteration 4822
    loss_train 0.10768815105644683
    iteration 4823
    loss_train 0.10768141533790129
    iteration 4824
    loss_train 0.10767468163532364
    iteration 4825
    loss_train 0.10766794994775877
    iteration 4826
    loss_train 0.10766122027425232
    iteration 4827
    loss_train 0.10765449261385036
    iteration 4828
    loss_train 0.1076477669655998
    iteration 4829
    loss_train 0.1076410433285481
    iteration 4830
    loss_train 0.10763432170174324
    iteration 4831
    loss_train 0.10762760208423404
    iteration 4832
    loss_train 0.1076208844750698
    iteration 4833
    loss_train 0.10761416887330043
    iteration 4834
    loss_train 0.10760745527797652
    iteration 4835
    loss_train 0.10760074368814934
    iteration 4836
    loss_train 0.10759403410287066
    iteration 4837
    loss_train 0.10758732652119289
    iteration 4838
    loss_train 0.10758062094216922
    iteration 4839
    loss_train 0.10757391736485325
    iteration 4840
    loss_train 0.1075672157882993
    iteration 4841
    loss_train 0.1075605162115624
    iteration 4842
    loss_train 0.107553818633698
    iteration 4843
    loss_train 0.10754712305376234
    iteration 4844
    loss_train 0.10754042947081224
    iteration 4845
    loss_train 0.10753373788390506
    iteration 4846
    loss_train 0.10752704829209889
    iteration 4847
    loss_train 0.10752036069445235
    iteration 4848
    loss_train 0.10751367509002473
    iteration 4849
    loss_train 0.10750699147787593
    iteration 4850
    loss_train 0.10750030985706643
    iteration 4851
    loss_train 0.1074936302266574
    iteration 4852
    loss_train 0.10748695258571053
    iteration 4853
    loss_train 0.10748027693328824
    iteration 4854
    loss_train 0.10747360326845341
    iteration 4855
    loss_train 0.10746693159026967
    iteration 4856
    loss_train 0.1074602618978012
    iteration 4857
    loss_train 0.10745359419011286
    iteration 4858
    loss_train 0.10744692846627
    iteration 4859
    loss_train 0.1074402647253387
    iteration 4860
    loss_train 0.10743360296638559
    iteration 4861
    loss_train 0.10742694318847792
    iteration 4862
    loss_train 0.10742028539068356
    iteration 4863
    loss_train 0.10741362957207097
    iteration 4864
    loss_train 0.10740697573170928
    iteration 4865
    loss_train 0.1074003238686681
    iteration 4866
    loss_train 0.10739367398201784
    iteration 4867
    loss_train 0.10738702607082933
    iteration 4868
    loss_train 0.1073803801341741
    iteration 4869
    loss_train 0.10737373617112429
    iteration 4870
    loss_train 0.1073670941807526
    iteration 4871
    loss_train 0.10736045416213241
    iteration 4872
    loss_train 0.10735381611433767
    iteration 4873
    loss_train 0.10734718003644286
    iteration 4874
    loss_train 0.10734054592752314
    iteration 4875
    loss_train 0.1073339137866543
    iteration 4876
    loss_train 0.10732728361291266
    iteration 4877
    loss_train 0.10732065540537522
    iteration 4878
    loss_train 0.10731402916311951
    iteration 4879
    loss_train 0.10730740488522371
    iteration 4880
    loss_train 0.10730078257076653
    iteration 4881
    loss_train 0.10729416221882741
    iteration 4882
    loss_train 0.10728754382848629
    iteration 4883
    loss_train 0.1072809273988237
    iteration 4884
    loss_train 0.10727431292892084
    iteration 4885
    loss_train 0.10726770041785944
    iteration 4886
    loss_train 0.10726108986472188
    iteration 4887
    loss_train 0.1072544812685911
    iteration 4888
    loss_train 0.10724787462855066
    iteration 4889
    loss_train 0.10724126994368476
    iteration 4890
    loss_train 0.10723466721307807
    iteration 4891
    loss_train 0.10722806643581596
    iteration 4892
    loss_train 0.10722146761098438
    iteration 4893
    loss_train 0.10721487073766982
    iteration 4894
    loss_train 0.10720827581495948
    iteration 4895
    loss_train 0.10720168284194095
    iteration 4896
    loss_train 0.10719509181770272
    iteration 4897
    loss_train 0.10718850274133354
    iteration 4898
    loss_train 0.10718191561192299
    iteration 4899
    loss_train 0.10717533042856117
    iteration 4900
    loss_train 0.1071687471903387
    iteration 4901
    loss_train 0.10716216589634682
    iteration 4902
    loss_train 0.10715558654567753
    iteration 4903
    loss_train 0.10714900913742313
    iteration 4904
    loss_train 0.10714243367067675
    iteration 4905
    loss_train 0.10713586014453196
    iteration 4906
    loss_train 0.107129288558083
    iteration 4907
    loss_train 0.1071227189104247
    iteration 4908
    loss_train 0.10711615120065242
    iteration 4909
    loss_train 0.10710958542786214
    iteration 4910
    loss_train 0.10710302159115039
    iteration 4911
    loss_train 0.10709645968961438
    iteration 4912
    loss_train 0.1070898997223518
    iteration 4913
    loss_train 0.107083341688461
    iteration 4914
    loss_train 0.10707678558704087
    iteration 4915
    loss_train 0.10707023141719087
    iteration 4916
    loss_train 0.10706367917801109
    iteration 4917
    loss_train 0.10705712886860215
    iteration 4918
    loss_train 0.10705058048806533
    iteration 4919
    loss_train 0.10704403403550247
    iteration 4920
    loss_train 0.10703748951001586
    iteration 4921
    loss_train 0.10703094691070862
    iteration 4922
    loss_train 0.10702440623668415
    iteration 4923
    loss_train 0.10701786748704674
    iteration 4924
    loss_train 0.10701133066090097
    iteration 4925
    loss_train 0.1070047957573522
    iteration 4926
    loss_train 0.10699826277550635
    iteration 4927
    loss_train 0.10699173171446977
    iteration 4928
    loss_train 0.10698520257334955
    iteration 4929
    loss_train 0.10697867535125331
    iteration 4930
    loss_train 0.10697215004728919
    iteration 4931
    loss_train 0.106965626660566
    iteration 4932
    loss_train 0.106959105190193
    iteration 4933
    loss_train 0.10695258563528016
    iteration 4934
    loss_train 0.10694606799493794
    iteration 4935
    loss_train 0.1069395522682774
    iteration 4936
    loss_train 0.10693303845441017
    iteration 4937
    loss_train 0.10692652655244843
    iteration 4938
    loss_train 0.10692001656150502
    iteration 4939
    loss_train 0.10691350848069324
    iteration 4940
    loss_train 0.10690700230912697
    iteration 4941
    loss_train 0.10690049804592076
    iteration 4942
    loss_train 0.10689399569018969
    iteration 4943
    loss_train 0.10688749524104936
    iteration 4944
    loss_train 0.10688099669761593
    iteration 4945
    loss_train 0.10687450005900624
    iteration 4946
    loss_train 0.1068680053243376
    iteration 4947
    loss_train 0.10686151249272788
    iteration 4948
    loss_train 0.10685502156329564
    iteration 4949
    loss_train 0.10684853253515983
    iteration 4950
    loss_train 0.10684204540744013
    iteration 4951
    loss_train 0.10683556017925668
    iteration 4952
    loss_train 0.10682907684973024
    iteration 4953
    loss_train 0.10682259541798209
    iteration 4954
    loss_train 0.10681611588313417
    iteration 4955
    loss_train 0.10680963824430881
    iteration 4956
    loss_train 0.10680316250062907
    iteration 4957
    loss_train 0.10679668865121852
    iteration 4958
    loss_train 0.10679021669520132
    iteration 4959
    loss_train 0.10678374663170206
    iteration 4960
    loss_train 0.10677727845984611
    iteration 4961
    loss_train 0.10677081217875917
    iteration 4962
    loss_train 0.10676434778756769
    iteration 4963
    loss_train 0.10675788528539859
    iteration 4964
    loss_train 0.10675142467137935
    iteration 4965
    loss_train 0.10674496594463805
    iteration 4966
    loss_train 0.1067385091043033
    iteration 4967
    loss_train 0.10673205414950422
    iteration 4968
    loss_train 0.10672560107937062
    iteration 4969
    loss_train 0.10671914989303274
    iteration 4970
    loss_train 0.10671270058962147
    iteration 4971
    loss_train 0.10670625316826816
    iteration 4972
    loss_train 0.10669980762810481
    iteration 4973
    loss_train 0.10669336396826397
    iteration 4974
    loss_train 0.10668692218787863
    iteration 4975
    loss_train 0.10668048228608253
    iteration 4976
    loss_train 0.10667404426200974
    iteration 4977
    loss_train 0.10666760811479503
    iteration 4978
    loss_train 0.10666117384357374
    iteration 4979
    loss_train 0.10665474144748165
    iteration 4980
    loss_train 0.1066483109256552
    iteration 4981
    loss_train 0.10664188227723134
    iteration 4982
    loss_train 0.10663545550134758
    iteration 4983
    loss_train 0.10662903059714192
    iteration 4984
    loss_train 0.10662260756375301
    iteration 4985
    loss_train 0.10661618640031995
    iteration 4986
    loss_train 0.1066097671059826
    iteration 4987
    loss_train 0.10660334967988104
    iteration 4988
    loss_train 0.10659693412115614
    iteration 4989
    loss_train 0.10659052042894926
    iteration 4990
    loss_train 0.10658410860240229
    iteration 4991
    loss_train 0.10657769864065772
    iteration 4992
    loss_train 0.1065712905428585
    iteration 4993
    loss_train 0.10656488430814819
    iteration 4994
    loss_train 0.10655847993567087
    iteration 4995
    loss_train 0.1065520774245712
    iteration 4996
    loss_train 0.10654567677399436
    iteration 4997
    loss_train 0.10653927798308604
    iteration 4998
    loss_train 0.10653288105099254
    iteration 4999
    loss_train 0.10652648597686074
    iteration 5000
    loss_train 0.10652009275983784

Let's see how our model did while learning!

```python
# Produce the Loss Function Visualization Graphs
fig, ax = plt.subplots(figsize=(18,8))
ax.plot(losses[0], color='blue', label='Training', linewidth=3);
ax.plot(losses[1], color='orange', label='Validation', linewidth=3);
ax.legend();
ax.set_ylabel('Log Loss')
ax.set_xlabel('Iterations')
ax.set_title('Loss Function Graph')
ax.autoscale(axis='x', tight=True)
fig.tight_layout();

```

```python
# Let's get predictions from our model for the training, validation, and testing
# datasets
y_hat_train = (predict(X_train, coeffs)>=.5).astype(int)
y_hat_valid = (predict(X_valid, coeffs)>=.5).astype(int)
y_hat_test = (predict(X_test, coeffs)>=.5).astype(int)

y_sets = [ [y_hat_train, y_train],
           [y_hat_valid, y_valid],
           [y_hat_test, y_test] ]

def accuracy_score(y_hat, y):
    assert(y_hat.size==y.size)
    return (y_hat == y).sum()/y.size

[accuracies.append(accuracy_score(y_set[0],y_set[1])) for y_set in y_sets]

printout= (f'Training Accuracy:{accuracies[0]:.1%} \n'
           f'Validation Accuracy:{accuracies[1]:.1%} \n')

# Add the testing accuracy only once you're sure that your model works!

print(printout)
```

Congratulations on training a logistic regression algorithm from scratch! Once you're done with the upcoming environmental science applications notebook, feel free to come back to take a look at the challenges 😀

## Challenges

* **C1)** Add L2 Regularization to training function

* **C2)** Add early stopping to the training algorithm! Stop training when the accuracy is >=90%

* **C3)** Implement a softmax regression model (It's multiclass logistic regression 🙂)

```python
a=np.array([1,2,6,4,7])
b=a<4
print(b)

```

    [ True  True False False False]

```python

```