# <markdowncell>

# # Convolutional Neural Network Workshop
#
# The aim of this workshop is to demonstrate the basics of using [TensorFlow](https://www.tensorflow.org/) for
# convolutional neural networks.
#
# TensorFlow is an open source system developed by Google Brain for working with computation graphs. It supports
# automatic differentiation, making it suitable for implementing backpropagation.
#
# Optimized implementation of convolutional network is not trivial, especially on GPU. Therefore, in your projects
# you are much more likely to use frameworks like TensorFlow than to code from scratch.
#
# This notebook will cover the basics, providing references to additional resources where needed. You are welcome
# to use many excellent tutorials on the web, including the [one from Google](https://www.tensorflow.org/get_started/get_started).

# Overview:
# * [Preparing data](#Preparing-data)
# * [Defining and examining a simple network](#Defining-and-examining-a-simple-network)
# * [Adding batch normalization](#Adding-batch-normalization)
# * [Training a more complex network](#Training-a-more-complex-network)
# * [Saving and loading models](#Saving-and-loading-models)
# * [Transfer learning](#Transfer-learning)

# <markdowncell>

# **Recommended TensorFlow version is 1.9.0.** If you are working on a machine with GPU, it is recommended that you
# **install GPU version of TensorFlow** (`tensorflow-gpu` Python package). You will also need `matplotlib`.
#
# We begin by importing all modules that we will need, including TensorFlow. This also verifies TensorFlow installation.

# <codecell>

import os
import tensorflow as tf
import numpy as np
from matplotlib import pyplot as plt
get_ipython().run_line_magic('matplotlib', 'inline')
from utils import load_cifar10, visualize_grid

# <markdowncell>

# ## Preparing data
#
# We will use CIFAR-10 dataset. Please download it from [here](https://www.cs.toronto.edu/~kriz/cifar.html)
# (alternatively from [here](https://1drv.ms/u/s!ApgLrbSPQF9XirQ0fEqljE5_IT0q6g)
# or [here](https://1drv.ms/u/s!ApgLrbSPQF9XirQ16VxWmz_x0bRwMg)), and make sure to specify correct location
# in `cifar10_dir` below. We load the dataset by calling a utility function.

# <codecell>

cifar10_dir = 'cifar-10-batches-py'
x_dev, y_dev, x_test, y_test = load_cifar10(cifar10_dir)

# <markdowncell>

# The dataset is divided into part that can be used for training and validation (which we will call
# *development*), and test part.
#
# There are 50,000 dev images, and 10,000 test images. test images. All images are 32 x 32 pixels, and
# have 3 channels. Each image is assigned to one of 10 classes.

# <codecell>

print('Dev data shape: ', x_dev.shape)
print('Dev labels shape: ', y_dev.shape)
print('Test data shape: ', x_test.shape)
print('Test labels shape: ', y_test.shape)
print('Class labels: ', np.unique(np.concatenate([y_dev, y_test])))

# <markdowncell>

# We define some useful constants to be used throughout

# <codecell>

num_dev = x_dev.shape[0]
num_test = x_test.shape[0]
num_classes = len(np.unique(np.concatenate([y_dev, y_test])))

# <markdowncell>

# ### Training/validation split
#
# From dev data we take 10% for validation, and use the rest for training.
# To make sure that train and dev set have the same distribution, we randomize
# dev set first.

# <codecell>

num_val = num_dev // 10
num_train = num_dev - num_val

perm = np.random.permutation(num_dev)
x_dev = x_dev[perm]
y_dev = y_dev[perm]

x_train = x_dev[:num_train]
y_train = y_dev[:num_train]
x_val = x_dev[num_train:]
y_val = y_dev[num_train:]

# <markdowncell>

# ### Data normalization
#
# We normalize all data by subtracting mean image from each sample.
# Note that the mean image is computed from training set only.

# <codecell>

mean_image = np.mean(x_train, axis=0)
x_train -= mean_image
x_val -= mean_image
x_test -= mean_image

# <markdowncell>

# ### Visualizing training examples

# Let's look at `sample_size` random examples from each class.

# <codecell>

sample_size = 10
sample = x_train[[i for c in range(num_classes)
                  for i in np.random.permutation(np.flatnonzero(y_train == c))[:sample_size]]]
sample += mean_image
plt.figure(figsize = (12, 12))
plt.imshow(visualize_grid(sample))
plt.axis("off")
plt.show()

# <markdowncell>

# Now that we know what the classes represent, we can create an array of human-readable names.

# <codecell>

class_names = ['airplane', 'automobile', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']

# <markdowncell>

# ## Defining and examining a simple network
#
# We define a simple network consisting of basic layers:
# * convolutional layer,
# * max-pooling layer,
# * fully connected layer, and
# * ReLU activation function.
#
# TensorFlow supports many other layer types and activations. See https://www.tensorflow.org/api_guides/python/nn for official API documentation.

# <markdowncell>

# The following line clears any network that might already exist in memory.

# <codecell>

tf.reset_default_graph()

# <markdowncell>

# ### Create TensorBoard log file
#
# We will use TensorBoard to visualize various data about our network. TensorBoard parses log files (also called event files) generated by TensorFlow. We will be placing those files in a separate dir.

# <codecell>

log_dir = './logs/'

# <markdowncell>

# A new event file is created by instantiating a `tf.FileWriter` class.

# <codecell>

writer = tf.summary.FileWriter(os.path.join(log_dir, 'simple_net'))

# <markdowncell>

# ### Placeholders for data
# First we define placeholders for input data (input image and its label) using `tf.placeholder`.
# We will eventually bind these to actual numerical data values.
#
# We choose to represent input data as 4D tensors whose shape is N x H x W x C, where:
# * N is the number of examples in a batch (batch size)
# * H is the height of each image in pixels
# * W is the height of each image in pixels
# * C is the number of channels (usually 3: R, G, B)
#
# This is the right way to represent the data for spatial operations like convolution. For fully connected layers, however, all dimensions except batch size will be collapsed into one.
#
# In `tf.placeholder`, if a dimension has value `None`, it will be set automatically once actual data is provided.

# <codecell>

def setup_input():
    x = tf.placeholder(tf.float32, [None, 32, 32, 3], name='x')
    y = tf.placeholder(tf.int64, [None], name='y')
    is_training = tf.placeholder(tf.bool, name='is_training')
    return x, y, is_training

# <codecell>

x, y, is_training = setup_input()

# <markdowncell>

# ### Convolutional and pooling nodes
# Next we start defining the main "body" of the network.
# We start by adding a single convolutional layer with bias and ReLU activation.
#
# We use [tf.layers API](https://www.tensorflow.org/api_docs/python/tf/layers) to generate a whole layer by a single function call.
# It is also possible to create each parameter and operation node separately, and connect them together, but that quickly becomes
# cumbersome for bigger networks.
#
# TensorFlow also provides other high-level APIs, such as [TensorFlow-Slim](https://github.com/tensorflow/tensorflow/tree/master/tensorflow/contrib/slim), which
# has become very popular, with a large number of [pretrained models](https://github.com/tensorflow/models/tree/master/research/slim) written in it.
# Fortunately, TF-Slim models can be quite easily combined with other TensorFlow APIs such as tf.layers.
#
# Convolutional layer is created by calling `tf.layers.conv2d`. Returned object is of type `tf.Tensor` and represents output activations of the layer.
#
# Bias is enabled by default, so it is not explicitly specified. `padding='SAME'` means that we allow padding of roughly half the kernel size
# (TensorFlow computes this value automatically), to avoid reduction in output size due to boundary effects. The other option is `padding='VALID'`, which means
# that padding is zero.

# <codecell>

conv1 = tf.layers.conv2d(inputs=x, filters=32, kernel_size=[7, 7], strides=2, padding='SAME', activation=tf.nn.relu, name='conv1')

# <markdowncell>

# Next we add a max-pooling node.

# <codecell>

pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2, padding='SAME', name='pool1')

# <markdowncell>

# ### View default graph in TensorBoard
#
# We can write graph data to event file we create above. A graph can be passed to `FileWriter` constructor as well, in which case it is written to file immediately after the file is created.

# <codecell>

writer.add_graph(tf.get_default_graph())

# <markdowncell>

# Now you should be able to run `tensorboard --logdir=./logs` from console (with your Python environment activated), and see the graph visualized in browser at `http://localhost:6006`.
#
# For more details please see official tutorial on [graph visualization](https://www.tensorflow.org/get_started/graph_viz).
#
# Note: graph visualization seems to work best in Google Chrome.

# <markdowncell>

# ### Examining static information
# Most information about the network which is static (i.e. independent of input data)
# can be seen from TensorBoard graph visualization -- node names, connectivity, tensor sizes,
# some layer parameters, but not tensor values.
#
# This data is also acessible from Python code, which is useful in some situations.
#
# If we already have handles on nodes, for example because we just created them, such as `conv1`
# and `pool1` above, we can query their properties.

# <codecell>

print(conv1.shape)
print(pool1.shape)

# <markdowncell>

# Note that shapes of activation tensors are computed automatically. Also, these tensors
# may have unknown dimensions which become known only when acutal input is presented.

# We may not have a handle on some nodes, for example internal parameters of `conv1` (kernel and bias).
# Those tensors are "hidden" inside the layer, because we used layers API. We can still access them
# using name lookup.

# <codecell>

conv1_kernel = tf.get_default_graph().get_tensor_by_name('conv1/kernel:0')
conv1_bias = tf.get_default_graph().get_tensor_by_name('conv1/bias:0')

# <markdowncell>

# The names can be found from TensorBoard visualization, or by listing all operations (nodes) in the network.

# <codecell>
tf.get_default_graph().get_operations()

# <markdowncell>

# The `conv1` prefix in both names (*name scope*) refers to the `name` parameter specified when creating
# the layer, and NOT the Python variable `conv1` that we assigned the result to. The `:0` suffix means that
# the tensor is the first (index 0) output of the node that produces it (`conv/kernel` and `conv1/bias`, respectively).
#
# We can get the shapes of kernel and bias as follows

# <codecell>

print(conv1_kernel.shape)
print(conv1_bias.shape)

# <markdowncell>

# Finally, we can also access internal parameters of operations (rather than their tensor outputs) using
# `get_operation_by_name` followed by `get_attr`.

# <codecell>

print(tf.get_default_graph().get_operation_by_name('conv1/Conv2D').get_attr('strides'))

# <markdowncell>

# ### Fully connected layers
# Next we append a fully connected layer with 1024 output neurons and ReLU activation.
# In order to determine the shape of its parameter tensor, we need to know the number of input neurons, which depends on the shape of the `relu1` activation tensor.

# <codecell>

fc1_input_count = int(pool1.shape[1] * pool1.shape[2] * pool1.shape[3])
fc1_output_count = 1024
print([fc1_input_count, fc1_output_count])

# <markdowncell>

# In order to append a fully connected layer, we need to flatten the spatial dimensions of `relu1`.

# <codecell>

pool1_flat = tf.reshape(pool1, [-1, fc1_input_count])

# <markdowncell>

# Now we are ready to add a fully connected layer.

# <codecell>

fc1 = tf.layers.dense(inputs=pool1_flat, units=fc1_output_count, activation=tf.nn.relu, name='fc1')

# <markdowncell>

# Finally, we add another fully connected layer with bias to output scores for 10 output classes. This layer has no nonlinearity following it, but it will be followed by a softmax function to convert scores to probabilities.

# <codecell>

fc2 = tf.layers.dense(inputs=fc1, units=num_classes, name='fc2')

# <markdowncell>

# ### Final classification
# We append a softmax layer to convert the scores coming from `fc2` into probabilities, as well as a "top-k" layer to get the three most probable guesses.

# <codecell>

prob = tf.nn.softmax(fc2)
(guess_prob, guess_class) = tf.nn.top_k(prob, k=3)

# <markdowncell>

# ### Visualizing parameters and activations
# TensorBoard supports visualizing tensors as images using `tf.summary.image` function.
# This function adds a subnetwork that computes images for a given tensor.

# <codecell>

def setup_image_summary(tensor, name):
    with tf.variable_scope(name):
        # Normalize to [0 1].
        x_min = tf.reduce_min(tensor)
        x_max = tf.reduce_max(tensor)
        normalized = (tensor - x_min) / (x_max - x_min)

        # Display random 3 slices.
        return tf.summary.image('tensor', normalized, max_outputs=3)

# <markdowncell>

# Using this we visualize some of the weights in `conv1_kernel`. Since each filter has 3
# input channels, it be visualized as an RGB image. Note that we need to transpose the
# tensor from (H, W, C, N) to (N, H, W, C) layout.

# <codecell>

conv1_kernel_summary = setup_image_summary(
    tf.transpose(conv1_kernel, [3, 0, 1, 2]),
    name='conv1_kernel_summary')

# <markdowncell>

# Similarly, we can visualize some of the activations in `conv1`. Use `setup_image_summary`
# in code below to visualize a subset of channels for example 0 in minibatch. Each channel
# should be visualized as a grayscale image. You can use [tf.slice](https://www.tensorflow.org/api_docs/python/tf/slice)
# to get example 0. If necessary, transpose into appropriate layout.

# <codecell>

# TODO

# <markdowncell>

# Summaries can also be *merged*. We add a special `image_summaries` node that outputs the union of our two image summaries.
# Evaluating this node causes both summaries to be computed.

# <codecell>

image_summaries = tf.summary.merge([conv1_kernel_summary, conv1_output_summary])

# <markdowncell>

# ### Update graph visualization
# We have added some new nodes, and we need to check if the new graph is OK.
# To update TensorBoard visualization, we just add a new graph to the event file.
# The visualizer will pick up the latest graph when its browser tab is refreshed.

# <codecell>

writer.add_graph(tf.get_default_graph())

# <markdowncell>

# ### Forward pass
# Next we run one CIFAR-10 frame through the network.

# <codecell>

def choose_random_image():
    index = np.random.randint(0, num_train)
    return index, x_train[[index]], y_train[[index]]

# <codecell>

random_index, random_image, random_label = choose_random_image()

# <markdowncell>

# A TensorFlow graph is executed by creating a `tf.Session` object and calling its `run` method.
# A session object encapsulates the control and state of the TensorFlow runtime.
# The `run` method requires a list of output tensors that should be computed, and a mapping of input tensors to actual data that should be used. For more information, see the TensorFlow [Getting started](https://www.tensorflow.org/get_started/get_started) guide.
#
# Optionally we can also specify a device context such as `/cpu:0` or `/gpu:0`. For documentation on this see [this TensorFlow guide](https://www.tensorflow.org/tutorials/using_gpu). The default device is a GPU if available, and a CPU otherwise, so we can skip the device specification from now on.
#
# Note: if GPU is explicitly specified, but not available, a Python exception is thrown; current graph is invalidated, and needs to be cleared and rebuilt.

# <codecell>

with tf.Session() as sess:
    with tf.device("/cpu:0") as dev: #"/cpu:0" or "/gpu:0"
        # Initialize weights.
        sess.run(tf.global_variables_initializer())

        # Map inputs to data.
        feed_dict = { x : random_image, y : random_label }

        # Set up variables we want to compute.
        variables = [guess_prob, guess_class, image_summaries]

        # Perform forward pass.
        guess_prob_value, guess_class_value, img_summ_value = sess.run(variables, feed_dict=feed_dict)

# <markdowncell>

# First let's see the image that was chosen, and networks predictions for it.

# <codecell>

def visualize_classification(image, guess_class, guess_prob):
    plt.imshow(image)
    plt.axis("off")
    plt.show()
    for i in range(3):
        ind = guess_class[0, i]
        prob = guess_prob[0, i]
        print("Class: {0}\tProbability: {1:0.0f}%".format(class_names[ind], prob * 100))
    print("Ground truth: {0}".format(class_names[random_label[0]]))

# <codecell>

visualize_classification((random_image[0] + mean_image) / 255.0, guess_class_value, guess_prob_value)

# <markdowncell>

# We write generated images to file. After running the next cell the images should be visible in TensorBoard.

# <codecell>

writer.add_summary(img_summ_value)

# <markdowncell>

# ### Loss and metric(s)
#
# We append more nodes to compute loss value, and the number of correctly predicted pixels.
# For loss we use `tf.sparse_softmax_cross_entropy_with_logits`. For other loss functions available
# out of the box in TensorFlow, see https://www.tensorflow.org/api_guides/python/nn#Losses and
# https://www.tensorflow.org/api_guides/python/nn#Classification.
# Of course, you can always build your own custom loss functions from simpler operations.

# <codecell>

def setup_metrics(y, y_out):
    # Define loss function.
    total_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(labels=y, logits=y_out)
    mean_loss = tf.reduce_mean(total_loss)

    # Add top three predictions.
    prob = tf.nn.softmax(y_out)
    (guess_prob, guess_class) = tf.nn.top_k(prob, k=3)

    # Compute number of correct predictions.
    is_correct = tf.equal(tf.argmax(y_out, 1), y)
    accuracy = tf.reduce_mean(tf.cast(is_correct, tf.float32))

    return mean_loss, accuracy, guess_prob, guess_class

# <markdowncell>

# We will be reusing this function later for other architectures.
# Now we create metrics for our current network.

# <codecell>

mean_loss, accuracy, guess_prob, guess_class = setup_metrics(y, fc2)

# <markdowncell>

# ### Visualizing loss and metric(s)
# We would like to use TensorBoard to visualize loss value and correct count.
# We add special nodes that generate those logs.

# <codecell>

def setup_scalar_summaries():
    mean_loss_summary = tf.summary.scalar('mean_loss', mean_loss)
    accuracy_summary = tf.summary.scalar('accuracy', accuracy)
    return tf.summary.merge([mean_loss_summary, accuracy_summary])

# <codecell>

scalar_summaries = setup_scalar_summaries()

# <markdowncell>

# ### Optimizer
#
# Finally, we define the optimization algorithm to be used for training. We use the Adam optimizer with learning rate 5e-4. For other choices see https://www.tensorflow.org/api_guides/python/train#Optimizers.
#
# Optimizer's `minimize` method essentially generates a network that performs backward pass based on the forward pass network that we defined, and passed to the optimizer via argument to `minimize`.
# The result of this method is a dummy node `train_step` which, when evaluated triggers execution of backward pass.

# <codecell>

def setup_optimizer(loss, learning_rate):
    optimizer = tf.train.AdamOptimizer(learning_rate)
    # Batch normalization in TensorFlow requires this extra dependency
    extra_update_ops = tf.get_collection(tf.GraphKeys.UPDATE_OPS)
    with tf.control_dependencies(extra_update_ops):
        train_step = optimizer.minimize(loss)
    return train_step

# <markdowncell>

# We will be reusing this function for other architectures. Now we create optimizer for our current network.

# <codecell>

train_step = setup_optimizer(mean_loss, 5e-4)

# <markdowncell>

# ### Adding an optional backward pass
# Above we saw how to execute forward pass using `tf.Session.run`. Now we wrap that into a function
# (since we will be calling it in a loop to train the network). We also add an option to execute a
# backward pass by passing the extra argument `training`. That way we can use the same function for
# both training (forward + backward), and evaluation (forward only).

# <codecell>

def run_iteration(session, variables, x_data, y_data, training=None):
    if training != None:
        variables += [training]

    # Map inputs to data.
    feed_dict = {x : x_data, y : y_data, is_training : training != None}

    # Compute variable values, and perform training step if required.
    values = session.run(variables, feed_dict=feed_dict)

    # Return loss value and number of correct predictions.
    return values[:-1] if training != None else values

# <markdowncell>

# ### Main training/evaluation loop
# The following is a simple function which trains or evaluates current model for a given number
# of epochs by repeatedly calling the `run_iteration` function defined above. It also takes care of:
# * aggregating loss and accuracy values over all minibatches
# * plotting loss and accuracy values over time.
#
# The code below assumes that `mean_loss`, `accuracy`, and `scalar_summaries` are defined externally.
# Optionally, the caller may also define `image_summaries`, if they want some image summaries (like kernel
# and activation visualizations in the form of images) to be also included.
# All those definitions will change depending on the network.

# <codecell>

def run_model(session, x, y, epochs, batch_size, print_every, training):

    # Number of examples in dataset.
    dataset_size = x.shape[0]

    # Count iterations since the beginning of training.
    iter_cnt = 0

    for e in range(epochs):
        # Randomize training examples for each epoch.
        train_indices = np.random.permutation(dataset_size)

        # Keep track of performance stats (loss and accuracy) in current epoch.
        total_loss = 0
        total_correct = 0

        # Iterate over the dataset once.
        for start_idx in range(0, dataset_size, batch_size):

            # Indices for current batch.
            idx = train_indices[start_idx : min(start_idx + batch_size, dataset_size)]

            # Get batch size (may not be equal to batch_size near the end of dataset).
            actual_batch_size = y[idx].shape[0]

            # Set up variables that we want to compute.
            variables = [mean_loss, accuracy, scalar_summaries]
            has_image_summaries = image_summaries != None
            if has_image_summaries:
                variables += [image_summaries,]

            # Compute loss, accuracy, and scalar summaries, and optionally perform backward pass.
            results = run_iteration(session, variables, x[idx], y[idx], training)

            # Unpack results.
            loss, acc, scl_summ = results[:3]
            if has_image_summaries:
                img_summ = results[3]

            # Update performance stats.
            total_loss += loss * actual_batch_size
            total_correct += acc * actual_batch_size

            # Add scalar summaries to event file.
            if (training is not None):
                writer.add_summary(scl_summ, iter_cnt)

            # Print status, and add image summaries (if any).
            if (training is not None) and (iter_cnt % print_every) == 0:
                print("Iteration {0}: with minibatch training loss = {1:.3g} and accuracy of {2:.2f}%".format(iter_cnt, loss, acc * 100))
                if has_image_summaries:
                    writer.add_summary(img_summ, iter_cnt)

            iter_cnt += 1

        # Compute performance stats for current epoch.
        avg_accuracy = total_correct / dataset_size
        avg_loss = total_loss / dataset_size

        print("Epoch {2}, Overall loss = {0:.3g} and accuracy of {1:.2f}%".format(avg_loss, avg_accuracy * 100, e + 1))

# <markdowncell>

# ### Training the model for one epoch

# <codecell>

sess = tf.Session()
sess.run(tf.global_variables_initializer())
print('Training')
run_model(sess, x=x_train, y=y_train, epochs=1, batch_size=64, print_every=100, training=train_step)
print('Validation')
run_model(sess, x=x_val, y=y_val, epochs=1, batch_size=64, print_every=100, training=None)

# <markdowncell>

# ### View summaries in TensorBoard log
# Now you should be able to refresh your TensorBoard tab and see the summaries.
# For more details please see [official tutorial](https://www.tensorflow.org/get_started/summaries_and_tensorboard) on summaries.
# TensorFlow also supports other kinds of summaries, such as [histograms](https://www.tensorflow.org/get_started/tensorboard_histograms).

# <markdowncell>

# ### Visualize some predictions
# Accuracy should be somewhat better now.

# <codecell>

random_index, random_image, random_label = choose_random_image()
guess_class_value, guess_prob_value = run_iteration(sess, [guess_class, guess_prob], random_image, random_label)
visualize_classification((random_image[0] + mean_image) / 255.0, guess_class_value, guess_prob_value)

# <markdowncell>

# ## Adding batch normalization
#
# Now we will modify the simple architecture by adding batch normalization. We expect this network to train faster, and achieve better
# accuracy for the same number of weight updates.
#
# Your task is to implement `bn_net` function below so that it creates the same network as in the previous section (with convolution, pooling,
# and two fully connected layers), except that there should also be a batch normalization layer after the convolution layer.
#
# Arguments of `bn_net` are placeholders for data (`x`), labels (`y`), and boolean flag (`is_training`) for indicating if the network should
# perform training or validation version of the computation on the input data. While `is_training` was completely ignored in the previous
# network, *here it should actually matter* (why?). `bn_net` should return the output (prediction) tensor of the network.
#
# API reference for batch normalization is at https://www.tensorflow.org/api_docs/python/tf/layers/batch_normalization.

# <codecell>

def bn_net(x, y, is_training):
    # TODO
    pass

# <markdowncell>

# Input, metrics and optimizer are the same as before, so we can assemble the whole network.
# If you implemented `bn_net` correctly, this code should run without modification.

# <codecell>

tf.reset_default_graph()
writer = tf.summary.FileWriter(os.path.join(log_dir, 'bn_net'))
x, y, is_training = setup_input()
y_out = bn_net(x, y, is_training)
mean_loss, accuracy, guess_prob, guess_class = setup_metrics(y, y_out)
scalar_summaries = setup_scalar_summaries()
image_summaries = None
train_step = setup_optimizer(mean_loss, 5e-4)

# <markdowncell>

# Now we are ready to train and validate the network with batch normalization as before.

# <codecell>

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print('Training')
    run_model(sess, x=x_train, y=y_train, epochs=1, batch_size=64, print_every=100, training=train_step)
    print('Validation')
    run_model(sess, x=x_val, y=y_val, epochs=1, batch_size=64, print_every=100, training=None)

# <markdowncell>

# ## Training a more complex network
#
# Please download TensorBoard event file from [here](https://1drv.ms/u/s!ApgLrbSPQF9XirQrPV7L1GOhxkc-ig).
# Your task is to build the architecture contained in the event file `cifar10_net_log\events.out.tfevents.1500743084.localhost`, and
# train it on CIFAR-10. You should train for 8 epochs with batch size 100 and learning rate 0.001.

# <codecell>

def cifar10_net(x, y, is_training):
    # TODO
    pass

# <codecell>

tf.reset_default_graph()
writer = tf.summary.FileWriter(os.path.join(log_dir, 'complex_net'))
x, y, is_training = setup_input()
y_out = cifar10_net(x, y, is_training)
mean_loss, accuracy, guess_prob, guess_class = setup_metrics(y, y_out)
scalar_summaries = setup_scalar_summaries()
image_summaries = None
train_step = setup_optimizer(mean_loss, 1e-3)

# <codecell>

writer.add_graph(tf.get_default_graph())

# <codecell>

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    print('Training')
    run_model(sess, x=x_train, y=y_train, epochs=8, batch_size=100, print_every=100, training=train_step)
    print('Validation')
    run_model(sess, x=x_val, y=y_val, epochs=1, batch_size=100, print_every=100, training=None)

# <markdowncell>

# ## Saving and loading models
#
# Saving is done using `tf.train.Saver` class:
# * `save` method saves both network definition and weights.
# * `export_meta_graph` method saves only network definition.
#
# Loading is done in two stages:
# * `tf.train.import_meta_graph` function loads network definition, and returns a saver object that was used to save the model.
# * `restore` method of the returned saver object loads the weights.
#
# Note that since weights are available only inside a session, `save` and `restore` methods above require a session object as a parameter.
#
# Official TensorFlow documentation: [Saving and Restoring Variables](https://www.tensorflow.org/api_guides/python/state_ops#Saving_and_Restoring_Variables), [tf.train.Saver class](https://www.tensorflow.org/api_docs/python/tf/train/Saver), [tf.train.import_meta_graph function](https://www.tensorflow.org/api_docs/python/tf/train/import_meta_graph).
#
# Useful unofficial tutorial on saving and loading: http://cv-tricks.com/tensorflow-tutorial/save-restore-tensorflow-models-quick-complete-tutorial/
#
# Many pretrained models found online do not contain network definition (meta-graph) files. The authors instead provide TensorFlow
# code to create the network graph. For such graphs, weights can also be restored from checkpoint file. Network definition file is
# just a way to avoid copying the code.

# <markdowncell>

# ## Transfer learning
#
# In this section we will take a model which is pretrained on ImageNet for 1000-class image classification task, and
# finetune it for our 10-class CIFAR-10 classification task.
#
# Pretrained model is given by meta-graph file (containing network definition), and checkpoint file (containing weights).
# Please download pretrained model from [here](https://1drv.ms/u/s!ApgLrbSPQF9XirQpuN32y0577zey8g).

# <codecell>

pretrained_meta_graph = os.path.join('mobilenet_v2_1.0_224', 'mobilenet_v2_1.0_224.meta')
pretrained_checkpoint = os.path.join('mobilenet_v2_1.0_224', 'mobilenet_v2_1.0_224.ckpt')

# <markdowncell>

# For the CIFAR-10 task we need to perform the following two modifications to the pretrained model at the very minimum:
# * Process CIFAR-10 images so that their size becomes what pretrained model expects
# * Adapt the layers which perform final classification so that the number of output neurons is 10 (the number of
# classes in the CIFAR-10 classification task)

# <markdowncell>

# ### Get names of relevant nodes
#
# Modifying input part of a pretrained network is somewhat cumbersome. It must be done simultaneously with loading network
# definition, by passing to `tf.train.import_meta_graph` a mappping from input tensors of the pretrained network to new input tensors.
#
# First we load pretrained network definition only to get the names of input placeholder nodes that we want to replace.
# This step can be skipped if these names are already known.

# <codecell>

tf.reset_default_graph()
_ = tf.train.import_meta_graph(pretrained_meta_graph)

# <markdowncell>

# The easiest way to get the nodes' names is using TensorBoard. It can also be done programmatically, as explained above
# (for example, using `tf.get_default_graph().get_operations()`).

# <codecell>

writer = tf.summary.FileWriter(os.path.join(log_dir, 'transfer_net'))
writer.add_graph(tf.get_default_graph())

# <markdowncell>

# After inspecting TensorFlow graph visualization, we find that
# * the input nodes are `image` and `is_training`
# * final classification is performed in the node group (subgraph) `MobilenetV2/Logits/Conv2d_1c_1x1` which implements 1x1 convolution.

# <markdowncell>

# ### Modify input and output

# Next we clear the default graph, and start creating new one, with modified input subnetwork which upsamples input image
# to match the size expected by pretrained network.

# <codecell>

tf.reset_default_graph()
writer = tf.summary.FileWriter(os.path.join(log_dir, 'transfer_net'))
x, y, is_training = setup_input()
x_upsampled = tf.image.resize_images(x, [224, 224])

# <markdowncell>

# Finally, we reload pretrained network definition, replacing pretrained input placeholders with new tensors we just created.

# <codecell>

saver = tf.train.import_meta_graph(pretrained_meta_graph, input_map={'image:0' : x_upsampled, 'is_training:0' : is_training})

# <markdowncell>

# We want to replace the node group `MobilenetV2/Logits/Conv2d_1c_1x1` by a 1x1 convolution with different number of output
# channels. To that end, we get a handle to the tensor immediately preceding `MobilenetV2/Logits/Conv2d_1c_1x1` in the graph.

# <codecell>

feat = tf.get_default_graph().get_tensor_by_name('MobilenetV2/Logits/Dropout/cond/Merge:0')

# <markdowncell>

# Attach a new prediction layer for modified task.

# <codecell>

with tf.variable_scope('MobilenetV2/Logits'):
      conv_1x1 = tf.layers.conv2d(inputs=feat, filters=num_classes, kernel_size=[1, 1], strides=1, name='conv2d_1x1_modified')
      output = tf.squeeze(conv_1x1, [1, 2], name='output_modified')

# <markdowncell>

# ### Complete network definition
# Add metrics and optimizer as before.

# <codecell>

mean_loss, accuracy, guess_prob, guess_class = setup_metrics(y, output)
scalar_summaries = setup_scalar_summaries()
image_summaries = None
train_step = setup_optimizer(mean_loss, 5e-4)

# <markdowncell>

# Once again write out the graph to make sure surgery succeeded.

# <codecell>

writer.add_graph(tf.get_default_graph())

# <markdowncell>

# ### Train and validate network (at your own risk!)

# <markdowncell>

# Only now we can restore weights from checkpoint, because weights exist only inside a session.
#
# Then we train the network as before. Note that this training is *extremely slow on CPU*, due to bigger network.
# Also observe that we still need to initialize variables, because we have introduced new ones, i.e. not all
# variables that exist in the network are restored from checkpoint.

# <codecell>

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())
    saver.restore(sess, pretrained_checkpoint)
    print('Training')
    run_model(sess, x_train, y_train, epochs=1, batch_size=64, print_every=100, training=train_step)
    print('Validation')
    run_model(sess, x_val, y_val, epochs=1, batch_size=64, print_every=100, training=None)


#from src.utilities.craft_pytorch.detect import detect_text_per_file as detect_text_per_page
#from src.services.ocr import get_text

import cv2
import config
#import imutils
import pandas as pd
import numpy as np

# Version 1.0 delivery in this week!

class Orientation:

    def __init__(self, image_path, get_text, conf_threshold=50, lang='eng'):

        self.image_path = image_path
        self.image = cv2.imread(image_path)
        self.get_text = get_text

        self.conf_threshold = int(conf_threshold)

        self.timer = {'net': 0, 'restore': 0, 'nms': 0}
        self.text = {}
        self.lang = lang

        # self.re_orient()

    def augment_df(self, line_list):
        dic = []
        for i, box in enumerate(line_list):
            # print(box)
            vertices = box["boundingBox"]["vertices"]
            dic.append({'x1': vertices[0]['x'], 'y1': vertices[0]['y'], 'x2': vertices[1]['x'], 'y2': vertices[1]
                       ['y'], 'x3': vertices[2]['x'], 'y3': vertices[2]['y'], 'x4': vertices[3]['x'], 'y4': vertices[3]['y']})

        df = pd.DataFrame(
            dic, columns=['x1', 'y1', 'x2', 'y2', 'x3', 'y3', 'x4', 'y4'])
        df['height'] = df['y4'] - df['y1']
        df['width'] = df['x2'] - df['x1']
        df['ymid'] = (df['y4'] + df['y3']) * 0.5
        df['area'] = df['width'] * df['height']
        df = df.sort_values(by=['ymid'])
        df['group'] = None
        df['line_change'] = 0
        return df

    def dump_out(self, bbc, rot):
        im = self.image.copy()
        for box in bbc:
            # print(box)
            cv2.polylines(im, [np.array(box).astype(np.int32).reshape((-1, 1, 2))], True, color=(255, 255, 0),
                          thickness=1)
        cv2.imwrite('tmp/' + str(rot) + '.png', im)

    def get_rotaion_angle(self, text_cor_df):

        bbox_df = text_cor_df.copy()
        bbox_df = self.augment_df(bbox_df)

        bbox_df['delta_x'] = bbox_df['x2'] - bbox_df['x1']
        bbox_df['delta_y'] = bbox_df['y2'] - bbox_df['y1']
        box_dir = [bbox_df['delta_x'].mean(), bbox_df['delta_y'].mean()]
        # print(box_dir)
        x_axis = [1, 0]
        try:
            cosine = np.dot(box_dir, x_axis) / \
                (np.linalg.norm(box_dir) * np.linalg.norm(x_axis))
        except Exception as e:
            print('ERROR in finding angle of rotaion!!!')
            print(e)
            cosine = 1
        angle = np.arccos(cosine) * 180 / np.pi
        avrage_height = bbox_df['height'].mean()
        avrage_width = bbox_df['width'].mean()
        # if avrage_height > avrage_width:
        #    angle = 90 - angle

        return angle * np.sign(box_dir[1])

    def rotate_bound(self, image, angle):
        # grab the dimensions of the image and then determine the
        # center
        (h, w) = image.shape[:2]
        (cX, cY) = (w / 2, h / 2)

        # grab the rotation matrix (applying the negative of the
        # angle to rotate clockwise), then grab the sine and cosine
        # (i.e., the rotation components of the matrix)
        M = cv2.getRotationMatrix2D((cX, cY), -angle, 1.0)
        cos = np.abs(M[0, 0])
        sin = np.abs(M[0, 1])

        # compute the new bounding dimensions of the image
        nW = int((h * sin) + (w * cos))
        nH = int((h * cos) + (w * sin))

        # adjust the rotation matrix to take into account translation
        M[0, 2] += (nW / 2) - cX
        M[1, 2] += (nH / 2) - cY

        # perform the actual rotation and return the image

        # return cv2.warpAffine(image, M, (nW, nH),flags=cv2.INTER_LANCZOS4)
        # return cv2.warpAffine(image, M, (nW, nH),flags=cv2.INTER_LANCZOS4, borderMode=cv2.BORDER_REPLICATE)
        return cv2.warpAffine(image, M, (nW, nH), flags=cv2.INTER_CUBIC, borderMode=cv2.BORDER_REPLICATE)

    def re_orient(self, page_dict, font_info):

        page_dict, lines = self.get_text(self.image_path, page_dict, font_info)
        angle = self.get_rotaion_angle(lines)
        print("Angle of tilt detected {} ".format(angle))
        if abs(angle) >= 2.5:
            self.image = self.rotate_bound(self.image, -angle)
            cv2.imwrite(self.image_path, self.image)
            page_dict, lines = self.get_text(
                self.image_path, page_dict, font_info)
            angle = self.get_rotaion_angle(lines)
            print("Angle of tilt after correction {} ".format(angle))

        return page_dict

    #
    # def re_orient(self):
    #     lang = 'hi'
    #     lines = detect_text_per_page([self.image],\
    #                                  network=True, \
    #                                  text_threshold=config.LANGUAGE_LINE_THRESOLDS[lang]['text_threshold'], \
    #                                  low_text_threshold=config.LANGUAGE_LINE_THRESOLDS[lang]['low_text'],
    #                                  link_threshold=config.LANGUAGE_LINE_THRESOLDS[lang]['link_threshold'])[0]
    #
    #     angle = self.get_rotaion_angle(lines)
    #     print("Angle of tilt detected {} ".format(angle) )
    #     rotations = 1
    #     # self.dump_out(lines,rotations)
    #     # Orientation correction
    #     if config.ALIGN_MODE == 'FAST':
    #      tolerance = 0.1
    #     if config.ALIGN_MODE == 'ACCURATE':
    #         tolerance = 0.1
    #
    #     while (abs(angle) > tolerance) and rotations < 3:
    #         self.image = self.rotate_bound(self.image, -angle)
    #
    #         if rotations > 1:
    #             # Remove rotaion artifacts
    #             contours = cv2.findContours(cv2.cvtColor(self.image, cv2.COLOR_BGR2GRAY), cv2.RETR_EXTERNAL,
    #                                         cv2.CHAIN_APPROX_SIMPLE)
    #             contours = contours[0] if len(contours) == 2 else contours[1]
    #             if len(contours) > 0:
    #                 x, y, w, h = cv2.boundingRect(contours[0])
    #                 # print('cropped area reduced ')
    #                 self.image = self.image[y:y + h, x:x + w, :]
    #
    #         lines = detect_text_per_page([self.image], \
    #                                      network=True, \
    #                                      text_threshold=config.LANGUAGE_LINE_THRESOLDS[lang]['text_threshold'], \
    #                                      low_text_threshold=config.LANGUAGE_LINE_THRESOLDS[lang]['low_text'],
    #                                      link_threshold=config.LANGUAGE_LINE_THRESOLDS[lang]['link_threshold'])[0]
    #         angle = self.get_rotaion_angle(lines)
    #         print("Angle of tilt after correction {} ".format(angle))
    #         rotations += 1
    #         # self.dump_out(east_cor,rotations)
    #     cv2.imwrite(self.image_path,self.image)
    #     words = detect_text_per_page([self.image], \
    #                                      network=False, \
    #                                      text_threshold=config.LANGUAGE_LINE_THRESOLDS[lang]['text_threshold'], \
    #                                      low_text_threshold=config.LANGUAGE_LINE_THRESOLDS[lang]['low_text'],
    #                                      link_threshold=config.LANGUAGE_LINE_THRESOLDS[lang]['link_threshold'])[0]
    #     # if config.ALIGN_MODE not "FAST":
    #     #     upside_down = self.check_orientation(bbox1.gr_cordinates)
    #
    #
    #     return words, lines
    #

  # def check_orientation(self, group_cordinates, margin=5):
    #     upside_down = False
    #     orientation = []
    #     for index, block in enumerate(group_cordinates):
    #         crop = self.image[block[0][1] - margin: block[1][1] + margin,
    #                block[0][0] - margin: block[1][0] + margin]
    #         try:
    #             osd = pytesseract.image_to_osd(crop)
    #             angle = osd.split('\nRotate')[0].split(': ')[-1]
    #             orientation.append(int(angle))
    #         except:
    #             pass
    #     orientation = np.array(orientation)
    #     chk_orientation = orientation > 170
    #
    #     # Taking vote of regions
    #     if chk_orientation.sum() > (len(orientation) * 0.5):
    #         # print ('Image is upside down')
    #         upside_down = True
    #         return upside_down
    #
    #     return upside_down
    #
    #


import datetime as dt
from typing import ClassVar, Optional, cast

from cuenca_validations.types import (
    Country,
    CurpValidationRequest,
    Gender,
    State,
)
from cuenca_validations.types.identities import CurpField
from pydantic.dataclasses import dataclass

from ..http import Session, session as global_session
from .base import Creatable, Retrievable

@dataclass
class CurpValidation(Creatable, Retrievable):
    _resource: ClassVar = 'curp_validations'

    created_at: dt.datetime
    names: str
    first_surname: str
    second_surname: Optional[str]
    date_of_birth: dt.date
    country_of_birth: Country
    state_of_birth: State
    gender: Gender
    nationality: Optional[Country]
    manual_curp: Optional[CurpField]
    calculated_curp: CurpField
    validated_curp: Optional[CurpField]
    renapo_curp_match: bool
    renapo_full_match: bool

    @classmethod
    def create(
        cls,
        names: str,
        first_surname: str,
        date_of_birth: dt.date,
        country_of_birth: str,
        state_of_birth: State,
        gender: Gender,
        second_surname: Optional[str] = None,
        manual_curp: Optional[CurpField] = None,
        *,
        session: Session = global_session,
    ) -> 'CurpValidation':
        req = CurpValidationRequest(
            names=names,
            first_surname=first_surname,
            second_surname=second_surname,
            date_of_birth=date_of_birth,
            state_of_birth=state_of_birth,
            country_of_birth=country_of_birth,
            gender=gender,
            manual_curp=manual_curp,
        )
        return cast(
            'CurpValidation',
            cls._create(session=session, **req.dict()),
        )


#!/usr/bin/env python3
# -*- coding=utf-8 -*-

import os, json

def main():
    d = r'D:\My Projects\Python Projects\listening_and_writing'
    f = r'g5s2u8.json'
    p = os.path.join(d, f)
    file = open(p, encoding='utf-8')
    t = file.read()
    file.close()
    j = json.loads(t)
    n = '{'
    for k, v in j.items():
        n += '"' + k + '":{'
        for k1, v1 in v.items():
            if k1 != 'items':
                n += '"' + k1 + '":"' + v1 + '",'
            else:
                n += '"' + k1 + '":['
                for k2, v2 in v1.items():
                    n += '{"en":"' + k2 + '","zh":"' + v2 + '","category":""},'
                n = n[:-1]
                n += '],'
        n = n[:-1]
        n += '},'
    n = n[: -1]
    n += "}"
    print(n)

    f = r'new_' + f
    p = os.path.join(d, f)
    file = open(p, 'w', encoding='utf-8')
    file.write(n)
    file.close()

if __name__ == "__main__":
    main()


from augmentation import *
from data_utils import prepare_data
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.metrics import precision_recall_fscore_support, confusion_matrix
from sklearn.utils import shuffle
import pandas as pd
import numpy as np

AUGMENT = 'NO'

"""## Helper Methods"""

def scale_datasets(scaler, train_set, to_scale):
  scaler = scaler().fit(train_set)
  res = []
  for dataset in to_scale:
    res.append(scaler.transform(dataset))
  return res

def display_results(precision_recall_fscore_support):
  disp_df = pd.DataFrame(data=precision_recall_fscore_support, columns=['Pos Class'], index=['Precision', 'Recall', 'F-score', 'support'])
  print(disp_df)

X, y = prepare_data()

"""### Train-Test-Validation Split"""
SEED = 0
DATA_SPLIT_PCT = 0.20

X_train, X_test, y_train, y_test = train_test_split(np.array(X), np.array(y), test_size=DATA_SPLIT_PCT, random_state=SEED)

classifier = LogisticRegression(class_weight='balanced', max_iter = 10000, penalty = 'l1', solver = 'saga', C = 0.01, verbose=1)

"""## No Augmentation"""
train_x_scaled, test_x_scaled = scale_datasets(StandardScaler, X_train, [X_train, X_test])

if AUGMENT == 'NO':
    classifier.fit(train_x_scaled, y_train)
    y_hat_train = classifier.predict(train_x_scaled)
    train_res = precision_recall_fscore_support(y_train, y_hat_train)#, average='binary')

elif AUGMENT == 'GRAD':
    augmented_train_x, augmented_y_train = gradient_augment(train_x_scaled, y_train)
    augmented_train_x_rescaled, augmented_y_train = shuffle(augmented_train_x, augmented_y_train)

    classifier.fit(augmented_train_x_rescaled, augmented_y_train)
    y_hat_train = classifier.predict(augmented_train_x_rescaled)
    train_res = precision_recall_fscore_support(augmented_y_train, y_hat_train, average='binary')
    test_x_scaled, _ = gradient_augment(test_x_scaled, y_test)

elif AUGMENT == 'GRADRESAMP':
    augmented_train_x, augmented_y_train = gradient_augment(train_x_scaled, y_train)
    augmented_train_x, augmented_y_train = resample_augment(augmented_train_x, augmented_y_train)
    augmented_train_x, augmented_y_train = shuffle(augmented_train_x, augmented_y_train)
    augmented_train_x_rescaled = scale_datasets(StandardScaler, augmented_train_x, [augmented_train_x])[0]
    classifier.fit(augmented_train_x_rescaled, augmented_y_train)
    y_hat_train = classifier.predict(augmented_train_x_rescaled)
    train_res = precision_recall_fscore_support(augmented_y_train, y_hat_train, average='binary')
    test_x_scaled, _ = gradient_augment(test_x_scaled, y_test)

else:
    augmented_train_x, augmented_y_train = resample_augment(df_train_x, y_train)

    augmented_train_x_rescaled = scale_datasets(StandardScaler, augmented_train_x, [augmented_train_x])[0]

    augmented_train_x_rescaled, test_x_scaled = scale_datasets(StandardScaler, augmented_train_x, [augmented_train_x, df_test_x])
    classifier.fit(augmented_train_x_rescaled, augmented_y_train)
    y_hat_train = classifier.predict(augmented_train_x_rescaled)
    train_res = precision_recall_fscore_support(augmented_y_train, y_hat_train, average='binary')

#display_results(train_res)
print(train_res)

y_hat_test = classifier.predict(test_x_scaled)
test_res = precision_recall_fscore_support(y_test, y_hat_test)#, average='binary')
#display_results(test_res)
print(test_res)
test_res = confusion_matrix(y_test, y_hat_test, labels=[1, 0])
print(test_res)


# add modules to Python's search path
import sys
from pathlib import Path
from os.path import dirname, realpath
script_dir = Path(dirname(realpath(__file__)))
root = str(script_dir.parent)
print(root)
sys.path.insert(0, root + '/modules')
sys.path.insert(0, root + '/custom_dists')
# import required modules
import fp_lstm as fp
import numpy as np
import scipy.stats as ss
import tensorflow as tf
import nn_plotter as pltr
import gaussian_circle as gc
import vegas
import fp_solver as fps
import derivative as dr

beta = 200.0
ens_file = 'data/sde_evolve_test_2d_n_001.h5'
cost_file = 'data/sde_evolve_test_2d_n_cost_2_001.h5'

dtype = tf.float64
dimension = 2
num_components = 10
domain = 2.0*np.array([[-1.0, 1.0], [-1.0, 1.0]])

class DiffOp(tf.keras.layers.Layer):
    def __init__(self, f):
        super().__init__(name='DiffOp', dtype=dtype)
        self.f = f

    def call(self, x, y):
        r2 = x*x + y*y
        z = 4.0*(r2 - 1.0)
        with tf.GradientTape() as outer_x, tf.GradientTape() as outer_y:
            outer_x.watch(x)
            outer_y.watch(y)
            with tf.GradientTape() as inner:
                inner.watch([x, y])
                f_ = self.f(x, y)
            grad = inner.gradient(f_, [x, y])
            f_x = grad[0]
            f_y = grad[1]
        f_xx = outer_x.gradient(f_x, x)
        f_yy = outer_y.gradient(f_y, y)
        a = (x*z) * f_x
        b = (y*z) * f_y
        c = 4.0 * (z + 2.0) * f_
        return a + b + c + (f_xx + f_yy) / beta

solver = fp.FPDGM(50, 4, DiffOp, ens_file, domain, None, sinkhorn_iters=10, sinkhorn_epsilon=0.01, dtype=dtype)
solver.summary()

num_components = 10
cov = 0.1*np.identity(dimension)
weights = np.ones(num_components)
rv = gc.GaussianCircle(cov, weights)

class CustomDensity(tf.keras.models.Model):
    def __init__(self, dtype=dtype):
        super().__init__(dtype=dtype)

    def call(self, x):
        return tf.convert_to_tensor(rv.pdf(x), dtype=self.dtype)

    def call_2(self, x, y):
        X = tf.concat([x, y], axis=1)
        return rv.prob(X)

class InitialPDF(tf.keras.layers.Layer):
    def __init__(self, dtype=dtype):
        super().__init__(dtype=dtype)
        #self.dist = tfp.distributions.MultivariateNormalTriL(loc=mean, scale_tril=tf.linalg.cholesky(cov))
        #self.pdf = self.dist.prob
        self.c = tf.cast(tf.math.sqrt((2.0 * np.pi * delta) ** dimension), dtype=dtype)
        self.d = tf.cast(delta**dimension, dtype=dtype)
    def sample(self, size):
        return tf.convert_to_tensor(np.random.multivariate_normal(mean=mean, cov=cov, size=size), dtype=dtype)
    def call(self, x):
        return tf.math.exp(- 0.5 * tf.reduce_sum(x**2, axis=1, keepdims=True) / self.d ) / self.c

real_density = CustomDensity()
partials_rd = dr.FirstPartials(real_density.call_2, 2)
#"""
domain = 2.5 * np.array([[-1.0, 1.0], [-1.0, 1.0]])
plotter = pltr.NNPlotter(funcs=[solver], space=domain, num_pts_per_dim=50)
plotter.plot('images/fp_lstm_before.png')

#"""
ensemble = tf.convert_to_tensor(rv.sample(size=100), dtype=dtype)
weights = tf.convert_to_tensor(np.ones(100), dtype=dtype) #rv.pdf(ensemble)
solver.learn_density(ensemble, weights, domain, epochs=200, initial_rate=0.001)
region = fps.Domain(domain, dtype)
for _ in range(50):
    ensemble = region.sample(1000)
    weights = tf.convert_to_tensor(rv.pdf(ensemble), dtype=dtype)
    solver.learn_function(ensemble, weights, epochs=500, initial_rate=0.001)
    #solver.compute_normalizer(domain)

#"""
ensemble = tf.convert_to_tensor(rv.sample(size=100), dtype=dtype)
a = tf.reshape(solver(ensemble), (-1))
b = rv.pdf(ensemble.numpy())
c = tf.math.abs(a - b)
print(c)
print(tf.reduce_mean(c))
plotter = pltr.NNPlotter(funcs=[solver], space=domain, num_pts_per_dim=50)
plotter.plot('images/fp_lstm_after.png', wireframe=True)
#"""


import os
import sys
import pickle

import argparse
import numpy as np
from numpy.lib.format import open_memmap

from ntu_read_skeleton import read_xyz

#代表这些人的动作用于训练，其他人的动作用于验证，即Cross-Subject划分准则
training_subjects = [
    1, 2, 4, 5, 8, 9, 13, 14, 15, 16, 17, 18, 19, 25, 27, 28, 31, 34, 35, 38
]
#代表只用2号、3号相机产生的数据进行训练，1号相机采集的数据用于验证，即Cross-View划分准则
training_cameras = [2, 3]
#最多两个人
max_body = 2
#总共关节数为25
num_joint = 25
#最多300帧
max_frame = 300

'''
输出过程信息
'''
toolbar_width = 30

def print_toolbar(rate, annotation=''):
    # setup toolbar
    sys.stdout.write("{}[".format(annotation))
    for i in range(toolbar_width):
        if i * 1.0 / toolbar_width > rate:
            sys.stdout.write(' ')
        else:
            sys.stdout.write('-')
        sys.stdout.flush()
    sys.stdout.write(']\r')

def end_toolbar():
    sys.stdout.write("\n")

'''
data_path:数据存放路径

out_path:输出路径

ignored_sample_path:用于存放忽略文件名的地方，如果为空则代表不忽略文件

benchmark:('xview','xsub')
    若为xview则代表按照不同相机来划分训练集和验证集（即相机1采集的数据作为验证集，相机2、3采集的数据作为训练集）
    若为xsub则代表按照不同的人来划分训练集和验证集（即在training_subjects中的人对应的数据为训练集，其他的人对应的数据为验证集）

part:('eval','train')，为eval代表验证集，为train代表训练集
'''
def gendata(data_path,
            out_path,
            ignored_sample_path=None,
            benchmark='xview',
            part='eval'):

    # step1.看是否有需要忽略的样本
    # ignored_sample_path中出现的文件名即代表忽略对应样本
    if ignored_sample_path != None:
        with open(ignored_sample_path, 'r') as f:
            ignored_samples = [
                line.strip() + '.skeleton' for line in f.readlines()
            ]
    else:
        ignored_samples = []

    # step2.依次遍历数据集目录下的文件名，通过是否需要忽略样本以及样本是否符合当前状态（是在训练还是在验证）以将需要读取的样本以及标签加入sample_name和sample_label中
    sample_name = []
    sample_label = []
    for filename in os.listdir(data_path):
        # 如果该文件名字在忽略样本中，则忽略该文件
        if filename in ignored_samples:
            continue
        # 文件名A开始的部分代表动作的分类
        action_class = int(
            filename[filename.find('A') + 1:filename.find('A') + 4])
        # 文件名P开始的部分代表人物的id
        subject_id = int(
            filename[filename.find('P') + 1:filename.find('P') + 4])
        # 文件名C开始的部分代表相机id
        camera_id = int(
            filename[filename.find('C') + 1:filename.find('C') + 4])

        # 看基准是xview还是xsub，若为xview则代表按相机来划分训练集和验证集，则看当前文件的文件名中对应的相机是否为训练相机
        # 若为xsub则代表以人为基准，则看文件名中对应的人是否在训练集对应的人中
        # istraining用于代表是否为训练部分
        if benchmark == 'xview':
            istraining = (camera_id in training_cameras)
        elif benchmark == 'xsub':
            istraining = (subject_id in training_subjects)
        else:
            raise ValueError()

        # 看传入的模式是在训练还是在验证，若在训练则取istraining的值以看当前文件是否为训练样本
        # 若在验证则对istraining的值取反以看当前文件是否为验证样本
        # issample代表当前文件是否需要读取（采样）
        if part == 'train':
            issample = istraining
        elif part == 'val':
            issample = not (istraining)
        else:
            raise ValueError()

        # 如果当前文件需要读取，则加入到列表中
        # action_class为由文件名给出的ground_truth动作类别号
        if issample:
            sample_name.append(filename)
            sample_label.append(action_class - 1)

    # 将以上获得的需要读取的样本文件名以及ground_truth动作类别号存入out_path的train_label.pkl（训练集）或eval_label.pkl（验证集）中
    with open('{}/{}_label.pkl'.format(out_path, part), 'wb') as f:
        # 将待取样的sample_name和sample_label以ascii形式存入f指向的文件中
        pickle.dump((sample_name, list(sample_label)), f)
    # np.save('{}/{}_label.npy'.format(out_path, part), sample_label)

    # 再打开一个numpy类型的存数据文件流
    fp = open_memmap(
        '{}/{}_data.npy'.format(out_path, part),
        dtype='float32',
        mode='w+',
        shape=(len(sample_label), 3, max_frame, num_joint, max_body))

    # 依次读取样本文件并写入
    # 数据的第0维是序号即代表是哪一个样本，第一维存的是关节的x,y,z坐标，第二维用于标识是哪一帧、第三维用于标识是哪个关节、第四维用于表示是哪个身体
    for i, s in enumerate(sample_name):
        print_toolbar(i * 1.0 / len(sample_label),
                      '({:>5}/{:<5}) Processing {:>5}-{:<5} data: '.format(
                          i + 1, len(sample_name), benchmark, part))
        data = read_xyz(
            os.path.join(data_path, s), max_body=max_body, num_joint=num_joint)
        # data:第一维存的是关节的x,y,z坐标，第二维用于标识是哪一帧、第三维用于标识是哪个关节、第四维用于表示是哪个身体
        fp[i, :, 0:data.shape[1], :, :] = data
    end_toolbar()

# 只有当运行该文件时才执行，其中的参数需要改为自己数据集的位置
if __name__ == '__main__':

    parser = argparse.ArgumentParser(description='NTU-RGB-D Data Converter.')
    parser.add_argument(
        '--data_path', default='D:/ntu-rgb+d/raw_txt')
    parser.add_argument(
        '--ignored_sample_path',
        default='D:/ntu-rgb+d/missing_list/samples_with_missing_skeletons.txt')
    parser.add_argument('--out_folder', default='D:/ntu-rgb+d/cooked_dat')

    benchmark = ['xsub', 'xview']
    part = ['train', 'val']
    arg = parser.parse_args()

    for b in benchmark:
        for p in part:
            out_path = os.path.join(arg.out_folder, b)
            if not os.path.exists(out_path):
                os.makedirs(out_path)
            gendata(
                arg.data_path,
                out_path,
                arg.ignored_sample_path,
                benchmark=b,
                part=p)


from silo import log, config
import io

def test_logging():
    stream = io.BytesIO()
    config(file=stream, handler=lambda x: x['message'].encode('utf-8'))

    def reset():
        val = stream.getvalue()
        stream.truncate(0)
        stream.seek(0)
        return val

    log('')
    assert reset() == b''

    log('hello')
    assert reset() == b'hello'

    log('hello {name}', name='world')
    assert reset() == b'hello world'

    log('{missing}')
    assert reset() == b'{missing}'


import dash
from dash import html, dcc
import dash_bootstrap_components as dbc
from dash import html, Input, Output, State, callback_context, dash_table,callback,ctx
import json, operator

dash.register_page(__name__, path='/lastPage')

layout = html.Div(children=[
    html.H1(children='Please enter your user testing ID.'),

    html.Div(
        [
            html.P("Please enter your user testing ID."),
            dbc.Input(id="userTestingID", type="text"),
        ],
        id="estimateInputDiv"),
    html.Div([
        html.A(dbc.Button('Submit', id='submitUserId', n_clicks=0),id="thankYouPageLink",href="/thankYouPage")
    ],style={'display':'flex','justifyContent':'center'}),
    # html.Div(children='''
    #     Thank you for taking this survey! Please paste the following code into Prolific to receive compensation: 'uniqueCode'
    # '''),
],style={'margin-left':'150px','margin-right':'150px','margin-top':'50px'})

def write_json(new_data, filename='assets/files/studyFiles/userDataInteractions.json'):
    print("inside write json")
    with open(filename, 'r+') as file:
        # First we load existing data into a dict.
        file_data = json.load(file)
        # Join new_data with file_data inside emp_details
        file_data["interaction_details"].append(new_data)
        # Sets file's current position at offset.
        file.seek(0)
        # convert back to json.
        json.dump(file_data, file, indent=4)

@callback(
    Output('thankYouPageLink', 'href'),
    [Input('submitUserId', 'n_clicks_timestamp'),
     Input("userIDStore", "data"),
     Input("userTestingID", "value"),
     ])
def saveUserTestingID(n, userIDStore,userTestingID):

    elementTriggered = ctx.triggered_id

    if elementTriggered == "submitUserId":
        matchingIDRecord= {"dashUserID":userIDStore['userID'],
                           "userTestingID":userTestingID
        }
        #open file and find record with same user ID
        write_json(matchingIDRecord)

    return "/thankYouPage"


import Pyro4

#from StringUtil import StringUtil

#util = StringUtil()

util = Pyro4.Proxy("PYRONAME:StringUtil")    # use name server object lookup uri shortcut

s = util.toUpperCase("hello world")
print(s)

s = util.reverse(s)
print(s)


import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

df = pd.read_csv("tp3.csv",sep=r'\s*,\s*',header=0, engine='python')
Y = df['Tiempo promedio'].tolist()
X = df['RC(FΩ)'].tolist()
yerr=df['error t'].tolist()
xerr=df['error RC'].tolist()

print(xerr)
z, residuals, rank, singular_values, rcond = np.polyfit(X, Y, 1, full=True)
print("residuals RC")
print(residuals)
p = np.poly1d(z)
plt.plot(X,p(X),"r--",color='red')
plt.errorbar(X, Y,xerr=xerr,yerr=yerr)
#plt.set_title('Resistores A y B en Serie', fontsize=14)
plt.xlabel('RC (FΩ = s)',fontsize=18)
plt.ylabel('Tiempo promedio (s)',fontsize=18)
plt.title('Tiempo característico', fontsize=20)
plt.show()


import requests
import json
import re
import time
import random
import redis
redis=redis.Redis(host="127.0.0.1",port=6379,db=2)
url="http://fundgz.1234567.com.cn/js/{}.js?rt=1589463125600"
r = requests.get('http://fund.eastmoney.com/js/fundcode_search.js')
print(r)
cont = re.findall('var r = (.*])', r.text)[0]  # 提取list
ls = json.loads(cont)  # 将字符串个事的list转化为list格式
print(ls)
daihao=[i[0] for i in ls]
num=0
redis.delete("errorFundCode")
for i in daihao:
    try:
        r=requests.get(url.format(i),timeout=3)
        text = re.findall('\((.*?)\)', r.text)[0]
        dic=json.loads(text)
        print("[{}] ".format(num),text)
    except Exception as e:
        print("wrong daihao: ",i)
        redis.lpush("errorFundCode",i)
        continue
    latest=redis.lindex(i,-1)
    if latest==None:
        redis.rpush(i,text)
        num+=1
        print("[{}] new add: ".format(num),i)
    else:
        latest=json.loads(latest)
        if latest["gztime"]!=dic["gztime"]:
            redis.rpush(i,text)
            num += 1
            print("[{}] add: ".format(num), i)
    time.sleep(random.random())


import logging
import os

from datetime import datetime

import monasca_agent.collector.checks as checks

log = logging.getLogger(__name__)

class Crash(checks.AgentCheck):

    def __init__(self, name, init_config, agent_config):
        super(Crash, self).__init__(name, init_config, agent_config)
        self.crash_dir = self.init_config.get('crash_dir', '/var/crash')
        log.debug('crash dir: %s', self.crash_dir)

    def check(self, instance):
        """Capture crash dump statistics
        """
        dimensions = self._set_dimensions(None, instance)
        dump_count = 0
        value_meta = {'latest': ''}

        # Parse the crash directory
        if os.path.isdir(self.crash_dir):
            for entry in sorted(os.listdir(self.crash_dir), reverse=True):
                if os.path.isdir(os.path.join(self.crash_dir, entry)):
                    try:
                        dt = datetime.strptime(entry, '%Y%m%d%H%M')
                    except ValueError:
                        continue

                    # Found a valid crash dump directory
                    log.debug('found crash dump dir: %s',
                              os.path.join(self.crash_dir, entry))
                    dump_count += 1

                    # Return the date-/timestamp of the most recent crash
                    if dump_count == 1:
                        value_meta = {'latest': unicode(dt)}

        log.debug('dump_count: %s', dump_count)
        self.gauge('crash.dump_count', dump_count, dimensions=dimensions,
                   value_meta=value_meta)


#!/usr/bin/env python
# -*- encoding: utf-8 -*-
# vim: set et sw=4 ts=4 sts=4 ff=unix fenc=utf8:
# Author: Binux<i@binux.me>
#         http://binux.me
# Created on 2014-02-22 23:20:39

import socket
import time
import datetime
from six import iteritems, itervalues
from flask import render_template, request, json

try:
    import flask_login as login
except ImportError:
    from flask.ext import login

from .app import app

index_fields = ['name', 'group', 'status', 'comments', 'rate', 'burst', 'updatetime']

@app.route('/')
def index():
    projectdb = app.config['projectdb']
    projects = sorted(projectdb.get_all(fields=index_fields),
                      key=lambda k: (0 if k['group'] else 1, k['group'] or '', k['name']))
    return render_template("index.html", projects=projects)

@app.route('/queues')
def get_queues():
    def try_get_qsize(queue):
        if queue is None:
            return 'None'
        try:
            return queue.qsize()
        except Exception as e:
            return "%r" % e

    result = {}
    queues = app.config.get('queues', {})
    for key in queues:
        result[key] = try_get_qsize(queues[key])
    return json.dumps(result), 200, {'Content-Type': 'application/json'}

@app.route('/update', methods=['POST', ])
def project_update():
    projectdb = app.config['projectdb']
    project = request.form['pk']
    name = request.form['name']
    value = request.form['value']

    project_info = projectdb.get(project, fields=('name', 'group'))
    if not project_info:
        return "no such project.", 404
    if 'lock' in projectdb.split_group(project_info.get('group')) \
            and not login.current_user.is_active():
        return app.login_response

    if name not in ('group', 'status', 'rate'):
        return 'unknown field: %s' % name, 400
    if name == 'rate':
        value = value.split('/')
        if len(value) != 2:
            return 'format error: rate/burst', 400
        rate = float(value[0])
        burst = float(value[1])
        update = {
            'rate': min(rate, app.config.get('max_rate', rate)),
            'burst': min(burst, app.config.get('max_burst', burst)),
        }
    else:
        update = {
            name: value
        }

    ret = projectdb.update(project, update)
    if ret:
        rpc = app.config['scheduler_rpc']
        if rpc is not None:
            try:
                rpc.update_project()
            except socket.error as e:
                app.logger.warning('connect to scheduler rpc error: %r', e)
                return 'rpc error', 200
        return 'ok', 200
    else:
        return 'update error', 500

@app.route('/counter')
def counter():
    rpc = app.config['scheduler_rpc']
    if rpc is None:
        print(11111111111111111)
        return json.dumps({})

    result = {}
    try:
        data = rpc.webui_update()
        for type, counters in iteritems(data['counter']):
            for project, counter in iteritems(counters):
                result.setdefault(project, {})[type] = counter
        for project, paused in iteritems(data['pause_status']):
            result.setdefault(project, {})['paused'] = paused
    except socket.error as e:
        app.logger.warning('connect to scheduler rpc error: %r', e)
        return json.dumps({}), 200, {'Content-Type': 'application/json'}

    return json.dumps(result), 200, {'Content-Type': 'application/json'}

@app.route('/run', methods=['POST', ])
def runtask():
    rpc = app.config['scheduler_rpc']
    if rpc is None:
        return json.dumps({})

    projectdb = app.config['projectdb']
    project = request.form['project']
    project_info = projectdb.get(project, fields=('name', 'group'))
    if not project_info:
        return "no such project.", 404
    if 'lock' in projectdb.split_group(project_info.get('group')) \
            and not login.current_user.is_active():
        return app.login_response

    newtask = {
        "project": project,
        "taskid": "on_start",
        "url": "data:,on_start",
        "process": {
            "callback": "on_start",
        },
        "schedule": {
            "age": 0,
            "priority": 9,
            "force_update": True,
        },
    }

    try:
        ret = rpc.newtask(newtask)
    except socket.error as e:
        app.logger.warning('connect to scheduler rpc error: %r', e)
        return json.dumps({"result": False}), 200, {'Content-Type': 'application/json'}
    return json.dumps({"result": ret}), 200, {'Content-Type': 'application/json'}

@app.route('/robots.txt')
def robots():
    return """User-agent: *
Disallow: /
Allow: /$
Allow: /debug
Disallow: /debug/*?taskid=*
""", 200, {'Content-Type': 'text/plain'}

@app.route('/health')
def health():
    """
    health check, service:
    webui
    scheduler
    fetcher
    processor
    result_worker
    :return:
    """
    # all service return dict
    good = {
        "status": "OK",
        "type": "HARD",
        "checked_at": gettime(),
        "spent_time": "1ms",
        "info": "ok"
    }
    bad = {
        "status": "ERROR",
        "type": "HARD",
        "checked_at": gettime(),
        "spent_time": "1ms",
        "info": "ERROR"
    }

    _health = {
        "status": "OK",
        "dependencies": {
            "webui": good.copy()
        }
    }

    # scheduler fetcher procsssor result
    mapping = {
        'fetcher2processor': 'processor',
        'newtask_queue': 'scheduler',
        'processor2result': 'result_worker',
        'scheduler2fetcher': 'fetcher'
    }

    result = get_queues()
    queue_info = dict(json.loads(result[0]))
    # redis down
    try:
        alert_info = {k: int(v) for k, v in queue_info.items()}
        _health['dependencies']['redis'] = good.copy()
    except:
        _health['dependencies']['redis'] = bad.copy()
        _health['dependencies']['redis']['info'] = 'redis down'
        return json.dumps(_health)

    # check num
    for k, v in alert_info.items():
        service = mapping.get(k)
        if service:
            if v >= 100:
                _health['dependencies'][service] = bad.copy()
                _health['dependencies'][service]['info'] = '{} down'.format(service)
            else:
                _health['dependencies'][service] = good.copy()

    # check error table
    resultdb = app.config['resultdb']
    project = 'error'
    offset = int(request.args.get('offset', 0))
    limit = int(request.args.get('limit', 20))
    try:
        results = list(resultdb.select(project, offset=offset, limit=limit))
        _health['dependencies']['db'] = good.copy()
    except:
        # db down
        _health['dependencies']['db'] = bad.copy()
        _health['dependencies']['db']['info'] = 'db down'

    # scheduler xmlrpc detect
    try:
        app.config['scheduler_rpc_client'].ping()
    except:
        _health['dependencies']['scheduler']['status'] = 'ERROR'
        _health['dependencies']['scheduler']['info'] = 'scheduler server down'

    try:
        app.config['fetcher_rpc_client'].ping()
    except:
        _health['dependencies']['fetcher']['status'] = 'ERROR'
        _health['dependencies']['fetcher']['info'] = 'fetcher server down'

    return json.dumps(_health)

@app.route('/ping')
def ping():
    return 'pong'

def gettime():
    return datetime.datetime.now().strftime('%Y-%m-%dT%H:%M:%S') + str(time.time() % 1)[1:5] + 'Z'


import subprocess

def url_xpath(url, path):
    import scrapelib
    import lxml.html
    doc = lxml.html.fromstring(scrapelib.urlopen(url))
    return doc.xpath(path)

def convert_pdf(filename, type='xml'):
    commands = {'text': ['pdftotext', '-layout', filename, '-'],
                'text-nolayout': ['pdftotext', filename, '-'],
                'xml':  ['pdftohtml', '-xml', '-stdout', filename],
                'html': ['pdftohtml', '-stdout', filename]}
    try:
        pipe = subprocess.Popen(commands[type], stdout=subprocess.PIPE,
                                close_fds=True).stdout
    except OSError:
        raise EnvironmentError("error running %s, missing executable?" %
                               ' '.join(commands[type]))
    data = pipe.read()
    pipe.close()
    return data

def pdf_to_lxml(filename, type='html'):
    import lxml.html
    text = convert_pdf(filename, type)
    return lxml.html.fromstring(text)


# This script is to find the probability of obtaining any one of the 256
# 8-bit binary unmbers which could constitute a changepoint configuration.
# We define \Gamma as the 8 bit number, and \gamma_i as each bimary element
# of \Gamma.
# We can define the probability of \gamma_i as
# P_gamma = (1/(((2*pi)^-0.5)*sigma))*exp((-(D-gamma_i*h)^2)/(2*sigma^2));
# Now we can define some other things. for example, D is fed forward from
# the CW analysis. Sigmma may be given (look into feed forward from CW
# analysis), h fed forward from CW analysis
# We compute the probability for each of the 256 8-bit numbers

from __future__ import division
import numpy as np
import matplotlib as mpl
from argparse import ArgumentParser
import inspect
import os
import datetime
mpl.rcParams['agg.path.chunksize'] = 10000
import matplotlib.pyplot as plt
from binary_structure import binary_structure
from logaddexp import logaddexp
from logaddexpvect import logaddexpvect
import math
import matplotlib as mpl
import plot_functions

def nCr(n,r):
    f = math.factorial
    return f(n) / f(r) / f(n-r)
	#There should exist a scipy nCr function

parser = ArgumentParser()

parser.add_argument("-d", "--data_list", dest = "data",
                  help = "List of data written as [x, x, x ...].", metavar = "LIST", default = [])
parser.add_argument("-i", "--input_file", dest = "infile",
		  help = "Location of the data file to be used as input, as a string.", metavar = "STRING")
parser.add_argument("-o", "--Output_path", dest = "outpath",
                  help = "Location of the data file to be used to print output.", metavar = "STRING")

args = parser.parse_args()

infile = args.infile
if not os.path.exists(infile):
        os.makedirs(infile)
        print('Please double check input file')

data = args.data
#CP_prior = args.CP_prior
output = args.outpath

if output[-1] != '/':
        output += '/'
if not os.path.exists(output):
        os.makedirs(output)
        print(output + ' created.')

data = []
def read_data(infile):
	data = []
	with open(infile, mode='r') as file:
		content = file.read()
	for line in content.split('\n'):
		string_line = line.split(',')
		for item in range(len(string_line)):
			string_line[item] = string_line[item].strip()
			string_line[item] = string_line[item].strip('\t')
			string_line[item] = string_line[item].strip('[')
			string_line[item] = string_line[item].strip(']')
		data_line = []
		for item in string_line:
			if item != '':
				data_line.append(float(item))
		if data_line != []:
			data.append(data_line)
	return data

data = read_data(infile)

all_data = np.asarray(data)
h_sd = 500 # 1e-24
hs = np.linspace(0, h_sd, 1001)
sigma = 50

l_prior = 0.001*np.ones(1000)

h_vals = np.linspace(h_sd/1001.0,h_sd,num=1000)
log_dh = np.log(h_vals[6] - h_vals[5])

#create a list of binary numbers of length len(data)
numlist = np.linspace(0, (2**len(all_data[0]))-1, num=2**len(all_data[0]))
bin_list = []

for itt in range(len(numlist)):
	number = bin(int(numlist[int(itt)]))
	x = []
	for item in number[2:]:
		x.append(int(item))
	while len(x)<len(all_data[0]):
		x = [0] + x
	bin_list.append(x)

bin_array = np.array(bin_list)

l_likelihood =  np.zeros(len(bin_list))

tile_var = [all_data.shape[0], 1]
big_h_vals = np.transpose(np.tile(h_vals, tile_var))
big_prior = np.transpose(np.tile(l_prior, tile_var))
#big_data = np.tile(data, [1000,1])
P_gamma = np.zeros([len(bin_list), len(bin_list[0])])
l_evidence = []
config = 0
n_changepoints = []
l_norm = []

for itt_number in range(len(bin_array)):
	config = itt_number
	binary_number = bin_list[itt_number]
	binary_struc =  binary_structure( binary_number )
	block_length = binary_struc["block_length"]
	block_numbers = binary_struc["block_numbers"]
	n_breaks = binary_struc["n_breaks"]
	n_changepoints.append(binary_struc["n_changepoints"])
	binary_number = np.append(binary_number, 0)
	each_h1 = -1*np.inf * np.ones(big_h_vals.shape)
	prefactor = np.log(np.sqrt(2.0*np.pi)*sigma)
	index = 0
	used_index = []
	data = all_data[itt_number]
	big_data = np.tile(data, [1000,1])
#	print(data)
	print(len(binary_number)-1)
	while index < len(binary_number)-1:
		if binary_number[index] == 1 and binary_number[index+1] ==1:
#			# Sum the data from the start to the end of the block,
			# also marginalise over h define variable end_of_block
			# to define where coherent marginalisation ends
			end_of_block = index + block_length[int(block_numbers[index])-1]
			exponent = -(((big_data[:,index:end_of_block] - big_h_vals[:,index:end_of_block])**2)/(2*sigma*sigma)) # -((D-h)/sqrt(2)*sigma)**2

			each_h1[:,index:end_of_block] = exponent + big_prior[:,index:end_of_block] + log_dh - prefactor
			# P_gamma is the sum of these values, log10(sum(prior * gaussian))
			# Calculate P_gamma chunk by chunk for each chunk in the block

			for row in range(index,end_of_block):
				if row == index:
					P_gamma[config, row] = logaddexpvect(each_h1[:,row])
					#used_index.append(row)
			index = end_of_block - 1 # Progess the index to end of block

		elif binary_number[index] == 1 and binary_number[index + 1] == 0:
			exponent = -(((big_data[:, index] - big_h_vals[:,index])**2) / (2 * sigma * sigma)) # -((D-h)/(sqrt(2)sigma))**2
			each_h1[:,index] = exponent + big_prior[:,index]  + log_dh - prefactor
			P_gamma[config,index] = logaddexpvect(each_h1[:,index]) # There is a numpy logaddexp function

		elif binary_number[index] == 0:
		    	P_gamma[config,index] = (-((data[index])**2)/(2.0*sigma*sigma)) - prefactor

		index = index + 1
		l_likelihood[config] = np.sum(P_gamma[config,:])
#	l_likelihood[config] = np.sum(data)
	l_norm.append(np.log((1/(len(binary_number)-1))*(nCr(len(data)-1,n_changepoints[config]))))
	l_evidence.append(l_likelihood[config] + l_norm[config])
	config += 1

l_odds = l_evidence - l_evidence[1]

sorted_evidence = sorted(l_evidence)
evidence_index = [i[0] for i in sorted(enumerate(l_evidence), key=lambda x:x[1])]
sorted_n_CP = [n_changepoints[i] for i in evidence_index]
print(l_evidence)

# Define the odds of one config vs all other configs
# Denominator is sum of all that are not that index
odds_all = np.zeros(len(l_evidence));
evidence = np.ma.array(np.exp(l_evidence), mask=False)
for index in range(len(l_evidence)):
	evidence.mask[index] = True
	odds_all_denom = np.sum(evidence)
	evidence.mask = False
	odds_all[index] = evidence[index] / odds_all_denom;

sorted_odds_all = sorted(odds_all)
odds_index = [i[0] for i in sorted(enumerate(odds_all), key=lambda x:x[1])]

sorted_odds = sorted(l_odds)
sort_index = [i[0] for i in sorted(enumerate(l_odds), key=lambda x:x[1])]

sorted_binaries = [bin_list[i] for i in odds_index]
sorted_priors = [l_norm[i] for i in odds_index]

sorted_data = [all_data[i] for i in odds_index]

sanity_check = [odds_index[i] - sort_index[i] for i in range(len(sort_index))]

out = output + 'outfile.txt'
o = open(out, 'w+')
o.seek(0)
o.write("Sanity check result: " + str(sum(sanity_check)) + ".\n")

results_dict = []
for i in range(len(sorted_odds_all)):
	results_dict.append({"sorted_binaries":sorted_binaries[i], "sorted_odds_all":sorted_odds_all[i], "sorted_evidence":sorted_evidence[i] , "sorted_priors":sorted_priors[i], "data used":sorted_data[i]})

for i in range(len(results_dict)):
	print(str(results_dict[i]) + "\n")
	o.write(str(results_dict[i]) + "\n")

# Find the right data instance to plot to show data atoms
i = len(all_data[0])
running_total = 0
while i>0:
	running_total += int(2**(i-1))
	i-=2
print(running_total)

plot_functions.barcode_plot(sorted_binaries,  sorted_odds, all_data[running_total],  output)
#for i in sorted_binaries:
#	print(str(i))

# Output key parameters to file.
# Plot barcode plots.

print("Sanity check result: " + str(sum(sanity_check)) + ".\n")


# -*- coding: utf-8 -*-
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
from bs4 import BeautifulSoup
import pandas as pd
import time
from camera_name_dict import make_dict

portal_sites = ['kakaku.com',
'www.yodobashi.com',
'biccamera.com',
'amazon.co.jp',
'mapcamera.com',
'rakuten.co.jp',
'shopping.yahoo.co.jp',
'sony.jp',
'shop.kitamura.jp',
'yamada-denkiweb.com',
'nojima.co.jp',
'joshinweb.jp',
'kojima.net',
'edion.com'
]

name_check_list = make_dict()

df_target_cameras = pd.read_csv('datasets/target_cameras.csv')
cam_names_list = df_target_cameras['name']

cam_list_del_id = 0

driver = webdriver.Chrome('./chromedriver')

driver_reset = 1

for model in cam_names_list:

  if driver_reset > 3:
    driver.quit()
    driver = webdriver.Chrome('./chromedriver')
    driver_reset = 1

  results =[]
  driver.implicitly_wait(5)
  driver.get('https://www.google.co.jp/search?q=' + model + 'レビュー')
  driver.implicitly_wait(5)

  try:
    while True:
      html = driver.page_source
      soup = BeautifulSoup(html, "html.parser")

      all_search = soup.find_all(class_='rc')

      for data in all_search:
        url = data.find('a').get('href')
        for pt_site in portal_sites:
          exit_flag = False
          if pt_site in url:
            exit_flag = True
          else:
            title = data.find('h3').string
            dis_element = data.find(class_='st')
            if dis_element is None:
              dis = ''
            else:
              dis = dis_element.string
            result = [title, dis, url]
            results.append(result)
            exit_flag = True
          if exit_flag is True:
            break

      driver.find_element_by_id('pnnext').click()

  except NoSuchElementException:
      print('最後のページまできたよ')

  index_of_results = 0
  del_list = []

  for item in results:
    dele = True
    stop_loop = False
    for check_words in name_check_list:
      if stop_loop is True:
        stop_loop = False
        break
      for check_word in check_words:
        if check_word in item[0]:
          dele = False
          stop_loop = True
          break
        else:
          pass
    if dele is True:
      results.pop(index_of_results)
    index_of_results += 1

  print(model)
  print(str(len(results)) + '件のデータが取れたよ')

  if len(results) ==0:
    print('スクレイピング失敗！処理を終えます！')
    driver.quit()
    break

  df_review = pd.read_csv('datasets/reviews.csv', index_col=0 )
  df_inter_table = pd.read_csv('datasets/camera_reviews.csv', index_col=0)

  if len(df_inter_table) == 0:
    last_cam_id = 0
  else:
    camera_ids = df_inter_table['camera_id']
    table_len = len(camera_ids)
    last_cam_id = int(camera_ids[table_len - 1])

  if len(df_review) == 0:
    review_index = 1
  else:
    review_index = len(df_review) + 1

  for result in results:
    df_result = pd.DataFrame()
    df_result['title'] = [result[0]]
    df_result['body'] = [result[1]]
    df_result['url'] = [result[2]]
    df_review = pd.concat([df_review, df_result])

    df_cam_rev = pd.DataFrame()
    df_cam_rev['camera_id'] = [last_cam_id + 1]
    df_cam_rev['review_id'] = [review_index]
    df_inter_table = pd.concat([df_inter_table, df_cam_rev])
    review_index += 1

  df_review = df_review.reset_index(drop=True)
  df_inter_table = df_inter_table.reset_index(drop=True)
  df_review.to_csv('datasets/reviews.csv')
  df_inter_table.to_csv('datasets/camera_reviews.csv')

  cam_names_list = cam_names_list.drop(cam_list_del_id)
  df_update_cameras = pd.DataFrame()
  df_update_cameras['name'] = cam_names_list
  df_update_cameras.to_csv('datasets/target_cameras.csv')
  cam_list_del_id += 1

  driver_reset += 1


# run python setup.py build_ext
#  in order to build cpp files from the pyx files

import os

import numpy

from distutils.core import setup
from distutils.extension import Extension
from Cython.Distutils import build_ext

distmetrics = Extension("distmetrics",
                        ["distmetrics.pyx"])

brute_neighbors = Extension("brute_neighbors",
                            ["brute_neighbors.pyx"])

ball_tree = Extension("ball_tree",
                      ["ball_tree.pyx"])

setup(cmdclass = {'build_ext': build_ext},
      name='distmetrics',
      version='1.0',
      ext_modules=[distmetrics],
      include_dirs=[numpy.get_include(),
                    os.path.join(numpy.get_include(), 'numpy')]
      )

setup(cmdclass = {'build_ext': build_ext},
      name='brute_neighbors',
      version='1.0',
      ext_modules=[brute_neighbors],
      include_dirs=[numpy.get_include(),
                    os.path.join(numpy.get_include(), 'numpy')]
      )

setup(cmdclass = {'build_ext': build_ext},
      name='ball_tree',
      version='1.0',
      ext_modules=[ball_tree],
      include_dirs=[numpy.get_include(),
                    os.path.join(numpy.get_include(), 'numpy')]
      )


from sklearn.externals import joblib
import json
import pickle
import pandas as pd
import thulac
import numpy as np
import json
# # 生成词向量
# thu1 = thulac.thulac(seg_only=True)  #默认模式
# text = thu1.cut("工商注册地、税务征管关系及统计关系在广州市南沙区范围内", text=True)  #进行一句话分词
# print(text)

def get_pretrained_vec():
    vocabu2id=joblib.load('../open_data/vocabulary2id.pkl')
    # df=pd.read_csv('../data/train_people_relation.txt',delimiter='\t')
    # print(df['rel'].value_counts())

    # for each in df['sentence'].head():
    #     print(each)

    # with open('../data/sgns.baidubaike.bigram-char',encoding='utf8') as f:
    #     all_lines=f.read()
    #     all_lines_split=all_lines.split('\n')
    #     print(len(all_lines_split))

    word_vec_new=[]
    with open('../data/sgns.baidubaike.bigram-char',encoding='utf8') as f:

        for index ,each0 in enumerate(f):
            word_dict = {}
            if index==0:
                print('index=0,each',each0)
                continue

            # if index>30:
            #     break
            #     print(each0)
            each=each0.strip().split(' ')

            # print('each:',len(each),each[0],len(each[1:]))
            # print(len(each),each[0],each[1:])
            if len(each[1:])!=300:
                print(index,len(each[1:]))
                print(each0)
                continue
            else:
                if each[0] in vocabu2id:
                    word_dict['word']=each[0]
                    word_dict['vec']=each[1:]
                    word_vec_new.append(word_dict)
                if index%50000==0:
                    print('index={},word_dict={}'.format(index,word_dict))
    print('len word_vec_new',len(word_vec_new))
    print('word_vec_new\n',word_vec_new[0:10])
    print('开始保存文件')

    with open('../data/word_vec_people.json','w',encoding='utf8') as f:
        #两种效果一样
        ## f.write(json.dumps(all_rel_data,indent=4))
        json.dump(word_vec_new,f,ensure_ascii=False)

    # with open('../data/word_vec_people.pkl','wb') as f:
    #     #两种效果一样
    #     ## f.write(json.dumps(all_rel_data,indent=4))
    #     # json.dump(word_vec_new,f,ensure_ascii=False)
    #     pickle.dump(word_vec_new,f)

    # with open('../data/word_vec_people.pkl', 'rb') as f:
    #      data = pickle.load(f)
    #      print(data)

def convert_people_train():
    path = '../open_data/sent_dev.txt'
    df=pd.read_csv(path,delimiter='\t',header=-1,names=['id','e1','e2','sentence'])
    print(df.head())

    df_rel=pd.read_csv('../open_data/sent_relation_dev.txt',delimiter='\t',header=-1,names=['rel_id','class'])
    print('df_rel.head',df_rel.head())
    with open('../data/rel2id_people.json',encoding='utf8') as f:
      rel2id=json.load(f)
      id2rel={}
      for k,v in rel2id.items():
          # print(type(k),type(v))
          id2rel[v]=k
    print('id2rel',id2rel)
    # df_rel['class'].apply(lambda row: print(type(row),row))
    def id2rel_process(class_input):
        # print('type',type(class_input),class_input)
        return  id2rel[int(class_input)]

    df_rel['rel']=df_rel['class'].apply(lambda row:id2rel_process(row))
    print('df_rel.tail\n',df_rel.tail())
    rel_class_id={}
    for index ,row in df_rel.iterrows():
        rel_class_id[row['rel_id']]=row['rel']

    # with open('../data/rel_class_id.json', 'w', encoding='utf8') as f:
    # # #两种效果一样
    #     ## f.write(json.dumps(all_rel_data,indent=4))
    #     json.dump(rel_class_id,f,ensure_ascii=False,indent=4)

    print('rel_class_id\n',rel_class_id)
    all_train=[]
    for index,row in df.iterrows():
        print('row',row['id'],rel_class_id[row['id']])
        each_train={}
        each_train['head']={'id':row['id'],'word':row['e1']}
        each_train['relation']=rel_class_id[row['id']]
        each_train['sentence']=row['sentence']
        each_train['tail']={'id':row['id'],'word':row['e2']}
        # print('each_train',each_train)
        all_train.append(each_train)

    with open('../data/dev_people.json','w',encoding='utf8') as f:
        #两种效果一样
        ## f.write(json.dumps(all_rel_data,indent=4))
        json.dump(all_train,f,ensure_ascii=False,indent=4)
        #显示比较少时前面的关系确实都为NA，转换程序并没有错

    pass
def convert_people_rel2id():
    rel2id_people={}
    path = '../open_data/relation2id.txt'
    df = pd.read_csv(path, delimiter='\t',header=-1,names=['relation','class'])
    print('df',df)
    for index,each in df.iterrows():
        rel2id_people[each['relation']]=each['class']
    with open('../data/rel2id_people.json','w',encoding='utf8') as f:
        #两种效果一样
        ## f.write(json.dumps(all_rel_data,indent=4))
        json.dump(rel2id_people,f,ensure_ascii=False)

    pass
if __name__=='__main__':
    # convert_people_rel2id()
    convert_people_train()

    # with open('../data/train_people.json',encoding='utf8') as f:
    #     # 两种效果一样
    #     ## f.write(json.dumps(all_rel_data,indent=4))
    #     list_data=json.load(f)
    #     print(list_data[-10:-1])
    pass