#!/usr/bin/env python
# coding: utf-8

#
# # Projet: Investiguer la Base de Données des Films "The Movies Database"
#
# ## Table des Matières
#

#
Introduction

#
Traitement des Données

#
Analyse Exploratoire des Données

#
Conclusions

#

#
# ## Introduction
#
# L'ensemble de données sur les films contient des informations sur 10 000 films recueillies auprès de The Movie Database (TMDb). Cet ensemble de données comprend les avis des utilisateurs, les revenus, les acteurs, le genre, le directeur et la boîte de production comme principales variables à étudier.
#
# Compte tenu de tout cela, la principale question à se poser est la suivante : **Quels sont les critères de succès d'un film?**
#
# Comme il n'existe pas de définition précise d'un film à succès, nous pouvons citer deux variables cruciales à cet égard :
# 1. Le film est un succès commercial en termes de profits.
# 2. le film est hautement noté par le public - Populairité.
#
# Pour cela, nous allons étudier les différents critères qui font le succès d'un film du point de vue commercial et du point de vue du public.
#
# Nous pouvons décomposer cette question générale en questions spécifiques afin d'orienter notre analyse et d'obtenir la réponse précise:
# 1. Success Commerciale: Quels critères sont associés aux films qui ont des revenus élevés?
# 2. Popularité: Quels critères sont associés aux films qui ont des votes élevés?
# 3. Un film populaire signifie-t-il directement un film commercialement réussi?
#
# Enfin, il est particulièrement intéressant d'étudier l'évolution des bénéfices au fil des ans.
#
# La conclusion de toutes les analyses sera détaillée à la fin de ce rapport et répond à la question principale.

# ## Importation des Paquets et La Base de Données
# Dans cette premiere section du rapport, nous allons charger les pqauets necessaires aisni que la base de données des films.

# In[1]:

# Importing necessary packages
import pandas as pd

import numpy as np

import matplotlib.pyplot as plt

import seaborn as sns

#Importing the database we will work on
df = pd.read_csv('tmdb-movies.csv')

#
# ## Traitement des Données
# Dans cette section du rapport, nous allons charger les données, vérifier leur propreté, puis découper et nettoyer notre ensemble de données pour l'analyse.
# ### Propriétés Générales

# In[2]:

# Load the database
df.head()

# In[3]:

print(df.shape)
df.info()

# #### Insight
# Les informations sur les données indiquent que certaines colonnes ne seront pas utilisées dans notre analyse car elles ne sont pas directement liées à notre problématique, et que certaines valeurs manquantes doivent être corrigées pour garantir une analyse précise.
#
# Nous vérifierons également les lignes dupliquées et les types de données inexactes pour les corriger.

# ### Data Cleaning

# In[4]:

# Dropping unecessary columns
df.drop(['id', 'imdb_id','homepage','tagline', 'overview',  'keywords'], axis=1, inplace=True)
df.info()

# In[5]:

# Checking and removing duplicated rows
sum(df.duplicated())

# In[6]:

df.drop_duplicates(inplace=True)

# In[7]:

sum(df.duplicated())

# In[8]:

# Checking and fixing missing values
df.isnull().sum()

# In[9]:

df.fillna(0, inplace = True)

# In[10]:

df.isnull().sum()

# In[11]:

# Correcting the release date data type
df['release_date'] = pd.to_datetime(df['release_date'])

# Maintenant que nous avons nettoyé notre base de données des colonnes non essentielles, des valeurs manquantes, des lignes redondantes et des types de données erronés, nous allons créer trois nouvelles colonnes:
#
#
# 1. **profit**: ou la colonne de bénéfices: cette colonne est nécessaire pour comparer les bénéfices de chaque film afin de pouvoir extraire le critère du succès commercial (bénéfices élevés)
#
#
# 2. **profit_rate**: ou le taux_de_profits : Même si les bénéfices peuvent donner un aperçu des films qui font des profits, le taux de profits sera un indicateur plus précis car il souligne à quel point un film multiplie ses revenus par rapport au budget investi.
#
#
# 3. **release_month**: dans cette colonne, nous ne considérerons que le mois car le mois de sortie peut affecter le succès du film. Pour cela, nous allons supprimer la colonne release_date.

# In[12]:

df['profits'] = df['revenue_adj'] - df['budget_adj']
df['profits_rate'] = df['revenue_adj'] / df['budget_adj']
df.drop(['revenue_adj', 'budget_adj'], axis=1, inplace=True)

# In[13]:

df['release_month'] = df['release_date'].dt.month
df.drop(['release_date'], axis=1, inplace=True)

# In[14]:

print(df.shape)
df.head()

#
# ## Analyse Exploratoire des Données
# Dans cette section, nous analyserons nos données pour répondre à nos principales questions en détail. À la fin de cette section, nous comprendrons les facteurs critiques d'un succès commercial et d'un film populaire (point de vue du public).
#
# Une analyse de corrélation sera effectuée pour découvrir la dépendance entre les deux principales variables en question : le succès commercial et la popularité d'un film. En d'autres termes, nous aimerions savoir si un film très bien noté fera plus de bénéfices.
#
#
# ### Question de Recherche 1: Success Commerciale: Quels critères sont associés aux films qui ont des revenus élevés?
#
# #### Critère 1: Genres
# Nous allons étudier si le genre d'un film est un facteur de succès commercial. En d'autres termes, nous chercherons à savoir s'il existe un ou plusieurs genres spécifiques d'un film qui contribuent à faire plus de profits. Si ce n'est pas le cas, alors il existe d'autres facteurs de succès commercial différents des genres.
# Nous étudierons deux échantillons : un échantillon de top 50 films et un second échantillon de top 1000 films.
#

# In[48]:

def stat(sortBy, headCount, column):
    df_stat = df.sort_values(by=sortBy, ascending = False).head(headCount)
    df_stat.loc[:, column].head()
    df_stat[column] = df_stat[column].astype(str)
    df_stat_sep = df_stat[column].str.cat(sep='|')
    df_stat_sep = pd.Series(df_stat_sep.split('|'))
    df_stat_sep = df_stat_sep.value_counts(ascending = False).to_frame()
    return df_stat_sep

data = stat('profits', 50, 'genres')
data.head()

# In[49]:

data = stat('profits', 1000, 'genres')
data.head()

# #### Insight
# Sur la base des bénéfices et de deux échantillons différents, nous n'avons pas pu prendre une décision claire sur les genres qui contribuent à des bénéfices élevés, car lorsque la taille de l'échantillon a changé, les valeurs ont changé, ce qui est logique. Pour cela, nous utiliserons le taux de profits pour une étude plus approfondie.

# In[50]:

data = stat('profits_rate', 50, 'genres')
data.head()

# In[53]:

data = stat('profits_rate', 1000, 'genres')
data.head()

# #### Insight
# L'utilisation du taux de profits donne de meilleurs résultats car il met l'accent sur les profits, c'est-à-dire sur les revenus qu'un film génère par rapport au budget investi.
#
# Cette analyse montre que les genres Drame et Comédie contribuent le plus à la réalisation de bénéfices, et donc à un succès commercial.

# #### Critère 2: Cast
# Nous allons étudier si le cast d'un film est un facteur de succès commercial. En d'autres termes, nous chercherons à savoir s'il un acteur ou plusieurs acteurs spécifiques contribuent à faire plus de profits.
# data = stat('profits_rate', 1000, 'genres')
# data.head()
#
# Nous étudierons deux échantillons : un échantillon de top 50 films et un second échantillon de top 1000 films.

# In[56]:

data = stat('profits', 50, 'cast')
data.head()

# In[59]:

data = stat('profits', 1000, 'cast')
data.head()

# In[61]:

data = stat('profits_rate', 50, 'cast')
data.head()

# In[63]:

data = stat('profits_rate', 1000, 'cast')
data.head()

# #### Insight
# Sur la base de cette analyse, nous pouvons conclure deux points majeurs:
# 1. Si l'on considère quels sont les acteurs qui contribuent à faire plus de recettes, il s'agirait de Tom Cruise et Tom Hanks, ce qui est logique puisqu'ils sont principalement choisis pour leur notoriété.
#
# 2. Si l'on considère les films qui font plus de recettes tout en investissant moins de budget, ce qui signifie un taux de profit plus élevé, ce sont d'autres acteurs comme Clint Eastwood et John Cusack. Dans le même temps, Tom Cruise et Tom Hanks n'apparaissent pas dans la liste des 10 acteurs ayant les taux de profit les plus élevés. Cela peut refléter le fait que ces deux acteurs sont payés très cher et qu'ils contribuent effectivement à la réalisation de bénéfices, mais pas au taux de profit.

# #### Critère 3: Production Company
# Nous chercherons à savoir si la société de production peut être un facteur de succès commercial pour les réalisateurs de films. Y a-t-il vraiment une différence dans les revenus lorsqu'on travaille avec une société de production et pas une autre?

# In[65]:

data = stat('profits', 50, 'production_companies')
data.head()

# In[66]:

data = stat('profits', 1000, 'production_companies')
data.head()

# In[68]:

data = stat('profits_rate', 50, 'production_companies')
data.head()

# In[69]:

data = stat('profits_rate', 1000, 'production_companies')
data.head()

# #### Insight
# Qu'il s'agisse de profits purs ou de taux de profits, il y a trois grandes sociétés de production qui contribuent le plus aux revenus élevés des films : Warner Bros, Universal Pictures et Paramount Pictures.
# Par conséquent, travailler avec l'une de ces sociétés garantira les profits les plus élevés.

# #### Critère 4: Release Month
# Nous chercherons à savoir si le mois de diffusion de film peut être un facteur de succès commercial pour les réalisateurs de films. Y a-t-il vraiment une différence dans les revenus lorsqu'on diffuse le film dans un mois et pas un autre?

# In[70]:

data = stat('profits', 50, 'release_month')
data.head()

# In[71]:

data = stat('profits_rate', 100, 'release_month')
data.head()

# #### Insight
# Le 6e mois, qui est Juin, s'est avéré être le mois le plus rentable pour la diffusion des films, ce qui est logique puisque c'est le début des vacances d'été où les gens souhaitent profiter au maximum de leur temps.

# ### Question de Recherche 2: Popularité: Quels critères sont associés aux films qui ont des votes élevés?
# Dans cette section, nous allons d'abord étudier la corrélation entre la popularité et les films bien notés. En d'autres termes, nous aimerions savoir si un film populaire est nécessairement bien noté par le public.
#
# Ensuite, nous étudierons les facteurs qui motivent le public à donner de bonnes critiques à un film. Cela aidera les réalisateurs de films à se concentrer sur ces facteurs pour obtenir l'appréciation du public.

# In[29]:

df_loved_movies = df
df_loved_movies.drop(df_loved_movies[df_loved_movies['vote_count'] < 1000].index, inplace=True)
df_loved_movies = df_loved_movies.sort_values(by='vote_average', ascending = False).head(1000)
df_loved_movies.drop(df_loved_movies[df_loved_movies['production_companies'] == 'nan'].index, inplace=True)

# In[30]:

print(df_loved_movies.shape)
df_loved_movies.head()

# #### Correlation between Popularity and Reviews

# In[72]:

# Defining our plotting function that will be used for all plots
def plotting (title, x, y, x_label, y_label):
    plt.scatter(x, y, alpha = 0.5)
    plt.xlabel(x_label)
    plt.ylabel(y_label)
    plt.title(title)
    plt.show()

plotting ('Correlation Between Average Vote and Popularity',df_loved_movies['vote_average'], df_loved_movies['popularity'],"Average Vote","Popularity")

# #### Insight
# Sur la base du Scratter plot ci-dessus, nous pouvons conclure qu'il n'y a pas de corrélation entre la popularité et les notes moyennes. Cela nous amène à conclure que le vote moyen du public n'affecte pas la popularité du film et que ce sont des variables indépendantes.
#
# Ceci étant dit, nous pouvons travailler sur les avis des utilisateurs et détecter les facteurs qui conduisent à un plus grand nombre d'avis des utilisateurs.

# #### Criteria 1: Production Companies

# In[32]:

df_loved_movies['production_companies'] = df_loved_movies['production_companies'].astype(str)
df_loved_movies_companies = df_loved_movies['production_companies'].str.cat(sep='|')
companies = pd.Series(df_loved_movies_companies.split('|'))
count_loved_movies_campanies = companies.value_counts(ascending = False).to_frame()
count_loved_movies_campanies.head(10)

# #### Insight
# Les films réalisés par Warner Bros. sont remarquablement mieux notés par le public que les autres films.

# #### Criteria 2: Cast

# In[33]:

df_loved_movies['cast'] = df_loved_movies['cast'].astype(str)
df_loved_movies_companies = df_loved_movies['cast'].str.cat(sep='|')
companies = pd.Series(df_loved_movies_companies.split('|'))
count_loved_movies_campanies = companies.value_counts(ascending = False).to_frame()
count_loved_movies_campanies.head(50)

# #### Insight
# Sur la base de cette analyse, nous pouvons conclure qu'il n'y a pas de grande différence dans le score en fonction des acteurs. Par conséquent, le casting n'est pas un facteur de popularité du film du point de vue du public.

# #### Criteria 3: Genres

# In[34]:

df_loved_movies['genres'] = df_loved_movies['genres'].astype(str)
df_loved_movies_companies = df_loved_movies['genres'].str.cat(sep='|')
companies = pd.Series(df_loved_movies_companies.split('|'))
count_loved_movies_campanies = companies.value_counts(ascending = False).to_frame()
count_loved_movies_campanies.head(10)

# #### Insight
# Sur la base de cet aperçu, nous pouvons conclure que les films d'action et d'aventure sont les plus appréciés par le public.

# ### Research Question 3: Un film populaire signifie-t-il directement un film commercialement réussi?

# In[35]:

df.corr()

# In[44]:

plotting ('Correlation Between Profits Rate and Popularity',df_loved_movies['profits'], df_loved_movies['popularity'],"Average Vote","Popularity")

#

# In[73]:

plotting ('Correlation Between Profits Rate and Popularity',df_loved_movies['profits_rate'], df_loved_movies['popularity'],"Profits","Popularity")

# In[74]:

plotting ('Correlation Between Profits and Average Vote',df_loved_movies['vote_average'], df_loved_movies['profits'],"Average Vote","Profits")

# In[75]:

plotting ('Correlation Between Profits Rate and Average Vote',df_loved_movies['vote_average'], df_loved_movies['profits_rate'],"Average Vote","Profits Rate")

# #### Insight
# Le succès commercial ( plus de bénéfices) s'est avéré indépendant de la moyenne des votes et de la popularité du film.

# ### Evolution du Bénéfice par Année

# In[40]:

df['prft'] = df['revenue'] - df['budget']
plt.figure(figsize=(10,8))
sns.lineplot(x = df['release_year'], y = df['prft'])
plt.title('Evolution du Bénéfice par Année')
plt.xlabel("Year")
plt.ylabel("Bénéfice")
plt.xlim(1955,2025)
plt.show()

# #### Insight
# L'évolution des bénéfices au fil des ans ne suit pas un schéma clair : elle est marquée par de nombreux hauts et bas, avec un pic significatif dans les années 70. Pour comprendre cette tendance, nous devons soit examiner en profondeur l'histoire de l'industrie cinématographique, soit étendre nos recherches et essayer de comprendre quels sont les facteurs qui ont contribué à ce pic à cette période.

#
# ## Conclusions
#
# Tout au long de notre travail, nous avons conservé deux définitions principales du **succès**: Le succès commercial (bénéfices) et la popularité ( avis élevés des utilisateurs). Sur la base de ces deux définitions, nous avons travaillé à la détection des facteurs contribuant au succès global d'un film.
#
# L'analyse a montré que :
#
# 1. **Succès commercial** : Les critères affectant le succès commercial d'un film sont principalement le genre du film et la société de production. Par exemple, les genres **Drame et Comédie** ont prouvé qu'ils généraient des taux de profit significativement plus élevés que les autres genres. De même, le bénéfice dépend de la société de production, puisque **Warner Bros., Universal Pictures et Paramount Pictures** sont les sociétés de production qui produisent les films les plus rentables.
# En plus, le 6ème mois s'est avéré être le mois le plus rentable pour la sortie des films, les producteurs doivent donc en tenir compte pour viser des bénéfices plus substantiels.
# En ce qui concerne le facteur "cast", il n'y a pas eu d'acteur ou de casting particulier pour contribuer à faire plus de profits. Pourtant, gagner plus d'argent dans un film dépend du salaire de chaque acteur. En d'autres termes, l'intégration d'une figure célèbre dans un film coûtera plus cher que celle d'une figure moins célèbre, et rapportera sûrement plus de recettes, mais en ce qui concerne les bénéfices, il est plus efficace de travailler avec une figure moins célèbre pour que le taux de profit soit plus élevé.
#
#
# 2. **Popularité** : La popularité et les avis des utilisateurs moyens se sont avérés être indépendants. Pour cela, nous reflétons la popularité par des films très bien notés par le public. Les facteurs qui affectent les avis des utilisateurs sont les genres des films et les sociétés de production. Sur la base de notre analyse, nous concluons que le fait de travailler avec **Warner Bros** contribuera de manière significative à l'obtention de meilleures critiques. De même, les genres **Action et Aventure** sont les plus appréciés par le public. Le casting n'a pas été présenté comme un facteur important contribuant à des critiques plus élevées.
# Sur cette base, nous pouvons conclure que pour faire un film plaisant pour le public, nous devons nous concentrer sur la société de production et les genres.
#
#
# 3. Comme nous voulions démontrer une **relation entre la popularité (les critiques des utilisateurs) et le succès commercial d'un film**, la corrélation était proche de zéro, ce qui prouve qu'un film très bien noté ne signifie pas nécessairement un succès commercial et vice versa.
#
#
# 4. **Evolution des bénéfice au fil des ans**:  L'évolution du bénéfice au cours de l'année ne présente pas un schéma clair : elle comporte de nombreux hauts et bas avec quelques pics. Nous pouvons étendre notre recherche pour étudier en profondeur les facteurs qui contribuent à ce modèle instable et déterminer comment optimiser les profits.
#
# En conclusion, dans l'industrie cinématographique, les réalisateurs s'efforcent de faire le plus de profits possible et d'attirer le public. Pour cela, afin de réaliser un film populaire et à succès sur le plan commercial, ils doivent se concentrer sur deux facteurs principaux : **les genres et les sociétés de production**.
#
# Les acteurs peuvent être un facteur important pour le film et peuvent générer plus de revenus, mais ils ne contribuent pas à générer plus de bénéfices (car pour générer plus de revenus, il faut plus de budget dans le cas des acteurs). Pour cela, le choix du cast dépend vraiment de la perspective du décideur.
#
# ### Limitations
# Bien que ces données aient contribué à la compréhension de nombreux facteurs et relations, il leur manque encore certaines données qui pourraient rendre notre analyse et notre interprétation plus précises, comme l'importante quantité de données manquantes trouvées dans cette base de données. Plus important encore, il n'y avait pas de données claires pouvant être utilisées pour recueillir des informations sur l'évolution instable des bénéfices au fil des ans. Ce type de données aurait beaucoup aidé à conclure les facteurs et critères importants sur lesquels se concentrer dans l'industrie cinématographique et contribuerait énormément à sa croissance.
#
# En outre, il serait plus efficace de construire un algorithme d'apprentissage automatique qui automatisera le processus de prédiction des films susceptibles d'avoir du succès commerciale et/ou popularité.

# In[ ]:


"""Embedding utils for queries."""
import heapq
import math
from typing import Any, Callable, List, Optional, Tuple

import numpy as np

from llama_index.embeddings.base import similarity as default_similarity_fn
from llama_index.vector_stores.types import VectorStoreQueryMode

def get_top_k_embeddings(
    query_embedding: List[float],
    embeddings: List[List[float]],
    similarity_fn: Optional[Callable[..., float]] = None,
    similarity_top_k: Optional[int] = None,
    embedding_ids: Optional[List] = None,
    similarity_cutoff: Optional[float] = None,
) -> Tuple[List[float], List]:
    """Get top nodes by similarity to the query."""
    if embedding_ids is None:
        embedding_ids = list(range(len(embeddings)))

    similarity_fn = similarity_fn or default_similarity_fn

    embeddings_np = np.array(embeddings)
    query_embedding_np = np.array(query_embedding)

    similarity_heap: List[Tuple[float, Any]] = []
    for i, emb in enumerate(embeddings_np):
        similarity = similarity_fn(query_embedding_np, emb)
        if similarity_cutoff is None or similarity > similarity_cutoff:
            heapq.heappush(similarity_heap, (similarity, embedding_ids[i]))
            if similarity_top_k and len(similarity_heap) > similarity_top_k:
                heapq.heappop(similarity_heap)
    result_tups = sorted(similarity_heap, key=lambda x: x[0], reverse=True)

    result_similarities = [s for s, _ in result_tups]
    result_ids = [n for _, n in result_tups]

    return result_similarities, result_ids

def get_top_k_embeddings_learner(
    query_embedding: List[float],
    embeddings: List[List[float]],
    similarity_top_k: Optional[int] = None,
    embedding_ids: Optional[List] = None,
    query_mode: VectorStoreQueryMode = VectorStoreQueryMode.SVM,
) -> Tuple[List[float], List]:
    """Get top embeddings by fitting a learner against query.

    Inspired by Karpathy's SVM demo:
    https://github.com/karpathy/randomfun/blob/master/knn_vs_svm.ipynb

    Can fit SVM, linear regression, and more.

    """
    try:
        from sklearn import linear_model, svm
    except ImportError:
        raise ImportError("Please install scikit-learn to use this feature.")

    if embedding_ids is None:
        embedding_ids = list(range(len(embeddings)))
    query_embedding_np = np.array(query_embedding)
    embeddings_np = np.array(embeddings)
    # create dataset
    dataset_len = len(embeddings) + 1
    dataset = np.concatenate([query_embedding_np[None, ...], embeddings_np])
    y = np.zeros(dataset_len)
    y[0] = 1

    if query_mode == VectorStoreQueryMode.SVM:
        # train our SVM
        # TODO: make params configurable
        clf = svm.LinearSVC(
            class_weight="balanced", verbose=False, max_iter=10000, tol=1e-6, C=0.1
        )
    elif query_mode == VectorStoreQueryMode.LINEAR_REGRESSION:
        clf = linear_model.LinearRegression()
    elif query_mode == VectorStoreQueryMode.LOGISTIC_REGRESSION:
        clf = linear_model.LogisticRegression(class_weight="balanced")
    else:
        raise ValueError(f"Unknown query mode: {query_mode}")

    clf.fit(dataset, y)  # train

    # infer on whatever data you wish, e.g. the original data
    similarities = clf.decision_function(dataset[1:])
    sorted_ix = np.argsort(-similarities)
    top_sorted_ix = sorted_ix[:similarity_top_k]

    result_similarities = similarities[top_sorted_ix]
    result_ids = [embedding_ids[ix] for ix in top_sorted_ix]

    return result_similarities, result_ids

def get_top_k_mmr_embeddings(
    query_embedding: List[float],
    embeddings: List[List[float]],
    similarity_fn: Optional[Callable[..., float]] = None,
    similarity_top_k: Optional[int] = None,
    embedding_ids: Optional[List] = None,
    similarity_cutoff: Optional[float] = None,
    mmr_threshold: Optional[float] = None,
) -> Tuple[List[float], List]:
    """Get top nodes by similarity to the query,
    discount by their similarity to previous results.

    A mmr_threshold of 0 will strongly avoid similarity to previous results.
    A mmr_threshold of 1 will check similarity the query and ignore previous results.

    """
    threshold = mmr_threshold or 0.5
    similarity_fn = similarity_fn or default_similarity_fn

    if embedding_ids is None or embedding_ids == []:
        embedding_ids = list(range(len(embeddings)))
    full_embed_map = dict(zip(embedding_ids, range(len(embedding_ids))))
    embed_map = full_embed_map.copy()
    embed_similarity = {}
    score: float = -math.inf
    high_score_id = None

    for i, emb in enumerate(embeddings):
        similarity = similarity_fn(query_embedding, emb)
        embed_similarity[embedding_ids[i]] = similarity
        if similarity * threshold > score:
            high_score_id = embedding_ids[i]
            score = similarity * threshold

    results: List[Tuple[Any, Any]] = []

    embedding_length = len(embeddings or [])
    similarity_top_k_count = similarity_top_k or embedding_length
    while len(results) < min(similarity_top_k_count, embedding_length):
        # Calculate the similarity score the for the leading one.
        results.append((score, high_score_id))

        # Reset so a new high scoring result can be found
        del embed_map[high_score_id]
        recent_embedding_id = high_score_id
        score = -math.inf

        # Iterate through results to find high score
        for embed_id in embed_map:
            overlap_with_recent = similarity_fn(
                embeddings[embed_map[embed_id]],
                embeddings[full_embed_map[recent_embedding_id]],
            )
            if (
                threshold * embed_similarity[embed_id]
                - ((1 - threshold) * overlap_with_recent)
                > score
            ):
                score = threshold * embed_similarity[embed_id] - (
                    (1 - threshold) * overlap_with_recent
                )
                high_score_id = embed_id

    result_similarities = [s for s, _ in results]
    result_ids = [n for _, n in results]

    return result_similarities, result_ids


import pandas as pd
from sklearn.model_selection import train_test_split
import librosa
import numpy as np
from scipy import signal
from tqdm import tqdm
import h5py
import sys
import argparse

parser = argparse.ArgumentParser(description='Preparing data.')

parser.add_argument('--run', help='Use this flag to run the prepare loop.', action="store_true")
parser.add_argument('--sample_rate', type=int, help='Desired sample rate.', default=220500)
parser.add_argument('--n_fft', type=int, help='Desired number of fft.', default=512)
parser.add_argument('--filename', help='Desired filename to write to.', default="prepared_signal.h5")

args, unknown = parser.parse_known_args()

sample_rate = args.sample_rate      # recordings are in 96 kHz, 24 bit depth, 1:10 TE (mic sr 960 kHz)
n_fft = args.n_fft                  # 23 ms * 22050 Hz ~ 512

# Smaller values improve the temporal resolution of the STFT at the expense of frequency resolution
# Shape: (1+nfft/2, n_frames = len/(n_fft/4))

# 1th order butterworth high-pass filter with cut-off frequency of 15,000 kHz
b, a = signal.butter(10, 15000 / 120000, 'highpass')

print(sample_rate)
def prepareData(y):
    filtered = signal.lfilter(b, a, y)                      # filter
    return filtered

def mergeClass(name):
    signals = []
    for filename in tqdm(classes[name]):
        y, _ = librosa.load("./data/" + filename, sr=sample_rate)
        sig = prepareData(y)
        signals.append(sig)

    if len(signals) >= 7:
        X_train, X_test, _, _ = train_test_split(signals, np.zeros(len(signals)), test_size=0.25, random_state=42)
        X_train, X_val, _, _ = train_test_split(X_train, np.zeros(len(X_train)), test_size=0.2, random_state=42)
        train = np.concatenate(X_train)
        test = np.concatenate(X_test)
        val = np.concatenate(X_val)
        train_set.create_dataset(name, data=train)
        test_set.create_dataset(name, data=test)
        val_set.create_dataset(name, data=val)

if args.run:
    df = pd.read_csv('./meta_n.csv', delimiter=';')
    hf = h5py.File(args.filename, 'a')  # will be ~13GB
    train_set = hf.require_group("train")
    test_set = hf.require_group("test")
    val_set = hf.require_group("val")

    classes = {}

    for index, row in df.iterrows():
        spec = row["MANUAL ID"]
        if spec not in classes:
            classes[spec] = []
        classes[spec].append(row["IN FILE"])

    print("sorted!")
    print(classes.keys())

    for classname in list(classes):
        if not classname in train_set or not classname in test_set or not classname in val_set:
            mergeClass(classname)
            print(classname + " prepared!")

    hf.close()


# -*- coding: utf-8 -*-
"""
Created on Thu Feb 28 10:16:47 2019

Script to convert text from an xml to geometries
and put the output into a postgres DB table

@author: crosic
"""

import pandas as pd
import xml.etree.cElementTree as et
import psycopg2
import configparser
import pandas.io.sql as psql

# Part 1
# read xml into pandas dataframe

def getvalueofnode(node):
    """ return node text or None """
    return node.text if node is not None else None

parsedXML = et.parse(r"C:\Users\crosic\Downloads\traffic-and-parking-by-law-schedules-data-march-2019\Ch_950_Sch_35_SpeedLimitOnPublicHighways.xml")
dfcols = ['ID', 'Chapter', 'Schedule', 'ScheduleName', 'Highway', 'Between', 'Speed_Limit_km_per_hr']
df_xml = pd.DataFrame(columns=dfcols)
for node in parsedXML.getroot():
    ID = node.find('ID')
    Chapter = node.find('Chapter')
    Schedule = node.find('Schedule')
    ScheduleName = node.find('ScheduleName')
    Highway = node.find('Highway')
    Between = node.find('Between')
    Speed_Limit_km_per_hr = node.find('Speed_Limit_km_per_hr')

    df_xml = df_xml.append(
            pd.Series([getvalueofnode(ID), getvalueofnode(Chapter), getvalueofnode(Schedule),
                       getvalueofnode(ScheduleName), getvalueofnode(Highway),
                       getvalueofnode(Between),  getvalueofnode(Speed_Limit_km_per_hr)],
                       index=dfcols),
            ignore_index=True)

# 6554 rows
print(df_xml.shape[0])

repealed = df_xml[df_xml.Between.isnull()]
# 1521
print(repealed.shape[0])

active_bylaws = df_xml[df_xml.Between.notnull()]
# 5033
print(active_bylaws.shape[0])

# Part 2
# match bylaws to geometry

CONFIG = configparser.ConfigParser()
CONFIG.read('db.cfg')
dbset = CONFIG['DBSETTINGS']
con = psycopg2.connect(**dbset)

from sqlalchemy import create_engine

engine = create_engine('postgresql://' + dbset['user'] + ':' + dbset['password'] + '@' + dbset['host'] + '/' + dbset['database'])

# get one row at a time
def text_to_centreline(highway, between):
   try:
      df = psql.read_sql("SELECT centreline_segments AS geom,  con AS confidence, notice, line_geom, oid1_geom, oid2_geom, street_name_arr, ratio FROM crosic.text_to_centreline('{}', '{}', {})".format(highway.replace("'", ""), between.replace("'", ""), 'NULL'), con)
   except Exception as e:
        print("------------------------")
        print("street: " + highway)
        print(" from: " + between)
        print("ERROR " + str(e))
        print("------------------------")
        raise Exception("Error in text_to_centreline: " + str(e))
        return  [highway, between, "Error", df['geom'].item(), df['notice'].item(), df['line_geom'].item(), df['oid1_geom'].item(), df['oid2_geom'].item(), df['street_name_arr'].item(), df['ratio'].item()]

   return [highway, between, df['confidence'].item(),  df['geom'].item(), df['notice'].item(), df['line_geom'].item(), df['oid1_geom'].item(), df['oid2_geom'].item(), df['street_name_arr'].item(), df['ratio'].item()]

def call_text_to_centreline_on_every_row(bylaws):
    rows = []
    total_rows = bylaws.shape[0]
    for index, row in bylaws.iterrows():

        # print messages to tell user where you are in process
        if index%100 == 0:
          print("Calling SQL function on index " + str(index))
        if index == total_rows-1:
          print("Processing last row with index value of " + str(index))

        # call text_to_centreline function
        try:
          row_with_wkt = text_to_centreline(row[4], row[5])
          # add ID and speed limit value to each row
          rows.append([row[0]] +  row_with_wkt + row[6])
        except Exception as e:
          rows.append([row[0], row[1], row[2], "No Match - ERROR", None, str(e), None, None, None, None, None, row[6]])
          continue

    return pd.DataFrame(data=rows, columns=['id','street_name', 'extents',  'confidence', 'geom', 'notice', 'line_geom', 'oid1_geom', 'oid2_geom', 'street_name_arr', 'ratio', 'speed_limit_km_per_hr'])  #gpd.GeoDataFrame(data=rows)

# convert list of rows to sql table
geom_data = call_text_to_centreline_on_every_row(active_bylaws)
geom_data.to_sql('posted_speed_limit_xml_open_data', engine)

con.close()


from datetime import datetime

class Guest(object):
    def __init__(
        self,
        first_name,
        last_name,
        phone_number,
        email_address,
        total_number_of_guests_attending,
        not_attending,
    ):
        self.first_name: str or None = first_name
        self.first_name_lower: str or None = first_name.lower()
        self.last_name: str or None = last_name
        self.last_name_lower: str or None = last_name.lower()
        self.phone_number: str or None = phone_number
        self.email_address: str or None = email_address
        self.total_number_of_guests_attending: int = total_number_of_guests_attending
        self.last_updated: datetime = datetime.now()
        self.not_attending: bool = not_attending


import os
from lib import Utils
from lib import Settings
from lib import SetupLogger

# ConfigurationFile class, contains parsed information of the configuration file
class ConfigFIle:

    def __init__(self, config_file_path):

        self.config_file_path = config_file_path
        self.alerts_dir = None
        self.documentation_dir = None
        self.upload_documentation = None
        self.documentation_plugin = None
        self.operations_dir = None
        self.upload_operations = None
        self.operations_plugin = None
        self.config_yml = None

        self.read_config_file()

    # Get alert dir variable from configuration file
    def read_config_file(self):

        # Read configuration file
        config_file = Utils.read_yml_file(self.config_file_path)

        # Check if the configuration file exist
        if config_file is None:
            print("[error] Configuration file not found: {}, exiting ...".format(self.config_file_path))
            exit(1)

        self.config_yml = config_file.get('config')

        # Validate if the value exist if don't use default
        if not self.config_yml.get('alerts_dir', False):
            self.alerts_dir = Settings.CONFIGURATION_FOLDERS["alerts"].get('folder')
            SetupLogger.logger.debug("Variable alerts_dir not defined, using default value: {}".format(self.alerts_dir))
        else:
            self.alerts_dir = self.config_yml.get('alerts_dir')
            SetupLogger.logger.debug("Variable alerts_dir defined, using value: {}".format(self.alerts_dir))
        # Validate if the dir is valid
        if not os.path.isdir(self.alerts_dir):
            SetupLogger.logger.fatal("Alerts directory is not valid, exiting ...")
            exit(1)

        # Validate if the value exist if don't use default
        if not self.config_yml.get('documentation_dir', False):
            self.documentation_dir = Settings.CONFIGURATION_FOLDERS["documentation"].get('folder')
            SetupLogger.logger.debug(
                "Variable documentation_dir not defined, using default value: {}".format(self.documentation_dir))
        else:
            self.documentation_dir = self.config_yml.get('documentation_dir')
            SetupLogger.logger.debug("documentation_dir defined, using value: {}".format(self.documentation_dir))
        # Validate if the dir is valid
        if not os.path.isdir(self.documentation_dir):
            print("[error] Documentation directory is not valid, exiting ...")
            exit(1)

        # Validate if the value exist if don't use default
        if not self.config_yml.get('operations_dir', False):
            self.operations_dir = Settings.CONFIGURATION_FOLDERS["operations"].get('folder')
            SetupLogger.logger.debug(
                "Variable operations_dir not defined, using default value: {}".format(self.operations_dir))
        else:
            self.operations_dir = self.config_yml.get('operations_dir')
            SetupLogger.logger.debug("operations_dir defined, using value: {}".format(self.operations_dir))
        # Validate if the dir is valid
        if not os.path.isdir(self.operations_dir):
            print("[error] Operations directory is not valid, exiting ...")
            exit(1)

        # Check if upload_documentation variable is available
        self.upload_documentation = self.config_yml.get('upload_documentation', False)

        # Check if documentation_plugin variable is available
        self.documentation_plugin = self.config_yml.get('documentation_plugin', Settings.DEFAULT_DOCUMENTATION_PLUGIN)

        # Check if upload_operations variable is available
        self.upload_operations = self.config_yml.get('upload_operations', False)

        # Check if operations_plugin variable is available
        self.operations_plugin = self.config_yml.get('operations_plugin', Settings.DEFAULT_OPERATIONS_PLUGIN)


# -*- coding: utf-8 -*-

import pickle
from sklearn.preprocessing import StandardScaler
import numpy as np

svm = pickle.load(open("..\\Algorithm Evaluation\\Models\\svm.sav", 'rb'))
rf = pickle.load(open("..\\Algorithm Evaluation\\Models\\random_forest.sav", 'rb'))
mlp = pickle.load(open("..\\Algorithm Evaluation\\Models\\mlp.sav", 'rb'))

new_register=[[50000, 40, 5000]]
new_register = np.asarray(new_register)
new_register = new_register.reshape(-1,1)

scaler = StandardScaler()
new_register = scaler.fit_transform(new_register)
new_register = new_register.reshape(-1,3)

response_svm = svm.predict(new_register)
response_random = rf.predict(new_register)
response_mlp = mlp.predict(new_register)

pay = 0
not_pay = 0

if response_svm == 1:
    pay += 1
else:
    not_pay += 1

if response_random == 1:
    pay += 1
else:
    not_pay += 1

if response_mlp == 1:
    pay += 1
else:
    not_pay += 1

# Estrategy: Plurality (Most Votes)
if pay >= not_pay:
    print("Client pays")
else:
    print("Client doesn't pay")


#!/usr/bin/env python
# -*- coding: utf-8 -*-
from __future__ import division
"""
Computing GC Content
"""
__author__ = 'caixia tian'
__date__= '2019/08/08'

import os
import argparse
import sys
import re

parser = argparse.ArgumentParser(description="input  parameters")
parser.add_argument('-i', '--fa',required=True, help=' a fasta file ')
parser.add_argument('-o', '--out',required=True, help='ouput file name')
args = parser.parse_args()

def gc_content( s ):
    seq = s.upper()
    total = len(seq)
    count = {}
    for base in seq:
        count.setdefault(base, 0) # 方法调用确保了键存在于 count 字典中(默认值是 0)
        count[base] = count[base] + 1
    gcnum = count['G'] + count['C']
    result = "%.2f%%" %(gcnum/total * 100)
    return result

with open(args.fa) as inp:
    stat={}
    for line in inp:
        if line.startswith('>'):
            seqid=line.replace('>','').split()[0]
            stat[seqid]=''
            #print(seqid)
            #break
        else:
            stat[seqid]+=line.replace('\n','').strip()

idlist=[]
gclist=[]
for name in stat:
    #print name,'\n',stat[name],'\n',gc_content(stat[name]),'\n'
    #break
    idlist.append(name)
    gclist.append(gc_content(stat[name]))

maxgc=max(gclist)
maxindex=gclist.index(maxgc)
output=idlist[maxindex] + '\n' + maxgc + '\n'
with open(args.out,"w") as oup:
    oup.write (output)

#########################################################
"""
Input fasta
>Rosalind_6404
CCTGCGGAAGATCGGCACTAGAATAGCCAGAACCGTTTCTCTGAGGCTTCCGGCCTTCCC
TCCCACTAATAATTCTGAGG
>Rosalind_5959
CCATCGGTAGCGCATCCTTAGTCCAATTAAGTCCCTATCCAGGCGCTCCGCCGAAGGTCT
ATATCCATTTGTCAGCAGACACGC
>Rosalind_0808
CCACCCTCGTGGTATGGCTAGGCATTCAGGAACCGGAGAACGCTTCAGACCAGCCCGGAC
TGGGAACCTGCGGGCAGTAGGTGGAAT

Output result
Rosalind_0808
60.92%
"""


import os
import time

from flask import Flask, request, render_template

from TravSHACL.core.GraphTraversal import GraphTraversal
from TravSHACL.core.ShapeSchema import ShapeSchema
from TravSHACL.sparql.SPARQLEndpoint import SPARQLEndpoint

app = Flask(__name__)
app.config['SCHEMA_PATH'] = os.environ.get('SCHEMA_PATH', '/path/to/your/shacl')
app.config['ENDPOINT'] = os.environ.get('ENDPOINT', 'https://example.org/sparql')

HEURISTICS = {
    'target': True,
    'degree': 'in',
    'properties': 'big'
}

@app.route('/validate', methods=['GET', 'POST'])
def validation():
    if request.method == 'GET':
        return render_template('validate.jinja2', schema_path=app.config['SCHEMA_PATH'], endpoint=app.config['ENDPOINT'])
    SPARQLEndpoint.instance = None  # Needs to be reset since it is a singleton and won't change otherwise
    schema_path = request.form.get('schemaDir', None)
    endpoint = request.form.get('external_endpoint', None)

    shape_schema = ShapeSchema(
        schema_dir=schema_path,
        schema_format='SHACL',
        endpoint=endpoint,
        graph_traversal=GraphTraversal.DFS,
        heuristics=HEURISTICS,
        use_selective_queries=True,
        max_split_size=256,
        output_dir=None,
        order_by_in_queries=False,
        save_outputs=False
    )

    start = time.time()
    result = shape_schema.validate()
    stop = time.time()
    result_html = travshacl_to_html_table(result, stop - start)
    return render_template("result.jinja2", html=result_html)

def travshacl_to_html_table(trav_result, time_used):
    parsed_result = []

    for shape, validation_dict in trav_result.items():
        for validation_result, instances in validation_dict.items():
            for instance in instances:
                finished_at = shape
                if finished_at[0] == '<':
                    finished_at = finished_at[1:]
                if finished_at[-1] == '>':
                    finished_at = finished_at[:-1]

                shape_ = instance[0]
                if shape_[0] == '<':
                    shape_ = shape_[1:]
                if shape_[-1] == '>':
                    shape_ = shape_[:-1]
                parsed_result.append({'shape': shape_, 'finished@shape': finished_at, 'validation result': validation_result.replace('_instances',''), 'instance': instance[1]})

    html = '
Trav-SHACL returned ' + str(len(parsed_result)) + ' validation results in ' + str(time_used) + ' seconds.

'

    order = []
    html += '
'
    for item in ['instance', 'shape', 'validation result', 'finished@shape']:
        order.append(item)
        html += '' + item + '	'
    html += '
'

    for res in parsed_result:
        html += '
'
        for item in order:
            if res[item]:
                if item == 'validation result':
                    if res[item] == 'valid':
                        html += '' + str(res[item]) + '	'
                    else:
                        html += '' + str(res[item]) + '	'
                else:
                    html += '' + str(res[item]) + '	'
        html += '
'

    html += '

'
    return html

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000, debug=True)


import base64
import datetime
import io
import plotly.graph_objs as go
import cufflinks as cf
import plotly.express as px

import dash
from dash.dependencies import Input, Output, State
# import dash_core_components as dcc
# import dash_html_components as html
# import dash_table

from dash import dcc
from dash import dash_table
from dash import html

import pandas as pd

external_stylesheets = ["https://codepen.io/chriddyp/pen/bWLwgP.css"]

app = dash.Dash(__name__, external_stylesheets=external_stylesheets)
server = app.server

colors = {"graphBackground": "#F5F5F5", "background": "#ffffff", "text": "#000000"}

app.layout = html.Div(
    [
        dcc.Upload(
            id="upload-data",
            children=html.Div(["Drag and Drop or ", html.A("Select Files")]),
            style={
                "width": "100%",
                "height": "60px",
                "lineHeight": "60px",
                "borderWidth": "1px",
                "borderStyle": "dashed",
                "borderRadius": "5px",
                "textAlign": "center",
                "margin": "10px",
            },
            # Allow multiple files to be uploaded
            multiple=True,
        ),
        dcc.Dropdown(id='x-axis', placeholder="X-Axis"),
        # html.Button('Add X-Axis Columns', id='add_x_axis_columns', n_clicks=0),
        dcc.Dropdown(id='y-axis', placeholder="Y-Axis"),
        # html.Button('Add Y-Axis Columns', id='add_y_axis_columns', n_clicks=0),
        dcc.Dropdown(id='color', placeholder="Color"),
        dcc.Graph(id="Mygraph"),
        html.Div(id="output-data-upload"),
    ]
)

# @app.callback(
#     Output('x-axis', 'options'),
#     columns)
# def update(n_clicks, existing_options):
#     print("Flag Updated")
#     return []

# @app.callback(
#     Output('x-axis', 'options'),
#     [Input('add_x_axis_columns', 'n_clicks')],
#     [State('x-axis', 'options')])
# def update_options(n_clicks, existing_options):
#     new_options = []
#     for choice in columns:
#         new_options.append({'label': choice, 'value': choice})
#     return new_options

# @app.callback(
#     Output('y-axis', 'options'),
#     [Input('add_y_axis_columns', 'n_clicks')],
#     [State('y-axis', 'options')])
# def update_options(n_clicks, existing_options):
#     new_options = []
#     for choice in columns:
#         new_options.append({'label': choice, 'value': choice})
#     return new_options

@app.callback(
    Output('y-axis', 'options'),
    [Input("upload-data", "contents"),
    Input("upload-data", "filename")],
    [State('y-axis', 'options')])
def update_y_columns(contents, filename, exsting_options):
    new_options = []
    if contents:
        contents = contents[0]
        filename = filename[0]
        df = parse_data(contents, filename)
        arr = df.columns
        for choice in arr:
            new_options.append({'label': choice, 'value': choice})
    return new_options

@app.callback(
    Output('color', 'options'),
    [Input("upload-data", "contents"),
    Input("upload-data", "filename")],
    [State('color', 'options')])
def update_y_columns(contents, filename, exsting_options):
    new_options = []
    if contents:
        contents = contents[0]
        filename = filename[0]
        df = parse_data(contents, filename)
        arr = df.columns
        for choice in arr:
            new_options.append({'label': choice, 'value': choice})
    return new_options

@app.callback(
    Output('x-axis', 'options'),
    [Input("upload-data", "contents"),
    Input("upload-data", "filename")],
    [State('x-axis', 'options')])
def update_x_columns(contents, filename, exsting_options):
    new_options = []
    if contents:
        contents = contents[0]
        filename = filename[0]
        df = parse_data(contents, filename)
        arr = df.columns
        for choice in arr:
            new_options.append({'label': choice, 'value': choice})
    return new_options

@app.callback(
    Output("Mygraph", "figure"),
    [Input("upload-data", "contents"),
    Input("upload-data", "filename"),
    dash.dependencies.Input('y-axis', 'value'),
    dash.dependencies.Input('x-axis', 'value'),
    dash.dependencies.Input('color', 'value')],
)
def update_graph(contents, filename, xaxis, yaxis, color):
    fig = {
        "layout": go.Layout(
            plot_bgcolor=colors["graphBackground"],
            paper_bgcolor=colors["graphBackground"],
        )
    }
    if contents:
        contents = contents[0]
        filename = filename[0]
        df = parse_data(contents, filename)

        # print("Called" + " X-axis: " + str(xaxis) + "Y-Axis: " + str(yaxis))
        if xaxis != None and yaxis != None:
            if color != None:
                fig = px.scatter(df, x=xaxis, y=yaxis, color=color)
            else:
                fig = px.scatter(df, x=xaxis, y=yaxis)
    return fig

    # if contents:
    #     contents = contents[0]
    #     filename = filename[0]
    #     df = parse_data(contents, filename)
    #     df = df.set_index(df.columns[0])
    #     fig["data"] = df.iplot(
    #         asFigure=True, kind="scatter", mode="lines+markers", size=1
    #     )

    # return fig

def parse_data(contents, filename):
    content_type, content_string = contents.split(",")

    decoded = base64.b64decode(content_string)
    try:
        if "csv" in filename:
            # Assume that the user uploaded a CSV or TXT file
            df = pd.read_csv(io.StringIO(decoded.decode("utf-8")))
        elif "xls" in filename:
            # Assume that the user uploaded an excel file
            df = pd.read_excel(io.BytesIO(decoded))
        elif "txt" or "tsv" in filename:
            # Assume that the user upl, delimiter = r'\s+'oaded an excel file
            df = pd.read_csv(io.StringIO(decoded.decode("utf-8")), delimiter=r"\s+")
    except Exception as e:
        print(e)
        return html.Div(["There was an error processing this file."])

    return df

@app.callback(
    Output("output-data-upload", "children"),
    [Input("upload-data", "contents"), Input("upload-data", "filename")],
)
def update_table(contents, filename):
    table = html.Div()

    if contents:
        contents = contents[0]
        filename = filename[0]
        df = parse_data(contents, filename)

        table = html.Div(
            [
                html.H5(filename),
                dash_table.DataTable(
                    data=df.to_dict("rows"),
                    columns=[{"name": i, "id": i} for i in df.columns],
                ),
                html.Hr(),
                html.Div("Raw Content"),
                html.Pre(
                    contents[0:200] + "...",
                    style={"whiteSpace": "pre-wrap", "wordBreak": "break-all"},
                ),
            ]
        )

    return table

if __name__ == "__main__":
    app.run_server(debug=True)


#coding:utf-8

"""
ID:          issue-5377
ISSUE:       5377
TITLE:       ISQL extract command looses COMPUTED BY field types
DESCRIPTION:
  Test creates database with empty table T1 that has computed by fileds with DDL appopriate to the ticket issues.
  Then we:
  1) extract metadata from database and store it to <init_meta.sql>;
  2) run query to the table T1 with setting sqlda_display = ON, and store output to <init_sqlda.log>;
  3) DROP table T1;
  4) try to apply script with extracted metadata (see step "1") -  it should pass without errors;
  5) AGAIN extract metadata and store it to <last_meta.sql>;
  6) AGAIN run query to T1 with set sqlda_display = on, and store output to <last_sqlda.log>;
  7) compare text files:
     <init_meta.sql> vs <last_meta.sql>
     <init_sqlda.log> vs <last_sqlda.log>
  8) Check that result of comparison is EMPTY (no rows).
JIRA:        CORE-5092
FBTEST:      bugs.core_5092
"""

import pytest
from difflib import unified_diff
from firebird.qa import *

init_script = """
    recreate table t1 (
         n0 int

        ,si smallint computed by( 32767 )
        ,bi bigint computed by ( 2147483647 )
        ,s2 smallint computed by ( mod(bi, nullif(si,0)) )

        ,dv double precision computed by (pi())
        ,fv float computed by (dv*dv)
        ,nv numeric(3,1) computed by (sqrt(fv))

        ,dt date computed by ('now')
        ,tm time computed by ('now')
        ,dx timestamp computed by ( dt )
        ,tx timestamp computed by ( tm )

        ,c1 char character set win1251 computed by ('ы')
        ,c2 char character set win1252 computed by ('å')
        ,cu char character set utf8 computed by ('∑')

        ,c1x char computed by(c1)
        ,c2x char computed by(c2)
        ,cux char computed by(cu)

        ,b1 blob character set win1251 computed by ('ы')
        ,b2 blob character set win1252 computed by ('ä')
        ,bu blob character set utf8 computed by ('∑')
        ,bb blob computed by ('∞')

        ,b1x blob computed by (b1)
        ,b2x blob computed by (b2)
        ,bux blob computed by (bu)
        ,bbx blob computed by (bb)
    );
    --insert into t1 values(null);
    commit;
"""

db = db_factory(charset='UTF8', init=init_script)

act = python_act('db')

@pytest.mark.version('>=3.0')
def test_1(act: Action):
    # Initial metadata
    act.isql(switches=['-x'])
    initial_metadata = act.stdout.splitlines()
    # SQLDA initial
    sqlda_check = 'set list on; set sqlda_display on; select * from t1; commit; drop table t1; exit;'
    act.reset()
    act.isql(switches=['-q', '-m'], input=sqlda_check)
    initial_sqlda = act.stdout.splitlines()
    # Apply extracted metadata
    act.reset()
    act.isql(switches=[], input='\n'.join(initial_metadata))
    # New metadata
    act.reset()
    act.isql(switches=['-x'])
    new_metadata = act.stdout.splitlines()
    # SQLDA new
    act.reset()
    act.isql(switches=['-q', '-m'], input=sqlda_check)
    new_sqlda = act.stdout.splitlines()
    # Check
    assert list(unified_diff(initial_sqlda, new_sqlda)) == []
    assert list(unified_diff(initial_metadata, new_metadata)) == []


from pathlib import Path
import time
from typing import List

import cv2
from loguru import logger
import numpy as np

from actions.action import Action
from actions.converter import Converter
from actions.indexer import GraspIndexer, ShiftIndexer, GraspShiftIndexer
from data.loader import Loader
from config import Config
from inference.planar import InferencePlanarPose
from picking.epoch import Epoch
from picking.image import clone, draw_around_box, draw_pose, get_area_of_interest, patch_image_at
from picking.param import SelectionMethod
from picking.planning_tree import PlanningTree
from orthographical import OrthographicImage

class Agent:
    def __init__(self, prediction_model):
        self.grasp_model = Config.grasp_model
        self.shift_model = Config.shift_model

        self.with_types = 'types' in self.grasp_model[1]

        self.output_layer = 'prob' if not self.with_types else ['prob', 'type']
        self.grasp_inference = InferencePlanarPose(
            Loader.get_model(self.grasp_model, output_layer=self.output_layer),
            box=Config.box,
            lower_random_pose=Config.lower_random_pose,
            upper_random_pose=Config.upper_random_pose,
            with_types=self.with_types,
            input_uncertainty=True,
        )
        self.grasp_inference.keep_indixes = [0, 1]
        self.grasp_indexer = GraspIndexer(gripper_classes=Config.gripper_classes)

        self.shift_inference = InferencePlanarPose(
            Loader.get_model(self.shift_model, output_layer='prob'),
            box=Config.box,
            lower_random_pose=Config.lower_random_pose,
            upper_random_pose=Config.upper_random_pose,
            with_types=False,
        )
        self.shift_inference.a_space = np.linspace(-3.0, 3.0, 26)  # [rad] # Don't use a=0.0
        self.shift_inference.size_original_cropped = (240, 240)
        self.shift_indexer = ShiftIndexer(shift_distance=Config.shift_distance)

        self.grasp_shift_indexer = GraspShiftIndexer(
            gripper_classes=Config.gripper_classes,
            shift_distance=Config.shift_distance,
        )

        self.converter = Converter(grasp_z_offset=Config.grasp_z_offset, shift_z_offset=0.007, box=Config.box)  # [m]

        self.prediction_model = prediction_model
        self.monte_carlo = 20

        self.actions_since_last_inference = 0
        self.actions: List[Action] = []

        self.output_path = Path.home() / 'Desktop'

        self.reinfer_next_time = True

        # First inference is slower
        self.prediction_model.predict([np.zeros((1, 64, 64, 1)), np.zeros((1, 1, 1, 1)), np.zeros((1, 1, 1)), np.zeros((1, 1, 1, 8))])

    def predict_images_after_action(
            self,
            images: List[OrthographicImage],
            action: Action,
            reward: float,
            action_type: int,
            uncertainty_images=None,
        ) -> List[OrthographicImage]:
        image = images[0]
        uncertainty_image = uncertainty_images[0]

        start = time.time()
        draw_around_box(image, box=Config.box)
        area = get_area_of_interest(image, action.pose, size_cropped=(256, 256), size_result=(64, 64))

        area_input = np.expand_dims(area.mat, axis=2).astype(np.float32) / np.iinfo(np.uint16).max * 2 - 1
        reward = np.expand_dims(np.expand_dims(np.expand_dims(reward, axis=1), axis=1), axis=1).astype(np.float32)
        action_type = np.expand_dims(np.expand_dims(action_type, axis=1), axis=1)

        use_monte_carlo = self.monte_carlo and self.monte_carlo > 1

        if not use_monte_carlo:
            area_result = self.prediction_model.predict([[area_input], [reward], [action_type], np.zeros((1, 1, 1, 8))])[0]
            area_result = np.array(np.iinfo(np.uint16).max * (area_result + 1) / 2, dtype=np.uint16)

        else:
            latent = np.random.normal(scale=0.05, size=(self.monte_carlo, 1, 1, 8))
            if self.monte_carlo > 3:
                latent[0, :, :, :] = 0.0

            predictions = self.prediction_model.predict([
                [area_input for _ in range(self.monte_carlo)],
                [reward for _ in range(self.monte_carlo)],
                [action_type for _ in range(self.monte_carlo)],
                latent,
            ])
            predictions = (predictions + 1) / 2

            predictions = np.array(predictions, dtype=np.float32)
            area_result = predictions[0]
            area_result = np.array(np.iinfo(np.uint16).max * area_result, dtype=np.uint16)

            predictions[predictions < 0.1] = np.nan
            area_uncertainty = np.nanvar(predictions, axis=0)
            area_uncertainty *= 7e3
            area_uncertainty[area_uncertainty > 1] = 1
            area_uncertainty = np.array(np.iinfo(np.uint16).max * area_uncertainty, dtype=np.uint16)

            uncertainty_image = patch_image_at(
                uncertainty_image,
                area_uncertainty,
                action.pose,
                size_cropped=(256, 256),
                operation='add',
            )

        result = patch_image_at(image, area_result, action.pose, size_cropped=(256, 256))

        logger.info(f'Predicted image [s]: {time.time() - start:0.3f}')

        if use_monte_carlo:
            return [result], [uncertainty_image]
        return [result]

    def plan_actions(
            self,
            images: List[OrthographicImage],
            method: SelectionMethod,
            depth=1,
            leaves=1,
            verbose=False,
        ) -> List[Action]:

        uncertainty_images = [OrthographicImage(
            np.zeros(i.mat.shape, dtype=np.uint16),
            i.pixel_size,
            i.min_depth,
            i.max_depth,
            i.camera,
            i.pose
        ) for i in images]

        tree = PlanningTree(images, uncertainty_images)

        for node, i in tree.fill_nodes(leaves=leaves, depth=depth):
            # Visited actions: node.actions

            for image in node.images:
                draw_around_box(image, box=Config.box)

            grasp = self.grasp_inference.infer(node.images, method, uncertainty_images=node.uncertainty_images)
            self.grasp_indexer.to_action(grasp)

            # Shift actions
            if Config.shift_objects and grasp.estimated_reward < Config.grasp_shift_threshold:
                shift = self.shift_inference.infer(node.images, method)
                self.shift_indexer.to_action(shift)

                bin_empty = shift.estimated_reward < Config.shift_empty_threshold

                if bin_empty:
                    action = Action('bin_empty', safe=1)
                else:
                    self.converter.calculate_pose(shift, node.images)
                    action = shift

            # Grasp actions
            else:
                estimated_reward_lower_than_threshold = grasp.estimated_reward < Config.bin_empty_at_max_probability
                bin_empty = estimated_reward_lower_than_threshold and Epoch.selection_method_should_be_high(method)
                new_image = False

                if bin_empty:
                    action = Action('bin_empty', safe=1)
                elif grasp.estimated_reward_std > 0.9:  # default=0.25
                    action = Action('new_image', safe=1)
                else:
                    self.converter.calculate_pose(grasp, node.images)
                    action = grasp
            logger.info(f'{i}: {action}')

            if verbose:
                image_copy = clone(images[0])
                uncertainty_image_copy = clone(uncertainty_images[0])

                draw_pose(image_copy, action.pose, convert_to_rgb=True)
                draw_pose(uncertainty_image_copy, action.pose, convert_to_rgb=True)

                cv2.imwrite(str(self.output_path / f'result-{i}.png'), image_copy.mat)
                cv2.imwrite(str(self.output_path / f'uncertainty-{i}.png'), uncertainty_image_copy.mat)

            if action.type == 'bin_empty' or action.type == 'new_image':
                break

            # Predict next image
            reward = action.estimated_reward > Config.bin_empty_at_max_probability if action.type == 'grasp' else action.estimated_reward
            action_type = self.grasp_shift_indexer.from_action(action)
            images = self.predict_images_after_action(
                node.images,
                action,
                reward=reward,
                action_type=action_type,
                uncertainty_images=node.uncertainty_images,
            )

            if isinstance(images, tuple):
                images, uncertainty_images = images
            else:
                uncertainty_images = None

            node.add_action(action, images, uncertainty_images)

        if verbose:
            cv2.imwrite(str(self.output_path / f'result-{i+1}.png'), node.images[0].mat)
            cv2.imwrite(str(self.output_path / f'uncertainty-{i+1}.png'), node.uncertainty_images[0].mat)

        actions, max_reward, mean_reward = tree.get_actions_maximize_reward(max_depth=depth)
        print(f'Max reward: {max_reward:0.3f}, Mean reward: {mean_reward:0.3f}, Length: {len(actions)}')

        # actions, max_steps, mean_steps = tree.get_actions_minimize_steps()
        return actions

    def predict_actions(
            self,
            images: List[OrthographicImage],
            method: SelectionMethod,
            N=1,
            verbose=True,
        ) -> List[Action]:

        actions: List[Action] = []

        uncertainty_images = [OrthographicImage(
            np.zeros(i.mat.shape, dtype=np.uint16),
            i.pixel_size,
            i.min_depth,
            i.max_depth,
            i.camera,
            i.pose
        ) for i in images]

        for i in range(N):
            for image in images:
                draw_around_box(image, box=Config.box)

            grasp = self.grasp_inference.infer(images, method, uncertainty_images=uncertainty_images)
            self.grasp_indexer.to_action(grasp)

            # Shift actions
            if Config.shift_objects and grasp.estimated_reward < Config.grasp_shift_threshold:
                shift = self.shift_inference.infer(images, method)
                self.shift_indexer.to_action(shift)

                bin_empty = shift.estimated_reward < Config.shift_empty_threshold

                if bin_empty:
                    actions.append(Action('bin_empty', safe=1))
                else:
                    self.converter.calculate_pose(shift, images)
                    actions.append(shift)

            # Grasp actions
            else:
                estimated_reward_lower_than_threshold = grasp.estimated_reward < Config.bin_empty_at_max_probability
                bin_empty = estimated_reward_lower_than_threshold and Epoch.selection_method_should_be_high(method)
                new_image = False

                if bin_empty:
                    actions.append(Action('bin_empty', safe=1))
                elif grasp.estimated_reward_std > 0.9:  # default=0.25
                    actions.append(Action('new_image', safe=1))
                else:
                    self.converter.calculate_pose(grasp, images)
                    actions.append(grasp)

            actions[-1].step = i
            action = actions[-1]
            logger.info(f'{i}: {action}')

            if verbose:
                image_copy = clone(images[0])
                uncertainty_image_copy = clone(uncertainty_images[0])

                draw_pose(image_copy, action.pose, convert_to_rgb=True)
                draw_pose(uncertainty_image_copy, action.pose, convert_to_rgb=True)

                cv2.imwrite(str(self.output_path / f'result-{i}.png'), image_copy.mat)
                cv2.imwrite(str(self.output_path / f'uncertainty-{i}.png'), uncertainty_image_copy.mat)

            if action.type == 'bin_empty' or action.type == 'new_image':
                break

            # Predict next image
            reward = action.estimated_reward > Config.bin_empty_at_max_probability if action.type == 'grasp' else action.estimated_reward
            action_type = self.grasp_shift_indexer.from_action(action)
            images = self.predict_images_after_action(
                images,
                action,
                reward=reward,
                action_type=action_type,
                uncertainty_images=uncertainty_images,
            )

            if isinstance(images, tuple):
                images, uncertainty_images = images
            else:
                uncertainty_images = None

        if verbose:
            cv2.imwrite(str(self.output_path / f'result-{i+1}.png'), images[0].mat)
            cv2.imwrite(str(self.output_path / f'uncertainty-{i+1}.png'), uncertainty_images[0].mat)

        return actions

    def infer(self, images: List[OrthographicImage], method: SelectionMethod, N=5, reinfer=False):
        if self.actions_since_last_inference == 0 or self.actions_since_last_inference >= N or self.reinfer_next_time or reinfer:
            logger.warning(f'Calculate {N} predictions.')

            # self.actions = self.predict_actions(images, method, N=(N+1))
            self.actions = self.plan_actions(images, method, depth=N, leaves=1)
            self.actions_since_last_inference = 0
            self.reinfer_next_time = False
        else:
            logger.warning(f'Saved action, last inference {self.actions_since_last_inference} actions ago.')
            if self.actions_since_last_inference == len(self.actions) - 2:
                self.reinfer_next_time = True

        self.actions_since_last_inference += 1
        return self.actions[self.actions_since_last_inference - 1]


from __future__ import annotations

from typing import Any, Dict, List, Type, TypeVar, Union, cast

import attr

from binarylane.models.network import Network
from binarylane.types import UNSET, Unset

T = TypeVar("T", bound="Networks")

@attr.s(auto_attribs=True)
class Networks:
    """
    Attributes:
        v4 (List[Network]): A list of the IPv4 networks for this server.
        v6 (List[Network]): A list of the IPv6 networks for this server.
        port_blocking (bool): Whether the default port blocking is enabled for this server.
        recent_ddos (bool): If this is true this server has been the target of a recent DDOS attack. An email will have
            been sent to your email address when the DDOS was detected (and if it has ended, when it ended) with more
            details.
        separate_private_network_interface (Union[Unset, None, bool]): Whether a separate private network interface is
            provided for the server's VPC traffic.
        source_and_destination_check (Union[Unset, None, bool]): If enabled, this server is only able to send and
            receive data packets directly addressed to an IP address assigned to this server.
        ipv6_reverse_nameservers (Union[Unset, None, List[str]]): Any configured IPv6 reverse nameservers for this
            server. Please see our documentation for how this interacts with IPv6 nameserver settings at the account level.
    """

    v4: List[Network]
    v6: List[Network]
    port_blocking: bool
    recent_ddos: bool
    separate_private_network_interface: Union[Unset, None, bool] = UNSET
    source_and_destination_check: Union[Unset, None, bool] = UNSET
    ipv6_reverse_nameservers: Union[Unset, None, List[str]] = UNSET
    additional_properties: Dict[str, Any] = attr.ib(init=False, factory=dict)

    def to_dict(self) -> Dict[str, Any]:
        v4 = []
        for v4_item_data in self.v4:
            v4_item = v4_item_data.to_dict()

            v4.append(v4_item)

        v6 = []
        for v6_item_data in self.v6:
            v6_item = v6_item_data.to_dict()

            v6.append(v6_item)

        port_blocking = self.port_blocking
        recent_ddos = self.recent_ddos
        separate_private_network_interface = self.separate_private_network_interface
        source_and_destination_check = self.source_and_destination_check
        ipv6_reverse_nameservers: Union[Unset, None, List[str]] = UNSET
        if not isinstance(self.ipv6_reverse_nameservers, Unset):
            if self.ipv6_reverse_nameservers is None:
                ipv6_reverse_nameservers = None
            else:
                ipv6_reverse_nameservers = self.ipv6_reverse_nameservers

        field_dict: Dict[str, Any] = {}
        field_dict.update(self.additional_properties)
        field_dict.update(
            {
                "v4": v4,
                "v6": v6,
                "port_blocking": port_blocking,
                "recent_ddos": recent_ddos,
            }
        )
        if separate_private_network_interface is not UNSET:
            field_dict["separate_private_network_interface"] = separate_private_network_interface
        if source_and_destination_check is not UNSET:
            field_dict["source_and_destination_check"] = source_and_destination_check
        if ipv6_reverse_nameservers is not UNSET:
            field_dict["ipv6_reverse_nameservers"] = ipv6_reverse_nameservers

        return field_dict

    @classmethod
    def from_dict(cls: Type[T], src_dict: Dict[str, Any]) -> T:
        d = src_dict.copy()
        v4 = []
        _v4 = d.pop("v4")
        for v4_item_data in _v4:
            v4_item = Network.from_dict(v4_item_data)

            v4.append(v4_item)

        v6 = []
        _v6 = d.pop("v6")
        for v6_item_data in _v6:
            v6_item = Network.from_dict(v6_item_data)

            v6.append(v6_item)

        port_blocking = d.pop("port_blocking")

        recent_ddos = d.pop("recent_ddos")

        separate_private_network_interface = d.pop("separate_private_network_interface", UNSET)

        source_and_destination_check = d.pop("source_and_destination_check", UNSET)

        ipv6_reverse_nameservers = cast(List[str], d.pop("ipv6_reverse_nameservers", UNSET))

        networks = cls(
            v4=v4,
            v6=v6,
            port_blocking=port_blocking,
            recent_ddos=recent_ddos,
            separate_private_network_interface=separate_private_network_interface,
            source_and_destination_check=source_and_destination_check,
            ipv6_reverse_nameservers=ipv6_reverse_nameservers,
        )

        networks.additional_properties = d
        return networks

    @property
    def additional_keys(self) -> List[str]:
        return list(self.additional_properties.keys())

    def __getitem__(self, key: str) -> Any:
        return self.additional_properties[key]

    def __setitem__(self, key: str, value: Any) -> None:
        self.additional_properties[key] = value

    def __delitem__(self, key: str) -> None:
        del self.additional_properties[key]

    def __contains__(self, key: str) -> bool:
        return key in self.additional_properties


# -- GEneral Packages --
from PyQt5.QtCore import Qt, pyqtSlot
from pyqtgraph.dockarea import *
import pyqtgraph as pg
from PyQt5.QtWidgets import *
from PyQt5 import QtGui, QtCore
from functools import partial
# -- My Packages --
from .action import Action
from tabs.experiment_tab.experiment import Experiment

class EmgDock(Experiment):
    def __init__(self, area, gv):
        super().__init__()
        self.area = area
        self.gv = gv

        self.actions = []
        self.action_name = 'ACTION'
        self.num_of_action = 100
        self.action_itt = 0

        self.end_experiment = False

        self.create_emg_dock()

        self.init_spawn_timer()
        self.init_plot_timer()

    def create_emg_dock(self):
        self.emg_dock = Dock('EMG experiment')
        self.area.addDock(self.emg_dock)

        self.layout = pg.LayoutWidget()
        self.emg_dock.addWidget(self.layout)

        self.plot = self.create_plot()
        self.layout.addWidget(self.plot, 1, 0, 1, 2)
        # Start and stop button
        self.create_start_b('EMG')
        self.create_stop_b('EMG')

    def create_plot(self, xs=(0, 20), ys=(0.7, 6.5)):
        plot = super().create_plot(xs, ys)
        # Horizontal delineation lines to activate an event
        hLine = pg.InfiniteLine(angle=0, pos=1.5, movable=False)
        plot.addItem(hLine, ignoreBounds=True)
        return plot

    def init_spawn_timer(self):
        self.spawn_timer = QtCore.QTimer()
        self.spawn_timer.timeout.connect(self.update_spawn)

    def init_plot_timer(self):
        self.plot_timer = QtCore.QTimer()
        self.plot_timer.timeout.connect(self.update_plot)

    def update_spawn(self):
        # Stop spawning value when we reach the number of experiment events
        if self.action_itt < self.num_of_action:
            # Create a new action:
            action = Action(
                    actn_txt=self.action_name, wait_txt='WAIT...', y_pos=6.5,
                    x_pos=10)
            # Plot this new action
            self.plot.addItem(action.plot)
            # Add it to the list of actions
            self.actions.append(action)
            self.action_itt += 1
        else:
            self.end_experiment = True

    def update_plot(self):
        for action in self.actions:
            # update the listed position of the action
            action.y_pos -= 0.04
            # If the action text event is bellow the horiz. activation line
            if 0 <= action.y_pos <= 1.5 and action.wait:
                self.gv.experiment_type = 1  # 1 meaning that an event happen
                                             # (binary scenario)
                action.activate_html()
                action.wait = False
            # update the position of the action
            action.plot.setPos(action.x_pos, action.y_pos)
            # If the action leave the screen remove it
            if action.y_pos < 0:
                self.plot.removeItem(self.actions[0].plot)
                self.actions.pop(0)
        # Stop spawning value when we reach the number of experiment events
        if self.actions == [] and self.end_experiment:
            print('End of EMG Experiment reached')
            self.show_end_txt()
            self.stop()

    def show_end_txt(self):
        self.end_txt = pg.TextItem(anchor=(0, 0), fill=(0, 0, 0, 0))
        self.end_txt_html = f"""

{'End of experiment...'}

"""
        self.end_txt.setHtml(self.end_txt_html)
        self.end_txt.setPos(7, 5)
        self.plot.addItem(self.end_txt)

    @pyqtSlot()
    def start(self):
        self.plot_timer.start(30)
        self.spawn_timer.start(1200)

    @pyqtSlot()
    def stop(self):
        self.plot_timer.stop()
        self.spawn_timer.stop()


# -*- coding: utf-8 -*-
import os
import subprocess

from django.http import HttpResponse

def recommend_achievements(request):
    params = dict(request.GET.items())
    user_id = params.get('user_id')
    achievements = params.get('achievements')
    dir_path = os.path.dirname(__file__)

    r = subprocess.Popen(
        "python2 %s %s %s" % (os.path.join(dir_path, 'recomendator.py'), user_id, achievements or '[]'),
        shell=True, stdout=subprocess.PIPE).stdout.read()

    return HttpResponse(r, content_type='application/json; charset=utf-8')


import bs4 as bs
import requests
import pickle

def GetHuStock():
	res = requests.get('https://www.banban.cn/gupiao/list_sh.html')
	# 防止中文乱码
	res.encoding = res.apparent_encoding
	# 使用bsoup的lxml样式
	soup = bs.BeautifulSoup(res.text, 'lxml')
	 # 从html内容中找到类名为'u-postcontent cz'的div标签
	content = soup.find('div', {'class': 'u-postcontent cz'})
	result = []
	for item in content.findAll('a'):
		result.append(item.text)
	with open('huStock.pickle', 'wb') as f:
		pickle.dump(result, f)


from django.conf.urls import include, url

from cmj.ouvidoria.views import DenunciaAnonimaFormView, \
    SolicitacaoDetailView, SolicitacaoManageListView, SolicitacaoListView, \
    SolicitacaoFormView, SolicitacaoInteractionView,\
    SolicitacaoMensagemRedirect, OuvidoriaPaginaInicialView,\
    SolicitacaoMensagemAnexoView

from .apps import AppConfig

app_name = AppConfig.name

urlpatterns_ouvidoria = [

    url(r'^fale-conosco/ouvidoria/denuncia-anonima/?$',
        DenunciaAnonimaFormView.as_view(), name='denuncia_form'),

    url(r'^fale-conosco/ouvidoria/solicitacao/(?P<pk>[0-9]+)/?$',
        SolicitacaoDetailView.as_view(), name='solicitacao_detail'),

    url(r'^fale-conosco/ouvidoria/solicitacao/minhas/?$',
        SolicitacaoListView.as_view(), name='solicitacao_list'),

    url(r'^fale-conosco/ouvidoria/solicitacao/manage/?$',
        SolicitacaoManageListView.as_view(), name='solicitacao_manage_list'),

    url(r'^fale-conosco/ouvidoria/solicitacao/registrar/?$',
        SolicitacaoFormView.as_view(), name='solicitacao_create'),

    url(r'^fale-conosco/ouvidoria/solicitacao/(?P<pk>[0-9]+)/interact$',
        SolicitacaoInteractionView.as_view(), name='solicitacao_interact'),

    url(r'^fale-conosco/ouvidoria/solicitacao/P(?P<hash>[0-9A-Fa-f]+)/(?P<pk>\d+)$',
        SolicitacaoInteractionView.as_view(), name='solicitacao_interact_hash'),

    url(r'^fale-conosco/ouvidoria/solicitacao/(?P<pk>[0-9]+)/mensagem',
        SolicitacaoMensagemRedirect.as_view(), name='mensagemsolicitacao_detail'),

    url(r'^fale-conosco/ouvidoria/mensagem/(?P<pk>[0-9]+)/anexo',
        SolicitacaoMensagemAnexoView.as_view(), name='anexo_mensagem_view'),

    url(r'^fale-conosco/ouvidoria/mensagem/P(?P<hash>[0-9A-Fa-f]+)/(?P<pk>\d+)/anexo',
        SolicitacaoMensagemAnexoView.as_view(), name='anexo_mensagem_view_hash'),

    url(r'^fale-conosco/ouvidoria',
        OuvidoriaPaginaInicialView.as_view(), name='ouvidoria_pagina_inicial'),

]

urlpatterns = [
    url(r'', include(urlpatterns_ouvidoria)),

]


from config import Config
from langchain.embeddings import OpenAIEmbeddings
from langchain.vectorstores import Qdrant
from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams

# See https://python.langchain.com/docs/integrations/vectorstores/qdrant
# See https://qdrant.tech/documentation/quick-start/
# See https://api.python.langchain.com/en/latest/vectorstores/langchain.vectorstores.qdrant.Qdrant.html
class VectorDBAccessor:
    # Use Qdrant as a vector database.
    # See https://python.langchain.com/docs/integrations/vectorstores/qdrant
    def __init__(self, path: str, collection_name: str, config: Config):
        self._path = path
        self._collection_name = collection_name
        self._config = config
        self.client = self.load_qdrant_client()

    def load_qdrant_client(self) -> Qdrant:
        client = QdrantClient(path=self._path, collection_name=self._collection_name)
        # Check if the collection exists
        collections = client.get_collections().collections
        collection_names = [collection.name for collection in collections]
        # Create a collection if it does not exist
        if self._collection_name not in collection_names:
            client.create_collection(
                collection_name=self._collection_name,
                vectors_config=VectorParams(size=1536, distance=Distance.COSINE),
            )
            print("Created a collection in sqlite.")

        return Qdrant(
            client=client,
            collection_name=self._collection_name,
            embeddings=OpenAIEmbeddings(openai_api_key=self._config.openai_api_key),
        )


from django.shortcuts import render
from django.http import Http404
from django.core.urlresolvers import reverse
from PersonalWebsite.settings import STATIC_DIR

import json

def gallery(request, page):
  images = []

  if page=="food":
    with open(STATIC_DIR+"/json/food.json") as f:
      images = json.load(f)

  elif page=="nature":
    with open(STATIC_DIR+"/json/nature.json") as f:
      images = json.load(f)

  else:
    raise Http404

  return render(request, "infoPage.html", {
    "template":"gallery.html",
    "content":{"images":images,
      "more": {
        "url":"http://cfalk.imgur.com/all/",
        "name":"Imgur"
      }

    }
  })


import openai
import json
import sys
import tempfile
import os
import threading
import pyttsx3
from nltk import sent_tokenize

# Put your URI endpoint:port here for your local inference server (in LM Studio)
openai.api_base = 'http://localhost:1234/v1'
# Put in an empty API Key
openai.api_key = ''

# Read context from the text file
with open('context.txt', 'r') as file:
    context = file.read()

# Adjust the following based on the model type
# Alpaca style prompt format:
prefix = "### Instruct:\n"
suffix = "\n### Response:"

# 'Llama2 Chat' prompt format:
# prefix = "[INST]"
# suffix = "[/INST]"

# Initialize sentences as an empty string
sentences = ""

# This is a simple wrapper function to allow you to simplify your prompts
def get_completion(prompt, model="local model", temperature=0.3):
    global sentences  # Declare sentences as a global variable
    formatted_prompt = f"{prefix}{prompt}{suffix}"
    messages = [{"role": "system", "content": context}, {"role": "user", "content": formatted_prompt}]

    # Use OpenAI's ChatCompletion to get a streaming response
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        stop="###",
        temperature=temperature,
        max_tokens=1024,  # Adjust this based on your requirements
        stream=True
    )

    # Process the streaming response

    for chunk in response:
        try:
            content = json.loads(str(chunk))
            content_string = content['choices'][0]['delta']['content']
            sys.stdout.write(content_string)
            sys.stdout.flush()

            # Append to sentences
            sentences += content_string

            # Check for sentence completion
            if sentences.endswith(".") or sentences.endswith("?") or sentences.endswith("!") or sentences.endswith(",") or sentences.endswith(";") or sentences.endswith(":") or sentences.endswith("--") or sentences.endswith("...") or sentences.endswith(")") or sentences.endswith("}") or sentences.endswith("]"):
                start_tts_thread(sentences)
                sentences = ""  # Reset sentences
        except (KeyError, json.JSONDecodeError):
            pass

# Create a lock to ensure only one TTS thread runs at a time
tts_lock = threading.Lock()

def text_to_speech(text):
    engine = pyttsx3.init()
    engine.say(text)
    engine.runAndWait()

def start_tts_thread(text):
    with tts_lock:
        tts_thread = threading.Thread(target=text_to_speech, args=(text,))
        tts_thread.start()
        tts_thread.join()  # Wait for the TTS thread to finish before starting a new one

while True:
    user_input = input("\n\nInput: ")
    if user_input.lower() == 'exit':
        break
    get_completion(user_input, temperature=0.3)


#scan function usable such that on running it, the barcode scanner will open, and
#on detecting the first barcode, it will print the data of the relevant person from
#the csv file.

import cv2
from pyzbar import pyzbar

def read_barcodes_for_each_frame(frame):
    barcodes_in_frame = pyzbar.decode(frame) #returns empty list if no barcodes found
    s = 0
    for barcode in barcodes_in_frame: #wont run if no barcodes found
        barcode_info = barcode.data.decode('utf-8')
        get_list_from_csv(barcode_info)
        s+=1
    return s,frame

def get_list_from_csv(adm):
    adm = adm[4:]
    import csv
    rows = []
    with open('class 12 project\dataset1.csv','r') as f:
        fc = csv.reader(f)
        for row in fc:
            rows.append(row)
            if row[0]==adm:
                #return row
                print(row)
        #return []
        print('No record found')

def scan():
    s = 0
    camera = cv2.VideoCapture(0)  #0 for iriun, 1 for laptop camera
    ret, frame = camera.read()    #2
    while ret and s==0:
        ret, frame = camera.read()
        s, frame_with_barcode = read_barcodes_for_each_frame(frame)
        cv2.imshow('Scan the ID card here', frame_with_barcode)
        if cv2.waitKey(1)==27:  #27 is the cv2 order of escape key
            break
    camera.release()
    cv2.destroyAllWindows()

scan()


from django.urls import path,include,re_path
from . import views
from django.conf import settings
from django.conf.urls.static import static
# from rest_framework.authtoken.views import ObtainAuthToken
from rest_framework import routers

router = routers.SimpleRouter()
router.register('patients', views.Patients,basename='patients')
router.register('phmedicines',views.Phmedicines,basename='phmedicines')
router.register('patmedicines',views.patmedicines,basename='patmedicines')
router.register('patdiets',views.patdiets,basename='patdiets')
router.register('drdiets',views.drdiets,basename='drdiets')
router.register('drmedicines',views.drmedicines,basename='drmedicines')

urlpatterns = [
    path('', include(router.urls)),
    path('register/',views.RegisterAPI.as_view(),name='register'),
    path('verify/<auth_token>',views.verification.as_view(),name='verify'),
    path('login/',views.LoginAPI.as_view(),name='login'),
    path('logout/',views.logoutt,name='logout'),
    path('oauth/', include('allauth.urls')),

    # re_path('api/register-by-access-token/' + r'social/(?P<backend>[^/]+)/$', views.register_by_access_token),
    # path('api/authentication-test/', views.authentication_test),
    # path('addviewpatient/',views.PatientsLC.as_view(),name='addviewpatients')
]
# if settings.DEBUG:
# urlpatterns += static(settings.MEDIA_URL,document_root=settings.MEDIA_ROOT)


from fastapi import APIRouter, Depends, HTTPException
from database import get_db
from models.cursos_api import CursoApi, CursoSinId, CursoList, CursoProfesor
from repos.cursos_repo import CursosRepositorio
from repos.profesores_repo import ProfesoresRepositorio

# Router:
cursos_api = APIRouter(prefix='/cursos', tags=['Cursos'])

# Repos:
cursos_repo = CursosRepositorio()
profesor_repo = ProfesoresRepositorio()

# Endpoint para traer lista de CursosSinId:

@cursos_api.get('', response_model=list[CursoList])
# Inyección de dependencias: La variable db existe en el ambito de este método y luego de cierra.
def get_all(db=Depends(get_db)):
    return cursos_repo.get_all(db)

# Enpoint para traer por Id:

@cursos_api.get('/{id}', response_model=CursoApi)
def get_by_id(id: int, db=Depends(get_db)):
    result = cursos_repo.get_by_id(id, db)
    # Si el result es None levanta una excepción con código de error.
    if result is None:
        raise HTTPException(status_code=404, detail='Curso no encontrado')
    return result

# Enpoint para traer por cursos de un profesor:
# TODO: VER RESPONSE MODEL @cursos_api.get('/profesor/{id_profesor}', response_model=CursoApi)

@cursos_api.get('/profesor/{id_profesor}')
def get_cursos_profesor(id_profesor: int, db=Depends(get_db)):
    if profesor_repo.get_by_id(id_profesor, db) is None:
        raise HTTPException(status_code=404, detail='El profesor no existe')
    else:
        result = cursos_repo.get_cursos_profesor(id_profesor, db)
        if not result:
            raise HTTPException(
                status_code=404, detail='El profesor no tiene cursos asignados')
    return result

# Endpoint para Crear Cursos. Viene sin Id y lo genera la BD, devuelve con Id:

@cursos_api.post('', response_model=CursoApi, status_code=201)
def post(datos: CursoSinId, db=Depends(get_db)):
    # Verifica que hay profesor titular y no se el mismo que el auxiliar
    if datos.profesor_titular_id == None:
        raise HTTPException(
            status_code=400, detail='El profesor titular no puede estar vacío')
    else:
        # Verifica que los profesores no sean el mismo
        if datos.profesor_titular_id == datos.profesor_auxiliar_id:
            raise HTTPException(
                status_code=400, detail='Los profesores titular y auxiliar no pueden ser los mismos')

    # Verifica que la fecha de inicio no sea la misma que la fecha de fin
    if datos.fecha_inicio == datos.fecha_fin:
        raise HTTPException(
            status_code=400, detail='La fecha de inicio no puede ser igual a la fecha de fin')
     # Verifica que la fecha de fin no sea anterior a la fecha de inicio
    if datos.fecha_fin < datos.fecha_inicio:
        raise HTTPException(
            status_code=400, detail='La fecha de fin no puede ser anterior a la fecha de inicio')

     # Si no se ha seleccionado un profesor auxiliar, establecerlo como None
    if datos.profesor_auxiliar_id == "":
        datos.profesor_auxiliar_id = None

    # Si valida, crea:
    result = cursos_repo.create(db, datos)
    return result

# Endpoint para Modificar Cursos. Viene sin Id y modifica la entidad, no su Id:

@cursos_api.put('/{id}', response_model=CursoApi)
def put(id: int, datos: CursoSinId, db=Depends(get_db)):

    # Verifica que hay profesor titular y no se el mismo que el auxiliar
    if datos.profesor_titular_id == "":
        raise HTTPException(
            status_code=400, detail='El profesor titular no puede estar vacío')
    else:
        # Verifica que los profesores no sean el mismo
        if datos.profesor_titular_id == datos.profesor_auxiliar_id:
            raise HTTPException(
                status_code=400, detail='Los profesores titular y auxiliar no pueden ser los mismos')

    # Verifica que la fecha de inicio no sea la misma que la fecha de fin
    if datos.fecha_inicio == datos.fecha_fin:
        raise HTTPException(
            status_code=400, detail='La fecha de inicio no puede ser igual a la fecha de fin')
     # Verifica que la fecha de fin no sea anterior a la fecha de inicio
    if datos.fecha_fin < datos.fecha_inicio:
        raise HTTPException(
            status_code=400, detail='La fecha de fin no puede ser anterior a la fecha de inicio')
    result = cursos_repo.modify(id, datos, db)

    # Si no se ha seleccionado un profesor auxiliar, establecerlo como None
    if datos.profesor_auxiliar_id == "":
        datos.profesor_auxiliar_id = None

    if result is None:
        raise HTTPException(
            status_code=404, detail='Curso no encontrado, no se puede modificar')
    return result

# Endpoint para Borrar Cursos:

@cursos_api.delete('/{id}', status_code=204)
def delete(id: int, db=Depends(get_db)):
    result = cursos_repo.delete(id, db)
    if result is None:
        raise HTTPException(
            status_code=404, detail='Curso no encontrado')
    return


'''
train mdn-rnn from pre-processed data.
also save 1000 initial mu and logvar, for generative experiments (not related to training).
'''

import numpy as np
import os
import json
import tensorflow as tf
import random
import time

from doomrnn import reset_graph, HyperParams, Model

os.environ["CUDA_VISIBLE_DEVICES"]="0"
np.set_printoptions(precision=4, edgeitems=6, linewidth=100, suppress=True)

model_save_path = "tf_rnn"
model_rnn_size = 512
model_num_mixture = 5
model_restart_factor = 10.

DATA_DIR = "series"

if not os.path.exists(model_save_path):
  os.makedirs(model_save_path)

initial_z_save_path = "tf_initial_z"
if not os.path.exists(initial_z_save_path):
  os.makedirs(initial_z_save_path)

def default_hps():
  return HyperParams(max_seq_len=500, # train on sequences of 500 (found it worked better than 1000)
                     seq_width=64,    # width of our data (64)
                     rnn_size=model_rnn_size,    # number of rnn cells
                     batch_size=100,   # minibatch sizes
                     grad_clip=1.0,
                     num_mixture=model_num_mixture,   # number of mixtures in MDN
                     restart_factor=model_restart_factor, # factor of importance for restart=1 rare case for loss.
                     learning_rate=0.001,
                     decay_rate=0.99999,
                     min_learning_rate=0.00001,
                     use_layer_norm=0, # set this to 1 to get more stable results (less chance of NaN), but slower
                     use_recurrent_dropout=0,
                     recurrent_dropout_prob=0.90,
                     use_input_dropout=0,
                     input_dropout_prob=0.90,
                     use_output_dropout=0,
                     output_dropout_prob=0.90,
                     is_training=1)

hps_model = default_hps()
hps_sample = hps_model._replace(batch_size=1, max_seq_len=2, use_recurrent_dropout=0, is_training=0)

# load preprocessed data
raw_data = np.load(os.path.join(DATA_DIR, "series.npz"))
raw_data_mu = raw_data["mu"]
raw_data_logvar = raw_data["logvar"]
raw_data_action =  raw_data["action"]

def load_series_data():
  all_data = []
  for i in range(len(raw_data_mu)):
    action = raw_data_action[i]
    mu = raw_data_mu[i]
    logvar = raw_data_logvar[i]
    all_data.append([mu, logvar, action])
  return all_data

def get_frame_count(all_data):
  frame_count = []
  for data in all_data:
    frame_count.append(len(data[0]))
  return np.sum(frame_count)

def create_batches(all_data, batch_size=100, seq_length=500):
  num_frames = get_frame_count(all_data)
  num_batches = int(num_frames/(batch_size*seq_length))
  num_frames_adjusted = num_batches*batch_size*seq_length
  random.shuffle(all_data)
  num_frames = get_frame_count(all_data)
  data_mu = np.zeros((num_frames, N_z), dtype=np.float16)
  data_logvar = np.zeros((num_frames, N_z), dtype=np.float16)
  data_action = np.zeros(num_frames, dtype=np.float16)
  data_restart = np.zeros(num_frames, dtype=np.uint8)
  idx = 0
  for data in all_data:
    mu, logvar, action=data
    N = len(action)
    data_mu[idx:idx+N] = mu.reshape(N, 64)
    data_logvar[idx:idx+N] = logvar.reshape(N, 64)
    data_action[idx:idx+N] = action.reshape(N)
    data_restart[idx]=1
    idx += N

  data_mu = data_mu[0:num_frames_adjusted]
  data_logvar = data_logvar[0:num_frames_adjusted]
  data_action = data_action[0:num_frames_adjusted]
  data_restart = data_restart[0:num_frames_adjusted]

  data_mu = np.split(data_mu.reshape(batch_size, -1, 64), num_batches, 1)
  data_logvar = np.split(data_logvar.reshape(batch_size, -1, 64), num_batches, 1)
  data_action = np.split(data_action.reshape(batch_size, -1), num_batches, 1)
  data_restart = np.split(data_restart.reshape(batch_size, -1), num_batches, 1)

  return data_mu, data_logvar, data_action, data_restart

def get_batch(batch_idx, data_mu, data_logvar, data_action, data_restart):
  batch_mu = data_mu[batch_idx]
  batch_logvar = data_logvar[batch_idx]
  batch_action = data_action[batch_idx]
  batch_restart = data_restart[batch_idx]
  batch_s = batch_logvar.shape
  batch_z = batch_mu + np.exp(batch_logvar/2.0) * np.random.randn(*batch_s)
  return batch_z, batch_action, batch_restart

# process data
all_data = load_series_data()

max_seq_len = hps_model.max_seq_len
N_z = hps_model.seq_width

# save 1000 initial mu and logvars:
initial_mu = []
initial_logvar = []
for i in range(1000):
  mu = np.copy(raw_data_mu[i][0, :]*10000).astype(np.int).tolist()
  logvar = np.copy(raw_data_logvar[i][0, :]*10000).astype(np.int).tolist()
  initial_mu.append(mu)
  initial_logvar.append(logvar)
with open(os.path.join("tf_initial_z", "initial_z.json"), 'wt') as outfile:
  json.dump([initial_mu, initial_logvar], outfile, sort_keys=True, indent=0, separators=(',', ': '))

reset_graph()
model = Model(hps_model)

hps = hps_model
start = time.time()

for epoch in range(1, 401):
  print('preparing data for epoch', epoch)
  data_mu, data_logvar, data_action, data_restart = 0, 0, 0, 0
  data_mu, data_logvar, data_action, data_restart = create_batches(all_data)
  num_batches = len(data_mu)
  print('number of batches', num_batches)
  end = time.time()
  time_taken = end-start
  print('time taken to create batches', time_taken)

  batch_state = model.sess.run(model.initial_state)

  for local_step in range(num_batches):

    batch_z, batch_action, batch_restart = get_batch(local_step, data_mu, data_logvar, data_action, data_restart)
    step = model.sess.run(model.global_step)
    curr_learning_rate = (hps.learning_rate-hps.min_learning_rate) * (hps.decay_rate) ** step + hps.min_learning_rate

    feed = {model.batch_z: batch_z,
            model.batch_action: batch_action,
            model.batch_restart: batch_restart,
            model.initial_state: batch_state,
            model.lr: curr_learning_rate}

    (train_cost, z_cost, r_cost, batch_state, train_step, _) = model.sess.run([model.cost, model.z_cost, model.r_cost, model.final_state, model.global_step, model.train_op], feed)
    if (step%20==0 and step > 0):
      end = time.time()
      time_taken = end-start
      start = time.time()
      output_log = "step: %d, lr: %.6f, cost: %.4f, z_cost: %.4f, r_cost: %.4f, train_time_taken: %.4f" % (step, curr_learning_rate, train_cost, z_cost, r_cost, time_taken)
      print(output_log)

# save the model (don't bother with tf checkpoints json all the way ...)
model.save_json(os.path.join(model_save_path, "rnn.json"))


import discord
from discord.ext import commands
from  battletime.db import db

class Prefix(commands.Cog):
    def __init__(self, bot):
        self.bot = bot

    @commands.command(name="cprefix", aliases=["ChangePrefix"], brief="Changes the prefix.")
    @commands.has_permissions(manage_guild=True)
    async def change_prefix(self, ctx, new: str):
        """Changes the prefix for the server.\n`Manage Server` permission required."""
        if len(new) > 10:
            await ctx.reply(
                "The prefix can not be more than 10 characters.", delete_after=10
            )

        else:
            db.execute(
                "UPDATE guild SET Prefix = ? WHERE GuildID = ?", new, ctx.guild.id
            )
            db.commit()
            embed = discord.Embed(
                title="Prefix Changed",
                description=f"Prefix has been changed to `{new}`",
            )
            await ctx.reply(embed=embed)

    @commands.command(name="prefix")
    async def prefix(self, ctx):
        prefix = db.field("SELECT Prefix FROM guild WHERE GuildID = ?", ctx.guild.id)
        await ctx.reply(prefix)

def setup(bot):
    bot.add_cog(Prefix(bot))