Hub documentation
Lakukan pencarian kemiripan vektor
Lakukan pencarian kemiripan vektor
Fitur Fixed-Length Arrays ditambahkan dalam versi DuckDB 0.10.0. Hal ini memungkinkan Anda menggunakan vector embeddings dalam tabel DuckDB, sehingga analisis data Anda menjadi lebih kuat.
Selain itu, fungsi array_cosine_similarity diperkenalkan. Fungsi ini mengukur kosinus sudut antara dua vektor, yang menunjukkan tingkat kemiripan mereka. Nilai 1 berarti keduanya sejajar sempurna, 0 berarti keduanya tegak lurus, dan -1 berarti keduanya berlawanan sepenuhnya.
Mari kita jelajahi cara menggunakan fungsi ini untuk pencarian kemiripan. Pada bagian ini, kami akan menunjukkan cara melakukan pencarian kemiripan menggunakan DuckDB.
Kami akan menggunakan dataset asoria/awesome-chatgpt-prompts-embeddings.
Pertama, mari kita pratinjau beberapa rekaman dari dataset:
FROM 'hf://datasets/asoria/awesome-chatgpt-prompts-embeddings/data/*.parquet' SELECT act, prompt, len(embedding) as embed_len LIMIT 3;
┌──────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬───────────┐
│ act │ prompt │ embed_len │
│ varchar │ varchar │ int64 │
├──────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼───────────┤
│ Linux Terminal │ I want you to act as a linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output insid… │ 384 │
│ English Translator… │ I want you to act as an English translator, spelling corrector and improver. I will speak to you in any language and you will detect the language, translate it and answer… │ 384 │
│ `position` Intervi… │ I want you to act as an interviewer. I will be the candidate and you will ask me the interview questions for the `position` position. I want you to only reply as the inte… │ 384 │
└──────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────┘
Selanjutnya, mari kita pilih embedding yang akan digunakan untuk pencarian kemiripan:
FROM 'hf://datasets/asoria/awesome-chatgpt-prompts-embeddings/data/*.parquet' SELECT embedding WHERE act = 'Linux Terminal';
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ embedding │
│ float[] │
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ [-0.020781303, -0.029143505, -0.0660217, -0.00932716, -0.02601602, -0.011426172, 0.06627567, 0.11941507, 0.0013917526, 0.012889079, 0.053234346, -0.07380514, 0.04871567, -0.043601237, -0.0025319182, 0.0448… │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Sekarang, mari gunakan embedding yang dipilih untuk menemukan catatan serupa:
SELECT act,
prompt,
array_cosine_similarity(embedding::float[384], (SELECT embedding FROM 'hf://datasets/asoria/awesome-chatgpt-prompts-embeddings/data/*.parquet' WHERE act = 'Linux Terminal')::float[384]) AS similarity
FROM 'hf://datasets/asoria/awesome-chatgpt-prompts-embeddings/data/*.parquet'
ORDER BY similarity DESC
LIMIT 3;
┌──────────────────────┬─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────────┐
│ act │ prompt │ similarity │
│ varchar │ varchar │ float │
├──────────────────────┼─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┼────────────┤
│ Linux Terminal │ I want you to act as a linux terminal. I will type commands and you will reply with what the terminal should show. I want you to only reply with the terminal output insi… │ 1.0 │
│ JavaScript Console │ I want you to act as a javascript console. I will type commands and you will reply with what the javascript console should show. I want you to only reply with the termin… │ 0.7599728 │
│ R programming Inte… │ I want you to act as a R interpreter. I'll type commands and you'll reply with what the terminal should show. I want you to only reply with the terminal output inside on… │ 0.7303775 │
└──────────────────────┴─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────┘
Itu saja! Anda telah berhasil melakukan pencarian kemiripan vektor menggunakan DuckDB.
Update on GitHub