From a5d658ac461b8a3d3d0791abf98311460c33e94c Mon Sep 17 00:00:00 2001 From: thorwebdev Date: Tue, 9 May 2023 17:14:58 +0800 Subject: [PATCH 1/2] chore: typos. --- docs/openai_embeddings.mdx | 6 +++--- supabase/migrations/20230418022251_init.sql | 2 +- supabase/seed.sql | 4 ++-- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/openai_embeddings.mdx b/docs/openai_embeddings.mdx index ef3aff6..4a3a562 100644 --- a/docs/openai_embeddings.mdx +++ b/docs/openai_embeddings.mdx @@ -14,10 +14,10 @@ Overall, OpenAI embeddings are a powerful tool for NLP tasks and have been used ## Storing Embeddings in Postgres -`pg_vector` is a Postgres extension that provides support for vector operations, including similarity search and nearest neighbor search. It allows you to store vectors as columns in a database table and perform vector operations on them using SQL queries. +`pgvector` is a Postgres extension that provides support for vector operations, including similarity search and nearest neighbor search. It allows you to store vectors as columns in a database table and perform vector operations on them using SQL queries. -One interesting use case of `pg_vector` is to use it in conjunction with OpenAI embeddings. Since OpenAI embeddings represent words and phrases as high-dimensional vectors, you can store these vectors in a pg_vector column and use the extension's similarity search capabilities to find similar words or phrases in a large dataset. +One interesting use case of `pgvector` is to use it in conjunction with OpenAI embeddings. Since OpenAI embeddings represent words and phrases as high-dimensional vectors, you can store these vectors in a pgvector column and use the extension's similarity search capabilities to find similar words or phrases in a large dataset. -For example, let's say you have a database of customer reviews and you want to find all the reviews that mention a particular product. You could use OpenAI embeddings to represent each review as a vector and store these vectors in a pg_vector column. Then, you could perform a similarity search to find all the reviews that are similar to the vector representing the product name. +For example, let's say you have a database of customer reviews and you want to find all the reviews that mention a particular product. You could use OpenAI embeddings to represent each review as a vector and store these vectors in a pgvector column. Then, you could perform a similarity search to find all the reviews that are similar to the vector representing the product name. This approach can be very powerful for applications that involve large amounts of text data, as it allows you to perform complex queries on the data using vector operations. However, it does require some additional setup and configuration compared to traditional SQL queries, so it may not be suitable for all use cases. diff --git a/supabase/migrations/20230418022251_init.sql b/supabase/migrations/20230418022251_init.sql index dd51676..06716c2 100644 --- a/supabase/migrations/20230418022251_init.sql +++ b/supabase/migrations/20230418022251_init.sql @@ -1,4 +1,4 @@ --- Enable pg_vector extension +-- Enable pgvector extension create extension if not exists vector with schema public; -- Create tables diff --git a/supabase/seed.sql b/supabase/seed.sql index 3623a1b..4f8be88 100644 --- a/supabase/seed.sql +++ b/supabase/seed.sql @@ -29,9 +29,9 @@ Overall, OpenAI embeddings are a powerful tool for NLP tasks and have been used ', 380, '[-0.011174448,-0.0019023881,-0.00038730755,-0.008162611,-0.0030584803,0.004581057,0.014726017,0.008129294,-0.011254408,-0.042165715,0.010967883,0.02565392,-0.019230444,-0.013180118,-0.00225388,0.020936264,0.011141131,0.017071517,0.014646056,-0.014339542,-0.020416522,-0.010394835,-0.024627764,-0.014845957,-0.008582402,-0.0069432165,0.032410562,-0.02787948,-0.02386814,-0.018164309,0.012540435,0.012567089,-0.0070831473,0.0009945058,-0.008016017,0.015165797,0.011654209,-0.023121845,0.028092708,0.0157655,0.007702839,0.031024585,0.014845957,-0.01997674,-0.0054806103,-0.0006946548,-0.020882957,-0.016431835,0.0031467697,0.015778827,-0.00028131856,0.050001822,-0.013886433,0.0041945823,-0.022402203,0.00088456046,-0.018430842,0.030384902,0.017418012,-0.0046576858,-0.008942223,-0.008309205,-0.009855104,0.015658885,-0.012500456,0.0061602723,0.000615111,0.021309411,0.004571062,-0.010088321,0.020962918,0.017871121,0.002533741,0.0041046273,0.02545402,-0.0020556452,-0.0072963745,-0.0152191045,-0.027373066,0.00054348,-0.009661866,-0.015672212,0.0010061667,0.022522142,0.009208757,-0.0220024,0.007829443,-0.0029635276,-0.0159654,0.005157437,0.015538946,0.013713187,0.01661841,0.007489612,-0.003216735,0.032650445,-0.02249549,0.001850747,0.00082917133,-0.032277297,-0.014499462,0.016072014,-0.022148995,-0.017111497,-0.030997932,0.02747968,0.024920952,-0.0031101212,0.005037497,0.022788677,0.0044444585,0.0159654,-0.0043778247,-0.022402203,-0.01783114,0.0037081575,0.039820213,-0.0200567,-0.01941702,0.01467271,0.019177139,0.01876401,0.020936264,-0.017577933,0.0147659965,-0.015712192,0.0053240214,-0.0050208387,0.004251221,-0.011680863,0.012673703,0.018190961,0.021456005,-0.015818806,-0.031024585,0.04101962,-0.0063535096,-0.007702839,-0.01580548,-0.019483652,0.02277535,0.011640883,-0.007096474,-0.011314378,-0.011274398,0.010361519,0.006660024,0.011281061,0.024987584,0.0042678793,0.016498469,-0.0021122838,-0.03627531,0.010394835,-0.003941375,-0.010294884,0.012873603,0.0062069157,-0.03611539,-0.017324725,0.010867934,0.015379026,-0.016325222,0.0071297907,0.013166791,0.023481665,0.0061069657,-0.0027386392,-0.01799106,-0.013626562,-0.007689513,0.017817814,-0.047043294,-0.0009112139,0.007642869,-0.0034949302,0.030944625,0.013233425,-0.011374349,-0.023188477,0.0065700687,0.0029518667,0.016111994,0.0075762356,-0.0057071643,-0.028759044,0.002678669,0.0047842893,0.010288222,-0.024014734,0.002428793,0.019083852,0.015059184,-0.01661841,-0.63072664,-0.013273405,0.028759044,-0.026720056,0.01568554,0.014712689,0.006113629,-0.0035848855,-0.0113077145,0.037901167,0.009395332,-0.028092708,-0.020229949,0.007116464,0.02654681,-0.025627267,-0.0056138774,-0.027586292,0.01775118,0.027532985,-0.02305521,0.012747,-0.016271915,-0.009395332,-0.0005672182,0.013313385,0.0016375197,-0.0065767323,0.009521935,0.010041677,-0.021136165,0.039793562,0.03278371,-0.011227755,0.039686948,0.013699859,-0.031051239,0.027986094,-0.0071497806,0.046616837,-0.0042145727,-0.020656403,-0.015658885,-0.0067999545,0.013346702,0.01981682,0.015632233,-0.008662363,-0.02172254,0.0021789174,-0.0009120468,0.026213642,-0.010814627,-0.014259581,-0.004704329,-0.002400474,0.02833259,-0.0012652046,0.030411556,-0.02613368,0.031957455,0.015205778,-0.021122837,-0.028385896,-0.024227962,0.015245758,-0.00889558,0.0134799685,-0.0036881675,-0.04765632,0.013686533,0.018084347,0.009102144,-0.009875094,-0.0011494289,0.030651437,0.012966891,-0.010508112,-0.014979224,0.008988867,0.022615429,-0.014659382,0.0036281971,-0.00784277,0.023308419,0.028679084,0.0010686357,0.0063535096,0.011314378,0.0056138774,0.00062760484,0.02581384,-0.035315786,-0.06817946,0.002513751,0.006546747,-0.012267238,-0.0077628097,0.0073563447,-0.0075362553,0.013380018,-0.027159838,0.014179621,0.027852828,0.017511299,0.009268728,0.016831636,0.006983197,-0.0026770032,-0.01848415,-0.014965897,0.00084749557,0.0015075842,-0.014259581,-0.0033333437,-0.022415528,-0.0016458489,-0.0057005007,0.028092708,-0.03270375,0.014379522,-0.0029901809,-0.015738847,-0.008129294,0.010967883,0.002535407,-0.012833623,-0.0065833954,-0.011627556,0.015139145,0.012853613,-0.0159654,-0.0063968217,-0.0059936885,0.02755964,0.022508817,0.03464945,0.0033599972,-0.00055805605,-0.010035014,0.0021939098,0.0076362058,0.017591259,-0.0014551104,-0.039020613,-0.012153961,-0.007249731,0.0045177555,-0.0015883774,-0.0143128885,0.003268376,-0.010254905,-0.008009354,0.03627531,0.010161618,-0.019230444,-0.009635213,-0.03675507,0.0054073134,-0.020203294,-0.003403309,0.028066054,-0.01624526,-0.010561419,0.0073563447,-0.0109812105,-0.0049808584,0.029078884,-0.017457992,-0.043018624,0.016711697,-0.028625777,0.013140137,0.013673206,-0.012747,0.019510306,-0.02825263,0.0013493296,-0.00076920114,-0.018430842,-0.003008505,-0.0140063735,-0.0036082072,-0.023308419,0.010874596,0.002865243,0.024747703,0.0157655,-0.01290692,0.009122134,0.0013218432,0.02857247,-0.025560632,0.031344425,5.6430294e-05,0.01803104,0.028625777,-2.3165572e-05,-0.004141276,0.015552272,0.0007375502,0.009655203,0.005254056,-0.0022372217,-0.0074296417,-0.0030734728,-0.0040080086,-0.022015728,0.0022005732,-0.0001826176,0.022762023,0.018177634,-0.028465856,-0.015792154,-0.0025970428,0.013180118,-0.007716166,0.011294388,-0.01373984,-0.011707516,0.0021489323,0.021082858,0.02184248,0.005647194,-0.011460972,0.024001408,0.0049508736,0.0038814049,0.0012452146,-0.03310355,0.021349393,0.02456113,0.010648043,0.0072897114,0.025374059,-0.0072763846,0.04336512,0.0073696715,0.03254383,-0.029372072,-0.006673351,0.013793146,0.0014875942,-0.025160832,0.039926827,0.0017016545,0.037181526,0.009122134,-0.019310405,0.0046710125,-0.02857247,-0.0009578574,-0.015778827,0.010128301,0.009555252,-0.017471319,0.018937258,-0.0108546065,0.022868637,0.027746214,-0.0063768313,-0.0069432165,0.013366692,0.016924923,0.021122837,-0.013953067,-0.0010461468,-0.02273537,-0.029212153,0.0034649451,-0.010628052,-0.022188975,0.0008554083,-0.022109015,0.031557653,0.02849251,-0.00046851722,0.017231438,0.014099661,0.056558564,-0.0076095522,-0.048935685,0.049335487,-0.0005143278,-0.006380163,-0.0041546025,-0.0416593,-0.014219602,-0.014379522,0.0022738702,-0.005947045,0.035795547,0.010794637,0.0031767548,-6.1011353e-05,0.015618906,0.022482162,0.002503756,0.025787188,-0.015179125,-0.003020166,0.0023854815,-0.0198568,-0.013486632,0.025094198,-0.0014784321,-0.018817317,0.002180583,-0.007456295,-0.011034518,0.005710496,0.00970851,0.02050981,-0.014206274,0.020922936,0.027119858,-0.00794272,-0.008675689,0.018937258,0.01183412,0.0038247663,-0.018910604,0.007749483,0.020003393,0.034409568,0.015072511,-0.016725022,0.008242571,-0.025933782,0.018937258,-0.01697823,0.0058604213,-0.016471816,0.021615926,-0.030491516,-0.026813343,0.01949698,-0.004584389,0.01393974,0.0007396325,-0.008055997,-0.036248654,-0.009048837,-0.024187982,-0.010974547,-0.012100654,-0.0001415963,0.010641379,0.03667511,-0.014539442,0.027266452,0.0152724115,-0.014392848,-0.02626695,0.007816116,-0.0065600737,-6.6438342e-06,0.006516762,-0.0143128885,0.0050608185,0.0046776757,0.03611539,0.008189265,0.006540084,0.014965897,0.022628756,-0.021749193,-0.018217614,-0.014925917,-0.02934542,0.005510595,0.00016543863,-0.0068565933,-0.011500952,-0.016311895,0.025294099,-0.03286367,-0.000517243,0.030944625,0.018550783,0.010055004,-0.016138649,-0.012607069,-0.041392766,0.008868926,-0.03635527,-0.005230734,-0.018364208,-0.009781807,-0.0038714097,0.0042378944,0.0010436481,-0.026919957,0.030224983,0.004714324,-0.02573388,-0.051521067,-0.031344425,-0.0030151685,0.017817814,0.010121637,-0.0022405535,-0.019536959,0.011341032,0.018257596,-0.016018707,0.01189409,-0.018470822,0.0011086158,-0.0007583732,-0.013753166,0.0071631074,-0.005503932,-0.014192948,0.014646056,-0.0039380435,-0.0072763846,0.0059870253,0.015205778,0.047203213,0.020283256,0.0029085549,0.01673835,-0.05256055,-0.022642083,-0.025760533,-0.011494289,-0.03067809,0.0052673826,-0.0034249648,0.019350385,0.027959442,0.011154458,-0.019070525,-0.0079760365,0.007616216,0.012413831,0.007043167,0.0046010474,-0.017444665,-0.0012185612,0.023481665,0.009122134,-0.01949698,-0.030971278,-0.012860277,0.023401706,0.025840493,-0.023534972,0.018817317,-0.0009895083,0.0047609676,-0.013833127,0.01840419,-0.016804984,0.022788677,-0.012213931,-0.0034616133,0.008902243,-0.0079760365,-0.031237813,0.0020489818,-0.027666254,0.0049741953,0.0067766327,-0.0021705881,-0.012367188,0.002493761,0.02082965,-0.017777834,-0.017231438,0.024441188,0.018017715,0.039447065,-0.009302044,0.03526248,-0.031157851,-0.03934045,-0.0058271047,-0.0088089565,0.0018240936,0.02168256,0.04032663,0.02525412,0.011947397,-0.014486136,0.03680838,0.011294388,-0.0029185498,0.0055605704,-0.0013801475,-0.009022184,-0.010621389,-0.01286694,-0.0011261072,0.003598212,0.015099164,-0.013320048,0.013373355,0.010001697,-0.0094419755,-0.009781807,-0.012167288,-0.021109512,-0.017924428,0.045017634,-0.017937753,0.023215132,-0.04541743,0.00293854,-0.0011127804,0.013080168,0.021149492,0.004137944,0.019603593,-0.0059836935,0.0019890117,0.006819945,-0.014979224,-0.02755964,-0.0005917893,-0.0552792,0.0026303597,0.0060969708,0.0075229285,0.022868637,-0.013699859,-0.012500456,0.0029818518,0.0085557485,-0.0041712606,-0.0018524129,0.011287725,-0.028119361,-0.024547802,-0.03424965,-0.02581384,-0.0013509954,-0.01689827,0.0074363053,0.0017924428,0.025267445,-0.025960434,0.0004839262,0.011367685,0.013320048,0.046270344,-0.003484935,0.022402203,-0.0042545525,0.012207268,-0.0045877206,0.017044865,-0.0042978646,0.0043378444,0.010541429,0.0028485847,0.016511796,-0.029931795,0.015578926,0.0046576858,0.00035149202,-0.02646685,0.039100572,-0.023841487,-0.010055004,0.0039147213,-0.022562122,-0.0220024,-0.0018840638,-0.015472312,-0.0067932913,5.203144e-05,-0.00486425,-0.009248738,0.013599909,0.023654914,0.011027854,0.00021489323,0.033823196,-0.0026520155,0.009488619,-0.013126811,-0.0039147213,0.020283256,0.012074001,-0.0019173806,0.0023821497,0.012580416,0.003528247,0.007343018,0.011007864,0.012213931,0.030438209,-0.040459897,-0.01685829,0.008342521,-0.013526612,0.014206274,-0.0005472281,-0.0020439844,0.0021272763,-0.010807963,0.006983197,0.03744806,-0.010594736,-0.024081368,0.0028185996,-0.0057871244,-0.027746214,-0.0068632565,-0.0057737976,0.018190961,-0.025267445,-0.025547307,0.013120147,-0.004757636,0.013646552,-0.007209751,-0.004207909,-0.0064967717,0.033023592,-0.00986843,-0.021096185,-0.008908907,0.010334865,-0.009901747,-0.010521439,-0.008562412,-0.0065001035,0.008402492,-0.02168256,-0.012986881,-0.0050341655,-0.013180118,0.01868405,0.0040979637,0.0082292445,-0.0007829443,-0.01467271,-0.0022488825,-0.012407169,-0.0052407295,0.007169771,0.022828657,-0.006163604,0.026933284,-0.025467345,-0.004634364,-0.016258588,0.006526757,0.00053181907,-0.05093469,-0.021402698,0.009608559,0.0031934134,0.010168281,-0.0036948307,-0.0060370006,-0.004141276,-0.02472105,0.026600117,0.0036981625,0.0097285,0.008155948,0.018057695,0.01665839,-0.016791657,-0.01900389,-0.012040684,-0.019896781,-0.012580416,-0.033583313,0.008315868,-0.012320545,0.030758051,0.0133533655,-0.038167704,-0.024827663,-0.006246896,-0.04227233,-0.0043878197,-0.0055272537,0.018350882,0.022562122,0.04571062,0.010581409,0.026253622,0.010934567,0.014046354,-0.03035825,0.01624526,-0.01463273,-0.008968877,-0.02456113,0.0033533338,-0.030118369,0.018710703,-0.009288718,-0.028199323,-0.0099284,0.02966526,-0.004147939,-0.0138597805,0.0048176064,0.02865243,0.025547307,0.023081865,0.029985102,-0.019963413,-0.00046560197,0.0051074624,-0.010961221,-0.012093991,-0.01783114,0.010674696,0.023641586,0.02156262,0.0064801136,0.0042445576,0.0015284072,0.007229741,0.04485771,0.00028235972,-0.025507325,0.013753166,0.015632233,0.003928048,-0.009901747,0.01104118,0.008549086,-0.0026120355,0.020336563,-0.029878488,-0.023375053,-0.026027068,0.015565599,-0.017418012,-0.0038247663,0.0040879687,0.012147298,-0.033530008,0.01641851,0.0028069387,-0.0010453139,0.009461965,-0.019323733,0.023388378,-0.008269224,-0.02966526,0.004217904,0.00048642498,0.008795629,0.0057737976,-0.0015600582,-0.02990514,0.01689827,0.033343434,-0.0123938415,0.019310405,0.1845483,-0.054159757,0.0009986705,0.0016808315,0.0027503,-0.01677833,0.0096418755,0.0011211096,-0.022988578,-0.0041546025,0.018097674,0.01491259,-0.016671717,0.00079876976,0.0073097013,-0.029398726,-0.034063075,-0.015992055,-0.0032850343,0.0032517177,-0.009688519,-0.0020872962,-0.0058370996,-0.003774791,0.0025470678,0.010568082,-0.00792273,-0.0026820006,0.017031537,0.014686036,-0.00095119403,0.011667536,0.018244268,0.00434784,-0.006110297,0.0022172316,0.008822283,0.025374059,0.015472312,-0.0025120853,0.030971278,0.009621886,-0.0037814544,-0.021362718,0.0023538305,0.026680076,-0.027199818,-0.017977733,-0.010035014,0.011454308,-0.008075987,-0.014726017,0.0026953274,0.0042645475,-0.009955053,0.015432332,-0.0035782221,0.010961221,0.022562122,0.005120789,-0.02301523,-0.0039013948,-0.007209751,0.0030601462,-0.013713187,0.0075229285,-0.0072164144,-0.018137654,0.018297575,0.006216911,-0.0066167125,-0.0012985214,-0.0055505754,-0.008709006,-0.032730404,-0.02646685,0.044031456,0.023081865,0.0120140305,0.03246387,0.012040684,-0.010068331,0.0028119362,-0.0068099494,-0.014192948,-0.043178547,0.018164309,-0.036728416,-0.0021839149,-0.01892393,-0.0016183625,-0.019203791,0.0062202425,0.011487626,0.012293891,-0.002946869,0.013833127,-0.0031334432,0.00173747,-0.023961427,-0.027932787,0.046830066,0.02448117,0.00027840334,-0.0053306846,0.019443672,-0.012480466,-0.0037781228,0.0129469,0.0056005507,0.004664349,-0.039527025,0.014939244,-0.0045777257,-0.0074096518,0.0045277504,0.008469125,0.0072364043,0.0071631074,-0.0028819013,-0.032517176,-7.5743614e-05,-0.01076132,-0.0011519276,0.00067758,-0.033769887,-0.033609968,0.01973686,-0.014126315,0.010068331,0.01568554,-0.02366824,0.0021456005,0.018950583,-0.013966394,0.0008033508,0.004124617,0.013313385,0.0059637036,0.0012593742,0.0070631574,-0.00023321745,-0.004147939,0.0046710125,0.0073630083,-0.0105281025,0.007749483,0.01072134,0.007056494,-0.0089355605,-0.0041012955,-0.005730486,0.009348689,0.00032088222,0.018097674,0.010874596,-0.0069365534,-0.020403195,-0.0062768813,0.006360173,-0.05965036,-0.014779323,0.00087539834,-0.017044865,-0.007949384,-0.024081368,-0.16695705,0.002420464,0.0016325222,-0.051414452,0.017844466,0.010134964,0.011947397,-0.013833127,-0.016471816,-0.011587576,0.018630743,-0.009981707,-0.0235483,-0.03488933,0.010234915,0.025947107,0.0052807094,0.0070698205,0.027173165,-0.0013001873,0.040913004,-0.035529014,0.0088089565,-0.0032433884,-0.011714179,-0.0078027896,0.0007221412,0.027906135,-0.0026403547,-0.037128218,0.0071297907,-0.01880399,0.048242696,0.0018007719,0.028759044,0.004067979,0.05045493,-0.02281533,-0.027586292,0.019563613,0.03472941,0.008009354,0.0019773508,0.0017624577,0.0038314296,-0.008549086,0.031344425,-0.010814627,0.029372072,-0.01292691,0.008669026,0.009615222,-0.00019469493,-0.011540933,0.0062768813,0.010907914,0.010488122,0.006836603,-0.007836106,-0.036435228,0.0134799685,-0.014885937,0.009028847,-0.006010347,-0.010548092,-0.0049808584,-0.030171676,0.027346412,-0.021322738,0.021882461,0.0073630083,0.01175416,-0.011820793,-0.009755153,0.028599123,0.03067809,-0.03169092,0.0077961264,0.025800513,0.0031800866,-0.01705819,0.023081865,-0.010907914,0.00038709934,0.004141276,-0.008482452,0.012953564,0.017817814,-0.040193364,0.0016575097,0.023348399,-0.025187485,0.0045144237,0.011847447,0.005274046,0.019616919,0.021216124,-0.030118369,0.018190961,-0.0062535596,0.003591549,0.00595704,0.0075162654,0.0062935394,0.028146015,0.014539442,-0.018097674,-0.019430345,0.030171676,-0.01665839,-0.0040213354,-0.013806473,-0.009501945,0.015618906,-0.018790664,0.011267735,-0.010101647,-0.019230444,0.031744227,-0.011007864,0.026160335,-0.003528247,-0.018284248,0.03318351,-0.006983197,-0.034969293,-0.11962056,-0.011347695,0.012540435,0.013093494,-0.011354358,0.017724527,0.019643573,0.02362826,-0.021749193,0.02865243,-0.03934045,-0.021242779,0.007956047,-0.015938748,-0.0010561419,-0.0061802627,-0.0030318268,-0.009568579,-0.037501365,0.022881964,-0.0013226761,0.00018834391,0.02755964,-0.013160128,0.00045185882,0.0015042525,-0.02172254,0.023175152,0.0075695724,-0.013913087,0.017138151,-0.026147008,0.00246211,-0.020363215,-0.0020323235,0.011974051,-0.0051840907,-0.0196569,0.010394835,-0.010194934,-0.0011369351,-0.0125204455,0.013366692,-0.002825263,-0.009715172,-0.013766493,-0.021056205,0.0198568,-0.018217614,-0.026839998,-0.03995348,-0.009741826,-0.043578345,0.0116142295,0.0076095522,0.021829154,0.008649036,-0.004281206,0.010274895,-0.010987874,-0.021309411,-0.008435808,-0.03326347,-0.009415322,-0.005170764,0.00087873003,-0.016991558,-0.014033027,-0.00881562,-0.008282552,-0.00024925114,0.018017715,-0.00094786234,0.009955053,-0.019403692,-0.0030351586,0.0019273757,-0.016525123,0.019896781,-0.0028219311,-0.013200108,-0.0014742675,0.00022322242,-0.0369683,-0.015605579,0.0048909034,0.0067866277,-0.024547802,0.0016799986,-0.040459897,0.0066966726,0.037821207,-0.0004183338,-0.020896284,-0.003341673,-0.0027736218,-0.036701765,-0.0016050359,0.0064368015,0.030331595,-0.00800269,-0.00050558214,-0.049548715,0.015339045,-0.0062002526,-0.01278698,-0.007996027,-0.017817814,0.010648043,-0.0033783214,-0.005710496,0.016218608,-0.021495987,-0.000253624,-0.005960372,0.01373984,-0.018617416,-0.01102119,0.028679084,0.00500418,0.014219602,0.0086823525,-0.012667039,-0.002657013,0.004207909,0.026027068,0.0017016545,-0.028066054,-0.027586292,0.024134675,-0.00067299895,-0.014126315,0.008802293,-0.01864407,-0.0040479884,-0.0052507245,-0.013573255,0.0010094984,0.0143128885,0.013993047,0.018577436,0.023375053,-0.037714593,-0.03675507,0.01661841,-0.017404685,-0.016458489,-0.024347901,0.012340535,-0.022841984,0.010048341,0.008742323,0.043471735,0.017964408,-0.02565392,-0.036088735,-0.025200812,0.005387323,0.02508087,0.0102215875,0.0034382916,0.007562909,0.060396656,0.019963413,0.007489612,-0.0019840142,-0.008469125,-0.003518252,-0.029638607,-0.017004885,-0.0022605434,-0.009228747,-0.0076495325,-0.0030867995,0.0011560923,0.0125204455,0.027612947,-0.00788275,0.013873107,-0.012953564,-0.017844466,0.03651519,0.010734666,-0.010774646,-0.009801797,0.006163604,0.023881467,0.009761816,-0.0030135026,-0.00895555,-0.023988081,-0.018737357,0.009428648,0.008782303,-0.0019523633,0.02464109,0.016631735,-0.0105281025,0.015352372,0.023561627,0.023108518,0.0011760823,0.02825263,-0.0055505754,-0.0028452529,-0.036568496,-0.0127736535,-0.0049775266,-0.0338765,-0.011074497,0.021975748,0.01844417,0.012860277,-0.009208757,-0.0131001575,-0.011800803,-0.014352868,0.014619403,0.00696987,-0.029212153,-0.034329608,0.0012069003,0.03145104,0.013186781,0.028599123,-0.003000176,0.033130206,-0.011520942,0.010128301,-0.004651022,0.032597136,0.01272701,-0.022348896,-0.022802003,-0.004624369,-0.012553763,-0.023268439,-0.014419502,-0.030731397,0.019030545,0.016711697,0.08475789,0.026919957,-0.0034816035,0.019083852,-0.010248241,0.005907065,0.011667536,0.0055472436,-0.01568554,0.010614726,-0.0103481915,-0.003300027,-0.030011754,-0.018364208,-0.0015867116,-0.023921447,0.011054507,0.017418012,-0.0074096518,0.025507325,0.036568496,0.0004210408,0.025094198,-0.0005505598,-0.031744227,0.0028735723,0.026613442,-0.008389165,-0.00998837,-0.025827168,0.04109958,0.021282759,-0.03987352,-0.0014642724,-0.01673835,-0.0007875254,-0.0046210373,-0.02545402,0.008848936,0.016032035,0.016551776,0.016538449,-0.008562412,-0.04989521,-0.021935767,0.01183412,-0.016445162,-0.022548797,-0.021309411]', 'openai-embeddings', 'OpenAI Embeddings'); INSERT INTO "public"."dfods_page_section" ("id", "page_id", "content", "token_count", "embedding", "slug", "heading") VALUES (2, 1, '## Storing Embeddings in Postgres -`pg_vector` is a Postgres extension that provides support for vector operations, including similarity search and nearest neighbor search. It allows you to store vectors as columns in a database table and perform vector operations on them using SQL queries. +`pgvector` is a Postgres extension that provides support for vector operations, including similarity search and nearest neighbor search. It allows you to store vectors as columns in a database table and perform vector operations on them using SQL queries. -One interesting use case of `pg_vector` is to use it in conjunction with OpenAI embeddings. Since OpenAI embeddings represent words and phrases as high-dimensional vectors, you can store these vectors in a pg\_vector column and use the extension''s similarity search capabilities to find similar words or phrases in a large dataset. +One interesting use case of `pgvector` is to use it in conjunction with OpenAI embeddings. Since OpenAI embeddings represent words and phrases as high-dimensional vectors, you can store these vectors in a pg\_vector column and use the extension''s similarity search capabilities to find similar words or phrases in a large dataset. For example, let''s say you have a database of customer reviews and you want to find all the reviews that mention a particular product. You could use OpenAI embeddings to represent each review as a vector and store these vectors in a pg\_vector column. Then, you could perform a similarity search to find all the reviews that are similar to the vector representing the product name. From 2892248c44bce37c71b2dfff3767c7fae1b59b16 Mon Sep 17 00:00:00 2001 From: thorwebdev Date: Tue, 9 May 2023 17:49:03 +0800 Subject: [PATCH 2/2] chore: final polish. --- README.md | 132 +++++++++++++++++++++++++++++++++++++-- islands/SearchDialog.tsx | 2 +- 2 files changed, 128 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index 924ee01..b999416 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,35 @@ -# deno-fresh-openai-doc-search +# Deno Fresh OpenAI Doc Search Starter Template for building your own custom ChatGPT style doc search powered by Fresh, Deno, OpenAI, and Supabase. -## Setup +This starter takes all the `.mdx` files in the `docs` directory and processes +them to use as custom context within +[OpenAI Text Completion](https://platform.openai.com/docs/guides/completion) +prompts. + +[![Made with Fresh](https://fresh.deno.dev/fresh-badge.svg)](https://fresh.deno.dev) + +## Run locally + +### Prerequisites + +- [Deno](https://deno.com/manual/getting_started/installation) +- [Docker](https://docs.docker.com/engine/install/) +- [Git](https://github.com/git-guides/install-git) +- [A free Supabase account](https://supabase.com) +- [Supabase CLI](https://supabase.com/docs/guides/cli) +- [An OpenAI Account](https://platform.openai.com/signup/) + +### Setup env vars: ```bash cp .env.example .env ``` -## Run locally +Set the required env vars as outlined in the file. -Start the project: +### Start the project: ```bash supabase start @@ -29,7 +47,7 @@ This will watch the project directory and restart as necessary. 2. Link your project: `supabase link --project-ref=your-project-ref` 3. Push up migration: `supabase db push` -## Setup GitHub Action +### Setup GitHub Action We're using a [GitHub Action](./.github/workflows/generate-embeddings.yaml) to generate the embeddings whenever we merge into the `main` branch. @@ -39,3 +57,107 @@ generate the embeddings whenever we merge into the `main` branch. them as Actions secrets in GitHub. 2. Set `OPENAI_KEY` as Actions secrets in GitHub. 3. Push or merge into `main` to kick off the GitHub action. + +### Automatic Deployment with Deno Deploy + +These steps show you how to deploy your app close to your users at the edge with +[Deno Deploy](https://deno.com/deploy). + +1. Clone this repository to your GitHub account. + +2. Sign into [Deno Deploy](https://dash.deno.com) with your GitHub account. + +3. Select your GitHub organization or user, repository, and branch + +4. Select "Automatic" deployment mode and `main.ts` as the entry point + +5. Click "Link", which will start the deployment. + +6. Once the deployment is complete, click on "Settings" and add the production + environmental variables, then hit "Save" + +Voila, you've got your own custom ChatGPT! + +## Technical Details + +Building your own custom ChatGPT involves four steps: + +1. [⚡️ GitHub Action] Pre-process the knowledge base (your `.mdx` files in your + `pages` folder). +2. [⚡️ GitHub Action] Store embeddings in Postgres with + [pgvector](https://supabase.com/docs/guides/database/extensions/pgvector). +3. [🏃 Runtime] Perform vector similarity search to find the content that's + relevant to the question. +4. [🏃 Runtime] Inject content into OpenAI GPT-3 text completion prompt and + stream response to the client. + +## ⚡️ GitHub Action + +Step 1. and 2. happen via a +[GitHub Action](./.github/workflows/generate-embeddings.yaml) anytime we make +changes to the `main` branch. During this time the +[`generate-embeddings`](./tools/generate-embeddings.ts) script is being executed +which performs the following tasks: + +```mermaid +sequenceDiagram + participant GitHub Action + participant DB (pgvector) + participant OpenAI (API) + loop 1. Pre-process the knowledge base + GitHub Action->>GitHub Action: Chunk .mdx pages into sections + loop 2. Create & store embeddings + GitHub Action->>OpenAI (API): create embedding for page section + OpenAI (API)->>GitHub Action: embedding vector(1536) + GitHub Action->>DB (pgvector): store embedding for page section + end + end +``` + +In addition to storing the embeddings, this script generates a checksum for each +of your `.mdx` files and stores this in another database table to make sure the +embeddings are only regenerated when the file has changed. + +## 🏃 Runtime + +Step 3. and 4. happen at runtime, anytime the user submits a question. When this +happens, the following sequence of tasks is performed: + +```mermaid +sequenceDiagram + participant Client + participant Edge Function + participant DB (pgvector) + participant OpenAI (API) + Client->>Edge Function: { query: lorem ispum } + critical 3. Perform vector similarity search + Edge Function->>OpenAI (API): create embedding for query + OpenAI (API)->>Edge Function: embedding vector(1536) + Edge Function->>DB (pgvector): vector similarity search + DB (pgvector)->>Edge Function: relevant docs content + end + critical 4. Inject content into prompt + Edge Function->>OpenAI (API): completion request prompt: query + relevant docs content + OpenAI (API)-->>Client: text/event-stream: completions response + end +``` + +The relevant files for this are the +[`SearchDialog` (Client)](./islands/SearchDialog.tsx) component and the +[`vector-search` (Edge Function)](./routes/api/vector-search.ts). + +The initialization of the database, including the setup of the `pgvector` +extension is stored in the +[`supabase/migrations` folder](./supabase/migrations/) which is automatically +applied to your local Postgres instance when running `supabase start`. + +## Learn More + +- Read the blogpost on how we built + [ChatGPT for the Supabase Docs](https://supabase.com/blog/chatgpt-supabase-docs). +- [[Docs] pgvector: Embeddings and vector similarity](https://supabase.com/docs/guides/database/extensions/pgvector) +- Watch [Greg's](https://twitter.com/ggrdson) "How I built this" + [video](https://youtu.be/Yhtjd7yGGGA) on the + [Rabbit Hole Syndrome YouTube Channel](https://www.youtube.com/@RabbitHoleSyndrome): + +[![Video: How I Built Supabase’s OpenAI Doc Search](https://img.youtube.com/vi/Yhtjd7yGGGA/0.jpg)](https://www.youtube.com/watch?v=Yhtjd7yGGGA) diff --git a/islands/SearchDialog.tsx b/islands/SearchDialog.tsx index 432cb07..973f253 100644 --- a/islands/SearchDialog.tsx +++ b/islands/SearchDialog.tsx @@ -44,7 +44,7 @@ export default function SearchDialog() {