compute/save embeddings into file/blob as a one-time operation, then read/load into memory going forward

[roalexan]

Hello. Is there a way to do the following:

1. one-time: calculate the embeddings - e.g. kernel builder set with AddAzureTextEmbeddingGenerationService then invoking kernel.Memory.SaveInformationAsync
2. save the embeddings from Semantic Kernel memory into a file
3. upload the file as a blob to a storage service (e.g. Azure blob storage)
4. then, get the blob from the storage service, and load it into Semantic Kernel memory?

I want to see if this is possible, so I can avoid taking the hit of computing the embeddings in memory every time, especially for those cases when the source of the embeddings changes very infrequently.

update The kernel.Memory.GetAsync method, using withEmbedding = True, to get the embeddings per collection per key is promising, and perhaps UpsertAsync could be used to load it into memory, but it is not clear to me now how to create a MemoryRecord - would be very helpful if any of the examples covered this.

update Getting closer, but still blocked on deserializing the float vector. If anyone has any ideas or samples on dealing with serializing/deserializing the embeddings vector that would be great. My current error: cannot convert from 'System.Collections.Generic.List<float> to 'Microsoft.SemanticKernel.AI.Embeddings.Embedding<float>

update Got that working. Downloaded source and looked through samples. This seems to do the job: new Embedding(floatVector.ToArray())

[craigomatic]

Related to your other question, you will find more success using the TextMemorySkill.

You can, and likely should generate embeddings offline. Some implementations you can use for persistence can be found under the Connectors.Memory.* namespace: https://github.com/microsoft/semantic-kernel/tree/main/dotnet/src/Connectors