Repository navigation
Add docs for Semantic Search using EmbeddingPipeline - #4174
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new documentation use-case for βMultimodal Embeddingβ intended to demonstrate semantic search / cross-modal retrieval concepts, plus a small edit in the existing text-embedding docs.
Changes:
- Added a new
multimodal-embeddingdocumentation page that introducesEmbeddingPipeline, including model conversion guidance. - Added βRun modelβ and βAdditional usage optionsβ sections with Python/C++ snippets and configuration notes.
- Tweaked wording/formatting in the existing text-embedding usage options section.
Reviewed changes
Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| site/docs/use-cases/text-embedding/_sections/_usage_options/index.mdx | Small formatting change in the batch size configuration text. |
| site/docs/use-cases/multimodal-embedding/index.mdx | New top-level Multimodal Embedding use-case page with conversion + section includes. |
| site/docs/use-cases/multimodal-embedding/_sections/_usage_options/index.mdx | New usage/options section describing inputs, prompts, config, and examples. |
| site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx | New βRun Modelβ section with device tabs. |
| site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx | New Python code snippet for embeddings. |
| site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx | New C++ code snippet for embeddings. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (4)
site/docs/use-cases/multimodal-embedding/index.mdx:15
- This guide introduces
EmbeddingPipelineas a replacement forTextEmbeddingPipeline, but the current codebase does not contain anEmbeddingPipelinesymbol nor the referenced C++ headeropenvino/genai/rag/embedding_pipeline.hpp(onlyopenvino/genai/rag/text_embedding_pipeline.hppexists). As-is, the documentation and examples appear to describe an API that is not present in this repo version.
`EmbeddingPipeline` computes embedding vectors for text, images, and videos with a single multimodal embedding model. It enables cross-modal use cases such as retrieving the most relevant image or video for a text query.
:::info
We recommend using `EmbeddingPipeline` instead of `TextEmbeddingPipeline`. It is more generic β it handles text, images, and videos β while providing all of the features that were available in `TextEmbeddingPipeline`.
:::
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:15
- This C++ snippet includes
openvino/genai/rag/embedding_pipeline.hppand usesov::genai::EmbeddingPipeline, but that header/class are not present in the repository (onlytext_embedding_pipeline.hpp/TextEmbeddingPipelineexist). Readers will not be able to compile this example against the current code.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"
int main(int argc, char* argv[]) {
std::string models_path = argv[1];
ov::genai::EmbeddingPipeline pipeline(models_path, "${props.device || 'CPU'}");
// Embed a text query
ov::genai::EmbedResult query = pipeline.embed("a photo of a cat");
// Embed an image (image is an ov::Tensor of shape [H, W, 3], uint8)
ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, {image});
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:12
- This Python snippet uses
ov_genai.EmbeddingPipeline(...), butEmbeddingPipelinedoes not appear to be exported by the current Python package in this repo (e.g., it is not present inpy_openvino_genai.pyi). This makes the example non-runnable for users of the current release artifacts built from this codebase.
pipeline = ov_genai.EmbeddingPipeline(models_path, "${props.device || 'CPU'}")
# Embed a text query
query_embedding = pipeline.embed("a photo of a cat").embeddings
site/docs/use-cases/multimodal-embedding/_sections/_usage_options/index.mdx:5
- The tip links to
samples/python/rag/image_video_embedding.py, but that file does not exist in this repository, so the link is broken.
:::tip
Check out the [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/rag/image_video_embedding.py) RAG samples for multimodal embedding examples.
:::
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (3)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:9
- Typo/grammar: "Cach" should be "Each", and the sentence should use a period instead of a comma before "The input can be" to read correctly in the rendered docs.
`EmbeddingPipeline` generates vector representations for text, images, and videos using multimodal embedding models. Because text and visual inputs are mapped into a shared embedding space, you can measure the similarity between a text query and an image or video (e.g. for cross-modal retrieval).
Cach call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_usage_options/index.mdx:8
- Grammar: "For text embedding example" is missing an article and comma; this reads awkwardly in the tip.
Check out the [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/rag/image_video_embedding.py) RAG samples for multimodal embedding examples.
For text embedding example check out [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/rag), [C++](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/rag), and [JavaScript](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/js/rag) samples.
:::
site/docs/supported-models/index.mdx:135
- The section was renamed to "Embedding Models", but the warning title still says "Text Embeddings Models Notes", which is inconsistent/misleading in the rendered docs.
:::warning Text Embeddings Models Notes
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (3)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8
- Grammar/punctuation issue: the sentence uses a comma splice and incorrectly capitalizes βTheβ ("vectors, The input..."). This reads awkwardly in the rendered docs.
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:10
argv[1]is used without validatingargc, so running the sample without arguments will crash. The docs example should guard against missing CLI args.
int main(int argc, char* argv[]) {
std::string models_path = argv[1];
site/docs/supported-models/index.mdx:145
- The section is now titled βEmbedding Modelsβ, but the warning block title still says βText Embeddings Models Notesβ, which is inconsistent and can confuse readers (and the anchor below is no longer text-only).
:::warning Text Embeddings Models Notes
Qwen3 Embedding models require `--task feature-extraction` during the conversion with `optimum-cli`.
:::
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (3)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8
- Grammar/punctuation: this sentence has a comma splice and an incorrect capital letter after the comma ("vectors, The").
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6
- This C++ snippet includes
load_video.hpp, but there is no such helper header in the repo. As written, the example will not compile and suggests a helper API that doesn't exist.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"
#include "load_image.hpp"
#include "load_video.hpp"
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22
- The snippet calls
utils::load_video(...), but there is noutils::load_videoimplementation in the repository (the onlyload_videohelper is a free function insamples/cpp/visual_language_chat/video_to_text_chat.cpp). This makes the example un-runnable as shown.
auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (3)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:9
- The sentence has incorrect punctuation/capitalization (", The") which reads like a typo in rendered docs. Please split it into two sentences (or use a semicolon) so it reads cleanly.
`EmbeddingPipeline` generates vector representations for text, images, and videos using multimodal embedding models. Because text and visual inputs are mapped into a shared embedding space, you can measure the similarity between a text query and an image or video (e.g. for cross-modal retrieval).
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/supported-models/index.mdx:145
- The warning block title uses inconsistent grammar ("Embeddings Models"). It should match the section heading "Embedding Models".
:::warning Embeddings Models Notes
Qwen3 Embedding models require `--task feature-extraction` during the conversion with `optimum-cli`.
:::
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:14
- The docs text describes per-image tensors as
[H, W, C], but this helper adds a batch dimension ([None]), making each image[1, H, W, C]. Whenread_images()returns a list, this becomes a list of batched tensors, which is likely not whatEmbeddingPipeline.embed(images=...)expects for multiple images and can confuse readers about shapes.
def read_image(path: str) -> ov.Tensor:
pic = Image.open(path).convert("RGB")
image_data = np.array(pic)[None]
return ov.Tensor(image_data)
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (2)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:9
- Sentence punctuation/capitalization is incorrect: "Each call to
embed()computes embedding vectors, The input can be:" should be split into two sentences (or use a semicolon) and "The" should not start after a comma.
`EmbeddingPipeline` generates vector representations for text, images, and videos using multimodal embedding models. Because text and visual inputs are mapped into a shared embedding space, you can measure the similarity between a text query and an image or video (e.g. for cross-modal retrieval).
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:14
- The image tensor shape in the comment is inconsistent with the EmbeddingPipeline docs/API and the repository sample (
samples/python/rag/image_video_embedding.pyusesov.Tensor(np.array(image)), which is[H, W, 3]). Listing[1, H, W, 3]may mislead users.
query_embedding = pipeline.embed("a photo of a cat").embeddings
# Embed images (image is an ov.Tensor of shape [1, H, W, 3], uint8)
images: list[ov.Tensor] = read_images("path/to/images")
image_embedding = pipeline.embed(images=images).embeddings
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (6)
site/docs/use-cases/multimodal-embedding/_sections/_usage_options/index.mdx:1
- Unused
OptimumCLIimport at the top of the page; it isnβt referenced anywhere in this MDX file and adds unnecessary bundle code.
import OptimumCLI from '@site/src/components/OptimumCLI';
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8
- Sentence punctuation/grammar issue: βEach call to
embed()computes embedding vectors, The input can be:β should be split into two sentences (comma + capitalized βTheβ reads as a typo).
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:13
- The image tensor shape in this comment is inconsistent with the API docs (single image is
[H, W, C]/[H, W, 3], not[1, H, W, 3]). This can confuse users about expected inputs.
# Embed images (image is an ov.Tensor of shape [1, H, W, 3], uint8)
images: list[ov.Tensor] = read_images("path/to/images")
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:7
load_video.hppis included here but doesnβt exist in the repository (onlyload_image.hppexists). This makes the C++ snippet misleading.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"
#include "load_image.hpp"
#include "load_video.hpp"
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22
- This snippet uses
pipeline.embed(std::string{}, images)/pipeline.embed(std::string{}, {}, {video}, ...)to work around the positional-args overload. Passing an empty text string is ambiguous and may accidentally embed an empty prompt; for image/video-only inputs the API supports the property-based overload (ov::genai::images(...),ov::genai::videos(...),ov::genai::videos_metadata(...)). Also,utils::load_videoisnβt defined anywhere in the repo.
ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images);
// Embed a video (video is an ov::Tensor of shape [F, H, W, 3], uint8)
auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});
site/docs/use-cases/multimodal-embedding/index.mdx:18
- Minor grammar/consistency issue in the introduction: βDownload and convert embedding model β¦ or text only model β¦β is missing an article and should use βtext-onlyβ for consistency elsewhere.
Download and convert embedding model (e.g. multimodal [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B), or text only model [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)) to OpenVINO format from Hugging Face:
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (4)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:9
- In this section header sentence, the comma and capitalization break the sentence flow ("vectors, The"). This reads like a grammar error in the rendered docs.
`EmbeddingPipeline` generates vector representations for text, images, and videos using multimodal embedding models. Because text and visual inputs are mapped into a shared embedding space, you can measure the similarity between a text query and an image or video (e.g. for cross-modal retrieval).
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6
- The code sample includes
load_image.hpp/load_video.hpp, but there is noload_video.hppin the repository, andload_image.hpponly exists under other sample directories. Copy/pasting this snippet will not compile as written.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"
#include "load_image.hpp"
#include "load_video.hpp"
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22
- For image-only / video-only embedding, the sample currently calls
embed(std::string{}, ...)as a placeholder "empty text" input and relies onutils::load_*helpers that aren't defined here. The API supports keyword properties likeov::genai::images(...)/ov::genai::videos(...), which avoids dummy text values and makes the example match the actual EmbeddingPipeline usage pattern.
// Embed images (image is an ov::Tensor of shape [H, W, 3], uint8)
std::vector<ov::Tensor> images = utils::load_images("path/to/images");
ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images);
// Embed a video (video is an ov::Tensor of shape [F, H, W, 3], uint8)
auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});
site/docs/use-cases/multimodal-embedding/index.mdx:19
- This sentence is missing articles and uses "text only" instead of the more common "text-only", which makes it read awkwardly in the docs.
<ConvertModelSection>
Download and convert embedding model (e.g. multimodal [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B), or text only model [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)) to OpenVINO format from Hugging Face:
<OptimumCLI model='Qwen/Qwen3-VL-Embedding-2B' outputDir='Qwen3-VL-Embedding-2B' task='feature-extraction'/>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.
Comments suppressed due to low confidence (4)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8
- In this sentence, the comma and capitalization break the grammar. Consider splitting it into two sentences for readability.
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6
- This C++ snippet includes
load_video.hppand usesutils::load_video(...), but there is no such header in the repo and noutils::load_videohelper. This makes the example non-actionable and potentially misleading. Consider using explicit placeholders forov::Tensorinputs andVideoMetadata, and remove the non-existent includes/helpers.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"
#include "load_image.hpp"
#include "load_video.hpp"
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:14
- The example uses
read_images(...)and documents an image tensor shape of[1, H, W, 3], but the pipeline expects each image tensor to be[H, W, C](or a batch[N, H, W, C]). Using a placeholder list of[H, W, 3]tensors (and linking to an existing sample loader) would be more accurate and easier to follow.
# Embed images (image is an ov.Tensor of shape [1, H, W, 3], uint8)
images: list[ov.Tensor] = read_images("path/to/images")
image_embedding = pipeline.embed(images=images).embeddings
site/docs/use-cases/multimodal-embedding/index.mdx:18
- Minor grammar: βconvert embedding modelβ is missing an article, and βtext onlyβ should be hyphenated as an adjective.
Download and convert an embedding model (e.g. multimodal [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B), or a text-only model [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)) to OpenVINO format from Hugging Face:
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (3)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8
- The sentence uses a comma splice and incorrect capitalization (", The"), which reads as a typo in the docs.
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22
- The example calls
embed(std::string{}, ...)to embed images/videos. This works but is awkward in docs because it requires passing a dummy empty text;EmbeddingPipelineprovides an overload that accepts keyword properties (images,videos,videos_metadata) and avoids the placeholder string.
ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images);
// Embed a video (video is an ov::Tensor of shape [F, H, W, 3], uint8)
auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:13
- The comment describes a single image tensor as shape
[1, H, W, 3], but the example uses a list of tensors (images: list[ov.Tensor]). This is a bit inconsistent with the API docs, which accept[H, W, C]or batched[N, H, W, C]tensors per image element.
# Embed images (image is an ov.Tensor of shape [1, H, W, 3], uint8)
images: list[ov.Tensor] = read_images("path/to/images")
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.
Comments suppressed due to low confidence (5)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8
- The sentence has incorrect punctuation/capitalization (comma before βTheβ and capitalized βTheβ after a comma). This reads like a typo in the rendered docs.
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:12
- The image tensor shape in this comment ([1, H, W, 3]) is inconsistent with the EmbeddingPipeline API docs (expects [H, W, C] or batched [N, H, W, C]). This can confuse users about whether a batch dimension is required.
# Embed images (image is an ov.Tensor of shape [1, H, W, 3], uint8)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6
- This include suggests a
load_video.hpphelper exists, but there is no such header in the repository. The only video-loading example is implemented directly insamples/cpp/visual_language_chat/video_to_text_chat.cpp.
#include "load_video.hpp"
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22
utils::load_video(...)is referenced here, but no such utility function exists (the sample implementation is a free functionload_video(...)insamples/cpp/visual_language_chat/video_to_text_chat.cpp). As written, this snippet wonβt compile if copied.
// Embed a video (video is an ov::Tensor of shape [F, H, W, 3], uint8)
auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});
tools/llm_bench/README.md:312
- This heading was changed to βText Embeddingβ, but later in the same section the README still says βSupported Text Embeddings model typesβ, which is inconsistent. Align the wording to avoid confusion (either keep both plural or both singular).
### Compare Text Embedding models
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (4)
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8
- Grammar/punctuation issue: this sentence has a comma splice and incorrect capitalization (", The").
Each call to `embed()` computes embedding vectors, The input can be:
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:16
- This section describes
EmbeddingPipeline(multimodal), but it includes a JS tab even though JS bindings currently expose onlyTextEmbeddingPipeline(text-only). Add an explicit note here so readers understand the JS example is text-only and not anEmbeddingPipelineexample.
Text, images, and videos are mapped into the same embedding space, so embeddings produced from different modalities can be compared directly (for example, with cosine similarity).
<LanguageTabs>
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6
#include "load_video.hpp"does not exist in the repository, so this example cannot compile as written.
#include "load_image.hpp"
#include "load_video.hpp"
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22
- The example calls
utils::load_video(...), but there is no such symbol in the repo (and no header to provide it). Also, usingpipeline.embed(std::string{}, ...)is unnecessarily unclear whenEmbeddingPipelinesupports property-based calls (e.g.ov::genai::images(...),ov::genai::videos(...)).
// Embed a video (video is an ov::Tensor of shape [F, H, W, 3], uint8)
auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});
as-suvorov
left a comment
There was a problem hiding this comment.
Please also apply API changes to cpp samples
|
|
||
| // Embed images (image is an ov::Tensor of shape [H, W, 3], uint8) | ||
| std::vector<ov::Tensor> images = utils::load_images("path/to/images"); | ||
| ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images); |
There was a problem hiding this comment.
| ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images); | |
| ov::genai::EmbedResult image_embedding = pipeline.embed(ov::genai::images(images)); |
|
|
||
| // Embed a video (video is an ov::Tensor of shape [F, H, W, 3], uint8) | ||
| auto [video, video_metadata] = utils::load_video("path/to/video.mp4"); | ||
| ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata}); |
There was a problem hiding this comment.
| ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata}); | |
| ov::genai::EmbedResult video_embedding = pipeline.embed(ov::genai::video(video), ov::genai::video(video_metadata)); |
| ov::genai::EmbeddingPipeline pipeline( | ||
| models_path, | ||
| "CPU", | ||
| ov::AnyMap{ | ||
| {ov::genai::text_embedding_config.name(), | ||
| ov::genai::TextEmbeddingPipeline::Config({ | ||
| {"pooling_type", ov::genai::TextEmbeddingPipeline::PoolingType::LAST_TOKEN}, | ||
| {"normalize", true}, | ||
| {"max_length", 512}, | ||
| {"pad_to_max_length", true}, | ||
| {"padding_side", "left"}, | ||
| })} | ||
| } | ||
| ); |
There was a problem hiding this comment.
Let's simplify a bit
| ov::genai::EmbeddingPipeline pipeline( | |
| models_path, | |
| "CPU", | |
| ov::AnyMap{ | |
| {ov::genai::text_embedding_config.name(), | |
| ov::genai::TextEmbeddingPipeline::Config({ | |
| {"pooling_type", ov::genai::TextEmbeddingPipeline::PoolingType::LAST_TOKEN}, | |
| {"normalize", true}, | |
| {"max_length", 512}, | |
| {"pad_to_max_length", true}, | |
| {"padding_side", "left"}, | |
| })} | |
| } | |
| ); | |
| auto text_embedding_config = ov::genai::TextEmbeddingPipeline::Config({ | |
| {"pooling_type", ov::genai::TextEmbeddingPipeline::PoolingType::LAST_TOKEN}, | |
| {"normalize", true}, | |
| {"max_length", 512}, | |
| {"pad_to_max_length", true}, | |
| {"padding_side", "left"}, | |
| }); | |
| ov::genai::EmbeddingPipeline pipeline( | |
| models_path, | |
| "CPU", | |
| ov::genai::text_embedding_config(text_embedding_config) | |
| ); |
Description
Add docs for Semantic Search using Mutlimodal Embedding /
EmbeddingPipelineLink to build docs https://pavel-esir.github.io/openvino.genai/docs/use-cases/multimodal-embedding/
CVS-186293
Checklist: