Skip to content

Add docs for Semantic Search using EmbeddingPipeline - #4174

Merged
yatarkan merged 21 commits into
openvinotoolkit:masterfrom
pavel-esir:add_embedding_pipeline_docs
Jul 25, 2026
Merged

yatarkan merged 21 commits into
openvinotoolkit:masterfrom
pavel-esir:add_embedding_pipeline_docs

Conversation

@pavel-esir

@pavel-esir pavel-esir commented Jul 17, 2026 •

Copy link
Copy Markdown
Contributor

Description

Add docs for Semantic Search using Mutlimodal Embedding / EmbeddingPipeline

Link to build docs https://pavel-esir.github.io/openvino.genai/docs/use-cases/multimodal-embedding/

CVS-186293

Checklist:

  • I have made corresponding changes to the documentation.

Copilot AI review requested due to automatic review settings July 17, 2026 11:59
@pavel-esir
pavel-esir requested a review from yatarkan as a code owner July 17, 2026 11:59
@pavel-esir pavel-esir added the documentation Improvements or additions to documentation label Jul 17, 2026
@github-actions github-actions Bot added the category: GH Pages Docs Github Pages documentation label Jul 17, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new documentation use-case for β€œMultimodal Embedding” intended to demonstrate semantic search / cross-modal retrieval concepts, plus a small edit in the existing text-embedding docs.

Changes:

  • Added a new multimodal-embedding documentation page that introduces EmbeddingPipeline, including model conversion guidance.
  • Added β€œRun model” and β€œAdditional usage options” sections with Python/C++ snippets and configuration notes.
  • Tweaked wording/formatting in the existing text-embedding usage options section.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
site/docs/use-cases/text-embedding/_sections/_usage_options/index.mdx Small formatting change in the batch size configuration text.
site/docs/use-cases/multimodal-embedding/index.mdx New top-level Multimodal Embedding use-case page with conversion + section includes.
site/docs/use-cases/multimodal-embedding/_sections/_usage_options/index.mdx New usage/options section describing inputs, prompts, config, and examples.
site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx New β€œRun Model” section with device tabs.
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx New Python code snippet for embeddings.
site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx New C++ code snippet for embeddings.

Copilot AI review requested due to automatic review settings July 17, 2026 12:06

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated 6 comments.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 19 out of 19 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (4)

site/docs/use-cases/multimodal-embedding/index.mdx:15

  • This guide introduces EmbeddingPipeline as a replacement for TextEmbeddingPipeline, but the current codebase does not contain an EmbeddingPipeline symbol nor the referenced C++ header openvino/genai/rag/embedding_pipeline.hpp (only openvino/genai/rag/text_embedding_pipeline.hpp exists). As-is, the documentation and examples appear to describe an API that is not present in this repo version.
`EmbeddingPipeline` computes embedding vectors for text, images, and videos with a single multimodal embedding model. It enables cross-modal use cases such as retrieving the most relevant image or video for a text query.

:::info
We recommend using `EmbeddingPipeline` instead of `TextEmbeddingPipeline`. It is more generic β€” it handles text, images, and videos β€” while providing all of the features that were available in `TextEmbeddingPipeline`.
:::

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:15

  • This C++ snippet includes openvino/genai/rag/embedding_pipeline.hpp and uses ov::genai::EmbeddingPipeline, but that header/class are not present in the repository (only text_embedding_pipeline.hpp / TextEmbeddingPipeline exist). Readers will not be able to compile this example against the current code.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"

int main(int argc, char* argv[]) {
    std::string models_path = argv[1];

    ov::genai::EmbeddingPipeline pipeline(models_path, "${props.device || 'CPU'}");

    // Embed a text query
    ov::genai::EmbedResult query = pipeline.embed("a photo of a cat");

    // Embed an image (image is an ov::Tensor of shape [H, W, 3], uint8)
    ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, {image});

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:12

  • This Python snippet uses ov_genai.EmbeddingPipeline(...), but EmbeddingPipeline does not appear to be exported by the current Python package in this repo (e.g., it is not present in py_openvino_genai.pyi). This makes the example non-runnable for users of the current release artifacts built from this codebase.
pipeline = ov_genai.EmbeddingPipeline(models_path, "${props.device || 'CPU'}")

# Embed a text query
query_embedding = pipeline.embed("a photo of a cat").embeddings

site/docs/use-cases/multimodal-embedding/_sections/_usage_options/index.mdx:5

  • The tip links to samples/python/rag/image_video_embedding.py, but that file does not exist in this repository, so the link is broken.
:::tip
Check out the [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/rag/image_video_embedding.py) RAG samples for multimodal embedding examples.
:::

Comment thread site/docs/supported-models/index.mdx Resolved
Comment thread site/docs/supported-models/index.mdx Resolved
Copilot AI review requested due to automatic review settings July 20, 2026 10:06
@github-actions github-actions Bot added category: visual language Visual language pipeline category: continuous batching Continuous batching category: LLM LLM pipeline (stateful, static) category: sampling Sampling / Decoding algorithms category: speculative decoding Speculative decoding labels Jul 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:9

  • Typo/grammar: "Cach" should be "Each", and the sentence should use a period instead of a comma before "The input can be" to read correctly in the rendered docs.
`EmbeddingPipeline` generates vector representations for text, images, and videos using multimodal embedding models. Because text and visual inputs are mapped into a shared embedding space, you can measure the similarity between a text query and an image or video (e.g. for cross-modal retrieval).
Cach call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_usage_options/index.mdx:8

  • Grammar: "For text embedding example" is missing an article and comma; this reads awkwardly in the tip.
Check out the [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/rag/image_video_embedding.py) RAG samples for multimodal embedding examples.
For text embedding example check out [Python](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/python/rag), [C++](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/cpp/rag), and [JavaScript](https://github.com/openvinotoolkit/openvino.genai/tree/master/samples/js/rag) samples.
:::

site/docs/supported-models/index.mdx:135

  • The section was renamed to "Embedding Models", but the warning title still says "Text Embeddings Models Notes", which is inconsistent/misleading in the rendered docs.
:::warning Text Embeddings Models Notes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (3)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8

  • Grammar/punctuation issue: the sentence uses a comma splice and incorrectly capitalizes β€œThe” ("vectors, The input..."). This reads awkwardly in the rendered docs.
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:10

  • argv[1] is used without validating argc, so running the sample without arguments will crash. The docs example should guard against missing CLI args.
int main(int argc, char* argv[]) {
    std::string models_path = argv[1];

site/docs/supported-models/index.mdx:145

  • The section is now titled β€œEmbedding Models”, but the warning block title still says β€œText Embeddings Models Notes”, which is inconsistent and can confuse readers (and the anchor below is no longer text-only).
:::warning Text Embeddings Models Notes
Qwen3 Embedding models require `--task feature-extraction` during the conversion with `optimum-cli`.
:::

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8

  • Grammar/punctuation: this sentence has a comma splice and an incorrect capital letter after the comma ("vectors, The").
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6

  • This C++ snippet includes load_video.hpp, but there is no such helper header in the repo. As written, the example will not compile and suggests a helper API that doesn't exist.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"
#include "load_image.hpp"
#include "load_video.hpp"

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22

  • The snippet calls utils::load_video(...), but there is no utils::load_video implementation in the repository (the only load_video helper is a free function in samples/cpp/visual_language_chat/video_to_text_chat.cpp). This makes the example un-runnable as shown.
    auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
    ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});

Comment thread site/docs/supported-models/index.mdx Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:9

  • The sentence has incorrect punctuation/capitalization (", The") which reads like a typo in rendered docs. Please split it into two sentences (or use a semicolon) so it reads cleanly.
`EmbeddingPipeline` generates vector representations for text, images, and videos using multimodal embedding models. Because text and visual inputs are mapped into a shared embedding space, you can measure the similarity between a text query and an image or video (e.g. for cross-modal retrieval).
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/supported-models/index.mdx:145

  • The warning block title uses inconsistent grammar ("Embeddings Models"). It should match the section heading "Embedding Models".
:::warning Embeddings Models Notes
Qwen3 Embedding models require `--task feature-extraction` during the conversion with `optimum-cli`.
:::

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:14

  • The docs text describes per-image tensors as [H, W, C], but this helper adds a batch dimension ([None]), making each image [1, H, W, C]. When read_images() returns a list, this becomes a list of batched tensors, which is likely not what EmbeddingPipeline.embed(images=...) expects for multiple images and can confuse readers about shapes.
def read_image(path: str) -> ov.Tensor:
    pic = Image.open(path).convert("RGB")
    image_data = np.array(pic)[None]
    return ov.Tensor(image_data)

Comment thread site/docs/supported-models/index.mdx Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:9

  • Sentence punctuation/capitalization is incorrect: "Each call to embed() computes embedding vectors, The input can be:" should be split into two sentences (or use a semicolon) and "The" should not start after a comma.
`EmbeddingPipeline` generates vector representations for text, images, and videos using multimodal embedding models. Because text and visual inputs are mapped into a shared embedding space, you can measure the similarity between a text query and an image or video (e.g. for cross-modal retrieval).
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:14

  • The image tensor shape in the comment is inconsistent with the EmbeddingPipeline docs/API and the repository sample (samples/python/rag/image_video_embedding.py uses ov.Tensor(np.array(image)), which is [H, W, 3]). Listing [1, H, W, 3] may mislead users.
query_embedding = pipeline.embed("a photo of a cat").embeddings

# Embed images  (image is an ov.Tensor of shape [1, H, W, 3], uint8)
images: list[ov.Tensor] = read_images("path/to/images")
image_embedding = pipeline.embed(images=images).embeddings

Comment thread tools/llm_bench/README.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (6)

site/docs/use-cases/multimodal-embedding/_sections/_usage_options/index.mdx:1

  • Unused OptimumCLI import at the top of the page; it isn’t referenced anywhere in this MDX file and adds unnecessary bundle code.
import OptimumCLI from '@site/src/components/OptimumCLI';

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8

  • Sentence punctuation/grammar issue: β€œEach call to embed() computes embedding vectors, The input can be:” should be split into two sentences (comma + capitalized β€œThe” reads as a typo).
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:13

  • The image tensor shape in this comment is inconsistent with the API docs (single image is [H, W, C] / [H, W, 3], not [1, H, W, 3]). This can confuse users about expected inputs.
# Embed images  (image is an ov.Tensor of shape [1, H, W, 3], uint8)
images: list[ov.Tensor] = read_images("path/to/images")

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:7

  • load_video.hpp is included here but doesn’t exist in the repository (only load_image.hpp exists). This makes the C++ snippet misleading.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"
#include "load_image.hpp"
#include "load_video.hpp"

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22

  • This snippet uses pipeline.embed(std::string{}, images) / pipeline.embed(std::string{}, {}, {video}, ...) to work around the positional-args overload. Passing an empty text string is ambiguous and may accidentally embed an empty prompt; for image/video-only inputs the API supports the property-based overload (ov::genai::images(...), ov::genai::videos(...), ov::genai::videos_metadata(...)). Also, utils::load_video isn’t defined anywhere in the repo.
    ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images);

    // Embed a video  (video is an ov::Tensor of shape [F, H, W, 3], uint8)
    auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
    ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});

site/docs/use-cases/multimodal-embedding/index.mdx:18

  • Minor grammar/consistency issue in the introduction: β€œDownload and convert embedding model … or text only model …” is missing an article and should use β€œtext-only” for consistency elsewhere.
    Download and convert embedding model (e.g. multimodal [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B), or text only model [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)) to OpenVINO format from Hugging Face:

@yatarkan yatarkan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Other changes LGTM

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:9

  • In this section header sentence, the comma and capitalization break the sentence flow ("vectors, The"). This reads like a grammar error in the rendered docs.
`EmbeddingPipeline` generates vector representations for text, images, and videos using multimodal embedding models. Because text and visual inputs are mapped into a shared embedding space, you can measure the similarity between a text query and an image or video (e.g. for cross-modal retrieval).
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6

  • The code sample includes load_image.hpp / load_video.hpp, but there is no load_video.hpp in the repository, and load_image.hpp only exists under other sample directories. Copy/pasting this snippet will not compile as written.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"
#include "load_image.hpp"
#include "load_video.hpp"

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22

  • For image-only / video-only embedding, the sample currently calls embed(std::string{}, ...) as a placeholder "empty text" input and relies on utils::load_* helpers that aren't defined here. The API supports keyword properties like ov::genai::images(...) / ov::genai::videos(...), which avoids dummy text values and makes the example match the actual EmbeddingPipeline usage pattern.
    // Embed images  (image is an ov::Tensor of shape [H, W, 3], uint8)
    std::vector<ov::Tensor> images = utils::load_images("path/to/images");
    ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images);

    // Embed a video  (video is an ov::Tensor of shape [F, H, W, 3], uint8)
    auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
    ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});

site/docs/use-cases/multimodal-embedding/index.mdx:19

  • This sentence is missing articles and uses "text only" instead of the more common "text-only", which makes it read awkwardly in the docs.
<ConvertModelSection>
    Download and convert embedding model (e.g. multimodal [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B), or text only model [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)) to OpenVINO format from Hugging Face:
    <OptimumCLI model='Qwen/Qwen3-VL-Embedding-2B' outputDir='Qwen3-VL-Embedding-2B' task='feature-extraction'/>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (4)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8

  • In this sentence, the comma and capitalization break the grammar. Consider splitting it into two sentences for readability.
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6

  • This C++ snippet includes load_video.hpp and uses utils::load_video(...), but there is no such header in the repo and no utils::load_video helper. This makes the example non-actionable and potentially misleading. Consider using explicit placeholders for ov::Tensor inputs and VideoMetadata, and remove the non-existent includes/helpers.
{`#include "openvino/genai/rag/embedding_pipeline.hpp"
#include "load_image.hpp"
#include "load_video.hpp"

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:14

  • The example uses read_images(...) and documents an image tensor shape of [1, H, W, 3], but the pipeline expects each image tensor to be [H, W, C] (or a batch [N, H, W, C]). Using a placeholder list of [H, W, 3] tensors (and linking to an existing sample loader) would be more accurate and easier to follow.
# Embed images  (image is an ov.Tensor of shape [1, H, W, 3], uint8)
images: list[ov.Tensor] = read_images("path/to/images")
image_embedding = pipeline.embed(images=images).embeddings

site/docs/use-cases/multimodal-embedding/index.mdx:18

  • Minor grammar: β€œconvert embedding model” is missing an article, and β€œtext only” should be hyphenated as an adjective.
    Download and convert an embedding model (e.g. multimodal [Qwen/Qwen3-VL-Embedding-2B](https://huggingface.co/Qwen/Qwen3-VL-Embedding-2B), or a text-only model [BAAI/bge-small-en-v1.5](https://huggingface.co/BAAI/bge-small-en-v1.5)) to OpenVINO format from Hugging Face:

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (3)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8

  • The sentence uses a comma splice and incorrect capitalization (", The"), which reads as a typo in the docs.
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22

  • The example calls embed(std::string{}, ...) to embed images/videos. This works but is awkward in docs because it requires passing a dummy empty text; EmbeddingPipeline provides an overload that accepts keyword properties (images, videos, videos_metadata) and avoids the placeholder string.
    ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images);

    // Embed a video  (video is an ov::Tensor of shape [F, H, W, 3], uint8)
    auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
    ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:13

  • The comment describes a single image tensor as shape [1, H, W, 3], but the example uses a list of tensors (images: list[ov.Tensor]). This is a bit inconsistent with the API docs, which accept [H, W, C] or batched [N, H, W, C] tensors per image element.
# Embed images  (image is an ov.Tensor of shape [1, H, W, 3], uint8)
images: list[ov.Tensor] = read_images("path/to/images")

Comment thread tools/llm_bench/README.md Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (5)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8

  • The sentence has incorrect punctuation/capitalization (comma before β€œThe” and capitalized β€œThe” after a comma). This reads like a typo in the rendered docs.
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_python.mdx:12

  • The image tensor shape in this comment ([1, H, W, 3]) is inconsistent with the EmbeddingPipeline API docs (expects [H, W, C] or batched [N, H, W, C]). This can confuse users about whether a batch dimension is required.
# Embed images  (image is an ov.Tensor of shape [1, H, W, 3], uint8)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6

  • This include suggests a load_video.hpp helper exists, but there is no such header in the repository. The only video-loading example is implemented directly in samples/cpp/visual_language_chat/video_to_text_chat.cpp.
#include "load_video.hpp"

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22

  • utils::load_video(...) is referenced here, but no such utility function exists (the sample implementation is a free function load_video(...) in samples/cpp/visual_language_chat/video_to_text_chat.cpp). As written, this snippet won’t compile if copied.
    // Embed a video  (video is an ov::Tensor of shape [F, H, W, 3], uint8)
    auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
    ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});

tools/llm_bench/README.md:312

  • This heading was changed to β€œText Embedding”, but later in the same section the README still says β€œSupported Text Embeddings model types”, which is inconsistent. Align the wording to avoid confusion (either keep both plural or both singular).
### Compare Text Embedding models

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:8

  • Grammar/punctuation issue: this sentence has a comma splice and incorrect capitalization (", The").
Each call to `embed()` computes embedding vectors, The input can be:

site/docs/use-cases/multimodal-embedding/_sections/_run_model/index.mdx:16

  • This section describes EmbeddingPipeline (multimodal), but it includes a JS tab even though JS bindings currently expose only TextEmbeddingPipeline (text-only). Add an explicit note here so readers understand the JS example is text-only and not an EmbeddingPipeline example.
Text, images, and videos are mapped into the same embedding space, so embeddings produced from different modalities can be compared directly (for example, with cosine similarity).

<LanguageTabs>

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:6

  • #include "load_video.hpp" does not exist in the repository, so this example cannot compile as written.
#include "load_image.hpp"
#include "load_video.hpp"

site/docs/use-cases/multimodal-embedding/_sections/_run_model/_code_example_cpp.mdx:22

  • The example calls utils::load_video(...), but there is no such symbol in the repo (and no header to provide it). Also, using pipeline.embed(std::string{}, ...) is unnecessarily unclear when EmbeddingPipeline supports property-based calls (e.g. ov::genai::images(...), ov::genai::videos(...)).
    // Embed a video  (video is an ov::Tensor of shape [F, H, W, 3], uint8)
    auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
    ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});

Comment thread site/docusaurus.config.ts Resolved

@as-suvorov as-suvorov left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also apply API changes to cpp samples


// Embed images (image is an ov::Tensor of shape [H, W, 3], uint8)
std::vector<ov::Tensor> images = utils::load_images("path/to/images");
ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ov::genai::EmbedResult image_embedding = pipeline.embed(std::string{}, images);
ov::genai::EmbedResult image_embedding = pipeline.embed(ov::genai::images(images));


// Embed a video (video is an ov::Tensor of shape [F, H, W, 3], uint8)
auto [video, video_metadata] = utils::load_video("path/to/video.mp4");
ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
ov::genai::EmbedResult video_embedding = pipeline.embed(std::string{}, {}, {video}, {video_metadata});
ov::genai::EmbedResult video_embedding = pipeline.embed(ov::genai::video(video), ov::genai::video(video_metadata));

Comment on lines +115 to +128
ov::genai::EmbeddingPipeline pipeline(
models_path,
"CPU",
ov::AnyMap{
{ov::genai::text_embedding_config.name(),
ov::genai::TextEmbeddingPipeline::Config({
{"pooling_type", ov::genai::TextEmbeddingPipeline::PoolingType::LAST_TOKEN},
{"normalize", true},
{"max_length", 512},
{"pad_to_max_length", true},
{"padding_side", "left"},
})}
}
);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's simplify a bit

Suggested change
ov::genai::EmbeddingPipeline pipeline(
models_path,
"CPU",
ov::AnyMap{
{ov::genai::text_embedding_config.name(),
ov::genai::TextEmbeddingPipeline::Config({
{"pooling_type", ov::genai::TextEmbeddingPipeline::PoolingType::LAST_TOKEN},
{"normalize", true},
{"max_length", 512},
{"pad_to_max_length", true},
{"padding_side", "left"},
})}
}
);
auto text_embedding_config = ov::genai::TextEmbeddingPipeline::Config({
{"pooling_type", ov::genai::TextEmbeddingPipeline::PoolingType::LAST_TOKEN},
{"normalize", true},
{"max_length", 512},
{"pad_to_max_length", true},
{"padding_side", "left"},
});
ov::genai::EmbeddingPipeline pipeline(
models_path,
"CPU",
ov::genai::text_embedding_config(text_embedding_config)
);

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category: GH Pages Docs Github Pages documentation category: llm_bench Label for tool/llm_bench folder category: RAG samples RAG samples category: WWB PR changes WWB documentation Improvements or additions to documentation no-match-files

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants