English | 中文

WeMM-Embedding is a family of universal multimodal embedding models developed by the WeChat Vision team. It provides unified representations for text, images, videos, visual documents, and interleaved multimodal inputs, achieving state-of-the-art performance across multiple benchmarks covering diverse tasks and domains.

All models support text, images, videos, visual documents, and interleaved multimodal inputs. Embeddings are obtained from the last-layer hidden state at the dedicated <embedding> token position, followed by L2 normalization. Audio input is not currently supported.

pip install -r requirements.txt

The snippet below is fully standalone: it does not use any code or files from this repository, and the image is loaded from a URL. Just install the dependencies, save the code as demo.py, and run python demo.py:

pip install torch transformers==5.2.0 "qwen-vl-utils==0.0.14"

import torch
import torch.nn.functional as F
from qwen_vl_utils import process_vision_info
from transformers import AutoModel, AutoProcessor

MODEL_ID = "tencent/WeMM-Embedding-2B"
IMAGE_URL = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/pipeline-cat-chonk.jpeg"

processor = AutoProcessor.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModel.from_pretrained(
    MODEL_ID, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda"
)
model.eval()


def encode(content: list[dict]) -> torch.Tensor:
    messages = [{"role": "user", "content": content}]
    text = processor.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=False
    )
    images, videos, video_kwargs = process_vision_info(
        messages,
        image_patch_size=16,
        return_video_kwargs=True,
        return_video_metadata=True,
    )
    if videos is not None:
        videos, video_metadata = zip(*videos)
        videos, video_metadata = list(videos), list(video_metadata)
    else:
        video_metadata = None
    inputs = processor(
        text=text,
        images=images,
        videos=videos,
        video_metadata=video_metadata,
        return_tensors="pt",
        **video_kwargs,
    ).to("cuda")
    with torch.inference_mode():
        return F.normalize(model.embedding(**inputs).float(), dim=-1)


image_embedding = encode([{"type": "image", "image": IMAGE_URL}])
texts = ["A chubby Pallas's cat walking in the snow.", "A dog is running on a beach."]
text_embeddings = torch.cat(
    [encode([{"type": "text", "text": text}]) for text in texts]
)

print("image embedding shape:", tuple(image_embedding.shape))  # (1, 2048)
print(
    "text-image similarities:",
    (text_embeddings @ image_embedding.T).flatten().tolist(),
)

The first text should have a higher similarity to the image than the second one.

We recommend using transformers==5.2.0 for inference and reproducibility, as newer versions may differ in preprocessing behavior.

python examples/transformers_inference.py \
  --model tencent/WeMM-Embedding-2B \
  --image /path/to/image.jpg \
  --video /path/to/video.mp4 \
  --dimension 2048

The example produces independent text, image, and video embeddings. Omit --dimension for the full embedding dimension.

python examples/sentence_transformers_inference.py \
  --model tencent/WeMM-Embedding-2B \
  --image /path/to/image.jpg \
  --video /path/to/video.mp4 \
  --dimension 2048

SentenceTransformer loads the model directly, so a Hugging Face model id such as tencent/WeMM-Embedding-2B also works in place of a local path. Text, image, and video inputs go through SentenceTransformer.encode(), and MRL is selected with --dimension.

Tested versions: vLLM 0.27.0 and SGLang 0.5.9.

vLLM:

MODEL_PATH=tencent/WeMM-Embedding-2B
CHAT_TEMPLATE=$(hf download "$MODEL_PATH" embedding_chat_template.jinja)
vllm serve "$MODEL_PATH" \
  --runner pooling \
  --chat-template "$CHAT_TEMPLATE"

SGLang:

MODEL_PATH=tencent/WeMM-Embedding-2B
python scripts/patch_sglang_video.py
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --is-embedding \
  --enable-precise-embedding-interpolation

Equivalent one-command wrappers are available in scripts/serve_vllm.sh and scripts/serve_sglang.sh.

For a supported dimension d, truncate the full embedding and normalize it again:

embedding = torch.nn.functional.normalize(embedding[..., :d], dim=-1)

On MMEB-v2, the 2B model at 256 dimensions retains 98.7% of its full-dimensional image and video performance.

Results on 78 datasets from Table 1 of the technical report. Image and video tasks use Hit@1, while visual-document tasks use NDCG@5. Higher is better.

† Closed-source leaderboard submission without publicly released model weights or a public inference endpoint.

Results on all 190 tasks; see the technical report for benchmark details. V3-All includes the 78 MMEB-v2 tasks, 53 text tasks, 47 agent tasks, 11 audio tasks, and MCMR. Unsupported tasks are assigned a score of zero.

Text results use NDCG@5; agent, MCMR, and audio results use Hit@1.

PeerQA and DeepPlanning scores and affected aggregates were updated on 2026-09-16 using task-wide global retrieval (#9).

mmeb_v3_eval/ contains the MMEB-v3 evaluation code used to produce our reported numbers. It is the official TIGER-AI-Lab/VLM2Vec pipeline with a minimal diff: multi-node multi-GPU inference (torchrun --nnodes=N), a wemm_embedding backbone implementing our preprocessing and batched inference, dataset instructions aligned with the released model, and 64-frame video sampling. Data download, single-node and multi-node commands are documented in mmeb_v3_eval/README.md.

cd mmeb_v3_eval
DATA_ROOT=/path/to/MMEB-V3 bash scripts/download_data.sh
MODEL_PATH=tencent/WeMM-Embedding-2B DATA_BASEDIR=/path/to/MMEB-V3 \
OUTPUT_DIR=exps/wemm_embedding bash scripts/run_eval.sh

If you find this repository useful, please consider giving a star ⭐ and citation

@article{wemm-embedding,
      title={WeMM-Embedding: WeChat Multi-Modal Embedding Technical Report}, 
      author={Junjie Zhou and Ke Mei and Lei Li and Tianyi Wang and Fengyun Rao and Jing Lyu},
      year={2026},
      eprint={2608.24053},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2608.24053}, 
}

Unless otherwise noted, Tencent-authored code in this repository is released under the Apache License 2.0.

Third-party components retain their original licenses and copyright notices. Please review the corresponding source files before use.