vLLM Embedder - Agno

The vLLM Embedder provides high-performance embedding inference with support for both local and remote deployment modes. All models are downloaded from HuggingFace.

Usage

Local Mode

You can load local models directly using the vLLM library, without any need to host a model on a server.

vllm_embedder.py

from agno.knowledge.embedder.vllm import VLLMEmbedder
from agno.knowledge.knowledge import Knowledge
from agno.vectordb.pgvector import PgVector

# Get embeddings directly
embeddings = VLLMEmbedder(
    id="intfloat/e5-mistral-7b-instruct",
    dimensions=4096,
    enforce_eager=True,
    vllm_kwargs={
        "disable_sliding_window": True,
        "max_model_len": 4096,
    },
).get_embedding("The quick brown fox jumps over the lazy dog.")

print(f"Embeddings: {embeddings[:5]}")
print(f"Dimensions: {len(embeddings)}")

# Use with Knowledge
knowledge = Knowledge(
    vector_db=PgVector(
        db_url="postgresql+psycopg://ai:ai@localhost:5532/ai",
        table_name="vllm_embeddings",
        embedder=VLLMEmbedder(
            id="intfloat/e5-mistral-7b-instruct",
            dimensions=4096,
            enforce_eager=True,
            vllm_kwargs={
                "disable_sliding_window": True,
                "max_model_len": 4096,
            },
        ),
    ),
    max_results=2,
)

Remote Mode

You can connect to a running vLLM server via an OpenAI-compatible API.

vllm_embedder_remote.py

# Remote mode (for production deployments)
knowledge_remote = Knowledge(
    vector_db=PgVector(
        db_url="postgresql+psycopg://ai:ai@localhost:5532/ai",
        table_name="vllm_embeddings_remote",
        embedder=VLLMEmbedder(
            id="intfloat/e5-mistral-7b-instruct",
            dimensions=4096,
            base_url="http://localhost:8000/v1",  # Example endpoint for local development
            api_key="your-api-key",  # Optional
        ),
    ),
    max_results=2,
)

Params

Parameter	Type	Default	Description
`id`	`str`	`"intfloat/e5-mistral-7b-instruct"`	Model identifier (HuggingFace model name)
`dimensions`	`int`	`4096`	Embedding vector dimensions
`base_url`	`Optional[str]`	`None`	Remote vLLM server URL (enables remote mode)
`api_key`	`Optional[str]`	`getenv("VLLM_API_KEY")`	API key for remote server authentication
`enable_batch`	`bool`	`False`	Enable batch processing for multiple texts
`batch_size`	`int`	`10`	Number of texts to process per batch
`enforce_eager`	`bool`	`True`	Use eager execution mode (local mode)
`vllm_kwargs`	`Optional[Dict[str, Any]]`	`None`	Additional vLLM engine parameters (local mode)
`request_params`	`Optional[Dict[str, Any]]`	`None`	Additional request parameters (remote mode)
`client_params`	`Optional[Dict[str, Any]]`	`None`	OpenAI client configuration (remote mode)

Developer Resources

View Cookbook