Embedding strategies for business data - why generic models fall short

General-purpose embedding models cost accuracy.

I know this because I’ve embedded everything from customer invoices to support tickets at Tallyfy. The pattern repeats: companies start with OpenAI or Cohere embeddings, get mediocre results, then wonder why their search returns irrelevant documents 40% of the time. It’s genuinely frustrating to watch teams blame their data when the actual issue is a model that was never trained on anything resembling it.

The problem isn’t the technology. It’s the mismatch between your data and what the model understands.

The mismatch problem

General-purpose models train on broad internet data. Wikipedia. News articles. GitHub repos. They get good at understanding common language patterns.

But your business doesn’t speak common language.

You have invoice numbers that mean something specific in your system. Product codes with internal logic. Customer support tickets using jargon only your team understands. Contract clauses with legal precision that matters in ways a general model simply can’t recognize.

There’s a paper testing embedding models on financial data that found something worth sitting with. State-of-the-art models struggled significantly on specialized domains. Performance on general benchmarks didn’t predict performance on your actual data at all.

That’s what makes this problem hard to catch. You can run every benchmark test and still walk away thinking your embeddings are fine. Then they hit real business data and fall apart.

Domain-specific models and fine-tuning

Testing on SEC filings data showed Voyage finance-2, a specialized model, hit 54% accuracy. OpenAI’s general model? 38.5%.

That’s roughly a 40% improvement from using embeddings trained on similar data. More recently, Voyage AI’s v4 series leads domain-specific benchmarks, with voyage-4-large outperforming OpenAI v3 Large by 14%, Cohere Embed v4 by 8%, and Gemini Embedding 001 by 4%.

The gap widened on specific query types. Direct financial questions saw specialized models reach 63.75% accuracy versus 40% for generic alternatives. Even on ambiguous questions where general knowledge might help, domain-specific embeddings held their edge.

Why such a difference? Specialized models learn actual relationships in your domain. A generic model sees invoice numbers as random strings. A finance-specific model recognizes patterns in how those numbers relate to transactions, dates, and entities. The semantic distance between concepts reflects real-world business logic rather than surface-level word similarity.

You have three paths: use what already exists, fine-tune something close to your domain, or train from scratch. Most companies should start with fine-tuning, and I think that’s probably the right call for 90% of teams reading this. Off-the-shelf embeddings work when your data looks like internet text. Fine-tuning gets you most of the benefit with a fraction of the effort. LlamaIndex published benchmarks showing 7-41% performance boosts with just 1,000-5,000 examples. Platforms like Google Cloud and Databricks make this process straightforward. Tools like LlamaIndex can generate synthetic training data from your own documents, which makes the whole thing easier than it used to be.

Training from scratch makes sense only when you have massive proprietary datasets and a truly unique domain. Genomics research. Highly specialized manufacturing processes. Not most businesses.

Open-source embedding models deserve a look too. BGE-M3 supports dense, lexical, and ColBERT retrieval simultaneously across 100+ languages with 8,192 token context. E5-Mistral-7B matches commercial offerings on many benchmarks. Newer contenders like Qwen3-Embedding and Google’s EmbeddingGemma-300M rival much larger models. For companies with privacy requirements or high embedding volumes, self-hosted open-source models deliver both compliance and real cost savings.

Chunking, metadata, and retrieval quality

The best embeddings mean nothing if you chunk your data wrong.

Start with 512 tokens per chunk and 50-100 tokens of overlap. Pinecone’s chunking strategy guide confirms this balances context with precision for most business data. Treat it as a starting point, not a fixed answer.

Your content type drives the optimal approach. NVIDIA benchmarks found page-level chunking achieves the highest accuracy (64.8%) with the lowest variance across document types, particularly for PDFs and formatted documents. Financial documents with dense information need smaller chunks around 250 tokens. Long-form analysis where context matters should push toward 1,024 tokens to maintain coherent meaning.

The overlap prevents you from cutting sentences or concepts in half. When one chunk ends mid-thought and the next begins with a fragment, retrieval suffers.

Metadata makes the difference between adequate and great retrieval. Effective metadata design means keeping things simple and standardized. Add document type, creation date, author, department, topic tags. Whatever helps filter before semantic search even runs.

Teams commonly tag support tickets with product area, severity, and resolution status. When someone searches for billing problems, metadata filtering narrows to relevant tickets first, then semantic search runs on a smaller, cleaner set. That approach can cut response time dramatically when done well.

Keep metadata lean, though. Too many tags slow processing and increase storage costs. Stick to fields that genuinely improve retrieval.

The case for hybrid search

Pure vector search isn’t enough anymore.

Hybrid search, which combines dense vector similarity with traditional keyword filtering, consistently achieves higher precision than vector search alone. This matters particularly for technical queries requiring exact terminology matches.

The approach combines BM25 keyword search with dense vectors. When someone searches for a specific product code or technical term, BM25 catches the exact match while vectors handle semantic similarity. Combine results using Reciprocal Rank Fusion.

Reranking reorders initial results so the most relevant information surfaces first. Without a reranker, cosine similarity rewards proximity, not usefulness. Cross-encoder reranking feeds the user query and each candidate chunk into a transformer model that scores how well they match. Accurate. Adds some latency. Worth it.

These techniques have become defaults in production systems. Anthropic’s Contextual Retrieval approach, where an LLM prepends context to each chunk before embedding, combined with hybrid search and reranking, achieves up to 67% reduction in retrieval failures. You can implement hybrid search confidently without extensive benchmarking. The improvement holds across most use cases.

Which vector database fits your situation

Your embedding strategy needs somewhere to live. The choice matters more than most teams expect.

Comparing the major options: Pinecone delivers production-ready infrastructure with consistent sub-50ms latencies at billion-scale. Their Dedicated Read Nodes, launched in December 2025, sustain 600 queries per second with P50 latency of 45ms and P99 of 96ms. The vector database market has grown rapidly, with pricing shifting from per-pod to serverless consumption.

Weaviate handles hybrid search, combining traditional database queries with vector operations. Version 1.34 added flat index support and rotational quantization. When you need both exact matches and semantic search, or when you’re working with multiple data types simultaneously, Weaviate makes sense. Companies running on-premise for compliance reasons tend to pick this.

Chroma works well for prototyping and smaller teams. Version 1.4.1 delivers fast search performance for moderate-scale datasets. Simple Python integration. Minimal setup. Perfect when you’re testing approaches before committing to production infrastructure.

Qdrant excels at complex metadata filtering with strong Rust performance and first-class multitenancy. It’s now SOC 2 Type II certified and HIPAA-ready for enterprise deployments. Milvus 2.6.x, which recently went GA, handles billion-scale deployments with tiered storage that reduces costs by 87% while maintaining sub-10ms latency.

Scale and budget drive the choice. Smaller teams benefit from Chroma’s simplicity. Enterprise applications with strict reliability requirements justify Pinecone’s costs. Hybrid search needs or on-premise requirements point to Weaviate. Complex filtering with cost sensitivity favors Qdrant. Billion-scale enterprise deployments lean toward Milvus/Zilliz.

The ROI calculation is fairly direct. If your team spends 2 hours daily searching for information and you reduce that by 30%, the productivity gains pay for infrastructure quickly. Some companies report substantial returns in the first year, though actual results depend heavily on implementation quality and how well the team actually adopts it.

Map your data types first. Financial records? Legal documents? Technical specifications? Each has different optimal approaches. Grab a few hundred representative documents, try both general-purpose and specialized embeddings, then measure retrieval accuracy on queries your team actually runs. Less than 60% accuracy with generic embeddings means specialization will help. Already hitting 80%+? You might be fine with what you have.

Chunk size needs real testing, not guesswork. Start at 512 tokens, then try 256 and 1,024. Your data will tell you what works.

Deploy incrementally. Pick one high-value use case, optimize embeddings for that workflow, measure improvement, then expand. Don’t rebuild everything at once.

One consideration that often gets overlooked: security. OWASP added Vector and Embedding Weaknesses as a new Top 10 entry in 2025. Embedding inversion attacks can reconstruct original text from vectors. Adversarial embeddings can poison search results at a mathematical level. Enforce access control at the retrieval layer, tag embeddings with access control metadata, and verify user permissions before returning results.

The companies getting semantic search right aren’t the ones with the fanciest models. They’re the ones who matched their embedding approach to their actual data.