Nearly every production RAG (Retrieval-Augmented Generation) pipeline starts the same way: a parser converts web pages and documents into plain text so they can be chunked and indexed. That first step, seemingly harmless, is usually the weakest link in the entire chain. Tables get flattened, layout is destroyed, and visual context disappears. According to the new research behind PixelRAG, that single conversion is responsible for the majority of wrong answers these systems produce.
The proposal is radical in its simplicity: if converting to text is the problem, stop converting to text. PixelRAG renders each page as a screenshot, indexes those images, and delivers the retrieved visual fragments directly to a vision-language model (VLM) that acts as the reader.
HTML parsing might be one of the most self-inflicted bottlenecks in web search. If fixing the parser is the question, perhaps the answer is to not use one at all.
PixelRAG pipeline diagram — available at the repository:
github.com/StarTrail-org/PixelRAG/blob/main/docs/assets/pipeline.png
01 The idea: treat the web the way humans do
When a person looks up a fact on a page, they don't mentally convert it to plain text — they look at it. They see the table, the header, the position of each figure. PixelRAG mimics that behavior. Instead of extracting text strings, it captures the page as it appears and lets a vision model interpret it.
Two pieces make this work. The first is the format shift itself: documents to images instead of documents to text, which preserves layout, structure, and tables. The second is an embedding model based on Qwen3-VL-Embedding, fine-tuned with LoRA on screenshot data, capable of converting page images into searchable vectors. The team found that a VLM, after continued pretraining on web screenshots, becomes a surprisingly strong visual retriever.
The most counterintuitive finding is that the advantage isn't limited to visual content. PixelRAG outperforms text RAG even on questions that could be answered with text alone, because the HTML-to-text conversion discards useful signals — layout, order, hierarchy — that the visual model preserves.
02 The numbers
The researchers tested PixelRAG on six benchmarks spanning factual Wikipedia questions, table-based queries, multimodal questions, and live news retrieval. It won all six — even on text-only tasks.
On SimpleQA it achieved 78.8% accuracy vs. 71.6% for the best text parser. The gap widens on structured table queries: 48.8% vs. 42.5%. The maximum improvement over text baselines reached +18.1%. All of this was measured on an index of 30 million tiles from screenshots covering the entirety of Wikipedia.
The team also reports that PixelRAG achieves higher accuracy than Google Search while maintaining a cost 2 to 4 times lower.
03 The economic argument for agents
Beyond accuracy, the strongest near-term use case is cost savings in AI agents. In tests, an agent using PixelRAG as its search engine consumed 3.6 million input tokens, compared to 37.5 million for text-based retrieval — roughly 10 times fewer. The reason is straightforward: a retrieved image gives the model just what it needs, instead of flooding the context with long text dumps.
And there's still room: image compression can trim that token budget by approximately a third more. For teams paying per token, that difference translates directly into more predictable and lower bills.
04 The fine print
The technique isn't free. To see the benefit, teams need models in the Qwen3-VL-4B class or above. Below that threshold, smaller models fall behind text retrieval by more than 12.5 percentage points. In other words: PixelRAG pays dividends when the reader has sufficient visual capacity; with tiny models, the classic text approach still wins.
There's also an implicit infrastructure cost — rendering, storing, and serving image indexes at scale isn't trivial — that each team will need to weigh against the inference token savings.
05 How it works inside
The repository describes PixelRAG as a general-purpose visual retrieval framework for building search engines from any document type — web pages, PDFs, or images. Wikipedia and its 8.28 million articles serve as the main test bench, but the design is content-agnostic. The architecture splits into five independently installable packages:
| Package | What it does |
|---|---|
| render | Converts documents into image tiles via Playwright/CDP and PDF. |
| embed | Transforms tiles into vectors and builds the FAISS index. |
| index | Orchestrates the full pipeline: source → ingest → embed → index. |
| serve | Exposes the search API over FAISS (FastAPI, CPU or GPU). |
| train | LoRA fine-tuning of the Qwen3-VL-Embedding model for web retrieval. |
Pre-trained adapters are published on Hugging Face, so retraining isn't required to use the model; the team also released the full training dataset so anyone can adapt other embedding backbones. The project is distributed under the Apache-2.0 license.
Try it in minutes
Searching over a pre-built Wikipedia index is a matter of installing the serve package, downloading the index, and launching the API:
# Install the search service uv sync --package pixelrag-serve # Launch the API pixelrag-serve --index-dir ./index --port 30001 # Query curl -X POST http://localhost:30001/search \ -H "Content-Type: application/json" \ -d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
Building your own index from local documents only requires a pixelrag.yaml file pointing to the source and embedding model, then running pixelrag-index build.
06 A plugin to give Claude Code eyes
One of the most practical details in the repository is a plugin for Claude Code that needs no MCP server or backend: it teaches the assistant to call pixelrag-render directly from Bash and read the resulting images. In practice, you just ask it to look at a page:
claude --plugin-dir ./plugin -p \
"take a screenshot of https://news.ycombinator.com and summarize the top stories"
The same approach works for visually reading a research paper or reviewing whether your own website looks correct, without going through any text parser.
07 Who's behind it
The work brings together researchers from UC Berkeley, Princeton, EPFL, and Databricks. Contributors include Yichuan Wang, Zirui Wang, and Matei Zaharia, CTO of Databricks and co-creator of Apache Spark. The code is open on GitHub and the corresponding paper was published on arXiv with identifier 2506.05209.
Beyond the concrete numbers, the underlying thesis is what's worth retaining: for years we've taken for granted that the first step of a RAG system should be converting everything to text. PixelRAG challenges that premise and suggests that sometimes the most efficient way for a machine to understand a page is the same way we humans do: by looking at it.
