GlyphScholar · Durgesh Gowda C

Brief

A PDF is where good information goes into hiding. Ctrl+F finds a word, never an idea; a copied table arrives as one long unreadable string; and the one diagram that actually explains the finding is invisible to almost every RAG pipeline ever built, because most of them were designed to read words, not pages.

GlyphScholar exists to make that someone else's problem — specifically, a machine's. It parses a document the way an attentive reader would: layout, OCR, tables, formulas, and page imagery, all at once. Everything gets indexed, and a plain question comes back with an answer traceable to the exact page it came from, rather than a confident-sounding guess.

A Chainlit interface handles the conversation, mounted inside a FastAPI backend, with a Next.js portal quietly taking care of the parts every app needs and nobody particularly enjoys building twice — auth, the landing page, that sort of thing. It's live at glyphscholar.vercel.app, and at any given moment is probably fielding more questions than its author has actually asked it.

Don’t take my word for it — the receipts are on GitHub

My Contribution

Built the ingestion pipeline end to end: PDFs go through MinerU for layout-aware parsing — the difference between recognizing a real table and three paragraphs pretending to be one — splitting into text chunks and visual crops (figures, tables, equations).

Added a caption-fallback step for crops MinerU leaves uncaptioned, generating one via Qwen3-VL rather than shipping a mute image into the index.

Designed the retrieval layer to fire three paths per question — dense vector search, BM25 sparse, and pixel/image retrieval — on the premise that no single retrieval method is right 100% of the time, then reranks the merged candidates before anything reaches the model.

Wired a raster fallback into the query path: when reranked page coverage still looks ambiguous, the pipeline renders the full page rather than answering from a guess about which chunk mattered.

Made the inference backend swappable on purpose — Ollama for local development, Modal’s serverless GPUs for actually serving people — so testing a one-line prompt fix doesn’t require waiting on a cold GPU start.

Instrumented the whole request path with Opik tracing, so a wrong answer is something you can actually go inspect, not a shrug.

Shipped the entire thing — portal, backend, retrieval, inference — without a single paid line item. Every service in the stack has a free tier generous enough to actually launch on, which turned out to be less of a budget constraint and more of a personal challenge.

System Architecture

Portal

The Next.js frontend — auth, landing page, the part with buttons.

Backend

A FastAPI app mounting the Chainlit app, handling auth, uploads, and ingestion.

Chainlit app

The chat itself, with session history kept in the app’s own Postgres (Neon in production) while Supabase minds the logins.

Ingestion

MinerU parsing → text chunks + visual crops → embeddings (Ollama locally, Modal in production) → Postgres/pgvector.

Retrieval

Dense + BM25 + pixel retrieval, merged and reranked before a single token reaches the answer model.

Modal services

Embedding, reranking, and answer models deployed as serverless GPU functions — scale to zero when nobody’s asking questions.

Technical Challenges

One retrieval method isn’t enough

Dense vector search alone misses lexical matches; BM25 alone misses semantic ones; neither sees a diagram. Running all three in parallel and reranking the merged set was the only way to stop losing whichever kind of match a single method is bad at.

Captionless crops going mute

A table or figure with no generated caption is invisible to text-based retrieval no matter how good the image embedding is. Solved with a Qwen3-VL caption-fallback step at ingestion time, so nothing enters the index silently.

Dev-cost vs. production-latency

Running everything on Modal’s GPUs during development would work, but burns through the free compute tier fast on typo fixes. Keeping Ollama as a local, no-GPU-bill inference path let iteration stay cheap without touching the production path.

Takeaways

The embedding model and the reranker are both well-understood, mostly-trustworthy-out-of-the-box components. The genuinely hard part was everything built around them — deciding when a caption needs generating, when page coverage is too thin to trust, when to burst to a GPU rather than wait on a local one. RAG gets pitched as "embed it, retrieve it, done"; the distance between that pitch and an answer actually worth trusting turned out to be almost entirely fallback logic.

The repo also keeps a public, unglamorous roadmap — tests not yet written, no CI, a setup process that's currently more "Homebrew and prayer" than one-command install — on the theory that a project honest about what it doesn't do yet is more trustworthy than one quietly pretending to be finished. Software rarely is; it just reaches a point where it's willing to say what's next.