Reading pull requests is one of those tasks that scales terribly with team size. When you're a team of 3, you can read every PR. At 10 people, you're skimming. At 50, you've given up and just approve things that look reasonable. The information is all there in the diff, but extracting the intent, the tradeoffs, the "why" behind the changes requires reading every file, understanding the context, and holding the whole thing in your head.
PullScan attacks this problem with AI. Point it at any GitHub repository and it generates comprehensive summaries of pull requests and individual commits. But the real power is the RAG-powered chat: it indexes the codebase and lets you ask questions like "what changed in the authentication module this week?" or "why was this function refactored?" and get context-aware answers grounded in the actual code.
The architecture is a Next.js 15 frontend with a backend powered by LangBase for the AI orchestration and UpStash for caching. LangBase handles the RAG pipeline, chunking the codebase, embedding it, and retrieving relevant context for each query. Google AI Studio provides the language model. The separation of concerns here matters: the frontend is just a thin UI layer, all the intelligence lives in the backend services.
This is the kind of tool that gets more valuable as the codebase grows. For a small project, you can just read the PRs. But for a large, active repository with dozens of contributors, having an AI that can synthesize changes across multiple PRs and explain them in plain English is genuinely useful.
Built with
Links
Features
- AI-powered PR summaries, comprehensive breakdowns of pull request changes without reading every line
- Commit-level summaries, understand individual commits with concise, AI-generated explanations
- RAG-powered codebase chat, ask questions about the code and get context-aware answers grounded in the actual repository
- LangBase integration, production-grade RAG pipeline for chunking, embedding, and retrieval
- Caching with UpStash, fast responses for repeated queries without redundant API calls
Challenges
The hardest problem in building PullScan was getting the RAG pipeline right. Naive chunking of source code produces terrible results, you end up with chunks that split functions in half or separate a class from its methods. The solution was to use AST-aware chunking that respects code boundaries, combined with metadata enrichment (file path, function name, class context) so the retrieval step can find semantically relevant code even when the query uses different terminology than the source.
What I learned
PullScan taught me that the gap between "AI demo" and "AI tool people actually use" is mostly about retrieval quality. The language model is the easy part, it's good at summarizing once you give it the right context. The hard part is finding the right context in a large codebase. RAG is not just "embed and retrieve", it's a whole pipeline of chunking strategies, metadata enrichment, re-ranking, and context window management that determines whether the output is useful or hallucinated.