I've noticed a shift in how I consume my token quotas. I no longer spend as much asking a model to spit out code, but rather to manipulate knowledge. More specifically: using LLMs to build and maintain personal knowledge bases on topics I'm studying and researching.
Recent models are surprisingly good at this. They treat text, markdown, and images as if they were Meccano pieces.
The basic concept: From RAG to the LLM as a librarian
When the industry talks about "chatting with your documents," they usually sell you a RAG (Retrieval-Augmented Generation) architecture. This typically involves chunking PDFs, calculating embeddings, stuffing them into a vector database, and crossing your fingers that semantic search returns the right snippets to the model.
Here I'm proposing something rawer, but which I control a hundred percent. Instead of hiding knowledge in vectors unreadable to a human, I force the LLM to read raw documents and write a plain text wiki (.md files). The model acts as a tireless librarian that reads, summarizes, catalogs, and links concepts in a visible folder structure. I only consume the final result.
The assembly line
The process starts with pretty basic data ingestion. I dump source documents into a raw/ directory. We're talking articles, papers, repos, datasets, and images. To capture web reads, I use the Obsidian Web Clipper extension and a keyboard shortcut to download images to the local disk. This way the model has quick access to visual references.
From there, a script asks the LLM to "compile" the wiki incrementally. The agent reads the raw/ directory, generates summaries, extracts key concepts, and writes new interlinked .md articles. It also updates indexes and manages backlinks.
The LLM writes and maintains all the data. I rarely touch the wiki directly.
Obsidian as an IDE
You need a frontend to see and manage this entire file system. Obsidian fulfills that role perfectly. It's my knowledge IDE. From there I see the raw data, the compiled wiki, and any derived visualizations.
Sometimes I play with additional plugins, like Marp, to render some of those markdown files as slides directly in the editor.
Questions, answers, and maintenance
Here comes the fun part. When the wiki grows (mine has about 100 articles and 400,000 words in a recent project), I throw complex questions at the agent. It goes away, reads, researches the answers, and comes back.
I thought I would need a sophisticated RAG for this. I was wrong. The LLM is capable of self-maintaining index files and short summaries of all documents. It reads the important data quite easily at this scale, which I still consider manageable with long context windows.
Instead of having it spit text at me in a terminal, I ask for the output to be a new markdown file, a presentation, or a chart generated with matplotlib. All of this ends up back in Obsidian. And often, I end up injecting those answers back into the wiki. Every query adds up and fattens the database.
Linting the wiki
Sometimes I run "health check" scripts over the repo. I unleash the LLM on the files to look for inconsistent data, fill in gaps (using web search), or discover weird connections that could become new articles. It's a continuous cleanup process. Models are very good at suggesting what else we should be asking ourselves about our own work.
A homemade search engine
To be able to navigate all this when the LLM isn't active, I instinctively coded a small search engine over the wiki. Sometimes I use it from a simple web interface, but usually I pass it to the LLM via command line so it can use it as a tool in larger queries.
Practical example: Compiling a concept
The theory sounds good. Let's get down to business. Imagine I've dropped a dense article about the architecture of a new language model in raw/. My script invokes the LLM with a command something like this:
python wiki_compiler.py --input raw/nuevo-modelo.md --output wiki/
Under the hood, the LLM reads nuevo-modelo.md and, based on its system prompt, generates a summary and extracts concepts, creating something like this in wiki/conceptos/mecanismo-de-atencion.md:
# Attention Mechanism
Data source: [[nuevo-modelo]]
Compilation date: 2026-03-07
The described attention mechanism differs from the traditional one in that...
Then, it silently updates the wiki/indice.md file to include the new link.
The next step
As the repo grows, momentum demands taking it a step further: using that data to generate synthetic information and finetune a model. The idea is that the neural network carries the embedded knowledge in its weights, not just depending on the text you pass it.
There's room here to build a serious product. Right now, my setup is nothing more than a weird collection of scripts.