Now that my Zettelkasten has over a thousand notes, I’d like to try to quite literally create the experience of a conversation with my second brain. The interface should be conversational rather than search queries. It should draw from the knowledge in my notes and respond in natural language. Finally, it should be useful in helping me make connections between ideas I hadn’t thought of before.
How should I build it?
You can find all the code for these experiments in org-ai repo on GitHub.
The first thing I tried was to build a model using txtai trained on my notes. This let me search for similarities between notes with a very simple shell script.
The results are decent for a search-like experience that does more than simple keyword matching. For example, searching for “semantic search” gives results for full text search and SEO (the latter doesn’t mention the word “search” at all).
$ Enter search query:
semantic search
-----------------------
SQLite can also be used for full text search using the FTS5 plugin making it a portable static site search engine like [[id:3D014256-4844-4591-8F9D-7878A1993210][stork-search]] for significantly larger datasets.
-----------------------
There are three kinds of SEO: technical, programmatic, and editorial. Technical SEO is about setting up the right infrastructure for pages including page speed, internal link architecture, redirects, and so on. Programmatic SEO generates pages automatically from a database (e.g. a page for each address on a real estate website) and sometimes comes from UGC, user generated content. Editorial SEO is content written by a person about a set of topics to drive traffic around relevant keywords.
-----------------------
[[https://stork-search.net][Stork Search]] is a full text search engine written in [[id:8F056F38-B79C-4D79-B10E-204D7A7952BF][Rust]] that compiles to [[id:515D33F5-7C7A-469A-89AB-5CD8E489AEEF][WebAssembly]]. That makes it compatible with static websites like notes.alexkehayias.com. The file size is relatively small (less than 1MB) for an index with hundreds of notes and the search results are fast.
While this does tick the box for helping to make connections (I hadn’t thought about how semantic search is also SEO), it’s not conversational and doesn’t really draw from my notes so much as cite the paragraphs with the highest relevance.
Next, I’ll try to train GPT-3 by OpenAI on my notes to see if I can get close to a ChatGPT experience.