Now that my Zettelkasten has over a thousand notes, I’d like to try to quite literally create the experience of a conversation with my second brain. The AI interface should be conversational rather than search queries. It should draw from the knowledge in my notes and respond in natural language. Finally, it should be useful in helping me make connections between ideas I hadn’t thought of before.
How should I build it?
You can find all the code for these experiments in org-ai repo on GitHub.
I tried several ways to do this and I’m still tinkering with the best way to do it. Here are my attempts so far.
Using txtai and semantic search
The first thing I tried was to build a model using txtai trained on my notes. This let me search for similarities between notes with a very simple shell script.
The results are decent for a search-like experience that does more than simple keyword matching. For example, searching for “semantic search” gives results for full-text search and SEO (the latter doesn’t mention the word “search” at all).
$ Enter search query:
semantic search
-----------------------
SQLite can also be used for full-text search using the FTS5 plugin making it a portable static site search engine like [[id:3D014256-4844-4591-8F9D-7878A1993210][stork-search]] for significantly larger datasets.
-----------------------
There are three kinds of SEO: technical, programmatic, and editorial. Technical SEO is about setting up the right infrastructure for pages including page speed, internal link architecture, redirects, and so on. Programmatic SEO generates pages automatically from a database (e.g. a page for each address on a real estate website) and sometimes comes from UGC, user generated content. Editorial SEO is content written by a person about a set of topics to drive traffic around relevant keywords.
-----------------------
[[https://stork-search.net][Stork Search]] is a full text search engine written in [[id:8F056F38-B79C-4D79-B10E-204D7A7952BF][Rust]] that compiles to [[id:515D33F5-7C7A-469A-89AB-5CD8E489AEEF][WebAssembly]]. That makes it compatible with static websites like notes.alexkehayias.com. The file size is relatively small (less than 1MB) for an index with hundreds of notes and the search results are fast.
While this does tick the box for helping to make connections (I hadn’t thought about how semantic search is also SEO), it’s not conversational and doesn’t really draw from my notes so much as cite the paragraphs with the highest relevance.
Using OpenAI GPT models
Next, I’ll try to train GPT-3 by OpenAI on my notes to see if I can get close to a ChatGPT experience. I’ll use conditional generation where the prompt is the title of the note and the response the note.
I need to prepare the data by converting from org-mode to jsonl formatted for fine-tuning.
- Extract the body of the text
- Replace links (with what??)
- Insert stop words
The results of fine-tuning the ada model were pretty abysmal (total cost $0.40):
% openai api fine_tunes.create -t tuning_prepared.jsonl -m ada --suffix "org-ai"
% python3 src/chat.py
> What's the connection between remote work and prosperity?
Self-regulating machines that can be programmed. There are two urgency bounds (e.g. self-soothing, work-life balance) as a way of disallowing automation and thus bringing more efficiency to work. People tend to focus on the immediate, but this is also important right now with robotics being highly automated and human interaction becoming heavily digitized—after all, AI shouldn’t need people’s full attention to function properly because it will save us from ourselves!
Trying it with the davinci model (total cost $34.12):
openai api fine_tunes.create -t tuning_prepared.jsonl -m davinci --suffix "org-ai"
The results were not much better. The answer just didn’t make sense and I couldn’t rely on them to be accurate.
Using langchain, embeddings, and a vector database
Next, I tried langchain along with ChatGPT. Rather than try to fine-tune a model, I wanted to use ChatGPT’s superior comprehension skills alongside a vector database by doing a similarity search based on the query and then summarizing the results with a large language models. Langchain makes (you guessed it) chaining models together easy.
A document index is built up using langchain’s built-in tools for embeddings. I created an index based on each org-roam note by extracting the metadata and body then using OpenAI to generate the embeddings. The index is then generated using FAISS (a local vector DB that’s easy to use).
The results are so much better!
% python3 src/chat.py
> What's the connection between remote work and prosperity? Use my notes.
Remote work has the potential to increase economic opportunity by decoupling work and place. Additionally, a survey of 12,000 people found that remote work can increase employee happiness by up to 20%. However, remote work can also have negative impacts, such as decreasing tax revenue for cities and creating inequality in access to high-speed internet and a conducive work environment. Remote work can result in a 30% increase in hours worked and a 20% decrease in productivity, and that half of Millennials and Gen Z would consider quitting if employers don't allow remote work.
If we look closer at the logging you can see the exact notes that were looked at:
> Entering new AgentExecutor chain… { “action”: “Find Notes”, “action_input”: “What’s the connection between remote work and prosperity?” } Observation: Remote work has the potential to increase economic opportunity by decoupling work and place, according to a note on decoupling work and place (/notes/2023-04-06–14-16-44Z–decoupling_work_and_place.org). Additionally, a survey of 12,000 people found that remote work can increase employee happiness by up to 20%, as noted in a study on remote work and happiness (/notes/2022-05-24–15-49-02Z–remote_work_is_correlated_with_happiness.org). However, remote work can also have negative impacts, such as decreasing tax revenue for cities and creating inequality in access to high-speed internet and a conducive work environment, as noted in a note on remote work becoming a political issue (/notes/2022-07-11–14-08-52Z–remote_work_will_become_a_political_issue.org). A note on the market value of remote work (/notes/2021-08-12–13-56-31Z–what_is_the_market_value_of_working_remote.org) provides additional insights, including that remote work can result in a 30% increase in hours worked and a 20% decrease in productivity, and that half of Millennials and Gen Z would consider quitting if employers don’t allow remote work.
SOURCES:
- /notes/2023-04-06–14-16-44Z–decoupling_work_and_place.org
- /notes/2022-05-24–15-49-02Z–remote_work_is_correlated_with_happiness.org
- /notes/2022-07-11–14-08-52Z–remote_work_will_become_a_political_issue.org
- /notes/2021-08-12–13-56-31Z–what_is_the_market_value_of_working_remote.org
Links to this note
-
Today, most large language models are run by making requests over the network to a provider like OpenAI which has several disadvantages. You have to trust the entire chain of custody (e.g. network stack, the provider, their subprocessors etc.). It can be slow or flakey and therefore impractical for certain operations (e.g. voice inference, large volumes of text). It can also be expensive—providers are charging per API call and experiments can result in a surprising bill (my useless fine-tuned OpenAI model cost $36).
-
Summarization With Chain of Density
A recent paper studied text summarization improvements using a chain of density prompt. The prompt improves over vanilla GPT responses and is close to human summarizations in informativeness and readibility.
-
The true value of taking notes is not for recall later, it’s to give you the safety to forget. The act of note taking creates safety by providing insurance for ideas—if you need this information again you can find it later and therefore don’t need to worry about keeping it in your head.
-
I started building AI for notes to help me chat with my library of notes. The result of that exploration is org-ai—my one of one software that helps me remember what I’ve previously written, summarize information. Under the hood it uses vector-based similarity search and LLMs and agent-based AI to extract useful information from my zettelkasten in a chat-based interface.