Here are the most useful metrics I’ve found for running a startup. These should be measured and reviewed regularly. The metrics that matter most the ones that raise questions and drive useful discussions. Try to avoid difficult to calculate metrics that are hard to build an intuition about (people don’t expect much from simple ideas).

MRR growth: You need to grow 6% monthly in order double revenue in a year. Of course, time is a factor, so venture-backed businesses need to be mindful of the time to the first $1MM of ARR.

Churned revenue: Monthly revenue churn is a headwind that slows growth and makes it difficult to sustain growth. High churn is also a signal that something else is going on which needs immediate attention.

Runway: By calculating burn rate each month you can estimate how much time you have before the business runs out of cash. Many important decisions need to be made with this in mind such as layoffs, seeking an aquisition, or spending to grow faster.

GTM efficiency: Net magic number breaks my rule a bit about avoiding complicated metrics, but it’s easy to reason about once it’s measured—for each dollar spent on marketing, how much revenue is it expected to return. A number less than 1.0 is bad (magic vending machine business model), a positive number greater than 1.0 is good, and multiples means you should spend more to grow faster.

Engineering velocity: By far the easiest number to track for engineering health is the number of commits to the main branch every week. There shouldn’t be a target set (Goodhart’s law) but if the number goes down (or up), it’s useful for the team to ask what’s going on so they can make adjustments if needed. It should also generally increase as more engineers are added to the team although sub-linearly as the company and codebase gets more complicated.

Defect rate: This will depend on the product, but having a way to know if delivery of the product is getting better or worse will help spot issues early. I’ve seen this measured as the number of bugs reported or count of a specific process that fails.

Support SLA met: No one wants to wait a long time when they need help. Tracking support SLAs weekly gives early warning if the team is falling behind or if there is a larger issue brewing that needs immediate attention.

Onboarding cycle time: Time to value is important because attention spans are short and failure to onboard is the leading cause of customer churn. This is even more important if there is an upfront implementation needed to adopt the product (what does customer success do at a startup?).

Talk to customers: I was tempted to put CSAT on this list, but it doesn’t quite capture whether or not you are doing a good job solving your customer’s problem. If a lagging indicator is churn, then a leading indicator is simply talking to your customers regularly.

This note does not have a description yet.

As it stands today, LLM applications have noticeable latency but much of the latency is output-size bound rather than input-size bound. That means the amount of text that goes into a prompt does not matter.

The way transformers and attention layers work, LLMs are basically predicting each successive word in the response so it makes sense that a longer response will take longer regardless of the input size. For example, a short prompt, “write me a 5 paragraph essay on computers”, will have higher latency than “write me one sentence about computers”.

However, when using techniques like RAG, the input might be used to fetch related documents that could introduce several more operations (including more LLM prompts) which would have an impact on overall latency.

Read What We’ve Learned From A Year of Building with LLMs.

See also:

• LLM applications need creativity
• Chatbots lack affordances
• How to build an intuition of what AI can do

Use the following prompt to return an org-ql s-expression based on the user’s input. This can be used along with calling org-ql from the command line to make an LLM tool that can query org-mode tasks and headings.

You are an AI assistant designed to help convert natural language questions into org-ql queries for use with Org mode in Emacs. org-ql queries are used to filter and search for specific entries in Org files based on various criteria like tags, properties, and timestamps.

Instructions:

1. Read the user’s question carefully.
2. Identify the key criteria mentioned in the question (e.g., tags, properties, deadlines, scheduled dates, priorities, etc.).
3. Construct an appropriate org-ql query that reflects the user’s criteria.
4. Only return the org-ql query with no other text or explanation

Examples:

User Question:

“Find all tasks that are due this week and have the tag ‘work’.”

Converted org-ql Query:

(org-ql-select (org-agenda-files) ‘(and (deadline :from today :to +7d) (tags “work”)) :action ‘(org-get-heading t t))

User Question:

“Show me notes with the tag ‘meeting’ that were created last month.”

Converted org-ql Query:

(org-ql-select (org-agenda-files) ‘(and (tags “meeting”) (timestamp :from -1m :to today)) :action ‘(org-get-heading t t))

User Question:

“List all tasks with priority ‘A’ that are scheduled for tomorrow.”

Converted org-ql Query:

(org-ql-select (org-agenda-files) ‘(and (priority “A”) (scheduled :on +1d)) :action ‘(org-get-heading t t))

User Question:

“Find all entries tagged ‘home’ that have a deadline next week.”

Converted org-ql Query:

(org-ql-select (org-agenda-files) ‘(and (tags “home”) (deadline :from +7d :to +14d)) :action ‘(org-get-heading t t))

User Question:

“Show tasks that are not done and have the tag ‘project’.”

Converted org-ql Query:

(org-ql-select (org-agenda-files) ‘(and (todo) (tags “project”)) :action ‘(org-get-heading t t))

User Question:

“Find all tasks with a priority of ‘B’ that were created this month.”

Converted org-ql Query:

(org-ql-select (org-agenda-files) ‘(and (priority “B”) (timestamp :from -1m :to today)) :action ‘(org-get-heading t t))

User Question:

{question}

Converted org-ql Query:

Most of my tasks and projects are organized using org-mode. I was looking for a way to query them from an LLM and, rather than recreate an index and a database, I can use what I normally use, org-ql using emacs in batch mode.

Here’s how to run org-ql from the command line.

emacs --batch -l $HOME/.emacs.d/init.el --eval "(message (with-output-to-string (princ (mapconcat 'identity (org-ql-select (org-agenda-files) '(todo \"WAITING\") :action '(org-get-heading t t)) \"\n\"))))"

Each matching result will be printed out on a newline. This can then be used for further processing by capturing stdout.

A recent study from Stanford found that LLM’s (GPT-4) and RAG-based AI tools (Lexis+ AI, Westlaw AI-Assisted Research, Ask Practical Law AI) hallucinate answers 16% to 40% of the time in benchmarking queries. GPT-4 had the worst performance while RAG-based AI tools did slightly better.

Hallucinations in this study wre defined as a response that is incorrect (factually inaccurate in some way) and misgrounded (propositions are cited but the source does not match the claim).

Read the preprint Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools.

The way to make everything in your codebase twice as complicated is to make data nullable.

foo: str | None

It starts out innocuous but the combination of nullables causes the state space of a program to explode, introducing hard to find bugs and making it hard to maintain over time due to high cyclomatic complexity.

What should you do about it?

By default, data should be non-nullable. Introduce nullables only if you have to (though it’s an indicator the code should be refactored to make more states invariant).

For database schema changes, maybe an optional column needs to be introduced because it can’t be backfilled right away. Great, make it optional, complete the data migration, then migrate the schema to make the column non-nullable.

See also:

• Novice software engineers tend to optimize code by decreasing duplication
• Compatibility is leverage
• Duplication over abstraction

While I don’t write code every day, I find it incredibly convenient.

At work, I can run projects locally to check progress on a project, fix a bug, or better understand engineering challenges. I can customize my tools to my liking like a spreadsheet that calls out to OpenAI. I can explore a problem or product area on my own and learn what’s possible.

Without knowing how to code, I would need to ask others to do things for me which increases the activation energy to get things done. Convenience is king.

Sometimes using Emacs you want to know what variables are being set in the current buffer. C-h v lists all the variables, including buffer-local variables in the minibuffer which you can then search and refine just like any other minibuffer session. This is really handy when you customized some settings using hooks and want to verify they took. For example, setting the org-download directory based on filetags.

See also:

• The Emacs completion stack also helps make everything more discoverable
• Emacs is the ultimate editor building material

I recently needed to change the behavior of a library (org-download) based on the tags of the current buffer. Most of the answers I found online are incorrect (and so was ChatGPT) for Org version 9.5.

Here’s how to get the filetags of an org-mode buffer or file programatically:

(org-collect-keywords '("FILETAGS"))

I used this to write a function to check whether or not the org-mode file was private or not.

(defun my/org-file-is-private ()
    (let ((tags (car (cdr (car (org-collect-keywords '("FILETAGS")))))))
      (if (stringp tags)
          (member "private" (split-string tags))
        nil)))

Many vector databases can find the top k most similar results to a query but are unable to sort by other document metadata. This is a pretty severe limitation for building LLM applications, especially for ones where time is dimension (meetings, calendars, task lists, etc.). For example, retrieving the 10 most similar results to the phrase “team meeting notes” but not being able to retrieve the team meeting notes from the last month.

To work around the limitation of sorting vector query results, the most common suggestion is to pre-process a query or run multiple queries with different strategies at once and then combine the results. For example, first query for documents with a string similar to “meeting notes” (maybe using fuzzy, full-text search) filtered by date, then constrain a vector query search based on the collection of filtered records, and then pass the results to an LLM to synthesize the results and respond.

See also:

• LLMs have difficulty with dates and time
• Chatbots lack affordances so the ability to gracefully handle filtering and sorting is even more important

Alfred does not have a mobile application that can make snippets available from your desktop to your phone. Here’s how to use Shortcuts on iOS to do that.

First, export your snippets in Alfred by going to Features -> Snippets -> right click the collection and select Export and save the file to Dropbox.

Next, create a shortcut on your phone with the following.

(Not pictured in the screenshot, add a step to copy the value to your clipboard or do something else with the value)

Now when you run the shortcut, you’ll be able access your snippets on your phone.

Make sure to replace the file path with the path to your snippets file stored in Dropbox. This also works if you store the file someplace else because the file that Alfred stores for snippets is a zipped json file.

One of the difficult parts of applying AI to existing processes and products is that people aren’t calibrated on what generative AI can and can’t do. This leads to both wild ideas that are not possible and missed opportunities to automate seemingly difficult work that is possible.

How do you build an intuition of what AI can do?

There’s no substitute for trying out these tools on your own. Not just fiddling around with ChatGPT but actually trying to solve a real problem using LLMs. Surprise plus memory equals learning, so the more things you try, the more surprises you will encounter, and the more you can learn about what it can and can’t do.

Speaking for myself, I’ve been quietly working on one of one software that helps me explore the latest AI advancements in a safe place. I built org-ai to try out RAG techniques over a large number of documents (my public and private notes). I then applied what I learned to a prototype at work using the same techniques. From there I spotted problems with prompts and retrieving relevant documents that helped me understand some limitations of vector databases and RAG.

I’m documenting a list of prompts that I can reuse for various problems. I wrote a script to use AI in a Google spreadsheet and wrote a prompt to detect specific kinds of spam signups. I wrote a prompt to help summarize the last week and communicate it to the team based on previous examples to match my writing style.

Once you build up the habit of applying AI to real problems it starts to become a default starting point for new solutions. Not everything can or should be solved with AI but as a way to rapidly explore the space and gain real-world experience, learning by doing is unmatched.

Writing a specification with sufficient detail to know exactly what software one should build is as much work as writing the code itself. In many cases, fully specifying the work beforehand is not possible because we don’t know enough about the problem or the domain to begin with. This is why our codebases are always in a state of flux and never complete systems.

This circular problem means most software development should be treated iteratively and course correct as more information becomes available. It’s worth investing more time in techniques for dealing with uncertainty rather than trying to figure out how to make better specification.

Read What Factors Explain the Nature of Software?

When asked to draw a random number between 1 and 10 and any color, most people choose the number seven and the color blue. Some potential explanations is that 7 is a lucky number in many cultures and blue is pleasant (or at least not unpleasant or unlucky).

See also:

• Blue-seven could be useful in finding a Schelling point
• The frequency range of hearing is probably the same in the universe

I want to be able to use generative AI in spreadsheets to solve unique problems. I want to call OpenAI from a cell that passes in a prompt and a value from a column then returns an answer I can easily parse.

Thankfully, you can do this in Google Sheets and Apps Script!

First, create the script using App Script. In your spreadsheet go to Extensions -> Apps Script then add the script.

function AI(prompt, value) {
  // Replace YOUR_API_KEY with your actual OpenAI API key
  const apiKey = 'YOUR_KEY';
  const apiURL = 'https://api.openai.com/v1/chat/completions';

  const payload = JSON.stringify({
    "model": "gpt-4-turbo",
    "frequency_penalty": 0,
    "presence_penalty": 0,
    "temperature": 0,
    "messages": [
      {"role": "system", "content": `${prompt}`},
      {"role": "user", "content": `${value}`}
    ]
  });

  const options = {
    "method": "post",
    "contentType": "application/json",
    "payload": payload,
    "headers": {
      "Authorization": "Bearer " + apiKey
    },
    "muteHttpExceptions": false
  };

  try {
    const response = UrlFetchApp.fetch(apiURL, options);
    const json = JSON.parse(response.getContentText());
    const resultText = json.choices[0].message.content;
    return resultText;
  } catch (e) {
    Logger.log(e.toString());
  }
}

Finally, call the App Script from a cell like you would any formula =AI(PROMPT, COLUMN).

(Based on this thread in the OpenAI forum)

See also:

• Building templates to use generative AI in tools you already use (like spreadsheets) is a another good way to get ready for AI
• AI is the next great interop layer just like spreadsheets

When selling a B2B SaaS product to a wide range of company sizes (e.g. SMB, mid-market, enterprise), you generally want smaller companies to pay less and larger companies to pay more. This makes pricing difficult—make it too one-size-fits-all and you can accidentally price out the smaller customers, make it too a la carte, and all customers will find your pricing confusing.

What are some pricing techniques you can use when you have a wide range of business sizes to sell to?

Pricing tiers using caps

If you design pricing plans around the expected size of customer, you can make it clear who should be buying which plan and make a natural upgrade path. One way to do this is to include caps around one of the variable like the number of employees. For example, plan A is for up to 50 employees, plan B up to 500 employees, and so on.

Simple and advanced

Providing versions of the same feature can be used to appeal to different size customers and signal to the bigger companies which plans they should be on. For example, you might provide hosting on the basic plan for SMBs but faster CDN for mid-market companies and the ability to set custom headers for enterprises.

Add-ons

Pricing with tiered plans (which is most common for SaaS) comes with the downside that their may be a few critical features that a customer really needs but is stuck between plans. Using add-ons allows customers to pull forward from higher plans to get what they need. Have 51 employees and the cap is 50? Provide an upsell for each additional employee after the cap. Really need a CDN because you are an e-commerce shop? Purchase the CDN add-on for the basic plan.

See also:

• Design the product around the price
• If you have channel partners, you also need to make sure your channel pricing models don’t accidentally introduce channel conflict
• The book Monetizing Innovation has many ideas for figuring out the best pricing

Selling something is already difficult enough but when you have to sell the problem first, it’s even hRder. That’s because convincing someone about a problem they didn’t know they have is like trying to push the bolder uphill. Sure it’s possible, it just takes a lot more effort.

Compare that to nothing sales—merely by showing up, you’ll make a sale 30% of the time. All things being equal, you want the market that looks like that.

Educating the market is expensive. If you have to convince someone of the problem, your best bet is to help shape the category altogether or hijack part of an existing one where customers are underserved.

See also:

• April Dunford’s book about positioning Obviously Awesome
• Don’t step into someone else’s frame

A lot of startups have a business model which roughly equates to: put a dollar in, get two dollars out. This “magic vending machine” is obviously unsustainable but easy to fool oneself into thinking you have excellent product market fit.

Building a magic vending machine happens in straightforward ways—using venture capital money to subsidize the prices of a commodity to gain market share (Uber, Lyft, DoorDash).

Building a magic vending machine happens in accidental ways—building a product that shifts cost from customers to operations of the business and not charging enough for the service.

See also:

• The easiest person to fool is yourself
• We saw many of these businesses during the zero interest rate policy (ZIRP) period

Global crises happening all at once re-inforce one another making the effects larger than any individual crisis alone. This polycrisis (originally coined by Edgar Morin), is an entanglement of events like pandemic, war, climate change, and inflation. For example, pandemic leads to inflation and greater political polarization giving rise to far-right movements and less action to address climate change, and so on.

See also:

• One of the interconnected factors of the polycrisis is globalization
• Zero interest rate policy (ZIRP) gives way to inflation creating an economic crisis
• Doughnut economic model
• Exacerbated by the fact that people are bad at long-term thinking

In an episode of Boy Meets World, Shawn is in a competition of who can wait longest on a billboard suspended high up in the air in order to win tickets to the Super Bowl. He stays there until the only two people left on a cold night are him and an Eskimo eating ice cream.

When Corey tries to talk him down (literally), Shawn tells him that he has to stay. All his life there was always something in his way, preventing him from what he wanted to achieve.

Shawn finally gives up on the contest, but goes to San Diego on his own, eventually finding a way into the Super Bowl.

While the lesson of the episode was that Corey can’t solve all of Shawn’s problems for him, I remember it differently—there will always be someone better than you no matter how hard you try, but that is not a reason to give up. There is more than one path.

The other day I noticed a tweet from Justin Duke which outlined a plan to get his company’s codebase ready for Devin—a programming focused generative AI product. While many are skeptical about AI taking over coding tasks, progress happening quickly and it seems likely that these tools will help software engineers, though maybe not replace the job outright).

If we think AI can positively impact many domains, the question becomes, how can we position ourselves today to take advantage of it in the future?

Writing more is almost certainly one way. Large language models operate on content. That content is mostly text. If you want to have a future co-pilot that can help you build useful and unique knowledge (and potentially avoid knowledge collapse), it’s reasonable to believe you need to be writing a heck of a lot so that AI can draw from what you already know. For example, in a video of Reid Hoffman interviewing himself as an AI, the AI was trained on every book Reid has written and every talk he’s given.

When it comes to code, writing comments explaining what the code is doing and why that’s important seems like it would help generative AI make better contributions that need less supervision. After all, reading someone else’s code is hard enough! I would love to read some research on performance improvements not just on writing code but about comprehension of business logic and how documentation helps or hurts.

Running a business, building a culture where writing is thinking and the primary way teams work asynchronously will amplify every future investment in AI. When Notion rolled out their AI-powered search, a new hire at Mosey started asking it questions about projects, tools, processes, and terms he didn’t understand—the responses helped him get up to speed quickly. FAQs don’t work, but being able to respond to almost any question however it’s formulated sure does!

Net magic number is a measurement of go-to-market (GTM) efficiency for SaaS businesses. A magic number less than 1.0 indicates the business will lose money on each customer. Top SaaS businesses making less than $25MM in revenue per year have a magic number of 1.7 according to benchmarks from Iconiq Capital as in their business returns 1.7x for every sales and marketing dollar they spent to acquire customers.

Net magic number is calculated by taking the current quarter’s revenue less previous quarter’s revenue times 4 divided by the previous quarter’s sales and marketing spend.

By looking at the previous quarter, net magic number accounts for the average sales cycle to complete which makes it a better measurement for enterprise SaaS that has a sales motion.

See also:

• More B2B benchmarks
• Burn multiples are another way to look at efficiency including
• Revenue per employee

Most managers, at best, are a form of benign neglect. Continual attention (micro management) usually results in the opposite of what a manager hopes for—less motivation from employees, learned helplessness, and worse performance. Benign neglect leaves room for employees to have agency to do their work but it’s kind of like the placebo effect, maybe most managers are better off doing nothing at all.

If that’s good, that’s an awfully low bar. When I first heard about this, I agreed it’s not such a bad outcome but surely it can be better.

I like to ask new hires who the best manager they’ve ever had in their career. No one described benign neglect.

See also:

• The line between micro management and leadership
• Delegation not abdication
• Principal-agent problem