• Why I Like Incidents

    A lot of tech company workers dread incidents. They are a high-pressure and often high-stakes ordeal that requires urgent attention. Iโ€™ve witnessed some incidents that lasted for months.

    Iโ€™ve come to like incidents because I see them as moments of progress. At a minimum, the team just learned how something works (or doesnโ€™t work). At its best, the incident process guarantees that improvements are made which prevent systems from failing the same way twice.

    So while I have that same feeling of dread whenever an incident is called, I remind myself that we are going to learn something important and this is the price of entry.


  • Use a Scenario Table to Organize Complicated Situations

    When thinking in scenarios, I find it useful to lay it out as a table. A table is the most compact way of sharing definitions with a team. A table helps you explain what’s going on in a way that is mutually exclusive and collectively exhaustive. This keeps the team organized and provides the ability to refer to each case.

    What goes into a scenario table?

    Columns and values

    Which columns determines which dimensions of the problem matter. If you are responding to an incident for example, the goal of the table is to summarize all of the ways a bad thing happened so each column would be contributing factors.

    Picking the right columns is a thinking aid to help you and the team reason about the problem. What other possible combinations of these columns exist in the system? Do we have examples and does it matter? Which can you rule out entirely?

    Labels for each case

    Since we’re in a table format, it’s easy to add a column for labeling each case (a row in the table) so they can be referred to easily. If your situation is very complicated (i.e. many columns), simply number the rows and refer to the case by number. If it’s not as complicated, use a very short name that is distinct and memorable for each case (I use the former for incidents and the latter for product brief).

    See also:


  • Emacs Sticky Buffer

    Sometimes I want an emacs buffer to always be visible but I want to ignore it when navigating between buffers.

    For example, I want a list of org-mode tasks I need to do today and I want it at the top of the window so it’s highly visible. Since I heavily use other-window and previous-multiframe-window to switch between buffers, it would slow me down if I have to visit the sticky buffer every time I’m cycling between buffers (I like to think of it as clockwise and counterclockwise).

    Here’s my first attempt at doing just that: sticky-buffer-mode.el.

    Now I can go to the buffer with the list of tasks and sticky-buffer-add and future navigation (I have previous and next bound to C-x p and C-x o) won’t visit it.


  • Manual Spam Filter

    I block every unwanted email I receive to keep my email inbox as signalful as possible (“block” is Gmail speak for “create a filter for this one email address and always send it to spam”).

    Isn’t that a lot of work?

    It’s certainly more than zero effort but the damage a low-signal email inbox is very high for me. Gmail’s built in spam filtering is good at catching the obvious stuff, but it flat out misses every cold email and recruiting agency. Missing a single important email can be the difference between closing a customer, fixing an issue, and making a hire.

    My manual email spam filter list


  • Four Levels of Product Market Fit

    First Round Capital has a helpful guide to product market fit that helps to orient founders so they can focus on the right things. Rather than a binary, yes/no, evaluation of product market fit, the guide discusses different levels.

    Level 1 - Nascent PMF is when you have some initial customers mostly from warm intros. Churn, gross margin, burn multiples, don’t really matter at this point so much as quickly discovering who the right customer is and solving their problem well.

    Level 2 - Developing PMF is when you have customers, the solution delivered is more repeatable, you have up to $1MM ARR, generating your own leads, and customers would be very dissapointed if you went away. Some metrics around go-to-market efficiency start to matter like net revenue retention, gross margin, and burn multiples because you are proving the business can effectively drive demand.

    Level 3 - Strong PMF is when demand is flooding in and this is often what founders refer to as “feeling the pull” and everything feels easier. There is strong growth of 3x revenue or higher, more leads are coming from word of mouth, and marketing/sales is very efficient with low CAC/burn multiples/churn and high gross margin and sales conversion rate.

    Level 4 - Extreme PMF is when companies earn permission to build new products that expand TAM, the brand starts to become synonymous with the product category, and the business is growing fast. Burn multiple is < 1 and there are more scalable customer acquisition channels.

    Read Levels of PMF by First Round Capital.


  • Don't Combine Domain Name Hacks

    I see a lot of startups with domain names like getbarai.co and trymspledword.io which combine multiple domain name hacks. These look untrustworthy, are hard to search, difficult to spell, and long. It’s hard enough to get anyone interested in what you are doing that introducing any friction can cause people to forget about it.

    Here’s my rule for startup domains:

    Use only one domain name hack (mispelling of a real word, made up word, prefix before a real word, non-dotcom top level domain, etc.) AND always use .com. Then, when you have the money, buy the clean .com domain name.

    For example, when I started Mosey, we started with getmosey.com (prefix hack) then mosey.so (non-dotcom TLD), then eventually bought mosey.com.

    See also:


  • Net Revenue Retention

    NRR measures the ability of a business to expand revenue over time. Itโ€™s only really useful if the company sells multiple products or one of them has some sort of usage based scaling factor that implies customers use it more over time.

    One of the misleading ways to look at NRR is when there is a single product that is usage based and scales with the growth of their customers. If the customers growth stops due to an economic condition outside of anyoneโ€™s control the NRR becomes stagnant or lower than 100%. All that measures is the growth of the economy rather than the effectiveness of the business.

    See also:


  • Personal Log

    I often find myself wondering what I did yesterday. I want to be able to reflect on the day before as I get ready in the morning for the day to come. I also donโ€™t want to make work for myself so the effort has to be useful.

    What would make a personal log useful?

    Facts about the day to jog my memory

    Looking at my calendar to see what meetings I had is a good clue about how my day was spent and who I met with.

    Seeing the tasks Iโ€™ve captured and completed also helps.

    Then there are life log things like where I went out to eat and how much time I spent looking at a screen. These are maybe not as useful to look at daily.

    Notes and observations

    Iโ€™d like to be better and inspecting my notes and thoughts. I write an enormous about of notes from meetings and tasks, but I donโ€™t review them as readily. Reviewing them more readily would help me turn more of what Iโ€™ve learned into permanent notes.

    Things I produced

    Concrete things Iโ€™ve produced would be great to catalog to reflect on how it went and what I learned now that itโ€™s complete. This includes writing docs, committing code, shipping a feature, closing a sale, or any other artifact.


  • How Does Tailscale Work Without Ports Open?

    If a home computer is running on a local network with no ports exposed, how are tools like tailscale working to connect to said computer? How would the computer know that another device is trying to connect to it?

    NAT traversal over UDP.

    Firewalls allow packets from ip:port s where it has observed packets sent to ip:port. A coordination server provides a mapping of devices and ip:port. To get through the Network Address Translation layer (NAT) which rewrites packets (e.g. a home router translating packets from local devices like your phone to come from one IP address on the internet), the STUN protocol informs each device what their ip:port is so they can send packets to the peer device.

    Altogether, by sending packets over UDP and being able to get replies to the correct ip:port, no ports need to be open (though UDP egress is required).

    See also: Why you still need an SSL certificate with tailscale


  • Dokku, Tailscale, and Letsencrypt

    To set up SSL certificates for use with HTTPs on dokku you can use the letsencrypt plugin.

    sudo dokku plugin:install https://github.com/dokku/dokku-letsencrypt.git

    Next, create an IAM user for dokku-letsencrypt with a custom policy scoped to the hosted zone in Route53. Configure dokku with IAM credentials to pass the DNS-01 challenge:

    dokku letsencrypt:set –global dns-provider route53 dokku letsencrypt:set –global dns-provider-AWS_ACCESS_KEY_ID your_key dokku letsencrypt:set –global dns-provider-AWS_SECRET_ACCESS_KEY your_secret dokku letsencrypt:set –global dns-provider-AWS_REGION aws_region dokku letsencrypt:set –global dns-provider-AWS_HOSTED_ZONE_ID your_hosted_zone dokku letsencrypt:set –global email <your@email.com>

    Enable letsencrypt for the app in dokku:

    dokku letsencrypt:enable <app>

    And set up cron job to auto renew certificates:

    dokku letsencrypt:cron-job –add


  • Why You Still Need an SSL Certificate With Tailscale

    I have a private network using Tailscale that runs a few local websites and services. Accessing the websites happens via the Tailscale client which connects nodes in the tailnet directly (e.g. my phone and a dokku hosted website) encrypting data from end to end. While this is a great way to secure the session itโ€™s not validating the identity of the website.

    Why does that matter and why does a certificate help?

    DNS can get spoofed and someone on the network you are connecting through could serve the same domain pointing to a malicious website. While unlikely, that means someone could trick you into sharing information you thought was happening on your private website like credentials, document uploads, or photos, or anything else you might normally interact with or share.

    An SSL certificate validates the identity of the private website so that you would receive a browser warning if it was being spoofed.


  • Fragility Is the Acceleration of Harm

    The definition of fragility (and its inverse antifragility) is the acceleration of harm. For example, if you plot speed of a glass cup hitting the floor and amount of harm to it, the curve rapidly accelerates as the speed goes up. Fragile things are harmed by disorder and stress.

    Therefore the definition of antifragility is the oppositeโ€”things that improve as disorder and stress grows. For example, natural selection results in an ecosystem that is more resilient as disorder increases (more species, more variability resulting in better fitness, etc.).

    See also:

    • Thinking in systems shows how multiple stocks has a stabilizing effect on the overall system
    • Antifragile systems exhibit a compounding effect for example, open source software that exponentially increases in popularity/value as it adds more contributors

  • Using Github Actions to Access Tailnet

    I want to access a private network behind Tailscale network so that I can make an API call to update my personal indexing service when a GitHub repo changes.

    I could use webhooks but Iโ€™ve set up Dokku on AWS to be completely private with no ports opened. Supporting webhooks would mean punching a hole in the network for the public internet. (Which could be done with Tailscale Funnel but thatโ€™s for later).

    To get notified on changes, I made a workflow in the repo that uses the Tailscale GitHub action.

    1. Create an oauth key with only write permission on the devices category from a tag specified in the workflow step (tag:ci in my case)
    2. Add the oauth client ID and key to the GitHub repoโ€™s Action secrets so it can be made available to the runner
    3. Create a GitHub actions workflow and add a step for setting up Tailscale
    4. Add a step to curl the API in the private tailnet

    Example workflow:

    name: Notify
    
    on:
      push:
        branches:
          - main
    
    jobs:
      notify-index:
        runs-on: ubuntu-latest
        steps:
        - name: Tailscale
          uses: tailscale/github-action@v2
          with:
            oauth-client-id: ${{ secrets.TAILSCALE_OAUTH_CLIENT_ID }}
            oauth-secret: ${{ secrets.TAILSCALE_OAUTH_SECRET }}
            tags: tag:ci
        - name: Call the private API
          id: call_api
          run: |
            #!/bin/bash
            curl -X POST http://my-private-api.com/do-something        
    

  • Maรฑana

    “Maรฑana, a lovely word and one that probably means heaven.”

    I love this line from On The Road by Jack Kerouac. It talks about a small group of friends who spend their day scraping by in the farmlands of California. Everything is pushed to tomorrow as they spend most of their time drinking.

    It perfectly captures that small bit of comfort we get from procrastination.


  • Dokku on Aws

    I’m setting up dokku as a personal infrastructure PaaS for running services like the personal indexing service.

    This was a confusing affair so I’m writing these notes to reference later if I ever need to set it up again.

    Notes:

    Create an EC2 instance and setup dokku

    • Has to be 2gb or more to avoid issues with dokku installation
    • Has to be Ubuntu (the default Amazon Linux distro will not work)
    • When you ssh into the newly created instance, you have to use the ubuntu default user ubuntu@ec2-address-here.region.compute.amazonaws.com
    • Make sure to add the .pem key to ssh-agent on your local machine or git push dokku main won’t succeed
    • Set up a domain by running dokku domains:set-global mydomain.com and setting up a Route53 CNAME record to point to the public domain name of the AWS EC2 instance (note: this will break if the EC2 instance is restarted, use an AWS Elastic IP to avoid this)

    Create a dokku app

    1. SSH into the dokku host server and run dokku apps:create my-project
    2. On local run git remote add dokku dokku@mydomain.com:my-project
    3. Push git push dokku main and trigger the build/deploy (this just works if you have a Dockerfile at the root of the project)

    Tailscale

    I followed the Tailscale app connector setup instructions to limit traffic to the dokku domain to my tailnet. That means I’m the only one that can access it and I must have tailscale running on my device to access dokku.

    On the dokku EC2 instance

    • Install tailscale curl -fsSL https://tailscale.com/install.sh | sh
    • Run the app connector sudo tailscale up --advertise-connector --advertise-tags=tag:indexer-app-connector
    • Now traffic to the domain is restricted to only go through tailscale

    Using GitHub deploy keys

    I sometimes need access to a GitHub repo at runtime from an application (e.g. pulling the latest from a repo, making a commit, etc.). GitHub has deploy keys for this (single repo key, read-only by default). Putting secret keys into a docker image would be insecure so instead, we can use dokku volume mounts to make them available to the app that needs it.

    1. Make the directory on the dokku EC2 instance that will become the mounted volume mkdir /var/lib/dokku/data/storage/my-app
    2. Copy or generate the deployment key to the directory that was just made
    3. Mount the volume dokku storage:mount my-app /var/lib/dokku/data/storage/my-app:/storage/path
    4. Access it from the running app under /storage/path