Something I’ve noticed about building products over the years is that it always feels like I’m doing it wrong. At the same time, it’s a clue about what’s really going on and what to do about it.

Here’s what I mean about product engineering feeling wrong.

It’s messy. Building a product is an annealing process and is in a constant state of change—adding features, removing them, tweaking them, breaking it, etc. People don’t enjoy the feeling of things being messy.

It relies on good taste and luck as much as analysis. Making decisions with incomplete information is worrying because what if you’re wrong?

Coordinating a team to build a new product is coordinating around uncertainty. That means priorities necessarily change which causes thrash. Writing code for uncertainty is exceedingly difficult even if you talk then code.

Finally, not many people have done product engineering before. It takes experience to find patterns and techniques for seeking good explanations.

So, embrace the messiness and uncertainty. It’s totally normal to go for a swirl!

See also:

• How to be a good product engineer

Smart people don’t like taking chances because they are afraid of being wrong.

Identity is a powerful motivator for behaviors. If you are told all your life that you are smart and special, being wrong is evidence that you aren’t who you thought you were. That dissonance makes you feel bad.

From How to do things if you’re not that smart and don’t have any talent by Adaobi.

See also:

• There are techniques to inoculate yourself from fear that other people will think you are wrong
• Despite the difficulties for smart people to take chances, it is essential to doing important work
• The quest for good explanations is error correcting so rather that thinking if of avoiding being wrong, think of it as a process for seeking better explanations
• Fear is bad advice

In a recent paper, researchers were able to use the properties of DNA self-assembly to classify images. First, they encoded an image into DNA tiles. Then they used self-assembly to generate the shape of a letter that corresponded to an input image. The image classification which was 80% accurate.

Biophysical processes happen all the time in our body and seeing how something as simple as DNA can perform complex calculations is fascinating. In a weird way, it’s easier to understand how these systems work by transposing what we would typically think of as computer algorithms into biophysical processes.

From Nature Vol 625.

See also:

• People don’t expect much from simple ideas
• Genetic isopoint

A study of 270 patients diagnosed with Long COVID found that 53% of patients had response times that were 2 standard deviations slower than the control mean after controlling for age-related decline and comorbidities. Cognitive slowing was also correlated with poor performance in sustained attention.

Read the study: Long COVID is associated with severe cognitive slowing: a multicentre cross-sectional study.

See also:

• List of studies about Long COVID
• 65 million people have long COVID
• 14% of US adults have had long COVID

Inspired by (Almost) Every infrastructure decision I endorse or regret, I thought it would be interesting to do the same for my startup.

AWS

Picking AWS

🟩 Endorse

There wasn’t really any discussion about this as it’s the default I’ve stuck with for the last decade. I’m sure there are things that GCP does better, but for an early-stage startup, I’d rather avoid anything exciting in this part of the stack. My only advice is to fully embrace as much of AWS as part of your stack, otherwise you will be spending a lot of time retrofitting and worrying about problems that don’t really exist (like having the optionality to change providers later).

ECS + Fargate

🟧 Weak Endorse

Early on, we decided we didn’t want to manage servers. We don’t want the complexity of k8, and deploys should be immutable (as much as possible). The deploy pipeline is understandable (build image, upload, replace Fargate tasks). While it took some finagling to get it in place, we basically haven’t touched it since.

RDS

🟩 Endorse

We’re not at any kind of scale where the costs would matter compared to managing a Postgres server directly. At a previous startup, I had to build all of the tooling for backups and replication, and that was a poor use of time.

KMS

🟩 Endorse

I wish I would have given KMS a try sooner! It’s tightly integrated with IAM and makes secrets much easier to manage. It seems daunting at first, but once you have a runbook, it’s easy-ish to use.

Terraform

🟩 Endorse

Using terraform from day 1 was a great idea. Configuration changes are checked into version control, which makes it transparent to everyone. Static analysis has made security audits easier, and automatic checks help us stay on top of security issues. It’s more likely that other engineers will make infra changes because the tooling is centralized and copy/paste-able.

Software and SaaS

Docker

🟧 Weak Regret

I’m not naive enough to believe there is an actual alternative to Docker, but the amount of grief Docker causes in local development on MacOS is very high. For deployment artifacts, I think it’s fine, but this is a clear case of Worse is Better and I accept that.

Postgres

🟩 Strong Endorse

Postgres is our primary DB and our early decision to centralize on this has paid off. At previous startups I stitched together multiple databases for different purposes thinking I was using the best tool for the job. This time around, we haven’t fallen into this well-known trap.

GitHub Actions

🟩 Endorse

It works out of the box, and it’s easy for engineers to fiddle with and extend. I just wish configuring beefier test runners was more comprehensible.

Sentry

🟩 Endorse

I’ve used Sentry at several companies now, and it’s pretty good as an early-stage error tracking, perf regression detector, and workflow for tracking runtime bugs. It takes some effort to dial in (associating errors to users, long stack traces), and there are some quirks (sampling, sometimes partial data from structured logs??).

Slack

🟩 Endorse

Starting with Slack alerts for most things is the easiest way to make sure you find issues before your customers tell you about them (if they tell you at all). We have several Slack bots to help with process and alerting. You can get pretty far with this setup without having a dedicated alerting/monitoring tool.

Notion

🟩 Endorse

I’m a huge proponent of writing things down, and for engineering in particular, documentation is the first step to automation. Most of our work is product engineering at this point, so having a culture of writing briefs and communicating asynchronously makes tools like Notion essential. It used to be slow, but thankfully, it’s getting better for business users.

Netlify

🟥 Regret

Almost all of my personal websites are static sites these days, and I mostly used Netlify to avoid setting up CloudFront (good lord, it’s complicated). Unfortunately, this has become a big thorn in our side because at this point, for security reasons, we use it as a glorified FTP. It’s the only part of our infra that is not in AWS, and only a few engineers have access to it. In hindsight, I probably would have put this in Terraform and CloudFront from the beginning.

1Password

🟩 Strong Endorse

It’s by far the best password manager I’ve used. Making sure all credentials are stored in 1Password from the beginning saved us a world of grief. There are decent admin tools for teams too.

aws-vault

🟩 Endorse

We use IAM for authentication anytime we are accessing infrastructure. The aws-vault CLI makes this very easy to set up. I just wish it integrated with touch ID or something else to make it less clunky.

Vanta

🟧 Endorse-ish

I haven’t used other providers for SOC2 to compare Vanta to, but the experience can be difficult to know what you need to do. The automated tests for infra integrated with AWS are good, as are the offboarding workflows.

The needs stack is a way of describing what a customer is trying to do when they are buying. There is always a hierarchy of needs the customer is trying to satisfy which businesses are competing to satisfy. For example, in using the Needs Stack for competitive strategy by A Smart Bear, the author talks about “buy infrastructure” vs “set up WordPress” and going up the stack to “get lucrative speaking gigs around the world”.

Buying a thing is a means to some end. If you know the target customer well, you can use this to find the right level of the needs stack to obviate competitors and better communicate value to the potential customers.

There are many packages for completions in Emacs. I recently switched over from ido and helm to the MOVEC stack: Marginalia, Orderless, Vertico, Embark, Consult.

What do these packages do?

• Marginalia appends additional information to completion results (e.g. adds file size to find-file results)
• Orderless enables fuzzy-ish search
• Vertico renders completions (vertically as the name implies) with some additional utilities for managing history, displaying the number of results, and more
• Embark is a whole other thing that applies actions to completion results (see how to use embark for emacs)
• Consult is the main framework around the built-in Emacs completing-read API with default wrappers around built-in Emacs commands like finding a file, switching or killing a buffer, etc.

I read The mistakes we make the second time by Harry Glaser. It points out that certain things are easier than they should be (raising money, hiring) and that friction is helpful. It’s easy to build in anticipation of scale you don’t have because you weren’t spending enough time getting to product market fit. It also talks about how the excitement and joy don’t the same thing they did the first time (e.g. seed round doesn’t feel like an accomplishment).

I’ve started a few companies and it’s easy to over correct based on prior, often painful, mistakes. It can be difficult to recognize the biases we have and the scar tissue that led to it. One thing I try to remind myself is that the number one job of a startup CEO is finding product market fit and to make contact with reality often.

Embark for Emacs seems like a useful way to further customize Emacs actions (M-x). I still don’t understand why I would want to use it, so this note is an exploration to try it out and see if it’s useful.

What can you do with embark?

I’m still figuring out what will be most useful for me, but I’m keeping a running log of what I find below.

Cherry-picking results

Select a custom set of results from a list of completions and export it into a new buffer to use later. For example, I can search org-roam for notes I need to complete that are tagged with #draft and then use embark to select a few notes I want to work on. Embark persists this new list in a buffer I can return to later.

Compose results

We can extend the above technique to build up a customized list of items from multiple searches. For example, I can start a search for notes about “AI”, select some results (C-. SPC), then search for LLMs and select those results, thereby accumulating the items across multiple searches. Invoking embark-export puts that in a new buffer to go through later just as before.

You can view the intermediate results by using embark-live.

Quirks:

• Unfortunately, the results won’t update the live list unless the search query is changed.
• I can’t seem to select all items from the search results without exporting a new buffer.
• I would love to be able to come back to an export and add more to it later, but there doesn’t seem to be a way to do that.

Narrow down results

Methodically narrow down a large list of results you don’t want to lose, export a sublist, and then work on that without losing your history.

Add contextual actions

I want to use embark to select a region and fix all the grammar and spelling mistakes using gptel (see ways to use AI with Emacs).

(I haven’t done this yet but am planning to try and extend the list of actions.)

Maintaining flow

Sometimes you’re in the minibuffer and realize there was something else you needed to do on the same item. Rather than exit (C-g), use embark to take another action instead and come back to what you were doing (C-u C-.).

You can even use another action that isn’t already defined by calling M-x and come right back to where you were.

Create a new note with a title from the selected region

Select some text and then use embark to insert a new org-roam note using the text as the title (C-. M-x org-roam-node-insert).

Other thoughts

Embark is close to a “universal interface” for invoking actions on items. The completions framework that Emacs introduced makes it possible for Embark to operate on anything using exactly the same pattern. What’s incredible is that means nothing in Emacs needs to be updated to work with Embark.

I’m starting to think this is a rather important discovery for UX. Search is the first step; what you do with it next is the second half.

Embark broke my brain at first, but it makes a lot more sense when you try to use it. Notice how it seems to just work everywhere and in ways that you would expect. Files, buffers, links, search, etc.

I was worried that this would all feel very modal and require rote memorization to do anything useful. Somehow it doesn’t feel like that because it’s all operating on text and a limited set of object types at the end of the day. Even if it is modal, there are not that many modes.

See also:

• Emacs is the ultimate editor building material
• Fifteen ways to use embark describes embark as the “actually” and “but first” command

I was thinking the other day that there are few objects that I use somewhat regularly that are more than 10 years old.

• Patagonia Better Sweater (brown)
• CX Cargo Guitar (carbon fiber travel guitar)
• Ray Bans (with the NYC subway map printed on the inside)
• A New York Knicks hat
• Midori Travelers notebook (a gift from my brother)
• Ikea plates (literally indestructible)

The list feels pretty sparse considering how much stuff I’ve accumulated over the years. It also reminds me to be mindful the next time I buy something in these every-day categories. Will I enjoy using it for 10 years?

(Surprisingly, there might be more digital things I use regularly that are that old e.g. gmail, dropbox, reddit!)

See also:

• Heirlooms foster long term thinking
• Living a rich digital life

In How to Live an Asymmetric Life, Graham Weaver talks about how to confront fear and get the most out of your life to a class of soon-to-be business school graduates.

• Do hard things (it gets worse first, working hard is required to do great work)
• Do your thing (suffering is inevitable so make it worth it, it’s hard to sustain someone else’s dream)
• Do it for decades (being excited about something for decades lets you do big things and it’s compounding)
• Write your story (put it out in the universe, give yourself permission to do it)

There doesn’t seem to be a good web-based query editor for an Amazon RDS hosted postgres. AWS has a query editor for Aurora but not for RDS. You could use AWS Redshift to run a federated query on RDS but it’s unclear if you can also use Query Editor V2 for that (ideally this would also use serverless Redshift so there is no infrastructure to maintain).

There are many theories put forth to explain human consciousness and experiments are running to test them. With all the discussion around AGI, it’s timely to keep an eye on them.

Higer-order theory

Humans become conscious of a thing (like what they are seeing) when there is a meta representation of it in the higher-order parts of the brain. This is kind of like the “symballs” analogy from I Am a Strange Loop.

Global workspace theory

Information enters consciousness when it is accessed and broadcast in a “workspace”, moving information from local processing units to the brain-wide workspace.

Integrated information theory

Concsciousness arises from the integration of information in a system where the greater degree of integration (more connections) the higher the level of consciousness.

See also: Consciousness is categories but as Douglas Hofstadter would argue, it is conscious or it is not, there is no unit of consciousness

Recurrent processing theory

There needs to be a loop of information flow from higher-order cognitive areas to lower sensory processing areas in both directions—top-down and bottom-up.

See also: Presence of a feedback loop shifts perception from mechanics to desires

I want to better utilize AI tools in my day-to-day work. I suspect there is much more I can be doing and using Emacs as building material to make it work for me.

What can I do with AI and Emacs?

Writing prose

• Rewrite and replace region to fix spelling and grammar but preserving as much of the original prose as possible
• Ask questions about the current document context
• Make a flow chart from the text (maybe graphviz?)
• Summarize a doc or region
• Rewrite and replace text for clarity
• Retry the last prompt to iterate on different instructions
• Select text, write custom instruction, send
• Create pre-canned prompts I use frequently (e.g. fix this grammar, make it shorter, etc.)
• Dispatch to org-ai on the section or file prompting for other things I’ve written which might be related
• Summarize a link

Writing code

• Write function from a comment and fill in the rest
• Debug section of code known to be wrong
• Fix badly formatted code
• Refactor a complicated function
• Generate a test from the function or module
• Write bash script for command line
• Ask question of log output or data from a SQL query

Management

• Construct follow up messages for delegated tasks in org-agenda for work that are not complete yet
• Write the basic outline of an investor update from updated information
• Write a weekly summary and priorities for the team

An English physician once described radium as “the unknown god”. This was at a time where radiation and it’s effects were still being discovered. Radium was being used to treat all manner of ailment, thinking that if it was helpful in large amounts for treating cancer, it must also keep you healthy in small amounts.

Characterizing power of something we don’t know as an unknown god is oddly fitting. There are many unknown gods in the physical sciences and entirely new fields like AI.

I first heard this on Behind the Bastards - William Bailey.

See also:

• Trying to know the unknowable leads to pessimism but early 20th century science knew just enough for irrational optimism
• Gödel’s incompleteness theorem tells us there must be many unknown gods

This note does not have a description yet.

I was watching an interview with Kobe Bryant where he talks about his workouts and approach to basketball. When asked if he ever gets nervous and what he does about it, he says, “the results don’t really matter” and that it’s much more important to focus on the process of improving every day. Moments like game winning shots come and go (even if you make them, the points go back to zero the next day) but the thing you can control is yourself.

That got me thinking about how focused we should be on results in everyday life. It’s easy to ignore results in basketball (at least short term) which has clear win/lose criteria (a literal scoreboard), but other domains not so much.

Still, experimenting and adjusting the underlying systems and processes that drive results feels like the right lesson about being productive when under enormous pressure to deliver. How people respond to the pressure in a productive way (like Kobe) makes a difference.

The internet is not some other thing, separate from life, it is an abstraction that is very real.

Fundamentally, I can inspect and see the packets and bits between my computer and the network. I can manipulate the pixels I see in the browser using code. These days more atoms are controlled by bits.

I can also make a life from the internet. I can read information. I can communicate with people and machines. I’ve made my livelihood from it by arranging bits just so.

If the internet is real life, how should our relationship with it change? I think we should care a lot more about living a rich digital life in the same manner we think about health, fitness, and gardening. Investing in quality of life ought to include being thoughtful about our digital skills, interests, and safety.

(Based on a conversation I had with Jake Stein)

The most reliable way to stop procrastinating is to get started. Not with the intention of completing the task at hand, but doing just a little bit.

By doing just a little bit to get started, the task you’re dreading becomes approachable. From there, it’s highly likely you’ll complete the whole darn thing.

The activation energy needed to get going always seems higher if you stare at the mountain ahead of you rather than the next step.

Even though I know the trick, it still amazes me that it works.

Asteroid City, Wes Anderson’s latest movie, is a movie about a play where the plot never quite make sense. There is a scene where the playwright is asking a room of actors to be asleep. The movie returns to this moment a few times. On the final time, the director, after telling the lead actor there are no answers for why he burned his hand on purpose, tells everyone “you can’t wake up if you don’t fall asleep.”

I think this is the whole point of the movie. The story is not the point, but a plot device to articulate the moment of “waking up”. The juxtaposition of sleep or dreaming (or watching a meandering Wes Anderson film), helps us realize reality and wake up. I caught myself noticing reality a little more clearly as that scene was happening.

Maybe not profound, but artful and interesting!

See also:

• Wakefulness is a form of determinate negation defined by the ceasation of asleep
• Mindfulness helps you notice these moments awareness is key to being present

It’s difficult to work with someone who always seems to be aggrieved. It’s a sign they will blame others rather than take responsibility themselves. It’s also a sign that they won’t be interested in improving their performance or making adjustments to collaborate better with you or the team.

If someone is perpetually aggrieved, it’s just a matter of time before they start blaming you for treating them unfairly. If that someone is particularly persuasive (it’s easy to get sucked into), it can have a far reaching, negative pull on everyone.

See also:

• Default optimism is rational
• This is usually a sign of narcissism so you can use the Narcissist’s Prayer to identify them

I used to feel like asking a list of discovery questions to qualify a prospect during a call was awkward and not useful. Now I realize it’s actually a key part to a consultative sales experience and should feel conversational.

Discovery questions should be asked because you genuinely want to know how to help them. It would be a disservice not to ask in the same way you would be wary of a doctor that jumped directly into treatment merely because you showed up in their office!

When you genuinely want to know how to help, it’s easy to make the whole exchange feel more conversational. When it’s conversational, everyone can put their guard down and be more forthcoming about their needs and whether your solution is a fit.

See also:

• Sending follow-up emails is worth billions per year so don’t forget to follow-up afterwards
• Getting comfortable with this process makes for more effective founder-led sales too
• A third of sales comes from doing nothing at all so stay out of the way when someone is ready to buy

Imagine a busy executive trying to work with multiple people. They’re coming in and out of meetings and working on several projects all day long. They have maybe 15 minutes to concentrate on a thing to try and keep things moving or prevent holding things up. Every time there is a broken link, spelling error, obviously incorrect statement, or something that needs to be double checked, it creates delays and more work.

A culture of sloppyness and lack of attention to detail multiplies the delays and increases the amount of work across everyone in the organization. Ostensibly small details become very important when multiplied by a large number of people.

See also:

• An organizational linter might automate away some issues rather than rely on training
• The Ringelmann effect shows groups become less productive as they grow
• If slowness begets more slowness, sloppyness begets more sloppyness because it was accidentally made okay

This chart, adapted from xkcd, tells you the amount of time saved over 5 years based on how much time you are able to shave off of the task. For example, a task that you do daily that you are able to reduce by 30 minutes will save 5 weeks of time over 5 years. This chart can also tell you how much time you should spend on trying to optimize so you don’t spend more time than it’s worth.

See also:

• Time saved using keyboard shortcuts is pretty high not because of the amount of time shaved off but because of how frequently I use them
• There’s a knock-on effect that’s not quantified here which is that automation reduces marginal cost of nonautomated tasks
• The second time, write it down because documentation is automation