The “the first AI software engineer” Devin, from Cognition Labs, was found to be fixing bugs of it’s own doing, solving problems in a roundabout way, and taking a long time, in a debunking video by a human software engineer.
As someone who codes, I was surprised to hear so many people saying Devin AI was a breakthrough. I’ve used generative AI tools quite extensively now and have a good feel for their current limitations. In real-life scenarios, they’re best at augmenting a knowledgeable person, but they always need to be checked for many of the reasons found in the video. I would love if we could get to the point where these systems can be unsupervised, but it appears we will need to wait a little bit longer.
See also:
Links to this note
-
The other day I noticed a tweet from Justin Duke which outlined a plan to get his company’s codebase ready for Devin—a programming focused generative AI product. While many are skeptical about AI taking over coding tasks, progress happening quickly and it seems likely that these tools will help software engineers, though maybe not replace the job outright).