In the last two months, my two-person team spent over $5,000 running roughly 1,000 tasks in Devin, the AI software engineer. We weren't spinning up 10 Claude Code instances to burn tokens as fast as possible. This is what actually happened when we integrated AI agents into our real-world workflow while building Hyprnote.
Here's what we learned.
Not a Reader? Watch the Video Instead
Timestamps throughout this post link to specific examples in the video.
Running AI Agents Where Your Team Already Works
The single most powerful decision we made was running Devin inside Slack, not our IDE.
Slack is where discussions already happen. We're already getting alerts from Zendesk, Sentry, and Discord. Launching an agent directly inside a thread where the problem is being discussed eliminates the friction of switching contexts.
Real example: John, my co-founder, identified an issue with our AI prompts and tagged Devin to fix it. Devin fixed it, but the approach was non-optimal. Since AI prompting is something I work on, I jumped into the same thread with more context. Devin figured it out based on my additional input, finished the PR, and it got merged. We stayed in the same Slack thread the whole time, with no copy-pasting between tools.
Another example: I tagged Devin about our 404 page not rendering properly. John, who works on our webpage primarily, pointed out reference files to look at in the same thread. Based on his input, we got a PR and merged it.
The agent participates in the same Slack threads where we discuss the problem.
AI Agents Enable Non-Technical Teams to Ship Code
Having an agent accessible from Slack opened up tasks that don't necessarily require technical skills. For instance, understanding what we're tracking in analytics or making small adjustments to better understand user behavior.
John attached some PostHog docs and asked questions about what we're tracking and what we should be tracking long-term. Devin made the changes. Now both John and I know we have analytics updates—super helpful for staying aligned.
Since we use GitHub as a CMS for Hyprnote, we can even update landing pages or blog content directly from Slack. John attached a PDF from an internal discussion, and Devin updated our docs based on the actual conversation we had.
Three Types of Tasks to Delegate to Devin AI
As a small early-stage startup, there's always a lot going on. We're handling day-to-day work while thinking about what's next—new features, product direction, how the codebase should evolve. Dumping all of this into an async coding agent lets us explore solutions without blocking other work.
Degree 1: Exploration (Not Shipping Anytime Soon)
This is work that isn't planned for the immediate future. We're not going to ship it or even merge it right now, but exploring it helps us understand what the work would look like, how complex it is, and roughly how long it might take.
Example: Even though we're focusing on our macOS desktop app, we had ideas around building a Chrome extension. I asked Devin to research how to make a Chrome extension that works with a desktop app. We learned how 1Password does it and got a rough plan.
Then we cloned the repository of a popular Chrome extension framework. Based on the docs and actual code examples, we implemented it to see how it would look. We didn't merge it, but it's useful to see how it'll look in the future.
Degree 2: Preparation (Relevant, But Not Right Now)
This is work we'll likely merge, but we're not pulling it into the IDE yet.
Example: Someone asked whether Hyprnote could import data from Apple Notes. That feels like a feature we could support in the future, but it's not a core focus at the moment.
We did research to see if there was any existing work on that. There was, so we cloned it, ported the test cases, and let Devin implement it. Tests passed, so we safely merged it for a future feature.
Degree 3: Production (Very Relevant Right Now)
This is work we'll definitely look at, but spawning the agent right now lets us avoid context switching. Maybe we're traveling or about to go to sleep.
Example: We needed to update test cases around our Tinybase utils—very relevant and important work. We asked Devin to clone the repo, inspect the codebase, and write the test cases.
One trick: we asked Devin to use the Claude CLI that we already installed on Devin's machine. This way we can offload some AI inference to our Anthropic account and use some credits.
Pro tip: We encoded this knowledge as an "offload agent" on how to use Claude CLI. Mentioning "consult smart friend" (something Devin uses as a prompt internally) helps Claude CLI get called at the right timing.
Good Documentation Enables AI Agents to Ship Code Faster
In Hyprnote, we focus on supporting multiple providers for language and speech model inference as part of our open-source effort. Early on, we spent time designing and documenting flexible, clean interfaces. This worked well for future contributions and community involvement, and the same benefits apply when working with coding agents.
Example: ElevenLabs Support
We support both WebSocket-based real-time transcription and file upload-based batch transcription. We had a very detailed prompt on how models should be handled, how language should be handled, and other API references in the docs.
Since we have end-to-end testing support in place, we sent the ElevenLabs API key as credentials (this can be passed in the prompt or through Infisical CLI). With all the documentation, test cases, and API key in place, Devin implemented it in almost one shot, and we safely merged it.
Example: Mistral Support
Same story for language models—even easier because there's no WebSocket involved. Since we have infrastructure to support any language provider, Mistral was supported in less than 10 minutes.
Example: OpenAI Support
This one was a little harder. We had errors in the client, so we passed the error message and credentials. After a few minutes—since we had API keys and test cases in place—Devin figured out that OpenAI only supports 24kHz sample rate. That's why it was failing. We fixed it without any engineering resources invested.
The pattern: good docs, clean interfaces, and test infrastructure make it possible for agents to ship working code.
Automating Code Maintenance with AI Agents
Once a codebase reaches a certain size and age, maintenance work alone can consume significant engineering time and slow the team down. Coding agents let us offload much of that work.
Single-Prompt Migrations
One common example is doing migrations that have clear documentation. In Hyprnote, we recently completed:
- AI SDK version 6 migration in a single prompt
- Tailwind v3 to v4 migration in a single prompt
Concurrent Multi-PR Migrations
Things can get more complicated and may require multiple PRs or concurrent work.
A good example is applying Vercel's recent React best practices agent skills. We attached Vercel's React best practices document, and Devin figured out what changes should be done. Since there was a lot of isolatable work, we prompted Devin to do this concurrently by spawning concurrent Devin sessions.
One way to do this is to ask Devin to make actual API calls. But there's a better way: use the analyze-session task. This lets you spawn concurrent Devin sessions to run work concurrently and generate separate PRs per task.
Daily Automated Linting
Migrations and new guidelines don't happen every day, but pairing an agent with an automated linting tool can be applied daily.
In Hyprnote, we have a large Rust codebase, and since Cargo Clippy is pretty good, we set up a GitHub Action to run Cargo Clippy daily and spawn Devin to apply any fixes based on the output.
This saves a lot of time applying these guidelines or Clippy warnings in an async manner, since running Clippy and Cargo check takes a long time.
When Devin Works and When It Doesn't
If you're expecting AI agents to replace developers, you'll be disappointed. We're not there yet.
But if you're looking to meaningfully extend what a small team can accomplish, it's absolutely worth it.
Devin AI is worth the investment when you:
- Have well-documented code with clean interfaces and test coverage
- Need to explore features before committing engineering time
- Want to offload maintenance work (migrations, linting, updates)
- Have non-technical team members who need to ship small changes
- Run concurrent work that would otherwise bottleneck your team
Devin AI is NOT worth it if you:
- Have poorly documented, tightly coupled code
- Expect it to understand context without clear instructions
- Want it to make architectural decisions
- Need it to work in complete isolation without human oversight
After 1,000 tasks, the pattern is clear: You're not buying code generation. You're buying the ability to delegate work and parallelize effort across your team.
The best ROI came from tasks we could delegate async—exploration work at 2 AM, maintenance work during travel, migrations while focusing on core features. The agent didn't replace our judgment; it multiplied our capacity to act on it.
Our recommendation: Start with one well-defined use case (like automated linting or simple migrations), measure the time saved, then expand. Don't try to use it for everything on day one.
