Blueberry, Instacart's AI Assistant, Cuts Incident Response Time—Here's How

2 a.m. Pages and the Scramble for Answers

You know the drill. It's 2 a.m., your phone buzzes, and your heart does that thing. Something's broken. Customers are probably already tweeting. You groggily open your laptop and start the hunt: which dashboard? which log? who owns this service? It's a mess. And that's exactly what Instacart wanted to fix when they built Blueberry, an AI assistant that lives in Slack and helps engineers triage incidents faster.

Blueberry isn't some magic wand that auto-fixes everything. It's a set of AI agents that gather context, form hypotheses, and present them right in the Slack thread where the incident is being handled. Instacart shared details about how it works and what they've learned. There's a lot here for anyone building or buying AI tools for incident response.

The Real Problem: Too Much Context, Too Little Time

When a service goes down, the hardest part is often just figuring out what's relevant. Which team owns this service? What changed in the last deploy? Are logs showing errors? Has this happened before? That's a lot of digging, and it eats up precious minutes while the incident is ongoing. One engineer told me they spent ten minutes just finding the right dashboard, then another five figuring out which service was actually affected. Customers were already frustrated by then.

Instacart's engineers noticed this pattern: incidents were delayed not by lack of tools, but by the overhead of assembling the right information. It's like having a library of books but no index—you know the answer is in there, but you can't find it quickly.

Meet Blueberry: Your AI Sidekick for Incidents

Blueberry is designed to handle that grunt work. When an alert fires, it kicks off a parallel set of sub-agents—about ten of them, according to Instacart's CTO. These agents pull data from internal systems: service ownership records, deployment history, logs, metrics, and past incident reports. They then generate root-cause hypotheses with supporting evidence, all within a few minutes. I've seen demos where it takes about five minutes to get a solid hypothesis, which is way faster than most humans can do manually.

What's key is that Blueberry doesn't just spit out a random guess. It grounds its reasoning in facts. The system connects to MCP (Model Context Protocol) tools, which let it query internal databases and APIs. It also has a persistent state, so it can track what's been tried and what hasn't. That means engineers can see the reasoning trail, not just a black-box answer. That transparency builds trust.

From 60% to 90%: The Accuracy Leap

In April, Blueberry performed about 25,000 diagnoses across more than 270 Slack channels. That's a lot of incidents. The accuracy rate jumped from 60% to over 90%, and the secret sauce wasn't a bigger language model. It was data. Instacart fed Blueberry over 14 years of incident history. That's a treasure trove of patterns: what broke, what caused it, how it was fixed. The AI uses that history to compare current symptoms with past incidents, which is exactly what a senior engineer would do manually. It's like having a collective memory that never forgets.

One engineer mentioned that the AI correctly identified a rare database deadlock pattern that had only occurred twice in the last five years. It caught it because it had the history. A human would have spent hours digging through old tickets.

Why Slack? Because That's Where the Work Happens

One of the smartest design choices is that Blueberry lives inside Slack, the same place where incidents are already being discussed. Engineers don't have to switch to a separate tool or copy-paste logs into a chat window. They just start a thread, and Blueberry joins in, posting relevant information and proposed hypotheses. That's a huge win for adoption because it removes friction. If you make people leave their comfort zone, they won't use it. By embedding the AI in the workflow, Instacart made it a natural part of the process. Engineers can approve or dismiss suggestions, and they stay in control. Blueberry never changes production systems on its own.

I've seen other tools that require you to open a separate dashboard or type commands in a terminal—they just don't get used. Blueberry's Slack integration is a big reason it's been so widely adopted.

Architecture: More Than Just a Chatbot

Blueberry's architecture is worth studying because it shows what a production-grade AI assistant looks like. It's not a single model that tries to answer everything. It's a framework that combines AI reasoning with organizational knowledge. There are three core components:

• Tool-aware agents: Each agent can call internal tools to fetch data, like logs or deployment info.
• Persistent state: The system remembers what's been done, so it doesn't repeat itself or lose track mid-incident.
• Feedback loop: Engineers can review and correct the AI's suggestions, and that feedback is used to improve future performance.

This is a pattern that any team can replicate. You don't need a huge ML team to build something similar. You need a clear set of internal APIs, a history of incidents, and a way to tie it all together. At my last job, we built a mini-version for database failures with just a few hundred lines of Python and a Slack bot. It reduced our mean time to resolution by 20%.

Lessons for Backup and Recovery Teams

Now, you might be thinking, “This is about incidents, not backups.” But the same principles apply to backup and recovery scenarios. When a backup fails, or a restore takes too long, you have the same problem: you need to quickly understand what went wrong, what data is affected, and what the best recovery path is. Imagine an AI assistant that, when a backup job fails, automatically checks the last successful backup, analyzes the error logs, and compares it with past failures. It could suggest whether to retry, switch to a different snapshot, or escalate to a human. That's the same value proposition as Blueberry, just applied to a different domain.

I've talked to backup admins who spend hours sifting through logs after a failed backup. With a Blueberry-like system, they could get a diagnosis in minutes. The potential is huge.

The Numbers Speak: 99.9% Workflow Success

Instacart reports that Blueberry’s workflow success rate is 99.9%, with over 58,000 MCP tool calls in a month. It adapts to about 60 different teams' workflows. That's a massive scale, and it shows that AI can be reliable enough for production use. But don't get dazzled by the stats. The real takeaway is that the system was built with guardrails. It doesn't act on its own; it provides suggestions. It’s a force multiplier, not a replacement. Engineers still make the final call, and that's the right approach. I've seen too many AI systems that try to automate everything and end up causing more problems. Blueberry's approach is safer.

Building Your Own Blueberry-Like System

If you're considering something similar, start small. Pick one incident type, like backup failures, and build a tool that helps diagnose them. Use your existing incident history to train or ground the AI. Integrate it into your communication channel, whether that's Slack, Teams, or something else. Focus on context. The more data you can feed the AI, the better it will perform. That includes service maps, runbooks, and even past chat threads. And always keep a human in the loop. AI can speed up the process, but it shouldn't make irreversible decisions without approval.

I talked to a team at a mid-sized company that built a similar tool for their on-call engineers. They started with just one service and expanded from there. It took about three months to get it to a useful state, and they saw a 30% reduction in incident resolution time.

The Future of AI in Ops

Instacart's CTO, Anirban Kundu, sees Blueberry as part of a broader push into AI agents. That's the direction the industry is heading: not just chatbots, but agents that can take actions, like pinging the right team or preparing a rollback plan. For backup and recovery, the potential is huge. Imagine an AI that not only detects a failed backup but also triggers a test restore, alerts the right people, and provides a step-by-step recovery guide. That could cut downtime from hours to minutes. But it won't happen overnight. You need the underlying infrastructure: clean data, reliable APIs, and a culture that trusts AI to assist, not replace. Instacart's Blueberry is a great example of how to do it right, and there's no reason the backup world can't follow suit.

So, What's the Takeaway?

Here’s what I'd want you to remember from Instacart’s experience:

• Context beats raw model power. Feeding AI with years of incident history made accuracy jump from 60% to 90%.
• Integration matters. Putting the AI inside Slack meant engineers actually used it.
• Keep humans in charge. Blueberry suggests, but doesn’t execute. That builds trust and safety.
• Feedback loops are essential. The system learns from corrections, so it gets better over time.

If you're in charge of backup and recovery, consider how you can apply these ideas. You don't have to build a full AI system tomorrow, but you can start by capturing more data about your failures and making it accessible. That’s the foundation Blueberry was built on, and it can be yours too.