I Build Reliable Agents From Unreliable Parts: My Dev Strategy

Hey everyone, Leo here from agntdev.com! Today, I want to talk about something that’s been on my mind a lot lately, especially as I’ve been wrestling with a few new agent projects. It’s about the “Dev” side of agent development, specifically, how we go about building reliable agents from unreliable parts. Yeah, you heard me. Because let’s be honest, that’s the reality for most of us, right?

We’re not usually working with perfectly curated, enterprise-grade APIs and services. More often than not, we’re stitching together open-source models, third-party APIs with questionable rate limits, and maybe even a few homegrown microservices that, let’s just say, have a personality. And yet, the expectation is always that our agents should just… work. Consistently. Reliably. Even when the underlying components are throwing a tantrum.

I’ve been down this road so many times. I remember a project last year where I was building an agent to help manage my open-source contributions. It needed to interact with GitHub’s API, a sentiment analysis model hosted on a free tier, and a custom notification service I’d whipped up in a weekend. Each of these had its quirks. GitHub would sometimes rate-limit me unexpectedly, the sentiment model would occasionally time out, and my notification service… well, let’s just say it had a habit of forgetting its manners after an hour of uptime. If I hadn’t built in some serious safeguards, the whole thing would have collapsed like a house of cards.

So, today, I want to share some practical strategies and patterns I’ve adopted to make my agents more resilient, even when the pieces they’re built from are anything but.

The Inevitable Truth: Things Will Break

First off, let’s just accept this as gospel. No API is 100% available. No model is 100% accurate. No network is 100% stable. Once you embrace this, you can start designing for failure, which, counterintuitively, makes your agent more successful.

The problem I see often, especially with newer developers exploring agent work, is that they assume success on every external call. They write code like this:

response = external_api.call_method(data)
# Assume response is always perfect and proceed
processed_data = process_response(response)

And then, when external_api.call_method throws a connection error, or returns a 500, or just sends back malformed JSON, the whole agent grinds to a halt. We can do better.

Strategy 1: solid Retries with Backoff

This is probably the most fundamental technique, and yet it’s often implemented poorly or not at all. Simply retrying immediately after a failure is usually a bad idea. If the external service is down, you’re just hammering it more, potentially making things worse or getting yourself rate-limited.

The key is exponential backoff. This means waiting progressively longer periods between retries. It gives the external service a chance to recover and reduces the load you’re putting on it.

Example: Python with Tenacity

For Python, my go-to library for this is Tenacity. It makes adding retry logic incredibly clean.

import random
import logging
from tenacity import retry, wait_exponential, stop_after_attempt, retry_if_exception_type

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class ExternalServiceError(Exception):
 """Custom exception for external service failures."""
 pass

# Simulate an unreliable external API call
def call_unreliable_api(data):
 if random.random() < 0.6: # 60% chance of failure
 logger.warning(f"API call failed for data: {data}")
 raise ExternalServiceError("Simulated API failure or timeout")
 logger.info(f"API call successful for data: {data}")
 return {"status": "success", "result": f"processed_{data}"}

@retry(wait=wait_exponential(multiplier=1, min=4, max=10),
 stop=stop_after_attempt(5),
 retry=retry_if_exception_type(ExternalServiceError))
def get_processed_data_with_retries(input_data):
 logger.info(f"Attempting to call API for: {input_data}")
 return call_unreliable_api(input_data)

if __name__ == "__main__":
 try:
 result = get_processed_data_with_retries("some_important_item")
 print(f"Final result: {result}")
 except ExternalServiceError as e:
 print(f"Failed after multiple retries: {e}")
 except Exception as e:
 print(f"An unexpected error occurred: {e}")

In this snippet:

• wait_exponential makes the waits longer with each retry (4s, then ~8s, then ~16s, etc., up to 10s max).
• stop_after_attempt(5) means it will try a maximum of 5 times.
• retry_if_exception_type(ExternalServiceError) ensures it only retries for specific errors, not for, say, a KeyboardInterrupt.

This pattern is a lifesaver. I use it for database connections, HTTP requests, and even for internal communication between agent modules when I know one might be temporarily overloaded.

Strategy 2: Circuit Breakers to Prevent Cascading Failures

Retries are great for transient errors. But what if the service is completely down? Repeatedly retrying will just exhaust your resources and potentially make the problem worse for the external service if it’s struggling to recover. This is where the Circuit Breaker pattern comes in.

Think of it like an electrical circuit breaker in your house. If there’s a fault (too many failures), it “trips,” preventing more current from flowing and protecting the system. After a while, it can be reset, but it won’t keep trying to send current through a shorted wire.

For agents, a circuit breaker monitors calls to an external service. If the failure rate crosses a certain threshold within a given time window, the circuit “opens.” When open, all subsequent calls to that service immediately fail without even attempting the call. After a configurable “timeout” period, the circuit moves to a “half-open” state, allowing a limited number of test calls to see if the service has recovered. If those succeed, it closes; if they fail, it opens again.

Why it matters for agents:

• Resource conservation: Your agent isn’t wasting time and resources trying to call a dead service.
• Faster failure: Instead of waiting for a timeout, your agent gets an immediate failure signal, allowing it to handle the situation (e.g., use a fallback, log the issue, notify an operator).
• Protects external services: Prevents your agent from DDOSing a struggling service.

I usually implement this using libraries. For Python, Pybreaker is excellent.

import time
import random
import logging
from pybreaker import CircuitBreaker, CircuitBreakerError

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

class ExternalAPIClient:
 def __init__(self):
 # Configure the circuit breaker:
 # 3 failures in a row within 60 seconds will open the circuit.
 # It stays open for 5 seconds.
 self.breaker = CircuitBreaker(fail_max=3, reset_timeout=5, exclude=[TypeError]) # Don't break on TypeErrors

 def _unreliable_call(self, data):
 if random.random() < 0.7: # 70% chance of failure
 logger.warning(f"Simulating internal API error for data: {data}")
 raise ConnectionError("Service unreachable")
 logger.info(f"API call succeeded for data: {data}")
 return {"result": f"processed_{data}"}

 def process_data(self, data):
 try:
 return self.breaker.call(self._unreliable_call, data)
 except CircuitBreakerError:
 logger.error(f"Circuit is open! Not calling API for data: {data}")
 # Fallback logic here: return cached data, default value, or raise a more specific error
 return {"result": "fallback_data", "source": "circuit_breaker"}
 except Exception as e:
 logger.error(f"Error during API call (not circuit breaker related): {e}")
 raise

if __name__ == "__main__":
 client = ExternalAPIClient()
 for i in range(15):
 print(f"\n--- Attempt {i+1} ---")
 try:
 result = client.process_data(f"item_{i}")
 print(f"Result: {result}")
 except Exception as e:
 print(f"Handled error: {e}")
 time.sleep(1) # Simulate some delay between calls

Run this, and you’ll see the circuit open after a few failures, then eventually try to half-open, and maybe even close again if the simulated service starts behaving.

Strategy 3: Idempotency for State-Changing Operations

This is crucial for any agent that modifies external state (e.g., creating a record, sending an email, initiating a payment). If your agent tries to perform an action, and the network blips, or the external service times out, how do you know if the action actually happened?

If you just retry without considering idempotency, you might accidentally perform the action twice. Imagine sending the same email twice, or worse, charging a customer twice. Not good.

An operation is idempotent if performing it multiple times has the same effect as performing it once. For example, setting a value (SET x = 5) is idempotent. Incrementing a value (x = x + 1) is not.

How to achieve idempotency:

• Use unique request IDs: When making a state-changing API call, include a unique, client-generated ID in the request header (e.g., X-Idempotency-Key). The external service can then use this key to detect duplicate requests and return the original response without re-processing.
• Design idempotent APIs: If you control the API, design endpoints that are naturally idempotent. For example, instead of a “create order” endpoint, have an “upsert order” endpoint that can create or update based on a unique order ID.
• Check status before retrying: After a failed state-changing operation, if the API supports it, query the status of the resource using the unique ID before attempting a retry.

While I don’t have a direct code snippet for this (it’s more about API design and client-side logic), here’s how your agent’s thought process might look:

# Agent's pseudo-code for an idempotent operation
transaction_id = generate_unique_id()
payload = {"data": "some_value", "idempotency_key": transaction_id}

try:
 response = external_payment_api.process_charge(payload)
 # Success! Store transaction_id and response.
except (ConnectionError, TimeoutError, APIError) as e:
 # Oh no, it failed. Did the charge go through anyway?
 logger.warning(f"Payment failed, checking status with ID: {transaction_id}")
 try:
 status_response = external_payment_api.get_transaction_status(transaction_id)
 if status_response.get("status") == "completed":
 logger.info(f"Payment {transaction_id} was actually successful on retry check.")
 # Treat as success, store info.
 else:
 logger.info(f"Payment {transaction_id} truly failed, attempting retry (with same idempotency key).")
 # This is where you'd retry with the *same* transaction_id
 # The payment API should recognize it and not double-charge.
 response = external_payment_api.process_charge(payload)
 # ... handle retry success/failure
 except Exception as check_e:
 logger.error(f"Could not even check transaction status for {transaction_id}: {check_e}")
 # Need to log for manual review, or move to dead-letter queue

This requires cooperation from the external service, but it’s a critical pattern for building truly reliable agents that handle financial or other sensitive operations.

Strategy 4: Fallbacks and Graceful Degradation

Sometimes, an external service is just completely unavailable, and there’s no hope of retrying or waiting. In these cases, a good agent doesn’t just crash; it finds a way to provide a degraded but still useful experience.

This could mean:

• Using cached data: If your agent needs specific data from a service, but the service is down, can you use a stale version from a cache?
• Providing default values: If an AI model for sentiment analysis is down, can you simply classify all input as “neutral” or “unknown” for a period, rather than failing the entire agent flow?
• Switching to a backup service: If your primary translation API is down, can you route requests to a secondary, perhaps less performant or more expensive, one?
• Skipping optional steps: If a non-critical enrichment step fails, can the agent just proceed without that enrichment, perhaps logging a warning?
• Notifying users/operators: At the very least, gracefully fail and clearly communicate the problem to the user or system operator.

My anecdote about the notification service failing? My fallback was simple: if my custom notification service went down, the agent would just log the event locally and send an email to *me* saying “Hey, your notification service is probably down again, check the logs.” Not ideal for end-users, but it prevented the entire agent from jamming up and ensured I knew something was wrong.

Actionable Takeaways for Your Next Agent Project

1. Assume failure: Design your agent from the ground up expecting external dependencies to fail.
2. Implement retries with exponential backoff: Use libraries like Tenacity (Python) or similar patterns in other languages for transient errors.
3. Deploy circuit breakers: Prevent cascading failures and conserve resources by “tripping” the circuit when a service is consistently failing. Pybreaker is a good start.
4. Prioritize idempotency for state changes: Ensure operations like payments or record creation don’t duplicate if a retry occurs. Use unique IDs.
5. Plan for graceful degradation: Identify critical vs. non-critical dependencies and build fallbacks. What’s the “least bad” thing your agent can do when a dependency goes kaput?
6. Monitor aggressively: All these strategies generate logs. Make sure you’re collecting and analyzing those logs to understand *why* things are failing and how often.

Building reliable agents isn’t just about clever algorithms or powerful models. It’s fundamentally about engineering solidness into every layer, especially when dealing with the messy reality of external dependencies. By applying these strategies, you’ll spend less time debugging mysterious agent crashes and more time building genuinely useful, dependable autonomous systems.

What are your go-to strategies for dealing with flaky external services? Drop a comment below, I’d love to hear your war stories and solutions!

🕒 Last updated: March 26, 2026  ·  Originally published: March 16, 2026