Multi-agent Coordination and Evaluation

When you try to get a single language model to write code, query the database, send emails, and analyze financial data all at once, failure is guaranteed. The hallucination rate skyrockets, and the system prompt becomes an unmanageable monster. The solution is divide and conquer.

The core concept: The end of the one-man band

If you run a company, you don't hire a single person to be the accountant, the programmer, the janitor, and the head of sales simultaneously. It's a disaster waiting to happen. You create departments and put a manager in place to coordinate.

In AI, this is called multi-agent architecture. Instead of an overloaded "one-man band agent," you deploy several specialist agents. One knows how to search for information, another knows how to write, and a supervisor agent decides who to assign what to. They pass the state between them until the supervisor determines the job is done.

Coordination patterns

I've tested everything from strict hierarchical structures to agent markets where they "bid" for tasks. In the end, what works best in production is the Supervisor-Worker (Routing) pattern. The user talks to the supervisor, the supervisor delegates to worker A, receives the result, passes it to worker B for review, and hands you back the answer.

The main challenge here is evaluation. How do you know the "Writer" agent didn't make up the data handed to it by the "Researcher"? The answer is LLM-as-a-judge or evaluations based on rigid assertions in your CI/CD pipeline.

Practical Example: A graph of Supervisor and Specialists

Imagine we use a graph engine to orchestrate delegation. Instead of putting in all the plumbing code, let's look at the supervisor's decision-making structure:

from typing import TypedDict, Literal

class MultiAgentState(TypedDict):
    task: str
    draft: str | None
    research_notes: str | None
    next_agent: str

def supervisor_node(state: MultiAgentState) -> dict:
    """
    The supervisor decides who has to act next.
    """
    if not state.get("research_notes"):
        # No data, we send it to the researcher
        return {"next_agent": "Investigador"}

    if not state.get("draft"):
        # We have data but no draft, we send it to the writer
        return {"next_agent": "Redactor"}

    # If we have a draft and notes, we assume we're done
    # In a real system, there would be an 'Evaluator' agent here checking quality
    return {"next_agent": "FIN"}

def investigador_node(state: MultiAgentState) -> dict:
    """Simulates the researcher agent"""
    notas = f"Datos encontrados sobre: {state['task']}..."
    return {"research_notes": notas, "next_agent": "Supervisor"}

def redactor_node(state: MultiAgentState) -> dict:
    """Simulates the writer agent"""
    texto = f"Basado en las notas ({state['research_notes']}), redacto este texto."
    return {"draft": texto, "next_agent": "Supervisor"}

# The system would queue these nodes based on the "next_agent" value.
# Supervisor -> Investigador -> Supervisor -> Redactor -> Supervisor -> FIN

This design allows you to swap the "Researcher" model for a cheaper or faster one without affecting the quality of the "Writer". It isolates problems, which is exactly what you want when moving from a prototype to a robust and auditable system.