Human-in-the-Loop Patterns

Knowledge

Not every decision should be made by an agent alone. An agent that automatically deploys code to production, sends customer emails, or executes database migrations needs human control at critical points. Human-in-the-loop (HITL) patterns define when and how a human intervenes.

When Must a Human Decide?

The decision of whether a human must intervene depends on two factors: risk and reversibility.

Examples:

• Autonomous: Code formatting, lint fixes, test execution
• Inform: Dependency updates, documentation changes
• Approval: Code deployment, API schema changes
• Active Confirmation: Database migration, deletion of production data, customer communication

The Three HITL Patterns

1. Approval Gate: The agent pauses at a defined point and waits for human approval before continuing.

2. Escalation: The agent recognizes it's reaching its limits and hands off to a human -- with full context.

3. Feedback Loop: The human gives the agent feedback after completing a task, which feeds into future decisions.

Understanding

Implementing an Approval Gate

import json
from enum import Enum
from dataclasses import dataclass, field
from datetime import datetime

class ApprovalStatus(Enum):
    PENDING = "pending"
    APPROVED = "approved"
    REJECTED = "rejected"
    TIMEOUT = "timeout"

@dataclass
class ApprovalRequest:
    task_id: str
    agent_name: str
    action: str
    context: dict
    risk_level: str
    status: ApprovalStatus = ApprovalStatus.PENDING
    reviewer: str = ""
    feedback: str = ""
    created_at: str = field(default_factory=lambda: datetime.now().isoformat())

class ApprovalGate:
    def __init__(self, notification_handler=None):
        self.pending_requests: list[ApprovalRequest] = []
        self.notify = notification_handler or self._default_notify

    def request_approval(
        self, task_id: str, agent_name: str, action: str,
        context: dict, risk_level: str = "high"
    ) -> ApprovalRequest:
        """Creates an approval request and notifies the reviewer."""
        request = ApprovalRequest(
            task_id=task_id,
            agent_name=agent_name,
            action=action,
            context=context,
            risk_level=risk_level
        )
        self.pending_requests.append(request)
        self.notify(request)
        return request

    def approve(self, task_id: str, reviewer: str, feedback: str = "") -> bool:
        """Approves a pending request."""
        for req in self.pending_requests:
            if req.task_id == task_id and req.status == ApprovalStatus.PENDING:
                req.status = ApprovalStatus.APPROVED
                req.reviewer = reviewer
                req.feedback = feedback
                return True
        return False

    def reject(self, task_id: str, reviewer: str, feedback: str) -> bool:
        """Rejects a request."""
        for req in self.pending_requests:
            if req.task_id == task_id and req.status == ApprovalStatus.PENDING:
                req.status = ApprovalStatus.REJECTED
                req.reviewer = reviewer
                req.feedback = feedback
                return True
        return False

    def _default_notify(self, request: ApprovalRequest):
        print(f"[APPROVAL NEEDED] {request.agent_name}: {request.action}")
        print(f"  Risk: {request.risk_level}")
        print(f"  Context: {json.dumps(request.context, indent=2)}")

Escalation Pattern

@dataclass
class EscalationRule:
    condition: str       # e.g., "confidence < 0.7"
    threshold: float
    target: str          # e.g., "senior-developer", "team-lead"
    message_template: str

ESCALATION_RULES = [
    EscalationRule(
        condition="confidence",
        threshold=0.7,
        target="senior-developer",
        message_template=(
            "Agent {agent} is uncertain about task {task}. "
            "Confidence: {confidence:.0%}. Please take over."
        )
    ),
    EscalationRule(
        condition="retries",
        threshold=3,
        target="team-lead",
        message_template=(
            "Agent {agent} failed to solve task {task} after {retries} "
            "attempts. Human intervention needed."
        )
    ),
    EscalationRule(
        condition="cost",
        threshold=5.0,
        target="engineering-manager",
        message_template=(
            "Agent {agent} has already spent ${cost:.2f} on task {task}. "
            "Budget limit reached."
        )
    ),
]

The HITL Workflow Visualized

The following diagram shows the workflow of a production agent with approval gates. Click on the checkpoints to see when the agent acts autonomously and when human approval is needed:

Apply

Reflect

Human-in-the-loop isn't a sign of weakness in an agent system -- it's a sign of maturity. The best production agents aren't the ones that do everything alone, but the ones that know when to involve a human. The art lies in finding the right balance: Too much human control makes the agent useless, too little makes it dangerous. The risk-reversibility matrix from this section is your tool for making this decision.