Data Consistency and High-Reliability Architecture for Multi-Agent Collaboration
TL;DR: Multi-agent systems contradict their own data because the communication architecture lacks state isolation and validation, not because the model isn't smart enough, so stable deployment takes centralized shared state, a validated task graph, 3-layer node validation, pre-delivery reconciliation, and gray-release evaluation.
Key ideas
- Errors like turning 500 yi into 1200 yi (yi is the Chinese unit meaning hundred million) come from Agents passing free text point to point: downstream Agents can't get the original facts and have to guess, and at aggregation they don't know whom to trust, so changing the prompt or the model can't fix the root cause.
- The data layer uses centralized shared state with strong typing, role-based write isolation and version records; at the task layer, the dependency graph generated by the large model is first checked programmatically, then each node is constrained by field trimming and a tool whitelist.
- Errors are stopped at the source: before writing back, a node passes 3 layers of validation (contract, evidence chain, consistency); on failure it retries in place, generally within 2 attempts, then takes a degraded branch so anomalies don't spill over; before delivery, global fact reconciliation flags unsupported data as pending verification or filters it out.
- When a core node fails, a state snapshot is saved and a human is notified; after the fix the run resumes from the point of interruption without restarting the whole chain; non-core tasks are degraded on timeout or repeated errors, so the main trunk runs through first.
- After launch, Trace ID tracking and quantified metrics give tuning an objective basis; any change to prompts, tools or the DAG must first pass evaluation regression and gray release, to keep a local change from dragging down the whole chain.
Section summaries
0:35Case Introduction and Four Structural Problems
Take report generation as an example: 4 Agents handle outline planning, data search, calculation and analysis, and report writing. The search found a baseline market size of 500 yi (yi is the Chinese unit meaning hundred million), the analysis Agent calculated 800 yi, and the writing Agent's summary said 1200 yi. The root cause is not how smart the model is, but that the communication architecture lacks state isolation and validation checkpoints. Breaking down the chain reveals 4 structural problems: a dilemma in passing context, dynamic task splitting without topology validation that is prone to deadlock, upstream errors spreading like a snowball, and a lack of local fault tolerance and fallback degradation.
▶ Watch from here
3:35Centralized Shared State and Three Rules
The solution is to stop letting Agents pass free text point to point and instead switch to a centralized shared state space for unified reads and writes, following 3 rules: constrain every business field with Pydantic strong typing, isolate write permissions by role (research can write only raw facts, analysis can write only its own analysis fields), and automatically record the version number and writing node on every read and write, so problems can be traced back to the source.
▶ Watch from here
5:05Task Dependency Graph Validation and Execution Safeguards
A task dependency graph generated by the large model can't be executed directly; it must first go through a programmatic validator that performs directed acyclic checks (ruling out circular dependencies and deadlocks), connectivity checks (every subtask has an input source and an output target), and merging of semantically duplicate tasks. At execution time, node behavior is further constrained: only the fields an analysis node needs are pushed to it to save tokens, and a tool whitelist limits the scope of calls (for example, a research node is given search only, with no code-execution permission), preventing overreach that would pollute the data.
▶ Watch from here
6:15Three-Layer Node Validation, Retry and Degradation
Before a node writes back to the shared state, it must pass 3 layers of validation: contract validation (whether the format matches the preset structure, with JSON parse failures intercepted on the spot), evidence-chain alignment (key data that can't be supported by the retrieved content is judged abnormal), and consistency comparison (whether the derived conclusion logically conflicts with facts already in the shared state). If any layer fails, the error reason is packaged as diagnostic information and sent back so the node retries locally, generally capped at within 2 attempts; if it still fails, it takes a degraded branch, such as filling in a default value or skipping a non-core step, keeping the anomaly locked inside a single node without affecting downstream.
▶ Watch from here
7:35Global Fact Reconciliation and Circuit-Breaker Fault Tolerance
Before the report is delivered, global fact reconciliation is performed first: every core figure and key conclusion in the draft is checked back against the shared state, and anything without support is flagged as pending verification or simply filtered out. Each node is also given a timeout and a retry cap: non-core tasks are degraded on timeout or repeated errors, so the main trunk runs through first; when a core node fails and the system can't handle it on its own, a state snapshot is saved and a human is notified, and after confirmation or correction execution can resume without rerunning the whole process from the start. Finally, it points out that a system to be maintained long term needs observability.
▶ Watch from here
8:55Observability Tracing and Quantified Metrics
Every task trigger generates a globally unique Trace ID, and each Agent's operations (calling tools, reading and writing state) are recorded as Spans, forming a clear timeline view. From this, quantified metrics are built: the data consistency rate shows how many anomalies the validation checkpoints intercepted, each node's share of token consumption reveals wasted spending, and the distribution of chain execution time pinpoints performance bottlenecks, so that tuning rests on objective data rather than blind guessing by feel.
▶ Watch from here
10:15Evaluation Regression and Gray-Release Process
Before a release, build a standardized evaluation dataset covering typical tasks, long-context multi-hop reasoning, and deliberately tricky abnormal inputs, and automatically run regression tests on every release to see whether consistency meets the target, whether the task completion rate has dropped, and whether the average per-task call cost has gotten out of control. Changes to prompts, tool orchestration or the DAG can't go straight to production: they must first pass static compilation checks, then be replayed against the benchmark dataset to confirm there is no regression; launch goes through gray release, first taking a small amount of real traffic and observing, then scaling up to full traffic gradually once the metrics are stable.
▶ Watch from here
11:15Summary of Multi-Agent Engineering Practice
A summary comparison: the communication mechanism changes from passing free text to structured shared state, paired with strong typing and permission isolation; task orchestration changes from leaving everything to the model's autonomous planning and black-box operation to a statically validated DAG with defined input and tool boundaries for each node; anomaly control changes from errors spreading all the way downstream to 3-layer validation at key nodes plus in-place retry and degradation that lock problems into a local scope; and delivery quality gains global fact reconciliation, circuit-breaker degradation and an automated evaluation pipeline. The value of multi-agent systems lies in decoupling tasks to handle complex business, and whether they land stably depends on using software engineering and state governance to build a deterministic, reliable foundation on top of model reasoning.
▶ Watch from here