Apply 23 settlement-policy documents to 18 operational CSVs, and pull out the transactions that broke the rules and the clawback amount

18 CSVs in the ERP, 23 policy documents in PDF

The day we imagined for this company is fragmented across three tools. Data gets looked up in the ERP, crunching happens in Excel, and rules get applied inside someone's head. For reference, an ERP is the backbone system that manages a company's orders, inventory, and accounting in one place.

• 18 kinds of operational-data CSVs. Tabular data files like orders, inventory, shipments, and settlement
• 23 kinds of documents, like settlement policies and related statutes. Documents a person opens as a PDF and reads
• The ERP screens. Sales looks at order status, CS at returns and claims, purchasing at inbound, logistics at outbound, finance at settlement. Each looks at their own screen, but those screens don't connect to one another on their own
• Excel sheets. The intermediate artifact you download out of the ERP and line up by hand against shipment and return data
• The lead's memory of which clause of the policy applies to which transaction. This connection lives only inside a person's head

Compare two data sets by eye, recall the rules line by line

Here's the procedure for cleaning up a single settlement. The lead queries the order data in the ERP, downloads it to Excel and manually lines it up against the shipment and return data, and finally opens the settlement-policy document to judge for themselves, "does this clause fit this transaction?" Connecting data to data, and connecting data to documents — it's all on the person.

There are two holes. First, there's no mechanism to automatically check consistency. With no tool to check on its own whether orders and shipments line up, or whether a given transaction properly went through the steps the settlement policy prescribes, a person has to put two data sets side by side, compare them by eye, and recall the rules line by line. Second, to answer a single question, you have to open the order, inventory, shipment, and settlement tables in turn and stitch them together by matching keys. Time melts away in this process, focus wears down, and mistakes slip in.

So couldn't you just bolt on the ordinary RAG everyone uses? RAG is a method that searches documents to compose an answer, but that alone falls short. RAG is strong at finding and reading out "what the rule says," but it can't apply that rule to actual transaction data and calculate "this transaction violated the rule, and the clawback amount is X." Reading a document and executing a document against data are different orders of work.

Put the 18 CSVs and the 23 documents into the same workspace

To check this hypothesis, there was one thing to prepare. Putting 18 kinds of operational-data CSVs and 23 kinds of documents — settlement policies, related statutes, and the like — into the same workspace is all of it.

Questions that cross tables, questions that execute the rules

"Line up orders against shipments and find the transactions that haven't gone out yet"
→ A multi-table reconciliation runs across orders and shipments.
   120 transactions come out looking unshipped.
   (a reference figure based on synthetic data)

"Are there transactions that skipped a step the settlement policy prescribes?
 If so, the clawback amount too."
→ It applies the settlement policy that came in as a document
   to the actual data, verifying whether each transaction went
   through the legitimate steps the policy prescribes.
   The transactions that violated policy get singled out,
   and the resulting clawback amount is computed.

"Pull this new-product line together from the data and make it a report"
→ Back comes an answer that aggregated the data, applied the rules,
   and then checked the result back against the data.
   It's the kind of answer that finding and reading documents alone won't produce.

These answers come out because the agent carries out queries that aggregate data and the work of cross-validating documents together within one flow. What a person used to do — reading the PDF and mapping it onto the data in their head — the agent executes directly on top of the data.

We didn't stop at one settlement verification. To see whether it truly goes beyond the limits of ordinary RAG, we also ran 21 hands-on, task-style scenarios — new-product analysis, demarketing, report writing, tax calculation, and the like. Demarketing refers to marketing that deliberately reduces a particular demand. What these tasks have in common is that finding and reading documents alone won't produce an answer. Only after you aggregate the data, apply the rules, and then check the result back against the data does a conclusion stand.

What changes

In this experiment, the order-to-shipment reconciliation and the rule application ran on top of the data inside the workspace, not on a person's memory. When there's a mechanism to automatically check consistency between data, and between data and documents, the gap for omissions and misapplications shrinks.