Green, Amber, Red: What an Automated Review Should Tell You
Including the checks it could not finish, and why those matter more than the passes.
TL;DR
Automated document review fails in two directions. It flags everything and people stop reading, or it quietly passes items it could not actually read, which is worse because the record now says verified. The fix is a third answer. A review that reports what it could not check is more useful than one that implies it checked everything, and the difference shows up the moment somebody asks the institution to prove what it looked at.
Fannie Mae's Selling Guide describes what happens when a quality control review turns something up.
"When a lender's review identifies discrepancies between the data used in underwriting and the data verified through the QC process, the lender must reassess the underwriting decision based on the newly verified information to determine whether the loan remains eligible as delivered to Fannie Mae."
The phrase worth pausing on is the data verified through the QC process. Verification is an affirmative act with a consequence attached. Something was established, and because it was established, a decision may have to be revisited. Which means a review that reports a field as passed when it could not actually read that field has done something quietly serious. It has asserted a verification nobody performed. The institution's record now says the data was checked. The population looks cleaner than it is. And the one thing the review was built to produce, a trustworthy account of what is known, has been undermined in a way that is almost impossible to detect afterward.
This is the most common design failure in automated document review, and it is easy to avoid once somebody names it.
The two ways automated review goes wrong
The first failure is the one everyone complains about. The system flags too much. Tolerances are set conservatively, every variance becomes an exception, and a reviewer opens a queue of four hundred items knowing that most of them are noise. Within a few weeks the team develops a working habit of skimming, and the exceptions that actually matter are buried among the ones that do not. Alert fatigue is a well understood problem and institutions recognize it quickly, because the pain is immediate and visible.
The second failure is quieter and does more damage. The system encounters something it cannot establish, a page that will not parse, a handwritten figure it cannot read with confidence, a document that never arrived, and it resolves that uncertainty as a pass. Nobody complains, because nothing appears in the queue. The report at the end of the cycle shows a clean population.
From our experience, institutions arrive at automation worried about the first failure and get caught by the second. The first one wastes reviewer hours. The second one produces a record that is wrong in a direction nobody is looking.
Pass, fail, and the answer nobody builds
Most review tooling offers two outcomes. The check passed or the check failed. Both are assertions about something the system was able to determine. The missing outcome is the honest one. The system attempted the check and could not complete it, and here is why.
A concrete case makes it obvious. Take a $430M credit union running post-close review on new member accounts. One account is missing its signature card. That is a finding in its own right, and any reasonable system reports it. But there is a second check in the procedure that compares the account holder's name on the signature card against the core record, and that check cannot run, because the document it depends on is not there.
A system with two outcomes has to pick one. If it reports that comparison as a pass, it has stated that the names match when nobody has seen one of them. If it reports a failure, it has manufactured a discrepancy that may not exist and sent a reviewer chasing it. Either answer is wrong, and both are wrong in ways the reviewer cannot see from the output.
With a third outcome the result is simply accurate. The signature card is missing, which is a break. The name comparison could not be performed, which is a separate item carrying a reason that points directly back at the first one. Two findings, one root cause, and a reviewer who understands the situation in about four seconds. Reported this way, a population resolves into three states. The accounts that cleared, the accounts with a genuine break, and the accounts where something could not be established. Green, amber and red, which happens to be the same vocabulary risk committees already use for everything else.
What a flag should show you
A finding a reviewer cannot trace is a finding they have to redo. Three things make a flag usable. The rule that failed, named the way the procedure names it rather than as an internal code. The document the finding came from. And the specific place in that document where the problem sits, so the reviewer can confirm it by looking rather than by searching.
That third element gets dismissed as a convenience feature, and it is the one that decides whether automation actually saves anything. A reviewer who has to open a forty page packet and hunt for the field the system is referring to is doing most of the work the system was supposed to remove. Multiply by a cycle's worth of findings and the time saving disappears entirely.
The same applies to the could-not-check items. An item reporting that a check did not run should say what it needed and did not have. Without that, the reviewer is left guessing whether the system was confused, the document was illegible, or something is genuinely absent, and those three situations call for completely different responses.
Where the a real human decides
None of this removes human judgment from the process, and a system that tried to would be solving the wrong problem. The useful division is that the machine narrows and the person decides. Fetching the documents, correlating them to accounts, running the comparisons, and recording what did not match is mechanical work that scales with volume. Deciding what a finding means is not. A reviewer looking at a flagged item has to determine whether it is a genuine break requiring remediation, a documentation gap somebody needs to chase, or a false positive that should be set aside with a note explaining why.
That decision is where an experienced reviewer's judgment actually lives, and it is the part most likely to be rushed when a cycle runs tight. An institution that automates the mechanical half has the same team spending a greater share of its time on the part that required them in the first place. We have written more fully about that shift in automating deposit account QC without adding headcount.
The record is the real output
The findings matter on the day. The record of how they were handled matters for considerably longer. A complete record says what was checked, what the result was, what could not be established and why, who made each disposition, and when. Assembled as a by-product of the review itself, that record exists before anybody asks for it. Assembled afterward from spreadsheets and recollection, it takes a week and is weaker for having been reconstructed.
This is also where the two-outcome problem comes home. An institution that cannot distinguish between verified and assumed in its own records has no way to answer a question about coverage. Asked how much of the population was actually examined, the honest answer becomes some of it, and we are not sure which parts, which is a difficult sentence to say out loud in a review meeting.
What to ask a vendor about indeterminate handling
So, what does your system do when it cannot read a document? Ask for a demonstration rather than a description, using a deliberately poor scan and an account where a required document is absent. Watch what the output says. If the system reports those as passed, that is the answer, and it tells you how the tool will behave on exactly the files most likely to carry a problem.
Two follow-ups are worth having ready. Can the "could-not-check" items be reported separately from passes, so coverage can be measured honestly. And does the reason travel with the item, so a reviewer knows what was missing rather than only that something was.
Have a Problem to Solve?
An institution does not need a platform decision to start here. Taking one month of completed reviews and asking how many checks actually ran, as opposed to how many were reported as clear, is usually enough to show whether this is a live problem. We provide automated QC and data quality review for community banks and credit unions, built so that a check that cannot run says so rather than passing quietly, with the institution's own reviewers making every disposition. If any of the above sounds like something you have run into, reach out to Shore Group and we can talk it through.
Frequently Asked Questions
What does indeterminate mean in an automated QC review?
It means the system attempted a check and could not complete it, so it is reporting neither a pass nor a failure. Common causes are an illegible scan, a field the system could not read with sufficient confidence, or a dependency on a document that is missing. Treated as its own category, these items are a direct measure of how much of the population was genuinely examined.
Why would a check not run?
Usually because something it depends on is unavailable. A comparison between two documents cannot be performed if one of them is absent, and a field cannot be validated if the page it sits on will not parse. The important part is that the reason should be reported alongside the item, because a missing document and an unreadable one call for different responses.
Is a false positive the same as a break?
No. A break is a genuine discrepancy between what the record says and what the documents show. A false positive is the system flagging something that turns out, on review, not to be a discrepancy at all. Both appear in the queue and only a person can tell them apart, which is why the disposition and the reasoning behind it should be recorded.
Who decides whether to waive a finding?
The institution's reviewers, working to its own written procedures. An automated system should narrow the population and present the evidence. Confirming a finding, waiving it, or marking it a false positive is a judgment, and the record should capture who made it and when, so the decision can be explained later.
What should a QC audit trail contain?
At minimum: what was checked, what each check returned including the items that could not be completed, which document each finding came from, what disposition a reviewer reached, the reasoning behind it, and the timestamps. Produced as part of the review rather than assembled afterward, it answers questions about coverage and handling without a reconstruction exercise.