How to Evaluate Automated QC Software for Banks and Credit Unions
The evaluation criteria that never make it into the RFP.
TL;DR
Most automated quality control tooling assumes an IT department and a team to operate it, which is why evaluations at community institutions stall after the demo rather than before it. The questions that decide whether a tool ever goes live are rarely the ones in the RFP. And whichever way an institution goes, federal guidance is clear that hiring a third party does not move the responsibility anywhere.
The federal banking agencies put it plainly in their guidance on third-party relationships. A banking organization's use of third parties does not diminish its responsibility to meet these requirements to the same extent as if its activities were performed by the banking organization in-house.
That sentence is the honest starting point for any conversation about buying quality control tooling, and it cuts against every vendor in the category including us. Nobody sells a product that transfers accountability. The institution still answers for the program, the findings, and what it did about them. Which is exactly why the evaluation matters more than the demo.
The reason QC tooling stalls at community institutions has very little to do with whether the software works. It has to do with questions that rarely appear in a request for proposal, and that only surface six months later when the project is sitting in a queue. It also matters because most institutions start this process one step too late. The first decision is not which product to buy. It is what kind of arrangement the institution is actually in a position to run, and the honest answer to that narrows the field faster than any feature comparison will.
Why QC tooling stalls at institutions this size
The common assumption is that price is the barrier. It usually is not.
The barrier is what the institution has to supply before anything runs. Most quality control platforms assume the buyer has an integration team, a data engineer who can map a core extract, and somebody whose job will include administering the platform afterward. At a $890M community bank with four people in IT and a core conversion already on the roadmap, every one of those assumptions is a problem.
So the evaluation goes well, the committee approves, and the work lands in an implementation queue behind the things that cannot wait. A year later the institution has a signed agreement, a partially configured system, and a QC team working the way they always did.
Top questions worth asking before the demo
The agencies describe third-party risk management as a life cycle with five stages: planning, due diligence and selection, contract negotiation, ongoing monitoring, and termination. The questions below map onto those stages deliberately, because an institution has to document this process anyway and it is easier to run one exercise than two.
At the planning and due diligence stage, these questions matter most:
What does onboarding actually require from your team, stated in people and weeks rather than in a phase diagram? Ask who at your institution has to do what, and when.
Whose rules are the rules? Some tools arrive with a fixed ruleset and expect the institution to adapt its procedures to match. Others take the institution's written procedures as the input. That distinction determines whether the tool will still be useful after the next policy change.
What happens when your procedure changes, and who operates the thing day to day? A tool needing a vendor ticket to adjust a rule will fall out of date quietly. A tool needing an internal administrator is a hiring decision wearing a software label, and that cost belongs in the comparison.
What does the system do when it cannot read a document? Every automated review encounters pages it cannot parse, fields that are illegible, and documents that never arrived. If a vendor tells you their tool is perfect, run. A tool that reports those as passed has made the population look cleaner than it is. A tool that reports them as a separate state has told the truth. Ask for a demonstration of that specific case rather than a description of it
What do you own if the relationship ends? Meaning the findings, the audit trail, the configured ruleset, and the underlying data, in a format somebody else can read.
What does coverage mean in the pricing model? because a tool priced per document behaves very differently from one priced per account once an institution decides to review the whole population rather than a sample.
Where licensed software is the better answer
A comparison that goes one way on every line is marketing rather than analysis, so it is worth being direct about where a licensed platform beats a managed service. An institution with genuine internal IT capacity and a team that wants to own the ruleset outright should buy software. The control is real and the knowledge stays in-house. An institution whose procurement rules favor perpetual licences will find a licence easier to approve. And an institution already running a platform that covers adjacent work should check whether the incumbent can extend, since every additional vendor relationship carries its own due diligence burden.
A managed arrangement makes more sense where the constraint is operating capacity rather than budget. If the honest answer to "who runs this day to day" is that nobody has the hours, a license buys a tool the institution cannot staff. That is the most common pattern we see at this size, and it is the reason the question matters more than the feature list. Neither model changes who answers to the examiner. Both require the same due diligence file.
A different approach worth understanding
The comparison above treats this as a choice between buying a license and having nobody to run it. There is a different approach worth understanding on its own terms. In a managed data quality service, the provider brings the technology and the people who operate it. The institution supplies a core extract and the supporting documents, and receives findings back. The mechanical half of the work moves. The judgment does not. Deciding whether a flagged item is a genuine break, a gap worth chasing, or a false alarm stays with the institution's reviewers, who also record why they decided what they decided.
The configuration question works differently in this model too. Rather than the institution adapting its procedures to a fixed ruleset, its own written standard operating procedures become the rules, assertion by assertion. When a policy changes, the provider changes the checks. For an institution that has spent years refining how it opens accounts or clears exceptions, that difference decides whether the tooling reflects how the place actually works or how a vendor assumed it would.
The commercial shape tends to differ as well. Work of this kind can start on a historical sample and a short engagement rather than a multi-year license. Ask any provider what the shortest honest commitment is, and treat a long minimum term as information about their confidence. Managed services are sold as an outcome rather than a license, which is why they tend to be priced against the work performed instead of against seats or modules.
There is a real trade-off, and it should be tested rather than assumed. In this model the institution depends on the provider's responsiveness when a procedure changes or a question comes up mid-cycle. That dependency is worth probing during a pilot, when it costs nothing to find out. And none of it changes the sentence this post opened with. The responsibility stays exactly where it was.
So Who Does All The Work?
The question to ask is not who provides the tool, and not who owns the license. Who, specifically, spends the hours fetching documents, running the comparison, and producing the report at the end of the cycle. If the answer after implementation is the same three people doing it now, with a system they also have to maintain, the institution has bought a tool and kept the problem.
It is the difference between buying a tool and buying an outcome. A tool is something the institution now owns and has to operate. An outcome is a result that arrives on a schedule, with somebody accountable for it arriving. Both are legitimate purchases. They are not the same purchase, and an evaluation that compares them as though they were will reliably pick the wrong one.
What proof before commitment should look like
Vendor demo data proves that a system works on documents chosen because the system works on them. A meaningful proof runs against the institution's own historical files, at real scan quality, including the ones that came out of archive storage sideways, the forms completed by hand, and the account where a document is simply missing. That last case is the most informative, because it shows how the system behaves at the edge of what it can establish.
Ask for a findings list rather than an accuracy percentage. The QC team should be able to read it, compare it against their own conclusions on the same files, and argue with it. If they cannot tell why the system reached a conclusion, the institution will re-review everything by hand.
Where this guidance is heading
The current rules are the interagency guidance issued jointly by the Federal Reserve, FDIC and OCC in June 2023. In late September 2026 those agencies, together with the National Credit Union Administration, proposed new guidance that would rescind and replace that 2023 version. It is a proposal and it has not been finalized, so the 2023 guidance governs today and an evaluation run against it is on solid ground.
Two things make it worth following. The proposal would bring NCUA in alongside the banking agencies, giving credit unions guidance in the same framework rather than adjacent to it. And any replacement will shape what a due diligence file is expected to contain, which is the document this exercise produces. The agencies publish proposals and supervisory letters on their own sites, and those are the places to watch rather than a vendor summary.
Have a Problem to Solve?
The institutions that get the most out of this exercise run it on one contained piece of work first. A single account type, a single month, a real sample. What that reveals beats a procurement cycle run on assumptions. We work with community banks and credit unions on exactly this kind of review, with the institution's own written procedures as the ruleset and its own team making every judgment call.
If an evaluation is underway and the questions above are useful, reach out to Shore Group and we are happy to be measured against them alongside anyone else.
Frequently Asked Questions
Does outsourcing QC transfer regulatory responsibility?
No. Federal guidance states that a banking organization's use of third parties does not diminish its responsibility to meet requirements to the same extent as if the activities were performed in-house. Whoever performs the review, the institution answers for the program and its results. That principle is the reason the due diligence and ongoing monitoring stages exist.
What should a QC vendor due diligence file contain?
At minimum, evidence covering the stages the agencies describe: what you considered at planning, what you reviewed during due diligence and selection, what the contract says about performance and termination, how you will monitor the relationship, and what happens to your data and findings if it ends. Security documentation and any independent audit reports belong in the same file.
How long does QC software implementation usually take?
It varies enough that any single number would be misleading. The more useful question is what the institution has to supply. Ask for onboarding requirements expressed in your people and your weeks. If a vendor cannot answer that specifically, the estimate is not grounded in your situation.
Can a managed service work without core integration?
It can, and for institutions without integration capacity that is usually the point. If the arrangement runs on a periodic extract from the core and a set of supporting documents, no interface build is required. It is worth confirming exactly what files are needed and who produces them, since an extract somebody has to assemble by hand each month is a recurring cost in disguise.
What is a managed data quality service?
It is an arrangement where a provider supplies both the review technology and the people who operate it, configured to the institution's own written procedures. The institution sends a core extract and supporting documents and receives findings back, while its own reviewers keep the judgment calls and the disposition record. It is bought as an outcome rather than as a licence, which is the main thing separating it from software an institution installs and staffs itself.
What happens to our data if we end the relationship?
That belongs in the contract rather than in a conversation. Establish what you receive on termination, in what format, and over what period, covering the findings, the audit trail, the configured rules, and the source data. The termination stage is part of the life cycle the agencies describe for a reason, and it is far easier to negotiate at the start.