First Financial Creates In-House AI/ML Lending Experience

There’s nothing artificial about the intelligence First Financial Federal Credit Union ($1.0B, Lutherville, MD) is applying to the new loan decisioning solution the credit union has created in-house.

The Baltimore-area cooperative is applying artificial intelligence and machine learning techniques to the new Anytime Express lending products it plans to launch this summer.

Michael Powers, the credit union’s chief innovation and strategy officer, says trial runs of the decisioning engine have resulted in similar, but much faster, results to those yielded by his shop’s traditional LOS software.

He says that’s especially true for applications that might not fit neatly into underwriting rules and otherwise would require human intervention, a capability that will improve as the system learns from its growing data store of credit scores and payment histories.

We’re making best use of the advantages of AI and human work processes.

To test run the tool, the credit union ran an in-house promotional offer for a $1,000 loan that 32 credit union employees took out.

“When those loans pay off, we’ll have the first complete cycle,” Powers says. “This summer, we’ll move toward exposing this to the general membership.”

First Financial plans to offer low-dollar, short-term loans at first, followed by revolving and auto loans.

Powers calls the new system a significant step up in LOS integration and a first for the credit union industry.

“As far as we know, we’re the first credit union to use machine learning and AI like this to approve a loan application, especially using an in-house system,” the executive says.

Here, Powers explains the thinking and processes behind the credit union’s improved loan decisioning project.

What inspired the creation of First Financial’s in-house AI/ML system for loan decisioning?

Michael Powers: We were motivated by the shortcomings of rules-based automatic decision engines, especially for applicants with less-than-pristine credit. We created an underwriting AI using a blended neural network and decision tree architecture that could work across the credit spectrum.

We used our historical data of unsecured personal loans to train the system, and we obtained good back-tested performance on those historical applications. We now can render a loan decision and turndown reasons, if applicable, in a fraction of a second.

We offered an employee-only loan to conduct an operational test in a benign environment. Applications included credit challenges such as high debt-to-income, delinquencies, and charge-offs. Out of 32 applications, our supervisory underwriting review agreed with the AI in 29 cases. Importantly, we tuned the system to a conservative setting and the AI did not approve a loan application that we would have otherwise denied through our usual process.

Please describe your background in aerospace and how it informs your work now.

MP: I’ve worked previously at a large aerospace and defense firm specifically on the subject of robotics and more recently for a Department of Defense laboratory. One of the many things the aerospace and defense industry does well is to take on game-changing technical challenges and manage the high risk by maturing a new concept through stages described by a technology readiness level (TRL).

That’s what we’ve done here. First, we demonstrated the basic principles in a synthetic, experimental environment. Now, we have completed our first test in a realistic environment, albeit one that is moderated and limited to employees. Next, we are completing integration into a real-time production system. Credit unions can follow this model to foster a rapid pace of innovation but with the care necessary in an environment where mistakes can be very costly.

What kind of automatic decision-making tools were you using before you created your own system, and what were its shortcomings?

MP: Our baseline for comparison is a standard rules-based automatic decision engine that’s included with most modern loan origination systems. Like others, we’ve found it isn’t practical to automatically, and instantly, approve a wide diversity of loan applications in a rules-based system without accepting some compromise in underwriting rigor.

Although rules-based systems work well for pristine or near-pristine credit, the many dimensions of creditworthiness and specific credit history frustrate efforts to write rules to cover substantially all circumstances. Our machine learning system adopted the scope and depth of rules observed in practice by learning thousands of application, credit report, and historical account behavior examples.

What technology and credit risks did you assume in this strategy, and how did you mitigate those?

MP: Technology-wise, we weren’t sure that this approach would work when we first started. We followed the TRL model to mitigate the risk of technical failure. We evaluated our progress at successive TRL stages to determine if the ultimate goal was still within reach. Importantly, our IT and lending departments saw this as an opportunity and a technical risk worth taking.

In terms of credit risk, we did not necessarily see our AI program as a finer underwriting instrument but rather as a faster one that can deliver our full underwriting treatment as fast as a simplistic rules system. It hasn’t been our intent to accept more credit risk with this system. As we continue to refine our approach, we might see decreased credit losses, but that’s not our focus at this point.

How much more flexibility does this system give you in approving lower-credit score loans?

MP: Previously, we would only automatically approve a sub-set of prime credit. This system will let us automatically and instantly approve in all credit tiers. In our employee test, AI/ML correctly approved sub-prime loans and those requiring exceptions, such as debt-to-income exceptions.

We observed examples of a need for underwriting logic that is impractical to manually write into a rules-based system. Importantly, the system did not approve a loan that would have been denied in our standard process.

Does this system use different underwriting criteria than you used to? What are they, if so?

MP: Our approach was to have the system act as close to our current underwriting practice as possible. Basically, the objective was to copy the risk profile we currently accept. In the future, we might use alternative data or specific loan performance data to adjust our underwriting AI practices based on observations.

Can underwriters and loan officers override the system’s recommendations? How does that work in your shop?

MP: Our concept is to start using AL/ML as a yes machine. In other words, to speed up approvals rather than speeding up denials. Our vision is to instantly approve every approvable loan. Applications that are denied by AI get a second look from a human underwriter before we issue an adverse action.

A second review could, for example, result in a counteroffer on a secured loan product if we would not meet the member’s needs with an unsecured product. Or, to see if we could get a co-borrower or guarantor on the loan. Or, find rare exceptional circumstances. We embrace this as an example of human/AI teaming. We’re making best use of the advantages of AI and human work processes.

Describe the process of building the underwriting AI system, including how long it took.

MP: The approach was considerably more iterative and incremental than other types of software development. It is less of a feature-by-feature, build-then-fix. In fact, I compare it somewhat to the way I see agriculture or farming, where you plant, observe the outcomes, make some adjustments, and repeat. This process itself took several months. There is some underwriting intuition involved, and the help of our underwriters was indispensable.

What other systems are involved, and how did you deal with integration issues?

MP: It involved our Symitar core and NetTeller, our online banking system. What we have now is a low-fidelity prototype. We are now working on a high-fidelity prototype to production system integrating into Alkami, our new online/mobile banking provider.

You used historical data from unsecured personal loans to train the system. How long did that take? When and how did you know the system was trained well enough to use?

MP: It took several months to cultivate our system to maximize the back-tested performance. Working on data quality issues took most of that time, and it was an iterative process involving IT and Lending.

What is good enough is a good question. When we looked at discrepancies in AI versus human outcomes, we were satisfied when the discrepancies were immaterial or fell within the limits of an underwriter’s judgement; i.e., they were not something for which a human underwriter would be reprimanded.

How tech-savvy would you need to be to replicate this in another shop your size?

MP: The general programming skill sets found in IT departments are only a part of what’s required to build AI/ML systems. Adding education and experience that is heavy in mathematics and statistics makes for a far more powerful toolbox and is the difference between getting something to work in a magical sort of way versus something that is robust, well understood, and well controlled. There are plenty of open-source libraries that implement the core routines.Having an experimental mindset helps a lot.

How does this solution compare to anything else on the market for like-sized credit unions?

MP: We’re not aware of something similar to our approach at the same state of progress, but there are other efforts out there. I think it’s important for AI technical solutions to come from within the credit union community, from both in-house development and CUSOs. Although I think the hype surrounding AI/ML is overblown, I have little doubt it will be essential to efficient and effective banking operations in this decade. It’s important to have some in-house AI/ML experience to be a competent buyer of a third-party technology.

How did you win senior management and board approval for this project?

MP: Our CEO and board maintain careful enthusiasm for next-generation technologies that will give us a competitive advantage. We clearly explained how AI/ML works in terms accessible to a broad audience, and we showed we were taking a step-by-step approach to proving the system works.

What’s the biggest takeaway you can share with other credit unions?

MP: There is one huge takeaway from this that practically all credit unions need to act on immediately: Credit unions need to maintain repositories of data now to have the option to build AI/ML applications in the future. We were able to do this because we had all of the necessary data inputs accessible and organized.

It’s far more difficult, even impossible, to reconstruct data sources needed to build machine learning systems at some later point in time. It’s practically impossible to speed up the collection of particular examples to train the system.Opportunities depend mostly on the quantity and quality of data available to learn and much, much less so on algorithms and software tools.