Instability of consumer credit models can be a significant source of risk for lenders – particularly for financial technology (fintech) firms and others, a paper released Wednesday by the federal insurer of bank deposits implied.
In the paper, “Why Do Models That Predict Failure Fail?” released by the Federal Deposit Insurance Corp. (FDIC), authors Hua Kiefer (of the FDIC) and Tom Mayock (of the University of North Carolina Charlotte) state that their research implies that model instability is a risk for fintech and other lenders that “rely heavily on predictive statistical models and machine learning algorithms for underwriting and account management.”
The researchers wrote that one of the primary contributions of their paper is that it provides evidence on the performance of both traditional statistical models and machine learning methods in the lending environment following the financial crisis of 2008-09. “For both types of models, we find that predictive accuracy deteriorates rapidly when models that are trained in one type of macroeconomic environment are used to predict loan performance in out-of-time samples characterized by much different economic conditions,” their conclusion stated.
Another major contribution of the effort, they wrote, is that the paper studies why that deterioration in predictive accuracy occurs. They stated that their findings “imply that the failure of models documented in previous research was not simply a by-product of information asymmetries in boom-era mortgage lending; rather, our results suggest that the poor out-of-time performance of both statistical and machine learning methods is structural in nature.”
The authors said that, in writing the first portion of the paper, they utilized millions of loan-level servicing records for mortgages originated between 2004 and 2016 to study the performance of predictive models of mortgage default. “We find that the logistic regression model – the traditional workhorse for consumer credit modeling – as well as machine learning methods can be very inaccurate when used to predict loan performance in out-of-time samples,” they wrote. “Importantly, we find that this model failure was not unique to the early-2000s housing boom.”
In the second portion, the researchers stated, they used the Panel Study of Income Dynamics in to provide evidence that “this model failure can be attributed to intertemporal heterogeneity in the relationship between variables that are frequently used to predict mortgage performance and the realized post-origination path of variables that have been shown to trigger mortgage default.”
While their paper offers no “fixes” for the model failure, the authors do write that their paper provides “evidence that model instability can be at least partially alleviated for traditional statistical models through the use of data from a wide mix of economic conditions.
“Interestingly, diversifying the macroeconomic conditions from which the training data are sampled did not generally improve the out-of-time performance of the machine learning methods that we considered,” they wrote.
The paper was sponsored by the FDIC’s Center for Financial Research