Agus Sudjianto Discusses Democratizing AI, Interpretable Machine Learning, and Lessons from Crises
On this episode of The Lending Link, host Rich Alterman sits down with Agus Sudjianto, Senior Vice President of Risk and Technology at h2o.ai, for a candid conversation about the evolution of AI and machine learning in finance. Agus discusses his mission to make advanced AI tools, like generative AI, accessible to more people—especially in high-stakes sectors like finance. He breaks down why it’s crucial for financial institutions to use machine learning models that are not only powerful but also transparent and easy to explain.
With decades of experience, Agus reflects on the lessons he’s learned from financial crises, including the 2008 crash and the COVID-19 pandemic, and how these experiences have shaped his approach to credit risk management. He also highlights the growing role of alternative data in smarter credit decisions and talks about the importance of rigorously testing AI models before they’re put to use.
Tune in for a compelling conversation on the future of AI in finance, the lessons learned from past crises, and how to prepare for the challenges ahead.
Catch the full episode now!
Episode Transcript
Rich Alterman: You're syncing up and tuning in to the Lending Link podcast powered by GDS Link, where the modern day lender can dive deeper into the future of data decisioning and credit risk solutions. Welcome to the show everyone. I'm your host Rich Alterman and today we are syncing up with Agus Sudjianto. Agus recently joined H2O.ai as a Senior Vice President of Risk and Technology. H2O.ai provides both generative and predictive AI solutions supporting multiple industries including financial services, government, insurance and many more. Agus joins us today with a career spanning over 20 years in banking, holding various positions including the Executive Vice President and Head of Corporate Model Risk at Wells Fargo, Director of Enterprise Analytics at Lloyds Banking Group in the United Kingdom, and Head of Quantitative Risk at Bank of America. Prior to his career in banking, Agus was an engineer and product design manager in the Powertrain division of Ford Motor Company for more than a decade. Agus is a co-creator of Python Interpretable Machine Learning, also referred to as PiML, a widely used integrated tool for developing and validating high risk machine learning models. Agus holds more than 25 patents, both issued and pending in both finance and engineering. He has published multiple technical papers on machine learning and statistics and is co-author of the book Design and Modeling for Computer Experiments, which you can purchase on Amazon. In July, Agus became an Executive in Residence for the Charlotte, North Carolina School of Data Science and will be co-teaching a class on Model Risk Management, Model Validation in the fall. Agus holds Master's and doctorate degrees in engineering and management from Wayne State University and the Massachusetts Institute of Technology. In this episode, Agus and I will be discussing some background on his journey from manufacturing to banking, PiML, machine learning and their impact in financial services, and so much more. But before we dive into the interview, please head over to our LinkedIn and Twitter pages at GDS Link. That's G D S L I N K and hit those like and follow buttons, if you've not done so already. Please subscribe to our podcasts on Apple Podcasts, Spotify or wherever you prefer to listen to your podcast. All right, now let's get synced with GDS Link. Welcome Agus. I hope you're having a great week. Where are you joining us from today?
Agus Sudjianto: Thank you very much, Rich. I am in Charlotte, North Carolina.
Rich Alterman: Okay, well not too far from the school then. Well, thank you for spending time with me today. But before we talk about business, let's get a bit personal. Please share you know some of the work you did at Ford in the powertrain division and the catalyst for transitioning from auto manufacturing to banking. Certainly not a typical transition path one might think of.
Agus Sudjianto: Oh yeah, that's an interesting one. As I joined Ford at the peak in the early 90s, that's when the time of quality movement in the US automotive manufacturing basically. So I focus a lot on the quality engineering and reliability engineering. I ended up in engine design. So my career spent a lot of time in designing V6, V8 engine for Ford Motor Company. So the—if you ask me what kind of car I drive, of course drive Ford. And which one? Of course the one with the 3.5 liter V6 engine. The engine that I designed, the last one. So that's where I came from. It's from really gasoline is in my blood. So designing car engine is what I was passionate about. And then in the early 2004, late 2003 decided to move south to North Carolina. Joining Bank of America at that time as the head of process engineering in risk management organization. So that's the time when Bank of America really focusing on the process. They want to improve the process, they want to work on process excellence. On Six Sigma—when I was at Ford, I was one of the first Six Sigma master black belt for Ford while I was leading engine design. That's when I came to Bank of America started as the executive for process improvement.
Rich Alterman: I'm going to force you to kind of go back in time. What was probably the most interesting interview question you got when somebody was saying you're coming from Ford and you want to come work for the Bank of America.
Agus Sudjianto: I told the people I know nothing about banking. I was recruited because Bank of America want to be able to grow organically, meaning they want to really focus on process improvement and process engineering. I told the people I know nothing about banking and they said don't worry about it. We're going to surround you with people that know banking very well. You're working with that and you're going to learn the whole thing. So there we go. Of course the first six, eight months was baptism under fire. It was very, very difficult. It was one of most difficult thing in my career I would say because I used to know everything what I do and people listen to me and suddenly you come to environment where you know nothing. You have to learn and it's not if you cannot something that you can pick up the phone and make it happen. So you have to really practice how to really in a new organization learning as fast as you can and get accepted by the senior management because I enter the organization as a senior executive.
Rich Alterman: A question for you. We hear so much debate today about electric cars and how quickly we as a country can go all electric. I think some of our politicians have rose colored glasses on. What's your view on how quickly this country truly is ready to adopt both from I think from a social standpoint, but also from a grid standpoint. How many years off are we really before we can go all electric in this country?
Agus Sudjianto: You have to be careful on this one because it's very polarized subject. It's a very, very polarized subject. I think it's very, very one side you want to go green. On the other side, in my opinion is the national strategic decision. If you go into choose certain technology, you want to make sure that you will be able to compete and you are in the forefront, you have the competitive advantage compared to other nations. It's a very hard decision because choosing technology where nationally it will be a strategic national advantage which people can, can draw a conclusion on that.
Rich Alterman: Okay, well, we'll leave it at that. I shared in your introduction that you recently joined the University of North Carolina School of Data Science. How do your academic endeavors complement your professional work and what impact do you hope to make in the field of AI education through relationship with the school?
Agus Sudjianto: Yeah, this is an interesting one because I had my career, I call it the most of my career the last 20 something years. I'm doing a lot of what I call it management, I call it management in entertainment. I have to entertain the senior executive. Right. I have to be able to speak with senior member and convince them and all those things speaking at the, at the very high level. But deep down my passion is really more on the academic side. I always interested in the technical side. So I'm one of the EVP at Wells Fargo. I report before then. When before, before I retire. I reported the chief risk officer who reported the CEO and the board. So very, very sen. But I probably among the very few who still code and do math because I love it. It's my personal passion. Personal things that I really love is dealing with math, algorithms and coding. So that's how I grew up while I'm doing my management duty as a leader and all of those things I never forget in terms of writing paper, advancing knowledge and all of those things. So my background when I did my PhD, I did PhD in machine learning back in the early 90s. Of course when you graduated machine learning in mid-90s, no such job is a machine learning engineer. It's no job as data scientist at that time. No term of data scientist. Either you're statistician or you are nothing. I was not a statistician but I always loved this coming from engineering. So when machine learning came back I thought because no job in machine learning I have to do. I have to get a real job which is a design real machine designing car engine. That's what I did. When machine learning came back very strong in the 2012, 2013 and 14 I would start thinking again. Okay, now this is coming back. This is now gaining adoption then becoming more serious. When I came back to the US in 2015 and 16 in banking in the US start looking into this very very seriously. In particular the concern banking need to be explainable need to be interpretable. So that's the whole area that I have been. I have been working in the last. In the last eight years really focusing in this area make model very transparent sophisticated machine learning so that banking can use it safely and without worry from from regulator in a point of view. So that's what the focus and I always always publish all along. So you can check my my h-index. So if you look at my Google Scholar h-index I'm not going to say the number you check yourself. It may be like more like a like professor. I'm not a professor. I'm working as an executive in banking. But publishing and writing paper is just. Just my. My first love. I I really enjoy it. So so that's when then start thinking about how can we help local university. That's very very important for the industry in Charlotte. That's what is coming is really okay model risk management. We banking use a lot of model and banking understand the implication when model wrong or wrongly used. So they manage it so well. We've been practicing in particularly US bank have been practicing model risk management for the last 12 years. So it's very very mature. But other industry have no such thing in model risk management today with AI people talk about responsible AI. At the end of the day the foundation of responsible AI is model risk management. That's when I feel like I need to help for other industry make it available broadly. So coming up with teaching with something that teachable that people can learn instead of jumping to the workplace have to learn from start at work. So we try to prepare students young generation talent to learn about this so that they can propagate the discipline of model risk management to other area that's really really important. For example medicine for healthcare and all of those things are super super important which they don't have practice of model risk management. Right.
Rich Alterman: Okay, great. Well thanks for that. Okay, well let's get down to business now. So you recently came out, kind of like came out of retirement. I think you had one of the shortest retirements in history from Wells Fargo and you joined H2O.ai. What was it that attracted you to H2O? I obviously you had other options on the table
Agus Sudjianto: before the last few years. In my career at Wells Fargo one thing that I'm very proud of is really teaching other making tool that's available for the industry. So I made PiML, Python Interpretable Machine Learning. This is tool to design interpretable machine learning. Sophisticated neural network, radial basis machine that are interpretable how to test them appropriately and we can talk about more detail on that. So we made the tool available so that the industry can do can use machine learning responsibly and doing it with confidence. So that's the tool that I released and that tool is for tabular data. Now when GenAI come in then we say okay this is interesting one because it's a powerful tool. GenAI is very powerful but they are very very unreliable. It's great to use GenAI if you want to use it for I call it mindless entertainment. If mistake has no implication I to create something to create the background for my PowerPoint slide. So that says no risk but now suddenly you want to use GenAI for making financial decision for healthcare and all of those things. This is very powerful tool but very unreliable. So people, we have to know how to control this tool, how to, how to test it, how to validate it. So I'm looking for partner engineer that can help me to make it into reality and build something quick. And that's when I myself and the team and my organization before we used H2O for quite a long time. I know the CEO very well, Sri Satish Ambati very, very well and the team very well and say okay, we have conversations, okay, we're going to build this, we're going to build model risk management as the foundation of responsible AI. We're going to build the tool and make it for everybody. So that's what motivates me really building this product that will come out in the end of September. Basically we're going to release end of September we're going to have a free workshop for the industry to learn how to test and how to validate so that we can control, we can use this powerful yet unreliable tool to be in high risk application.
Rich Alterman: So you kind of answered my next question, but I'll ask it anyway if you go visit the About Us page on H2O.ai website it says H2O.ai is the visionary leader in democratizing AI and that democratization AI isn't just an idea, it's a movement. So maybe just elaborate a little more. A lot of people talk about the democratization of financial investing. So let's talk about the democratization of AI.
Agus Sudjianto: I put it in a very simple word. Democratization is basically to make accessible people to broader audience to a lot of people that otherwise require specialized skill or expensive becoming exclusive group that can do it. No, no, no, we're going to make it. Everybody can do it. So if everybody can adopt GenAI. The difficulty in GenAI today is not building application. Yeah, it has technology challenge on that, but testing it, testing it rigorously and to be able to deploy tool with confidence so we don't have accident like the accident that happened with Air Canada. Even Google have accident before when they release it. So all kind of accidents happen with GenAI. So testing GenAI is not trivial and requires specialized skill and very tedious and expensive. So that's what we're trying to do. Okay, we're going to make tool that will be available broadly and people can deploy, people can apply it very, very easily. So democratization for me is basically make it accessible to more people, to a lot of people.
Rich Alterman: Well that's a good segue. Kind of going back on to talk about PiML or Python ML. So you're co-creator. So let's start first, you know who was your partner on this and when did you guys actually start this journey?
Agus Sudjianto: It was a very, very interesting idea at that time. We released it about two and a half years ago. I remember it's May 4th so we purposely put it May 4th. So May the 4th with you. So that was the release at that time was for the purpose of workshop. It's one big conference. Every year America has a conference. They would like to have a workshop on how to validate, how to test, how to validate, build and validate machine learning for financial application. Very highly regulated industry. So I and my team at Wells Fargo at the time, okay, we're going to conduct this workshop. We're going to make it the workshop to be very, very useful and very hands on. We need to build tool for the workshop. So we built this tool. It was very fast and furious in about five months prior to the workshop. So December prior, at that time we start working on it. We sketch out. This is really from the practice have the practice of testing and validating machine learning at Wells Fargo at that time. Let's build this tool, let's do this one. It's also accumulation of many years of research and publication that I have. So we built this tool, we conducted the workshop. The reaction is it was very, very positive. We have a lot of requests that we kept building the tool. And we have a lot of seminar workshop every year, four or five times a year. We're teaching other bank as well on this. So today it is very, very widely adoption. More than 300,000 download today on this. So. So a lot of people using it. And it's the only tool that if you want to build interpretable model, this is the place if you want to also test it. When we think, thinking about testing this is the problem that people have with machine learning. People, people think about just the. The performance of the model. What. Well, we're not thinking of just the performance. We need to think how the model will fail, in what situation the model will fail and how are we going to test it. So this is talking about testing, identifying model weakness. How reliable is the output, how robust is the outputs against noise in the input, how the model will perform, how resilient the model, how the model is going to perform under environment change, under distribution shift. So all of this thing, if you look at this, the language that I use is of course inspired by my background in engine design. Because when design engine, we don't design engine for perfect environment. The engine have to survive in very cold winter in Bemidji, Minnesota. The engine has to perform very well in a very hot temperature in the Death Valley. The engine has to perform very well in a hot summer day in Miami. So we're thinking about all of this thing and model is a product, model will fail. And so we need to know, we need to be able to test it. Test to failure. So model validation is about model hacking, test it to failure, understand how this is going to fail and how bad is it and what's the root cause so that we can fix it or we can manage the risk. So those kind of principle that we built around PiML thinking about, about model as a product, that we need to really test the function under different environment to identify model weakness and failure so that we can manage it. So that's how it's coming from it. Built by very very small team. We have three developers and one leader and five with me. I was very, very hands on including coding and working together on crafting the methodology. So that's how it is. So, so now it's been more than two years. It's widely used and across the board in the previous company that I was part of is widely used by model developer and model validator and widely used in the industry as well.
Rich Alterman: So for our listeners, the PiML toolbox is available at a website. It's arXiv.org and that's A R X I V dot org. If you want to go and pull it down and leverage it from there,
Agus Sudjianto: Google PiML, it'll bring you to GitHub and where you can download the code, you can install it. So you can do pip install and it's free to use and everybody can use it for free.
Rich Alterman: Great. So one of the things that I want to do is just give you an opportunity to speak to our audience today that may not be in the financial services sector. Could be college graduates just kind of thinking about careers. Maybe can you give some real world examples of how PiML is leveraged in financial services? Maybe just pick two, two examples that would be good to.
Agus Sudjianto: I pick one simple example. Everybody have credit card. So when you apply credit card your the decision will be will be done by model. So model will make decision. Okay. Machine learning model will make decision whether it's going to approve your credit card or going to approve the credit card. So decision like that will be made will be made by model. Now when decision financial decision made like that, it's a very very important because making decision about giving loan and not giving loan is about determining someone socioeconomic of the future. So if it's wrong, it can be. It can be very impactful to people. So with that if you apply model for high risk decision making like that, you make sure that your model is conceptually sound. You understand your model very well, you understand how the decision are being made. You understand when you decline application, you can explain very clearly why. So the model has to be very, very explainable. The model has to be conceptually sound and thoroughly tested, including tested for fairness. So make sure that your model is not discriminatory against protected class. It's not discriminatory, discriminatory against age again, gender against ethnics and all of those things. So those are the tool that we put in PiML. So and also when, when for for the bank side, when you make decision approve or not approve, you have, you're approving loans, you hold the loan until when an economy is going good or going bad, your DQ you have to live with that consequence. So your model has to be resilient in different environment has to function appropriately. So all those things that is need to be in mind. So that's what we built, that's what we put in PiML to make for people to build model for high risk application. So have you have confidence, you test it thoroughly and the model is completely transparent. And this is a big problem with the regular machine learning because they are black box. And with black box people try to explain it and then all kind of techniques that people call like people call it XAI, explainable AI. They try to explain black box. All those explainer they are not reliable because they are approximation try to explain what PiML is doing. No, no, no, we are not going that we are going build model from the ground up. Machine learning architected in a way that very very transparent and interpretable. So that's the true what I call it, the true XAI because you build model, you craft model that is inherently interpretable. So we to do that requires special skill. Now we put PiML. Everybody can do it.
Rich Alterman: You know, talking about explainability, you know, certainly over the years there's been concerns at certain financial institutions about adopting machine learning for that reason. So do you, where do we, where do you feel we are on the path of acceptance? Are most financial institutions now very comfortable with using machine learning models in originations where you have to have that explainability? Or would you say that, you know, the adoption in the industry is still. There's still a lot more room for growth if that makes sense.
Agus Sudjianto: Build machine learning that inherently interpretable like what we put in PiML, you should have no worry. You should have the very high confidence because it's explainable as traditional statistical model. So you can be very, very confident because it's very, very transparent. It's very explainable, just like like traditional statistical models. So it's no reason. So for tabular data I would call it tabular data is numerical data. Tabular data. The issue of interpretability explainability in my opinion is solved. And I we look at it, we can do it. Control the architecture of the model to be interpretable. That's what we put in PiML. So the is I don't think there is any, any excuse in terms of interpretability anymore for tabular data. Now language model or vision model is completely different. That is still a challenge how to make it really interpretable.
Rich Alterman: One of the misconceptions I've seen in the market is when people talk about AI and machine learning models and especially around originations, some people, the way they project it is that these models are continuously learning. Right. And almost to the point where if Richard applied for a loan at 2 o' clock versus applying again at 4 o' clock that the model may actually be different two hours later. So can you kind of talk about the whole concept around continuous learning and why you know once again where you have to do that? Explainability. It's not realistic to think this model is going to change all the time as it may in like authorization where that is a little easier to deal with. But maybe touch on that be great.
Agus Sudjianto: Yeah. If you do machine learning for origination and continuous learning, it's a completely the wrong thing. Because if you originate, you approve loan you stuck with it until the next recession. You're still holding the loan. Your model cannot change. If you have to change your model model, you have very very bad model. You have to build model that is resilient that your model will work any different environment. Because once you originate it, you stuck with it. So it's a completely bonkers. If people think about I am going to build origination model that continuous learning. That's a complete BS. It's completely wrong thing. Now if I build recommender system what video to watch if I'm will, if I'm TikTok what recommender things I'm going to change? Well, the data change all the time. The trend, the interest of people change all the time. So I need to adapt it because the selection people click different video and all those things. Yes, if you build something for recommender system like what video to watch, what things to purchase. Yeah, you build model that you probably going to retrain it every 15 minutes but still not continuous. So you're going to build it probably for every 15 minutes. You retrain it to give a better recommendation or every 5 minutes depend on computation cost. Because this retraining model is computationally expensive. Of course people can do something that incrementally using techniques like reinforcement learning and all of those things. But no, the conception of people that machine learning continuously learning it's a completely wrong misconception in my opinion. It really depends on what application is for lending. No, no, no, no. You don't want to change your model too frequently because once you originate, you better make the right decision environment change your model still need to work because otherwise you originate the wrong things. When the economy economic situation turn you all going to end up with all the garbage origination.
Rich Alterman: Let's kind of extend on that. And you didn't use these terms. But let's talk about champion challenger. When we're you know, talking to prospects about the capabilities of our GDS platform. One of the things that we'll talk talk about is champion challenger capability. But do you typically see that financial institutions are are really strong in adopting and implementing champion challenger as you know, taking percentages of their portfolio and running through different strategies.
Agus Sudjianto: Champion challenger I think the area that one thing is high impact model and the outcome is highly uncertain, then you do need champion challenger because all models are wrong and different model will be wrong differently. So I give an example. If we build model for the purpose of stress testing, I am going to stress my portfolio. I want to understand what's going to happen in the future, right under stress condition. Well, first of all, you don't have the data. The future stress scenario will be different from the past. So the outcome is very uncertain. So in that situation, because the outcome is uncertain, you cannot even consider it beyond just you do back testing. But back testing is looking back and your model need to look forward. Sandy, look forward. You have not get exposed to that environment. I gave an example when COVID-19 hit the situation where unemployment went to the roof and GDP dropped. Model never seen anything like that today. Environment high interest rate that we experience with your historical data. Never seen high interest rate like what we see today. So when that situation because the outcome is very uncertain, we don't know the model never experienced it before. You better off have champion challenger because different modeling approach will give you different answer.
Rich Alterman: You touched on fairness and disparate impact. In your opinion, has the market adopted fairness testing capabilities as strongly as they need to, or is there more work to be done? As you're aware, I did a podcast with Kareem Saleh. They have fairness as a service as one of their platforms and they're certainly getting a lot of good traction. Which tells me that there's more and more adaptability to that need to test for fairness. Where do you feel we are on that fairness testing paradigm?
Agus Sudjianto: It's a requirements law, right? So everybody have to do it. Particularly for lending product. Right, for lending product. Anything that touch lending product, you have to do a fairness testing. So that is a requirement. So large bank and more sophisticated bank have very established practice to do fairness testing before the deploy model. Now, fairness is very, very data dependent. So you can have model that's fair, but your model fairness fairness in your model is as a whole probably fair. But you can look at the pocket. This is in this area the model is weaker, less fair. In the other area is more fair and all of those because model performance will not be uniform necessary now the monitoring is very, very critical because of that. Because the customer, the application that coming to you, it can shift and it can shift to the region where your model is less fair. Then you got the problem get accumulated on that the scale that needs to be done not only before you deploy model, but also during monitoring when model are being used. You need to monitor the fairness of the fairness of your model because distribution shift all the time. So that's really what needs to be done. And that can be quite challenging from technology point of view because you have to deal with data, large data, a lot of portfolio and you have to monitor all aspect. Right. Not only origination but also your collection, also your marketing or your offer. So it's very, very, very broad about fairness testing that you need to be done. This is only talking about lending product now about—if that is a non lending product now because you're talking about fair banking. How about deposit? If you even for fraud detection. If you say I'm going to block this transaction, you make sure that you test for fairness too. So that when you block it's fair. So those kind of things that becoming the scale is so big what need to be done in the. In. In the fairness and and with that some company depending on their technology infrastructure can be better than the other.
Rich Alterman: Well, that's a nice segue into my. My next question. We were talking about the adoption of machine learning in originations. Where do you feel that the industry has gone in using machine learning techniques in the full lifecycle of a customer? So behavioral models, account management, line management collection. Is that still a. In your opinion from a machine learning perspective is that kind of an untapped opportunity at this point?
Agus Sudjianto: The progression that people do. People do machine learning more on the before the credit decision is the last one if you think about it. Yeah. In the value chains of the process. They apply machine learning everywhere, right. Fraud, they do it all day long. On the account management they do it. But when it comes to credit decision that was the last one that company adopted is that. But now today people are. People adopt it, people come comfortable with it because we can build really interpretable machine learning. And I'm very pleased to see that because that's the area that I've been working. If you haven't for company that haven't. I would suggest look at interpretable machine learning like the one in PiML so that you have high confidence that this model is as explainable, as interpretable as your logistic regression model. And it gives you a lot more capability that you can do because now you can have what we call it in the in the statistical lingo interaction you can have interacting variable that have you can, you can get benefit from that where the traditional model it really typically don't use any interaction. So I think that's what will be useful and I don't think it's any excuse anymore today. Like I said said this model is opaque and not transparent.
Rich Alterman: That's gone. A couple more questions keeping an eye on the time here. In my prior podcast, which was actually just last week, I think my guest was Jason Appel, EVP and Chief Risk Officer of goeasy Ltd. A Canadian based financial services company. Like you, Jason held key roles in credit risk management for multiple decades. Since my podcast with Jason has not been released to the public yet, I thought it'd be interesting to ask you two questions that I posed to him and then it will be interesting for our listeners down the road to compare the answers from both of these podcasts. So the first question is as follows, reflecting on your extensive experience in the credit risk analysis analytics space, what do you consider the three most significant innovations in this field over the past two decades? And if machine learning is one of them, then you need to come up with three other ones.
Agus Sudjianto: The most important thing is in today's environment is probably the alternative data. So people use other data other than the traditional credit bureau. That's important because now you get the richer information. I think that's probably one of the most things I'm looking at from analytics point of view here. So that's the things that become very important because now we can do a lot more with the adoption of alternative data. Data is not only important because you can do things, but also defensively because if you think about it today with all the proliferation of fintech, when people apply to fintech and approve and get loan from fintech, fintech don't have obligation like bank to report that loan to credit bureau. So if you don't use alternative data you get blindsided because you don't, you don't know that there are other trades and other borrowing that you should consider. So that's in very very very very important to have so that you're not blindsided because otherwise you will be approving, you will become too optimistic because your credit bureau you to pull from credit bureau. The information is normal. So that's why what becoming very important and necessary for people to adopt and use alternative data today, be it data from other sources or transactional data and all of those. So the advancement of information system, the database that we have more data, we can do all of this becoming very, very important. I think that's probably the biggest and the most important things now now on the frontier that we have today and then also is the processing. You know that loan processing from you look at the value chains from application to approval to account management and all of those things to servicing. It's a very, very heavy process and also very very document heavy. This is where the innovation yes people start with RPA, a lot of robotic process automation. But with all the GenAI today that's the opportunity if you can use it right? The language model that can help you on this document processing, automating some of the process. I think that is just started to get day to not only looking at the decision which is the domain of predictive AI, but the GenAI really helping on the process, make the process more efficient, make the process less prone to error.
Rich Alterman: So systematic reviewing documents that a person may upload like a mortgage document application. So using more generative AI to take that, interpret that type of information. So the second question I asked Jason was the following. During your career in risk and you touched on this. During your career in risk management, you've operated through the financial crisis of 2008 and of course the COVID-19 pandemic which you mentioned. What are some of the key lessons that you learned about credit risk management during each of these periods and how have these insights shaped your strategies for future economic disruptions?
Agus Sudjianto: To look at it from a little bit narrowly on the analytical side and it's interesting, I think the first always, every experience is always humbling, very, very humbling experience. Every time I've been through the 2008 and I went to Europe, you have Euro crisis, you have COVID. And now at the time, the last few period is also very interesting when you start seeing as well. So every time is always very very humbling experience. And I work in modeling a lot. Okay so and this is the the problem in modeling your model work when you don't need model in good time, your model work in the bad time when you really need your model to predict your loss and all of those things your model doesn't work okay, so that's always a very very humbling experience. This is what really shaped my, my my thinking in term of model risk management. You have to anticipate, you have to understand, you have to understand in model will be wrong. So when the situation you know exactly the playbook. So prepare for the worst, understand what can go wrong and hope for the best, but prepare for the worst. You have playbook, you know right away what to do instead of being reactive. And that's the value of risk management, to be able, thinking about what can go wrong, how this thing can go wrong and develop, develop playbook so that you can adapt very, very quickly and adjust very quickly instead of being very, very reactive. And so that's, that's, that's probably the most lesson learned that the, with the 20 years in risk management is like that.
Rich Alterman: Right? Right. So selfish plug is, you know, certainly software like GDS is required, right. To be able to more easily react and implement those changes. Right. If you're dealing with legacy applications with hard coding and you have to get in the queue and you know it's going to be done six months from now, that may be too late. So you need that type of technology that can allow you to very rapidly make those changes. So last question as we come up towards the top of the hour. You know, whenever I am talking to friends that have younger children and I ask them what, you know, what are their favorite subjects and they mention math. And I'll say to them, well, I got to tell you, you need to really emphasize and push, push their hunger for math. Because in my opinion when we talk about analytics, there's probably no other profession, although some people may say marketing, but no other profession that has applicability in pretty much every single industry. Right. Analytics is used in every single industry. If you are talking to some kids that are about to enter into college and they're trying to figure out, out, you know, what might be a good career path for the future, what would be some of your recommendations and wisdom that you'd like to partake or share with the group today?
Agus Sudjianto: Interesting subject and very, very personal because I experienced it five years ago with my younger daughter. Right. So she want to decide, okay, what decided, okay, you don't know what five years from now, you don't know what 10 years from now, what the world looks like, what kind of skill that's needed, what kind of things, what you do and generation we came from completely changed. I remember, you know, when you grew up and things to today are so, so totally different. I talk about when I studied machine learning, it's no job on machine learning. Who would have known 25 years later is the, the hardest job in the market. So that is a crazy, crazy thing. So you don't know what's in the future. So when you study, study, prepare yourself that you will be able to do lifelong learning, you can adjust. So, so and that is where STEM study, right? The science, the technology, the engineering is preparing you to do that. So if things change that you can adapt, you can learn and you can learn very very quickly. And for that the basic science and math is a requirement. So this is talking about from the personal thing. When I did my undergraduate, I never regret I did it in physics because physics is basically science and math. And then from that I began my graduate school in engineering and all those things I adopted. So I adapt along the way. My younger daughter at that time said okay, well something that very broad all kind of thing that you go inside did you pick in your graduate school, you pick something that you want to specialize or in the future you can learn. So she ended up studying mechanical engineering because it's basically applied physics and applied math. So and then now she's in graduate school studying artificial intelligence in graduate school in Carnegie Mellon. So you can adapt, you can adjust. This is the thing that you like. Finally you go to that. So I think preparing something that early, it will not shut you off for opportunity in the future. Study something that build a very, very good foundation that you can adapt and you can change, you can follow if you want to. So I think that's very, very important for that going to college. Pick an area that will prepare you, that will allow you, that will enable you to change to different area to study different area. Because when you just graduate from high school you will not know what you want to do. You don't know what the world looks like. So study something and math and science is that of course along you study some liberal art tools as a balance. But it's really all of this thing, the foundation that you can build for the future future.
Rich Alterman: Another point is when we, we talked earlier about continuous learning of machine learning. One of the things that I tell younger kids is that you have to be constantly learning, right? You have to be taking it upon yourself. And, and now I was talking to a 20 year old the other day and I said so how far along are have you been teaching yourself how to use ChatGPT? And he said not at all. I said that's a mistake. You need to start learning it pretty quick because it's not going anywhere and you know if you know how to use it, you're going to become more, more useful to other people. Well, I, I really appreciate your, your getting together with us today. And once again this is Rich Alterman and we've been meeting with Agus Sudjianto, co-creator of PiML and Executive in Residence for the Charlotte, North Carolina School of Data Science and Senior Vice President of Risk and Technology with H2O.ai. Thank you for joining me today and sharing your knowledge with us on various aspects of credit, risk management and machine learning. I wish you much success in your new role at H2O.ai and your class on machine learning monitoring this fall. Maybe if you give me an invitation I'll come up and sit through one of your classes. I'd like that. We hope you've all enjoyed the podcast and please stay connected with GDS Link and the Lending Link to listen to future podcasts and catch up on ones you've missed. Thank you and make it a great day. Thanks for listening. If you've enjoyed today's episode, please be sure to subscribe on Apple, Spotify, Google, or wherever you listen to your podcasts. And be sure to leave us a review, follow us on LinkedIn and connect with us on Twitter @GDSLink. That's GDS L I N K. Have a question for the show or have a specific topic you want us to cover? Hit the link in the description to drop the us a note. Thank you for lending us part of your day. Make it a great one.
About Agus Sudjianto
Agus Sudjianto is SVP Risk & Technology of H2O.ai. He has a span of 20+ years of experience in banking where he was an executive vice president, head of Model Risk Management at Wells Fargo, director of enterprise analytics at Lloyds Banking Group in the United Kingdom. Before joining Lloyds, he was an executive and head of Quantitative Risk at Bank of America.
Prior to his career in banking, he was an engineer and product design manager in the Powertrain Division of Ford Motor Company for more than a decade.
Agus is the creator of PiML (Python Interpretable Machine Learning), a widely used integrated tool for developing and validating high risk machine learning models. He holds numerous U.S. patents in both finance and engineering. He has published extensively technical papers in machine learning and statistics and is a co-author of Design and Modeling for Computer Experiments. His technical expertise and interests include machine learning/AI applications, risk management, and computational statistics.
He holds masters and doctorate degrees in engineering and management from Wayne State University and the Massachusetts Institute of Technology.
About GDS Link:
GDS Link empowers financial institutions to navigate the complexities of modern lending with precision and confidence. Our advanced data integration and automated decision-making tools enable smarter, faster decisions that drive growth and optimize risk management across the entire credit lifecycle.
With a platform that connects over 200 data sources, GDS Link streamlines complex decision-making by incorporating delivers the insights needed to stay ahead in a competitive market, ensuring you seize every opportunity.
Our comprehensive solution isn’t just about keeping up with the industry’s demands—it’s about getting ahead. By leveraging real-time data and advanced analytics, we help lenders make informed decisions that uncover new opportunities, enhance operational efficiency, and improve customer outcomes. GDS Link is the partner you need to transform your credit risk management processes with agility and precision.
About The Lending Link Podcast: The Lending Link, powered by GDS Link, is a podcast hosted by Rich Alterman and designed for the modern-day lender. Each episode dives deep into innovation within the financial services industry and transformation efforts, including AI/ML integration, modeling, risk management tactics, and redefining customer experiences.
GDS Link launched The Lending Link to explore unique strategies for the modern-day lender, dive into the innovative advancements GDS Link and our partners are currently developing and delivering, and gain insights from captivating guests within the fintech, banking, and credit union worlds.
We feature a wide range of guests from various lending institutions and diverse organizations who share their strategies, technology insights, and everything in between.