For those of you who missed last week’s post, this is part 2 of our ScienceOps feature spotlight series. In last week’s blogpost we talked about the importance of Systems Monitoring and the new features we’ve built out.
To back up one step further, if you’re not familiar with Yhat, our mission is to make it easy for data scientists to deploy predictive models. Our flagship product, ScienceOps, lets data scientists put predictive models into production applications via standard REST API requests, avoiding the hassle of translating code.
Today we’re introducing a new set of features for managing those models within ScienceOps called Prediction Analytics. Prediction Analytics gives you visibility into how your models are performing in your production applications.
Why it matters
So you’ve finally gotten your beloved Python model up and running in production, either by manually recoding your model into PHP/Java/the like, or by hosting your model on ScienceOps. Mom must be so proud! Now what?
What we built
In ScienceOps lingo, a prediction is a single request made to a model. In the Prediction Analytics tab of ScienceOps, you can view aggregate predictions over time, view success and error rates, filter across models and time periods, and query for individual predictions.
You can quickly catch any anomalies on the time-series predictions graph and inspect them in the predictions table below.
Spotted something out of the ordinary, like a surge in traffic or an increase in errors? Check out the Inputs/Outputs tabs to compare variable distribution data across date ranges to get to the root of the problem.
In (sorta) real life
Let’s dive into a use case using data from a fictitious peer-to-peer lending company. This company has built a model that predicts risk of default based on a variety of variables, such as credit score and home ownership status. They’ve used ScienceOps to deploy their model into a website and mobile app where you can apply for a loan.
A business analyst, Carl, notices something is out of the ordinary. Net new revenue was low last month, and Carl has also observed that loan applicant denial was higher than normal. He suspects that overall loan applicant quality must have decreased in March. He heads to the Outputs tab to test his hypothesis.
Carl wants to check how the data for March compares to the previous month. He selects the month of March as Date Range A (blue) and February as Date Range B (orange).
Since the number of applicants varied over the two time periods, he normalizes the data by showing percentages rather than raw count. As he suspected, the predicted probability of default was significantly higher in March.
Curious Carl wants to know why. He switches over to the Inputs tabs and notices that the majority of March loan applicants were renters, while the majority of February applicants had home mortgages. Carl can also see that the FICO ranges of applicants were significantly lower in March than February.
Now Carl has the insights he needs to work with his business team to investigate what’s behind these changes. As it turns out, the company launched a marketing campaign in San Fran on February 1 targeting recent college grads. While the marketing team viewed the campaign as a success thanks to a giant surge in loan applications, Carl was able to catch and pinpoint the downstream implication of the initiative.
This is just one example of how Prediction Analytics can help you uncover business insights based on the wealth of data coming into and out of your models.
I want it now!
Want to start deploying and managing your predictive models with ScienceOps?
Head over to our site where you can request a live demo or download the data sheet today!