A research team from IBM at Haifa built FreaAI to detect defects in machine learning models by automatically scrutinizing human-interpretable chunks of data to predict when algorithms are working and when they are not.
The researchers addressed the challenge of finding low-performing data slices to validate the ML performance of Machine Learning solutions. The team developed automated slicing heuristics and implemented them in FreaAI, such that the resulting pieces statistically significant, correct, and explainable.
“It might be acceptable in some cases to have an incorrect answer from time to time, but it’s vital that we can understand and control the extent of a mistake and the circumstances under which it could occur,” the IBM blog stated.
For example, take a health insurance firm, here FreaAI could be used to discover the error rate in insurance approval or rejection and fix the issue. In the future, FreaAI will automatically find inaccuracies in an AI model and suggest course corrections.
IGNITE is an IBM platform that pools automated testing services. The Big Blue incorporated FreaAI to IGNITE and other testing technologies from their collaborators in the India Research Lab to provide IBM clients with Full AI Cycle Testing (FACT) capabilities.
Today, machine learning models are the key drivers of business decisions. And, these ML models must be updated over time to neuter ‘Model Drift’ risks. Model drift can be grouped into two broad categories.
- Concept drift occurs when the statistical properties of the target variable change leading to a broken model.
- Data drift occurs when the statistical properties of the predictors change.
Testing has traditionally been an exercise in optimizing detection-based strategies for correcting business processes. Machine learning and neural networks offer apparent advantages in software testing compared to traditional testing:
- Traditional testing tools can test and provide results automatically, but they still require human oversight.
- Traditional test automation technologies can’t identify which tests to run without human supervision; thus, they wind up performing every test or a predetermined set of tests.
- AI in test automation tools can intelligently pick which tests to run and then trigger them automatically based on test status, recent code modifications, available data, and associated code metrics.