Training & Evaluating ML Models

Training Models

The first step in a typical machine learning workflow is training. Training is the process of passing training data with the target or target attribute to a model so that it can learn to identify patterns. The algorithm finds patterns in this data that maps the input to the target. The output is typically a model that captures these patterns.

In Supervised learning, a human will create a data set that connects different variables with known outputs. This training set will help the machine find relationship and concepts. 

In Unsupervised learning, a large data set is fed to a machine and the machine will study and observe the data itself. It then clusters the data and creates connections that might be otherwise unknown to a human.

There are also opportunities to cross between the two; where some set can be trained by a human, and additional data can be fed to the machine without that training. 

Evaluating Models

Once the model has been trained, we then need to test how our model performs on data outside of the training set to see its effectiveness in real world scenarios.  We may need to run training & evaluation multiple times, performing additional feature engineering and tweaking the model(s) as needed. 

The below metrics are typically used to understand the success of a model.

1. Data quality metrics
2. 1. Validation of configuration and results
   2. Understanding the schema, prevent drift, and prevent training-serving skew
3. Overall evaluation criteria
   1. Long-term business value and user satisfaction
4. Guardrail metrics
   1. SLO degradation thresholds
5. Local behavioral metrics
   1. Explain tradeoffs leaving to the overall evaluation 

Appendix & Resources

In general, I have years of experience in this area and some of the content on this page involves general subject matter expertise on my part. That said, I am constantly reading and continually learning in this space. 

Here are some Resources that I used to build this document. Please reach out to contact me if any questions or concerns. 

Online Resources.. 

https://developers.google.com/machine-learning/guides/rules-of-ml#training-serving_skew

https://wikidata.ca/docs/Dmitriev,%20P_Gupta,%20S_etal%20-%20A%20dirty%20dozen_Twelve%20common%20metric%20interpretation%20pitfalls%20in%20online%20controlled%20experiments%20-%202017.pdf

https://developers.google.com/machine-learning/guides/rules-of-ml#rule_26_look_for_patterns_in_the_measured_errors_and_create_new_features

https://docs.aws.amazon.com/machine-learning/latest/dg/training-ml-models.html

https://docs.microsoft.com/en-us/windows/ai/windows-ml/what-is-a-machine-learning-model

Physical Resources

https://www.oreilly.com/library/view/machine-learning-design/9781098115777/