{"id":8432,"date":"2021-08-18T00:14:20","date_gmt":"2021-08-18T00:14:20","guid":{"rendered":"https:\/\/wealthrevelation.com\/data-science\/2021\/08\/18\/why-your-data-science-team-needs-separate-testing-validation-training-sets\/"},"modified":"2021-08-18T00:14:20","modified_gmt":"2021-08-18T00:14:20","slug":"why-your-data-science-team-needs-separate-testing-validation-training-sets","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2021\/08\/18\/why-your-data-science-team-needs-separate-testing-validation-training-sets\/","title":{"rendered":"Why Your Data Science Team Needs Separate Testing, Validation & Training Sets"},"content":{"rendered":"
\n

Automated testing of machine learning models can help to dramatically reduce the amount of time and effort that your team has to dedicate to debugging them. Not applying these techniques properly, however, could potentially do a great deal of harm if the process is completely unmonitored and doesn\u2019t adhere to a list of best practices. Some tests can be applied to models after the training stage is over while others should be applied directly to test the assumptions that the model is operating under.<\/p>\n

Traditionally, it\u2019s proven somewhat difficult to test machine learning models because they\u2019re very complex containers that often play host to a number of learned operations that can\u2019t be clearly decoupled from the underlying software.. Conventional software can be broken up into individual units that each accomplish a specific task. The same can\u2019t be said of ML models, which are often solely the product of training and therefore can\u2019t be decomposed into smaller parts.<\/p>\n

Testing and evaluating the data sets<\/a> that you use for training could be the first step in unraveling this problem.<\/p>\n

Monitoring Data Sets in a Training Environment<\/h2>\n

ML testing is very different from testing application software because anyone performing checks on ML models will find that they\u2019re attempting to test something that is probabilistic as opposed to deterministic. An otherwise perfect model could occasionally make mistakes and still be considered the best possible model that someone could develop. Engineers working on a spreadsheet or database program wouldn\u2019t be able to tolerate even the slightest rounding errors, but it\u2019s at least somewhat acceptable to find the occasional flaw in the output of a program that processes data by way of learned responses. The level of variance<\/a> will differ somewhat depending on the tasks that a particular model is being trained to accomplish, but it may always be there regardless.<\/p>\n

As a result, it\u2019s important to at least examine the initial data that\u2019s being used to train ML models. If this data doesn\u2019t accurately represent the kind of information that a real-world environment would thrust onto a model, then said model couldn\u2019t ever hope to perform adequately when such input is finally given. Decent input specifications will help to ensure that the model comes away with a somewhat accurate representation of natural variability in whatever industry it\u2019s performing a study in.<\/p>\n

Pure performance measurements can come from essentially any test set, but data scientists will normally want to specify the hyperparameters of their model to provide a clear metric by which to judge performance while taking said measurements. Those who consistently use one model over another solely for its performance on a particular test set may end up fitting test sets to models to find something that performs exactly as they want it to.<\/p>\n

Those who are working with smaller data sets will need to find some way to evaluate them in spite of their diminutive size.<\/p>\n

Managing a Smaller Set of Data Safely<\/h2>\n

Those working with particularly large data sets have typically gone with 60-20-20 or 80-10-10 splits. This has helped to strike a decent balance between the competing needs of reducing potential bias<\/a> while also ensuring that the simulations run fast enough to be repeated several times over.<\/p>\n

Those working with a smaller data set might find that it simply isn\u2019t representative enough, but for whatever reason it isn\u2019t possible to increase the amount of information put into the test set. Cross-validation might be the best option for those who find themselves in this sort of situation. This is normally used by those in the applied ML field to compare and select models since it\u2019s relatively easy to understand.<\/p>\n

K-fold cross-validation algorithms are often used to estimate the skill of a particular ML model on new data irrespective of the size of the data in question. No matter what method you decide to try, though, your team needs to keep the concepts of testing and validation data separate when training your ML model. When you square these off in a training data vs. test data<\/a> free-for-all, the results should come out something like this:<\/p>\n