{"id":39,"date":"2020-08-04T12:05:55","date_gmt":"2020-08-04T12:05:55","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/08\/04\/data-drift-detection-for-image-classifiers\/"},"modified":"2020-08-04T12:05:55","modified_gmt":"2020-08-04T12:05:55","slug":"data-drift-detection-for-image-classifiers","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/08\/04\/data-drift-detection-for-image-classifiers\/","title":{"rendered":"Data Drift Detection for Image Classifiers"},"content":{"rendered":"
\n

This article covers how to detect data drift for models that ingest image data as their input in order to prevent their silent degradation in production. Run the example in a complementary Domino project<\/a>. <\/small><\/small><\/em><\/p>\n

In the real word, data is recorded by different systems and is constantly changing. How data is collected, treated and transformed impacts the data. Change may occur with the introduction of noise due to mechanical wear and tear to physical systems or if a fundamental shift in the underlying generative process occurs (e.g., a change in the interest rate by a regulator, a natural calamity, the introduction of a new competitor in the market or a change in a business strategy\/process, etc.). Such changes have ramifications for the accuracy of predictions and necessitate the need to check that the assumptions made during the development of models still hold good when models are in production.<\/p>\n

In the context of machine learning, we consider data drift1<\/sup> to be the change in model input data that leads to a degradation of model performance. In the remainder of this article, we shall cover how to detect data drift for models that ingest image data as their input in order to prevent their silent degradation in production.<\/p>\n

Given the proliferation of interest in deep learning in the enterprise, models that ingest non traditional forms of data such as unstructured text and images into production are on the rise. In such cases, methods from statistical process control and operations research that rely primarily on numerical data are hard to adopt and necessitates a new approach to monitoring models in production. This article explores an approach that can be used to detect data drift for models that classify\/score image data.<\/p>\n

Our approach does not make any assumptions about the model that has been deployed, but it requires access to the training data used to build the model and the prediction data used for scoring. The intuitive approach to detect data drift for image data is to build a machine learned representation of the training dataset and to use this representation to reconstruct data that is being presented to the model. If the reconstruction error is high then the data being presented to the model is different from what it was trained on. The sequence of operations is as follows:<\/p>\n

    \n
  1. Learn a low dimensional representation of the training dataset (encoder)<\/li>\n
  2. Reconstruct a validation dataset using the representation from Step 1 (decoder) and store reconstruction loss as baseline reconstruction loss<\/li>\n
  3. Reconstruct batch of data that is being sent for predictions using encoder and decoder in Steps 1 and 2; store reconstruction loss<\/li>\n<\/ol>\n

    If reconstruction loss of dataset being used for predictions exceeds the baseline reconstruction loss by a predefined threshold set off an alert.<\/p>\n

    The steps below show the relevant code snippets that cover the approach that has been detailed above. The complete code of the project is available and can be forked from the Image_Drift_Detection<\/em> project<\/a> on https:\/\/try.dominodatalab.com<\/a>.<\/p>\n

    \"\"<\/p>\n

    The notebook (Convolutional_AutoEncoder.ipynb<\/em>) that is available at Image_Drift_Detection<\/em> project<\/a><\/p>\n

    \"\"<\/p>\n

    \"\"<\/p>\n

    is laid out as follows:<\/p>\n

      \n
    1. Train a convolutional autoencoder on the MNIST dataset2<\/sup>; use the validation loss as the baseline to compare drift for new datasets.<\/li>\n
    2. Add noise to the MNIST dataset and attempt to reconstruct the noisy MNIST dataset; note reconstruction loss. (to be used as an alternative baseline if required).<\/li>\n
    3. Reconstruct the nonMNIST3 dataset using the convolutional autoencoder built in Step 1 and compare the reconstruction loss with the validation loss of the MNIST dataset or the reconstruction loss on the noisy MNIST data set.<\/li>\n<\/ol>\n

      We now dive into the relevant code snippets to detect drift on image data.<\/p>\n

      Step 1<\/strong><\/em>: Install the required dependencies for the project by adding the following to your Dockerfile. You can also use the Image_Drift_Keras_TF<\/em> environment in https:\/\/try.dominodatalab.com<\/a> as it has all the libraries\/dependencies preinstalled.<\/p>\n

      \r\n\r\nRUN pip install numpy==1.13.1\r\nRUN pip install tensorflow==1.2.1\r\nRUN pip install Keras==2.0.6\r\n\r\n<\/pre>\n
      Step 2:<\/strong><\/em> Start a Jupyter notebook workspace and load the required libraries.<\/p>\n
      \r\n\r\n#Load libraries\r\nfrom keras.layers import Input, Dense, Conv2D, MaxPool2D, UpSampling2D, MaxPooling2D\r\nfrom keras.models import Model\r\nfrom keras import backend as K\r\nfrom keras import regularizers\r\nfrom keras.callbacks import ModelCheckpoint\r\n\r\nfrom keras.datasets import mnist\r\nimport numpy as np\r\nimport matplotlib.pyplot as plt\r\n\r\n<\/pre>\n
      Step 3:<\/strong> <\/em>Specify the architecture for the autoencoder.<\/p>\n
      \r\n\r\n#Specify the architecture for the auto encoder\r\ninput_img = Input(shape=(28, 28, 1))\r\n\r\n# Encoder\r\nx = Conv2D(32, (3, 3), activation='relu', padding='same')(input_img)\r\nx = MaxPooling2D((2, 2), padding='same')(x)\r\nx = Conv2D(32, (3, 3), activation='relu', padding='same')(x)\r\nencoded = MaxPooling2D((2, 2), padding='same')(x)\r\n\r\n# Decoder\r\nx = Conv2D(32, (3, 3), activation='relu', padding='same')(encoded)\r\nx = UpSampling2D((2, 2))(x)\r\nx = Conv2D(32, (3, 3), activation='relu', padding='same')(x)\r\nx = UpSampling2D((2, 2))(x)\r\ndecoded = Conv2D(1, (3, 3), activation='sigmoid', padding='same')(x)\r\n\r\nautoencoder = Model(input_img, decoded)\r\nautoencoder.compile(optimizer='adadelta', loss='binary_crossentropy')\r\n\r\n<\/pre>\n
      Step 4:<\/strong> <\/em>Generate the test, train and noisy MNIST data sets.<\/p>\n
      \r\n\r\n# Generate the train and test sets\r\n(x_train, _), (x_test, _) = mnist.load_data()\r\n\r\nx_train = x_train.astype('float32') \/ 255.\r\nx_test = x_test.astype('float32') \/ 255.\r\nx_train = np.reshape(x_train, (len(x_train), 28, 28, 1))\r\nx_test = np.reshape(x_test, (len(x_test), 28, 28, 1))\r\n\r\n# Generate some noisy data\r\nnoise_factor = 0.5\r\nx_train_noisy = x_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_train.shape)\r\nx_test_noisy = x_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=x_test.shape)\r\n\r\nx_train_noisy = np.clip(x_train_noisy, 0., 1.)\r\nx_test_noisy = np.clip(x_test_noisy, 0., 1.)\r\n\r\n<\/pre>\n
      Step 5:<\/strong> <\/em>Train the convolutional autoencoder.<\/p>\n
      \r\n\r\n# Checkpoint the model to retrieve it later\r\ncp = ModelCheckpoint(filepath=\"autoencoder_img_drift.h5\",\r\nsave_best_only=True,\r\nverbose=0)\r\n\r\n# Store training history\r\nhistory = autoencoder.fit(x_train_noisy, x_train,\r\n                epochs=10,\r\n                batch_size=128,\r\n                shuffle=True,\r\n                validation_data=(x_test_noisy, x_test),\r\n                callbacks = [cp, plot_losses]).history\r\n\r\n<\/pre>\n
      The validation loss we obtained is 0.1011.<\/p>\n
      Step 6:<\/strong> <\/em>Get the reconstruction loss for the noisy MNIST data set.<\/p>\n
      \r\n\r\nx_test_noisy_loss = autoencoder.evaluate(x_test_noisy,decoded_imgs)\r\n\r\n<\/pre>\n
      \"\"
      Sample of noisy input data (top row) and reconstructed images (bottom row)<\/figcaption><\/figure>\n
      The reconstruction error that we obtained on the noisy MNIST data set is 0.1024<\/strong>. Compared with the baseline reconstruction error on the validation dataset this is a 1.3%<\/strong> increase in the error.<\/p>\n
      Step 7:<\/strong><\/em> Get the reconstruction loss for the nonMNIST dataset; compare with validation loss for MNIST dataset and noisy MNIST.<\/p>\n
      \r\n\r\nnon_mnist_data_loss = autoencoder.evaluate(non_mnist_data,non_mnist_pred)\r\n\r\n<\/pre>\n
      \"\"
      Sample of nonMNIST data (top row) and its corresponding reconstruction (bottom row)<\/figcaption><\/figure>\n
      The reconstruction error that we obtained on the nonMNIST data set is 0.1458<\/strong>. Compared with the baseline reconstruction error on the validation dataset, this is a 44.2%<\/strong> increase in the error.<\/p>\n
      From this example, it is clear that there is a spike in the reconstruction loss when the convolution encoder is made to reconstruct a dataset that is different than the one used to train the model. The next step as a result of this detection is to either retrain the model with new data or to investigate what led to a change in the data.<\/p>\n
      In this example, we saw that a convolutional autoencoder is capable of quantifying differences in image datasets on the basis of a reconstruction error. Using this approach of comparing reconstruction errors, we can detect changes in the input that is being presented to image classifiers. However, improvements can be made to this approach in the future. For one, while we used a trial-and-error approach to settle on the architecture of the convolutional autoencoder, by using a neural architecture search algorithm we can obtain a much more accurate autoencoder model. Another improvement could be using a data augmentation strategy on the training dataset to make the autoencoder more robust to noisy data and variances in rotations and translations of images.<\/p>\n
      If you would like to know more about how you can use Domino to monitor your production models, you can attend our Webinar on \u201cMonitoring Models at Scale<\/a>\u201d. It will cover continuous monitoring for data drift and model quality.<\/p>\n
      References<\/h3>\n
        \n
      1. Gama, Joao; Zliobait, Indre; Bifet, Albert; Pechenizkiy, Mykola; and Bouchachia, Abdelhamid. \u201cA Survey on Concept Drift Adaptation\u201d ACM Computing Survey Volume 1<\/em>, Article 1 (January 2013)<\/li>\n
      2. LeCun, Yann; Corinna Cortes; Christopher J.C. Burges. \u201cThe MNIST Database of handwritten digits<\/a>\u201c.<\/li>\n
      3. notMNIST dataset<\/a><\/li>\n<\/ol>\n
        \n<\/div>\n","protected":false},"excerpt":{"rendered":"
        https:\/\/blog.dominodatalab.com\/data-drift-detection-for-image-classifiers\/<\/p>\n","protected":false},"author":0,"featured_media":40,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[2],"tags":[],"_links":{"self":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/39"}],"collection":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/comments?post=39"}],"version-history":[{"count":0,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/posts\/39\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media\/40"}],"wp:attachment":[{"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/media?parent=39"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/categories?post=39"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wealthrevelation.com\/data-science\/wp-json\/wp\/v2\/tags?post=39"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}