{"id":8382,"date":"2021-07-20T01:33:21","date_gmt":"2021-07-20T01:33:21","guid":{"rendered":"https:\/\/wealthrevelation.com\/data-science\/2021\/07\/20\/attention-monitoring\/"},"modified":"2021-07-20T01:33:21","modified_gmt":"2021-07-20T01:33:21","slug":"attention-monitoring","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2021\/07\/20\/attention-monitoring\/","title":{"rendered":"Attention Monitoring"},"content":{"rendered":"
\n

LinkedIn<\/a> | GitHub<\/a> | Email<\/a> | Notebook<\/a><\/p>\n

\u00a0<\/p>\n

The Idea<\/h2>\n

Hate to break it to you, but you cannot trust people. Initially, there were no cabin-facing cameras in Tesla vehicles. It was thought that an agreement before using Autopilot and detecting whether adequate pressure was applied to the steering wheel was sufficient to ensure that the driver would pay attention. Even when cameras were added, starting with the Model 3, they were not intended to monitor the driver. Elon Musk, CEO and Product Architect of Tesla, stated they were implemented to prevent people from vandalizing cars when they were used as <\/span>robotaxis<\/span><\/a>. However, George Hotz, President of Comma.ai was convinced that Musk would ultimately have to add driver-monitoring cameras to their vehicles. His prediction that people would misuse Autopilot as Tesla\u2019s driver-assistance features advanced and gained in popularity turned out to be correct.<\/span><\/p>\n

\n

“Do I still need to pay attention while using Autopilot?<\/b><\/em><\/p>\n

Yes. Autopilot is a hands-on driver assistance system that is intended to be used only with a fully attentive driver. It does not turn a Tesla into a self-driving car nor does it make a car autonomous.<\/span><\/em><\/p>\n

Before enabling Autopilot, you must agree to \u201ckeep your hands on the steering wheel at all times\u201d and to always \u201cmaintain control and responsibility for your car.\u201d Once engaged, Autopilot will also deliver an escalating series of visual and audio warnings, reminding you to place your hands on the wheel if insufficient torque is applied. If you repeatedly ignore these warnings, you will be locked out from using Autopilot during that trip.<\/span><\/em><\/p>\n

You can override any of Autopilot\u2019s features at any time by steering, applying the brakes, or using the cruise control stalk to deactivate.”<\/span><\/em><\/p>\n

https:\/\/www.tesla.com\/support\/autopilot<\/em>
<\/span><\/i><\/p>\n<\/blockquote>\n

The many reports of people sleeping, reading, and engaging in other activities that showcased their utter disregard for the Autopilot agreement were proof enough that more obtrusive mechanisms had to be implemented to garner greater compliance among its drivers. Beginning in early 2020, Volvo (Pilot Assist) also began adding driver-monitoring cameras to all of their vehicles to combat distracted driving. They have joined the likes of Comma.ai\u2019s Openpilot, GM\u2019s Super Cruise, Ford\u2019s Co-Pilot360, and a myriad of other companies and their driver-assistance technologies that incorporate driver-monitoring.<\/span><\/p>\n

\u00a0<\/p>\n

The Implementation<\/h2>\n

\"\"<\/p>\n

Building a platform on a neural network that utilized a human-facing camera to determine whether or not the person was attentive seemed straightforward and scalable. Swap a camera mounted on a rearview mirror for a standard laptop camera, and you have my project, essentially.<\/span><\/p>\n

\u00a0<\/p>\n

The Steps<\/h2>\n

Take a look at my <\/span><\/i>notebook<\/span><\/i><\/a> to see how the steps below played out.<\/span><\/i><\/p>\n

    \n
  1. Understand Neural Networks \u2713<\/span><\/li>\n
  2. Outline Project<\/span> \u2713<\/span><\/li>\n
  3. Collect and Prepare Data<\/span> \u2713<\/span><\/li>\n
  4. Build, Test, Iterate<\/span> \u2713\u00a0<\/span><\/li>\n
  5. Final Results <\/span> \u2713<\/span><\/li>\n<\/ol>\n

    \u00a0<\/p>\n

    1. Understand<\/h3>\n

    I chose a convolutional neural network (CNN) because it is commonly used in image classification. CNNs are preferred over traditional neural networks because it reduces the number of input nodes, tolerates pixel shifts in the image, and takes advantage of the correlation observed in complex images where similarly colored pixels tend to be close together. I went with a pre-trained CNN as it would yield much better results than training the CNN from scratch, given my limited dataset. This process is called <\/span>transfer learning<\/span><\/a>.\u00a0<\/span><\/p>\n

    ResNet-152<\/span><\/a> was chosen due to its better performance compared to other pre-trained architectures offered by the <\/span>Keras<\/span><\/a> module in TensorFlow. Its depth is a big contributor to its effectiveness; the \u201c152\u201d stands for 152 layers. ResNet-152 is trained on a subset of the ImageNet dataset consisting of 1.2 million images with 1000 categories.<\/span><\/p>\n

    \u00a0<\/p>\n

    2. Outline<\/h3>\n
      \n
    1. Train a pre-trained CNN to classify images.<\/span>\n
      \"\"<\/p>\n

      Top Layer<\/em><\/p>\n<\/div>\n<\/li>\n

    2. Use CNN to classify video clips.<\/li>\n
    3. Use CNN to classify live video.<\/span><\/li>\n
    4. Learn how to train the model on AWS (TBD).<\/span><\/i><\/li>\n
    5. Deploy a web application (TBD).<\/span><\/i><\/li>\n<\/ol>\n

      \u00a0<\/p>\n

      3. The Data<\/h3>\n

      Ideally, I should have had a bunch of interns helping me with this, and I am only half joking. It was the longest and most tedious, hands-on portion of my project (the longest portion was training the model). I was unable to find a single dataset that met my requirements with regard to the subject\u2019s body position and diversity, accessibility, and data size. So I spent a great deal of time compiling and labeling the data. By the end, I had 10427 photos. I divided them manually into a 70-30 train-evaluation split, then further manually divided the 30 to 15-15 validation-test split. This had to be done manually so the photos from the various data sources were more-or-less evenly distributed. <\/span><\/p>\n

      \"\"<\/p>\n

      Here is how I partitioned the data:
      Train<\/span>: 7302 images belonging to 2 classes
      Validate<\/span>: 1561 images belonging to 2 classes.
      Test<\/span>: 1564 images belonging to 2 classes.<\/p>\n

      The training data was augmented to create a more diverse data set and make the model more generalizable. The first set of parameters I used achieved that goal but resulted in unrealistic training data:<\/span><\/p>\n

      \"\"<\/p>\n

      So I tweaked the parameters when I retrained the model:<\/p>\n

      \"\"<\/p>\n

      \u00a0<\/p>\n

      4. Build, Test, Iterate<\/h3>\n

      Initially I trained a ResNet-152 model with frozen base layers and unfrozen fully-connected layers I constructed. The base layers were frozen to prevent their weights from being updated during training.<\/p>\n

      \"\"\"\"<\/p>\n

      As mentioned, I tweaked the augmentation parameters to get more realistic images, and unfroze the last convolution block so that the last block could be trained with the head I constructed.<\/p>\n

      \"\"\"\"<\/p>\n

      1st model training results:
      loss: 0.4209 – accuracy: 0.7615<\/p>\n

      2nd model training results:
      loss: 0.3175 – accuracy: 0.8561<\/p>\n

      Although the second model reported a better loss and accuracy, one can see by the graphs that it has a greater degree of overfitting. Tweaking the augmentation parameters and using techniques such as regularization could mitigate that overfitting.<\/p>\n

      \u00a0<\/p>\n

      5. Results<\/h3>\n

      I utilized the OpenCV computer vision library for the video classification portion of my project. To my surprise and delight, there was no substantial difference in code between classifying video clips and classifying live video from my laptop\u2019s webcam.<\/p>\n

      As I tested the model, I noticed a couple things:<\/p>\n