{"id":2945,"date":"2020-10-07T14:39:43","date_gmt":"2020-10-07T14:39:43","guid":{"rendered":"https:\/\/data-science.gotoauthority.com\/2020\/10\/07\/a-step-by-step-guide-for-creating-an-authentic-data-science-portfolio-project\/"},"modified":"2020-10-07T14:39:43","modified_gmt":"2020-10-07T14:39:43","slug":"a-step-by-step-guide-for-creating-an-authentic-data-science-portfolio-project","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2020\/10\/07\/a-step-by-step-guide-for-creating-an-authentic-data-science-portfolio-project\/","title":{"rendered":"A step-by-step guide for creating an authentic data science portfolio project"},"content":{"rendered":"
\n

By Felix Vemmer<\/a>, Operational Intelligence Data Analyst at N26<\/b>.<\/p>\n

<\/p>\n

As an inspiring data scientist,\u00a0building interesting portfolio projects is key to showcase your skills.<\/strong> When I\u00a0learned coding and data science as a business student<\/a>\u00a0through online courses, I disliked that datasets were made up of\u00a0fake data or were solved before\u00a0like\u00a0Boston House Prices<\/a>\u00a0or the\u00a0Titanic dataset<\/a>\u00a0on Kaggle.<\/p>\n

In this blog post, I want to\u00a0show you how I develop interesting data science project ideas and implement them step by step<\/strong>, such as exploring Germany\u2019s biggest frequent flyer forum Vielfliegertreff. If you are\u00a0short on time, feel free to skip to the conclusion TLDR.<\/p>\n

\u00a0<\/p>\n

Step 1: Choose your passion topic that is relevant<\/h3>\n

\u00a0<\/p>\n

As a first step, I think about a potential project that fulfills the following three requirements to make it the most interesting and enjoyable:<\/p>\n

    \n
  1. Solving\u00a0my own problem or burning question.<\/strong>\n<\/li>\n
  2. Connected to somerecent event to be relevant<\/strong>\u00a0or especially interesting.<\/li>\n
  3. Has not\u00a0been solved or covered before.<\/strong>\n<\/li>\n<\/ol>\n

    As these ideas are still quite abstract, let me give you a rundown of how my three projects fulfilled the requirements:<\/p>\n

    Overview of my own data science portfolio projects fulfilling the three outlined requirements.<\/em><\/p>\n

    As a beginner,\u00a0do not strive for perfection, but choose something you are\u00a0genuinely curious about\u00a0and write down all the questions you want to explore in your topic.<\/p>\n

    \u00a0<\/p>\n

    Step 2: Start scraping together your own dataset<\/h3>\n

    \u00a0<\/p>\n

    Given that you followed my third requirement, there will be no dataset publicly available, and you will have to scrape data together yourself. Having scraped a couple of websites, there are\u00a03 major frameworks<\/strong> I use for different scenarios:<\/p>\n

    Overview of the 3 major frameworks I use for scraping.<\/em><\/p>\n

    For Vielfliegertreff, I used\u00a0scrapy<\/a>\u00a0as a framework for the following reasons:<\/p>\n

      \n
    1. There was\u00a0no JavaScript\u00a0<\/strong>enabled elements that were hiding data.<\/li>\n
    2. The website\u00a0structure was complex, <\/strong>having to go from each forum subject, to all the threads and from all the treads to all post website pages. With\u00a0scrapy you can easily implement complex logic\u00a0<\/strong>yielding requests that lead to new callback functions in an organized way.<\/li>\n
    3. There were quite a lot of posts, so crawling the entire forum will definitely take some time. Scrapy allows you to\u00a0asynchronously scrape websites at an incredible speed<\/strong>.<\/li>\n<\/ol>\n

      To give you just an idea of how powerful scrapy is, I quickly benchmarked my MacBook Pro (13-inch, 2018, Four Thunderbolt 3 Ports) with a 2,3 GHz Quad-Core Intel Core i5 that was able to\u00a0scrape around 3000 pages\/minute:<\/p>\n

      <\/p>\n

      Scrapy scraping benchmark. (Image by Author)<\/em><\/p>\n

      To be nice and not to get blocked, it is important that you scrape gently, by for example, enabling scrapy\u2019s\u00a0auto-throttle feature<\/a>. Furthermore, I also saved all data to an SQL lite database via an items pipeline\u00a0to avoid duplicates and turned on to log each URL request to make sure I do not put more load on the server if I stop and restart the scraping process.<\/p>\n

      Knowing how to scrape gives you the\u00a0freedom to collect datasets\u00a0by yourself and\u00a0teaches you important concepts about how the internet works, what a request is, and the structure of HTML\/XPath.<\/p>\n

      For my project, I ended up with\u00a01.47 GB of data which was close to 1 million posts in the forum.<\/p>\n

      \u00a0<\/p>\n

      Step 3: Cleaning your dataset<\/h3>\n

      \u00a0<\/p>\n

      With your own scraped messy dataset, the most challenging part of the portfolio project comes, where\u00a0data scientists spend on average 60% of their time<\/strong>:<\/p>\n

      <\/p>\n

      Image by\u00a0CrowdFlower<\/a>\u00a02016.<\/em><\/p>\n

      Unlike clean Kaggle datasets, your own dataset allows you to\u00a0build skills in data cleaning\u00a0and show a future employer that you are\u00a0ready to deal with messy real-life datasets. Additionally, you can explore and take advantage of the Python ecosystem by\u00a0leveraging libraries that solve some common data cleaning tasks\u00a0that others solved before.<\/p>\n

      For my dataset from Vielfliegertreff, there were a couple of common tasks like turning the\u00a0dates into a pandas timestamps, converting numbers from strings into actual numeric data types, and cleaning a very messy HTML post text\u00a0to something readable and usable for NLP tasks. While some tasks are a bit more complicated, I would like to\u00a0share my top 3 favourite librarie<\/strong>s that solved some of my common data cleaning problems:<\/p>\n

        \n
      1. \ndateparser<\/a>:\u00a0Easily parse localized dates in almost any string formats commonly found on web pages.<\/li>\n
      2. \nclean-text<\/a>:\u00a0Preprocess your scraped data with clean-text to create a normalized text representation. This one is also amazing to remove personally identifiable information, such as emails or phone numbers etc.<\/li>\n
      3. \nfuzzywuzzy<\/a>:\u00a0Fuzzy string matching like a boss.<\/li>\n<\/ol>\n

        \u00a0<\/p>\n

        Step 4: Data Exploration and Analysis<\/h3>\n

        \u00a0<\/p>\n

        When completing the Data Science Nanodegree on Udacity, I came across the\u00a0Cross-Industry Standard Process for Data Mining (CRISP-DM)<\/strong>, which I thought was quite an interesting framework to structure your work in a systematic way.<\/p>\n

        With our current flow, we implicitly followed the CRISP-DM for our project:<\/p>\n

        Expressing\u00a0business understanding<\/strong>\u00a0by coming up with the following questions in step 1:<\/p>\n

          \n
        1. How is COVID-19 impacting online frequent flyer forums like Vielfliegertreff?<\/li>\n
        2. What are some of the best posts in the forums?<\/li>\n
        3. Who are the experts that I should follow as a new joiner?<\/li>\n
        4. What are some of the worst or best things people say about airlines or airports?<\/li>\n<\/ol>\n

          And with the scraped data, we are now able to\u00a0translate our initial business questions from above into specific data explanatory questions:<\/p>\n

            \n
          1. How many posts are posted on a monthly basis? Did the posts decrease at the beginning of 2020 after COVID-19? Is there also some sort of indication that fewer people joined the platform not being able to travel?<\/li>\n
          2. What are the top 10 number of posts by the number of likes?<\/li>\n
          3. Who is posting the most and also receives, on average, the most likes for the post? These are the users I should regularly follow to see the best content.<\/li>\n
          4. Could a sentiment analysis on every post in combination with named entity recognition to identify cities\/airports\/airlines lead to interesting positive or negative comments?<\/li>\n<\/ol>\n

            For the Vielfliegertreff project, one can definitely say that there has been\u00a0a trend of declining posts over the years. With\u00a0COVID-19, we can clearly see a rapid decrease in posts from January 2020\u00a0onwards when Europe was shutting down and closing borders, which also heavily affected travelling:<\/p>\n

            Posts created by month. (Chart by Author)<\/em><\/p>\n

            Also,\u00a0user sign-ups have gone down over the years, and the forum seemed to see less and less of its rapid growth since the start in January 2009:<\/p>\n

            Sign up numbers of users over the months. (Chart by author)<\/em><\/p>\n

            Last but not least, I wanted to check what the most liked post was about. Unfortunately, it is in Germany, but it was indeed a very interesting post, where a German guy was allowed to\u00a0spend some time on a US aircraft carrier and experienced a catapult take off in a C2 airplane. The post has some very nice pictures and interesting details. Feel free to check it out\u00a0here<\/a>\u00a0if you can understand some German:<\/p>\n

            <\/p>\n

            Sample picture from the most liked post on Vielfliegertreff (Image by\u00a0fleckenmann<\/a>).<\/em><\/p>\n

            \u00a0<\/p>\n

            Step 5: Share your work via a blog post or web app<\/h3>\n

            \u00a0<\/p>\n

            Once you are done with those steps, you can go one step further and create a model that classifies or predicts certain data points. For this project, I did not attempt further to use machine learning in a specific way, although I had some interesting ideas about\u00a0classifying the sentiment of posts in connection with certain airlines.<\/p>\n

            In another project, however,\u00a0I modeled a price prediction algorithm that allows a user to get a price estimate for any type of tractor. The model was then deployed with the awesome\u00a0streamlit framework<\/a>, which can be found\u00a0here<\/a>\u00a0(be patient with loading, as it might load a bit slower).<\/p>\n

            Another way to share your work is like me through blog posts on Medium,\u00a0Hackernoon<\/a>,\u00a0KDNuggets<\/a>, or other popular websites. When writing blog posts about portfolio projects or other topics, such as\u00a0awesome interactive AI applications<\/a>, I always try to make them as\u00a0fun, visual, and interactive\u00a0<\/strong>as possible. Here are some of my top tips:<\/p>\n