{"id":8255,"date":"2021-05-10T03:16:59","date_gmt":"2021-05-10T03:16:59","guid":{"rendered":"https:\/\/wealthrevelation.com\/data-science\/2021\/05\/10\/we-dont-need-data-engineers-we-need-better-tools-for-data-scientists\/"},"modified":"2021-05-10T03:16:59","modified_gmt":"2021-05-10T03:16:59","slug":"we-dont-need-data-engineers-we-need-better-tools-for-data-scientists","status":"publish","type":"post","link":"https:\/\/wealthrevelation.com\/data-science\/2021\/05\/10\/we-dont-need-data-engineers-we-need-better-tools-for-data-scientists\/","title":{"rendered":"We Don\u2019t Need Data Engineers, We Need Better Tools for Data Scientists"},"content":{"rendered":"
\n <\/p>\n

By Devin Petersohn<\/a>, PhD Student at UC Berkeley<\/b>.<\/p>\n

\"Tools\"<\/p>\n

In most companies, Data Engineers support Data Scientists in various ways. Often this means translating or productionizing the notebooks and scripts that a Data Scientist has written.\u00a0A large portion of the Data Engineer\u2019s role could be replaced with better tooling for Data Scientists, freeing Data Engineers to do more impactful (and scalable) work.<\/strong><\/p>\n

\u00a0<\/p>\n

Why does this matter?<\/h3>\n

\u00a0<\/p>\n

There\u2019s a sentiment making its way around the internet (again): We don\u2019t need Data Scientists, we need Data Engineers.<\/p>\n

<\/p>\n

(source<\/a>)<\/em><\/p>\n

<\/p>\n

(source<\/a>)<\/em><\/p>\n

These articles focus on the\u00a0number of available job positions for the title of \u201cData Engineer\u201d vs. \u201cData Scientist.\u201d Let\u2019s put aside the fact that the hiring managers who post these positions often don\u2019t know the difference between the two jobs and use them interchangeably (or use whatever is in style at the moment). For the sake of this article, we can take the existence of the positions at face value. The question then becomes:\u00a0Is the surplus of available Data Engineer positions solely a personnel problem?<\/strong><\/p>\n

\u00a0<\/p>\n

Data Science is messy because it reflects the real world<\/h3>\n

\u00a0<\/p>\n

Data Scientists are domain experts (on top of knowing statistics), and they don\u2019t often have a strong background in programming. I\u2019ve seen this expertise discounted in multiple Twitter and forum threads, with software engineers and other \u201ctechnical people\u201d asking questions like \u201cWhy don\u2019t they just learn Spark?\u201d This type of mentality completely misses the fact that Data Scientists can already do what they want to do at smaller scales with their existing tools. Data Scientists want to gain insights, not worry about building elegant pipelines. Companies want something actionable, not beautiful.<\/p>\n

\n

Insights are more important than elegant pipelines.<\/em><\/p>\n<\/blockquote>\n

Popular Data Science tools are also criticized by more technical people and academics: \u201cWhy would anyone use pandas?\u201d\u00a0pandas must be the most popular tool to hate by people who have no use for it<\/strong>. It is loved (or at least appreciated) by the Data Scientists who use it daily, however.<\/p>\n

pandas, among other tools, was built to handle the\u00a0messiness<\/strong>\u00a0of the real world. Just look at how many parameters\u00a0read_csv<\/em>\u00a0has:<\/p>\n

<\/p>\n

(read_csv reference)<\/a><\/em><\/p>\n

If pandas is so bad, why has nothing unseated it as the standard dataframe for Python Data Science? Why does it continue to grow in adoption year after year? It\u2019s not the fastest, it\u2019s not the most robust, so why?<\/p>\n

\u00a0<\/p>\n

Data Engineers have to handle the messiness that scalable tools can\u2019t<\/h3>\n

\u00a0<\/p>\n

The scalable systems (e.g., Apache Spark) that are robust enough for production use can\u2019t handle the messiness of the real world as-is. It\u2019s difficult to scale without clean and simple assumptions, and the messier the problem, the harder it is to scale. Data Engineers handle the messiness because scalable tools can\u2019t.<\/p>\n

Messiness, in this case, can mean:<\/p>\n