Why it's your fault your data scientists keep quitting

Bummer. You worked so hard to find the perfect data scientist to magically transform all your data into (wait for it!) "actionable insights" and now she's quitting. Even worse, it's your fault, and you may not realize it.

It's not you, it's me

As Jonny Brooks highlighted in an insightful article, data scientists may be richly paid but many tend to be unhappy in their current jobs. As such, machine learning experts and data scientists top the list of technical folks actively searching for a new job. The reasons are several, but Brooks points to one in particular: Most companies aren't actually in a good position to make use of the data they have.

As Brooks wrote:

[M]any companies hire data scientists without a suitable infrastructure in place to start getting value out of AI. This contributes to the cold start problem in AI. Couple this with the fact that these companies fail to hire senior/experienced data practitioners before hiring juniors, [and] you've now got a recipe for a disillusioned and unhappy relationship for both parties. The data scientist likely came in to write smart machine learning algorithms to drive insight but can't do this because their first job is to sort out the data infrastructure and/or create analytic reports. In contrast, the company only wanted a chart that they could present in their board meeting each day. The company then gets frustrated because they don't see value being driven quickly enough and all of this leads to the data scientist being unhappy in their role.

Of course, companies can reset expectations, but the real problem starts with the data. Surveys have remained consistent in suggesting that roughly 80% of a data scientist's time is spent on cleaning and organizing data, the absolute last thing they want to be doing with their time. This isn't new, and it doesn't seem to be getting much better. Way back in 2009, Mike Driscoll coined the term "data munging" and detailed it as a "painful process of cleaning, parsing, and proofing one's data."

Not surprisingly, data scientists join a company to change the world through data, but quit when they realize they're merely taking out the data garbage. The company likely doesn't know just how grubby its data is, so can't fathom why the data scientist never quite gets around to those easy-to-say-but-hard-to-realize "actionable insights."

Other reasons to be unhappy

While mismatched expectations are the biggest miss in data scientist happiness, there are others. For one, data scientists may want to swim in data but first they must bathe in politics.

"The truth is the people in the business with the most clout need to have a good perception of you," Brooks said, which means that "you have to constantly do ad hoc work such as getting numbers from a database to give to the right people at the right time."

Perception matters, and in order to have adequate time to clean all that dirty data, you may have to pull some pseudo-data science tricks to placate the powers that be.

Also, given that there's a near-religious reliance on data these days, many companies expect their pet data scientist to be a unicorn. As Brooks wrote: "you'll be the analytics expert as well as the go-to reporting guy and let's not forget that you'll be the database expert too." Never mind that none of these were what you (thought you) were hired to do-you're suddenly the data expert. All of it. (Now if only you could get through prepping the data.)

Of course, some of these same issues plague people hired to do most any job. Poorly-aligned expectations wreck many a hire. In the case of data scientists, however, expectations (and attendant salaries) are so inflated that it behooves the data scientist to carefully calibrate expectations from the start. This is easier said than done, of course, but the alternative is an expensive series of getting the "wrong" hires.

The article was originally published here

Source: HOB

Share This On

Previous Story Next Story

Latest News Feed