Why I Left Data Science - And Picked Data Engineering Instead

Can you be more specific as to why you consider an endless loop of research questions undesirable so we can pinpoint the idiosyncrasies of your experience of data science work?

I would guess that this sort of loop would be candy for a voraciously curious person.

Perhaps I misunderstand. So correct me.

If we focus on the endless research questions vs. more sense of permanence in problem solving and broader set of problems….

What sort of person would prefer data science?

What sort of person would prefer data engineering?

Great video. Your channel has helped me a lot.

your story is exactly why I stopped pursuing data science. About 5 years ago there was a big push to start training people to move into data science. I went through a 16 week training program at my job and got to talk to experienced data scientists and they all talked about how many times you will fail as a data scientist. With that, I thought I'll probably never be able to deliver anything of any value to anyone. So, I gave up and I eventually my existing role sort of morphed into more of an analytics engineer. I'm def grateful I didn't keep pursuing data science because i'd probably be miserable

MLE is a Data Scientist or a Data Engineer that knows how to build a ML model ?
Im about to start my first job as Data Analyst and want to put some future plans.
If i want to be MLE.
Should i focus to get a job in Data Engineering role then switch to Machine Learning Engineering?

Love this channel! I have a question. I have been doing Helpdesk for the last 2 years. I just passed the DP-900 Azure Data Fundamentals also just graduated with my Software development degree. I know sql and python. What skills and roles should I focus on to be a good Data Engineer one day?

I was working as a data scientist for almost 2 years and I am currently having the same experience as you mentioned in the video an endless loop of questions and research without building a permanent product/solution.

I have just a silly question, I love coding, as a data engineer will a get the chance to write a lot of code?

I have the same experience working as a data scientist in healthcare after 3 years. Stuck in research instead of actually building a product. My question is does this feeling arise due to a particular domain (life science) vs eg. Ecommerce or is it the Data scientist role in general? Would love to know your thoughts on this.

its very wage...

How to switch from being a Data science freelancer to a Data Engineer job in corporate?

This whole Data revolution started after 2010. The problem was this revolution started even before data infrastructure was set up in many companies. Just having a database doesn't make you ready for data science. Before 2010, we used to call it Data mining. Now companies are seeing the importance of Data Engineers and are setting up data infrastructure. Data Science will boom again once this is sorted out. Moreover for most companies sophisticated data analytics is more than enough to improve their business. They don't need Data Science algorithms. Our data needs are growing at a rapid pace both personal as well as commercial. Right now, data Engineering is far more important than data science.

So you never mentioned any reason why you left DS or am i missing something?

My goal is to land a Data Analyst jobs then pivot towards a data engineer in like 3 years max.
If that's even possible.

I have seen you doing same video again and again. Request you to talk about some of the best practices about data engineering . I thought your channel is different but all you doing recently is talking same stuff over and over again . I guess channels do get boring after a certain time

wasn’t there a saying in one of the popular DE books that said, “data engineers are just recovering data scientists”?

I did the same. Started as a software engineer working on on data management systems to data scientist back to data engineering. My biggest problem with DS was that management always expect results and want to put you into the same agile structures with the software engineers. I found it almost impossible to correctly estimate the effort let alone the outcome of a DS project.

And also tooling. I neither like R nor pandas.

I can relate to this so much.

Love the new setup!

I think I currently re-live your journey just 8 years later ;) i am a data scientist that is becoming a data engineer due to the same reasons you mentioned.

I have been watching your videos for a long time but it feels like you keep circling back to similar topics such as this....maybe show us actual data engineering; what's under the hood. Just a suggestion. A portion of your followers are people looking to break into DE and honestly when I started all I saw were creators defining a star schema but never showed me an actual one from BigQuery or Snowflake. I made one wrong because my only reference was the diagram creators kept showing haha but all love!! Your content was one of the few things that pushed me to switch roles in my company to pursue an engineering track

Wow, this describes exactly my journey over the years: the growing disappointment about the lack of groundwork - that I did not know was a thing in the first place. I wish I had realized sooner!
I'm now in MLOps, but frankly want to acquire DE skills as it not only will it enable me to push initiatives forward but also give me more independence. And to be honest, a lot of MLOps is very much related to data quality and pipelines - so it feels like just another spin off of the data scientist. Who knows, might become a DE for good, if not a very well rounded ML/SWE engineer.
And to start a company in the data space, one would definitely need DE knowledge in order to lead and push forward.

Thanks for sharing your story! It was inspiring to hear it and I'm sure more people will realize they are not alone in the "data scientist disappointment".

what math do you do as a data engineer?