Think Less, Build More đŸ’ĄđŸƒđŸ»
All articles

December 18, 2021

What I learned from working as a Software Engineer, Machine Learning Engineer, and Data Scientist within Four Years.

Lessons learned from switching roles and navigating the career in tech

Views

  • career

Photo credit: Northwestern MSIA. Me inside a classroom doing something on my laptop.

Read the original article here.

Hi, I’m Jason.

I am an accountant turned data scientist, a data scientist turned software engineer.

Introduction

Hi there! 4 years ago, I got my first job as a data scientist (DS) and I talked about that experience on TDS. 2 years ago, I left my data scientist position to become a software engineer (SWE). Last month, I left my job as a machine learning engineer (MLE) at GoDaddy.

Fickle much? This is a story about how I switched into software engineering and left data science. It is also about the perspectives I got after having three distinct titles (DS, SWE, MLE) within a short span of 4 years.

For your information, below were my titles, team, tech stacks, and short role description:

  • Data Scientist (Marketing, Data Science): Python, Tensorflow, PySpark, SQL -> building a ton of ML models for predicting customer behaviors, building features, and conducting analysis.
  • Software Engineer (Data Platform): Java, Beam, Spark -> building and productionizing core data products for other teams.
  • Machine Learning Engineer (GoDaddy Machine Learning): Python, AWS -> building ML platform in AWS and deploying ML models to production.

As I am transitioning into a new company, I want to reflect on my experience and talk about the key takeaways in this article. What did I like and dislike about these roles? Do I regret these switches? If you ever wondered about switching, I hope these lessons can help you with your decision.

What I didn’t like in data science

The first few years as a data scientist were a wild ride. I learned how to build and deploy models, automate pipelines, analyze results, and more. Though there were many things I enjoyed about working as a data scientist, there were a few things that did bother me.

  • The messy, unorganized, and undocumented code/query: People usually don’t care too much about your code mostly because these codes aren’t usually going into production. It is also because many DS projects are solo projects. When it’s just you and the manager, you may not care as much about formatting and hardcoding variables. As long as the result is good, we may just move on because you are not going to revisit the same code.
  • The difficulty and inability to deploy machine learning models into production that faces customers: This will widely depend on your company’s internal tools. Companies like Facebook and Google have mature ML deployment tools that scientists can use to deploy models. In many other companies, a data scientist may need to figure it out on their own or depend on other software engineers to productionize them. As a data scientist, I felt limited in my own ability to own the entire pipeline and to deploy my own machine learning models.

Creating a customer facing ML model in a large company is not an easy task. You need to make sure the model has low latency, meets company coding standards, is reviewed and tested, and has on-call staff to deal with failures.

What I liked about software engineering

  • You get to build a product: It’s an awesome feeling when you build something and see others use it and benefit from it. It could be a data product, web application, or API. Companies like Lyft, Shopify, and Instagram are all examples of software products. A good product will last a long time and have a large impact in the world. Though an analysis is useful and can steer the direction of the company, it’s just used one time for a particular scenario. It’s a very different feeling when you actually build something.
  • Automation makes your life easier: Even though automations are not exclusive to software engineering, it’s not as common in data science. For example, there are tools like Jenkins that help you automatically build, unit test, and deploy your code (CICD) from Github. Using different tools, you can customize how you want to reformat and test the code. Virtually anything can be automated as long as it makes sense. Unlike the one-time analysis you do in data science, you revisit the same code many times in software engineering. I loved this aspect of SWE because I do not like to repeat the same task and love when things are efficient.
  • Collaboration through version control (e.g. Github) makes you and others better programmers: As a data scientist, I used Github mainly as a place to store and save my work. This was ok because no one else was really checking my code. But in software engineering projects with multiple engineers working on the same codebase, it is imperative you have a clear version controlling system. It was fun to see the codebase grow and evolve with everyone’s contribution. I also found it very constructive to provide and receive feedback through pull requests.

What I didn’t like about software engineering (as a data scientist)

  • On-call sucks, usually: If you are a data scientist, it would be rare to be on call. Being on-call is similar to how doctors are on call and how they need to be available during emergencies. Regardless, you will need to schedule your life around the on-call schedule and may get calls at 12am or 5am in the morning. It’s unavoidable because building and maintaining a software product is like fixing a car while it is driving. Because most software engineering products need to be available 24/7, on-calls are necessary.

  • A rigid scrum framework can be stifling if you are coming from data science: With some oversimplifications, scrum is part of an agile development process that uses two-week intervals to plan, track, and organize work, where the work is split into detailed ticket items. This style of planning requires a very rigid structure, detailed monitoring, and frequent meetings. It could be seen as an awesome organizational method or extreme micromanagement.

    As a data scientist, I was not used to this because data science problems are often vague. You might have an open-ended problem like “how do we increase metric Y in product X?” You may also have several weeks or months to solve this problem with some check-ins. Lots of uncertainties. So it’s harder to plan out the work weeks in advance like you can in software engineering.

  • Every detail matters: As a data scientist, you might be used to using tools and environments that are already set up and ready to go. But when you are building a product yourself, you need to set and maintain these environments yourself. This could be rotating clusters, setting up servers, making sure your code is compliant to the company’s security standards, and more. Not only that, you need to focus on little details, like formatting or choosing the right variable name (describe_var? var_describe? VAR? var?). Details are crucial. When you could have 10 people working on the same codebase, having a clean coding format and clear style is important.

What I wish I knew before switching

Even though I was excited, I was demoralized when I switched into SWE. As a data scientist, I was confident in my abilities and performance. I was good at what I did. But after I switched, I was a noob again, slower than everyone else. It was frustrating because I wanted to do well, but it was physically not possible as I had a lot of catching up to do.

Don’t think you can start performing from day one. It will take time. But be humble, open to feedback, and learn from your teammates, and be grateful for their help and time.

Secondly, because I switched into different roles and teams within a relatively short period of time, I didn’t get promoted as fast. Some of my peers who stayed in the same role and team were becoming managers or higher level individual contributors (ICs). If my goal was to get promoted as soon as possible and retire, I should have joined FANG as a DS and stayed there. Does this mean I regret my decision? No. It depends on what you are optimizing for.

Thirdly, there is only so much you can pick up and learn from work. You have to learn things on the side. Here are a few core concepts that are important for you to learn to get started:

Life of a SWE is a life of continuous learning and adapting to new technologies and tools.

Conclusion

What role should you choose? As you can guess, both have pros and cons. SWE focuses on building while DS focuses on discovering.

My experience as a MLE was that of a SWE working in ML space. It was 90% software engineering and 10% data science.

My final takeaway from this experience: there is no such thing as a perfect job! If you are unsure of which direction to choose, I suggest you use the famous regret minimization framework: will you have regretted more by trying or more by not trying on your deathbed?

When I switched, I didn’t know what to expect. But I knew that I would regret it if I didn’t try. Though it was painful at times, I am so glad I tried. Next week, I am actually starting a new role at Shopify! I wish you luck in your future endeavors and finding your way in this career jungle. Wish me luck too in my new role!

If you have any questions, please comment below or reach out to me here.