The Success Story of Microsoft’s Senior Data Scientist

Sakshi Khanna 09 Jul, 2023 • 9 min read

Introduction

In today’s digital era, the power of data is undeniable, and those who possess the skills to harness its potential are leading the charge in shaping the future of technology. Among these trailblazers stands an exceptional individual, Mr. Nirmal, a visionary in the realm of data science, who has risen to become a driving force at one of the world’s foremost technology giants, working as Microsoft’s Senior Data Scientist.

Meet Mr. Nirmal, the embodiment of perseverance, brilliance, and unwavering dedication. From humble beginnings, Mr. Nirmal embarked on a transformative journey that led them to the pinnacle of their career as a Senior Data Scientist at Microsoft. His meteoric rise serves as an inspiring success story, not only for aspiring data scientists but for anyone with a dream and the determination to achieve greatness.

In this success story article, we delve deep into Mr. Nirmal’s journey, tracing the key milestones, challenges, and triumphs that have shaped their extraordinary career. We explore the groundbreaking projects he has led, the transformative impact he made, and the invaluable lessons he learned along the way. Through Mr. Nirmal’s story, we discover the traits and mindset necessary to thrive in the ever-evolving world of data science.

Microsoft's Senior Data Scientist | Data Science

Let’s Begin with the Conversation!

AV: Please highlight your career trajectory, educational background, and how did it help you get your first data scientist job?

Mr. Nirmal: My career trajectory has never been a linear path. All of us have our own stories, and I am sure they all are interesting. Here is mine: I completed my Undergrad in IT Engineering from Nepal. I moved to the United States in 2007 for my Masters Degree. After completing my Master’s, I joined the US Army. Yes, it sounds very uncommon. Because of the great recession in the US around 2009 (which also happened to be my graduation year), the job market was very bad, especially for international students. There was a special pilot program run by the US Army, and I went through all the required processes to become a service member. Growing up, I had some passion to join the military. What a way to fulfill that.

While I was in the military, I completed my MBA. In 2014, after my first enlistment contract was completed, I left the US Army. In the same year, I got my first data role as a Cyber Security Analyst, working as a US federal government employee for the Department of Navy. I completed my 3rd Masters in Data Science while I was working on this job. After gaining some experience working as a Data Analyst, and building the academic credentials plus skills on Data Science, I transitioned to the private industry taking my first role as a Data Scientist title for Wells Fargo Bank in 2018. Since then I have been in data science, and currently working as Senior Data Scientist for Microsoft.

AV: Can you tell us about a project you worked on where you had to use data to solve a real-world problem and the impact it had on the business or product strategy?

Mr. Nirmal: There are many examples. First of all, we do not have to hold a ‘Data Scientist’ title to work and solve any data problems. There are some misconceptions like that. We can be working as Data Analysts, Data Engineers, Business Analysts or any titles working with data.

I mostly work in the cyber security domain. Two of the major focus areas for us are: investigation and detection. When dealing with cyber security problems, one of the very popular problem spaces is anomaly detection. I have worked in a data science team to build anomaly detention systems, helping the security analysts save time on what events/alerts to focus on. The impact is on saving their time and resources.

AV: What was the most challenging problem you have solved using data science? How did you approach the problem? What was the outcome?

Mr. Nirmal: I would say – the most challenging problem for me is yet to be solved. As we live in the world of highly innovative AI, we should always be aware that adversaries now have the most advanced tools than ever. However if I have to mention one interesting problem then I would pick the user behavior analysis or also called user entity behavior analysis , widely known as UEBA in the industry. UEBA is a type of cybersecurity feature that discovers the threats by identifying user activity that deviates from a normal baseline.

One simple example: We have a user who often logins from location A, and all of sudden we see login activity from location B. This could be normal related to travel, but it is still deviation from the normal behavior so must be looked at to confirm normality vs. maliciousness. The most challenging part of UEBA is to understand and create the baseline.

Data-driven Insights

Data Science vs Machine Learning | Microsoft's Senior Data Scientist | Data science

Mr. Nirmal: As a data scientist, we will come across multiple scenarios like these. Most of the business stakeholders are well versed with their problem and intended solutions. However sometimes it is hard to explain to them why some solutions make sense and why some don’t. I can share one example. We built a fraud detection model, it was a binary classifier with fraud vs. non fraud transactions. The fraud analysts know their domain well. But for us to explain the model results back to them was challenging to break it down into their language.

If we share details like – model tuning and hyper parameters or cross validation or sampling methods, these things will make less sense to them. However if we interpret into higher levels like what attributes we found useful based on the feature ranking, what are some challenges with classes being imbalanced, those things will make sense to them. Therefore it is always important for a data scientist to talk in business language as well.

AV: How do you ensure that the machine learning models your team builds are explainable and transparent to the end-users, particularly in the context of security and threat detection?

Mr. Nirmal: Like I mentioned in a previous example, model interoperability is very important when it comes to explaining it back to the business partners. This is important regardless of which domain you are working. In security and threat detection, it becomes more important because anything we build as a model, shall be explainable to the threat analysts so they can take appropriate actions. One good example that I can share here is the concept of Benign Positive. When I first heard about this term, I was a little confused, as I was only aware of true positives, and false positives. But in the security domain, benign positives are important. Here is the breakdown of those categories:

True positive (TP): A malicious action detected by a security tool.
Benign true positive (B-TP): An action detected by a security tool that is real, but not malicious, such as a penetration test or known activity generated by an approved application.
False positive (FP): A false alarm, meaning the activity didn’t happen.

AV: Have you ever encountered a situation where the data you were working with was messy or incomplete? How did you handle it, and what was the outcome?

Mr. Nirmal: This happens all the time. If a data scientist says he/ she got clean data to work with, then that will be like a lottery ticket winning for him/her. Real world projects are not like the Kaggle competition where data comes mostly clean as csv files. We spend more time on data needs, working with data owners for data contract, data collection. These are the things that come even before the exploratory data analysis (EDA) happens.

Most of the time, we encounter messy data with some discrepancies with schema. Data versioning is important, where we keep track of each version of data when we iterate multiple times to orchestrate the ETL pipeline until we get the right data. There is a concept of data observability which means exactly the same as I mentioned here. It deals with getting the right data to the right destinations, in the right formats, at the right time.

AV: Can you tell us about a project where you collaborated with a team to achieve a common goal? How did you contribute to the team’s success? What did you learn from the experience?

Mr. Nirmal: In Microsoft, we follow something called ‘One Microsoft’, which focuses on developing services and products that will embrace the culture of collaboration across the teams to innovate novel concepts and work on it together , rather than working in siloed ways. Almost all the projects that I have worked on are in collaboration with other teams- which could be engineering counterparts, or external teams. One good thing about Microsoft’s culture is- they make us focus on building systems on top of existing services, rather than re-inventing the wheels. This not only promotes building relationships with other teams, but also saves time and resources for the company. Personally I have learned many things working with different teams.

Data Security Projects

Mr. Nirmal: This is a great question. Thank you for bringing it up. Since data is everywhere, data science becomes applicable for all domains. I usually suggest the early career data scientists to try multiple paths, atleast have three domains of interest so you can do trial and error, just like training machine learning models, career path selection is an iterative process in the beginning of your career. Security and data science is one of the rare and unique combinations. The job market is in demand, and in the harsh economy, job security is also stronger in this domain.

To share my story, one of the best things for me being in security is that it is a constantly evolving field. Hackers are coming up with new strategies and tools, and we have to respond to that in no time. One of the simple and yet helpful projects from a business standpoint, that I was part of is – Alerts Classification. As the security researchers find various attack patterns, they help security engineers write detection rules, which in turn fires alerts if there is a match or hit with the rules. However the problem is that every system generates thousands of events which are converted to alerts. The false positive rate on those alerts are high.

To balance security and efficiency, we developed an ML model to categorize alerts into true positives, benign positives, and false positives, ranked by risk scores. This allows analysts to prioritize their queues and avoid overwhelming volumes of alerts while minimizing the risk of adversaries slipping through undetected.

Advice on Handling Unexpected Insights

AV: Have you ever encountered a situation where the data showed unexpected or surprising insights? What is your suggestion on dealing with those scenarios?

Mr. Nirmal: One of the things that we tend to miss during the exploratory data analysis (EDA) phase is that- we might not be asking the right questions to data. If we only follow the standard process of doing descriptive stats, uni- or multi variate analysis, correlation heat maps etc, which are basic steps of EDA, chances are we might miss finding key insights.

One example: The most common process to follow when we encounter outliers in our data is to drop them, because they will skew the distribution. However, dropping them is not always a good idea, and it depends on your project. What if we are doing an anomaly detection project, then the outliers can be those anomalies that we are trying to find. In this case dropping from the training data is not a wise decision. It’s always better to check with the domain experts before dropping any kind of data, even the missing data.

Advice to Become Successful Data Scientist

AV: What advice would you give to someone who wants to become a successful data scientist at a tech giant like Microsoft?

Mr. Nirmal: My suggestions are not only limited to Microsoft but apply in general to every industry and company. If I have to summarize in few points:

Stay Hungry for Learning New Things: The data science industry is always moving at a fast pace. Continuous learning is very important in this field.
Build your Network: Attend conferences, be part of community channels in linkedin, contribute to community by writing articles in popular data science platforms like medium, or towards data science. Networking helps a lot.
Focus on Impactful Projects: The data scientist title can put you in many responsibilities- some doing data engineering work, some doing data analyst work. Regardless, I suggest you focus on high impact projects where you can make your contributions more visible, and can be measured in tangible outcomes.

Conclusion

In closing, Mr. Nirmal’s success story serves as a shining example of the incredible heights that can be achieved when talent, opportunity, and unwavering dedication converge. Microsoft’s Senior Data Scientist has proven that the power of data, when harnessed with brilliance and purpose, has the potential to transform industries, shape the future, and create a legacy that will endure for generations to come.

Finally, I would like to thank Analytics Vidhya for giving me this opportunity to share my experience. To all my audience, please feel free to connect with me on LinkedIn.