Data Lineage, Data Quality, Data Observability & Data Governance — I could remember during my early days in learning about Data engineering, I came across all these different data terms I would have no idea what these terms meant, later only after delving into data engineering projects at scale it would start to make more sense to me. The data…

Deep dive into the most interesting date functions with clear examples — Pyspark and Spark SQL provide many built-in functions. The functions such as the date and time functions are useful when you are working with DataFrame which stores date and time type values. Sometimes you may be tempted to write a UDF(user-defined functions) only later to realize that it would have…

Understand the fundamentals behind SQL, NoSQL, Relational Databases, and much more… — This article will go through some of the fundamental questions on Data modeling which are most commonly asked at interviews. It is highly rewarding mastering data modeling in your data science journey, it can be overwhelming for beginners at first but with practice, it can produce huge pay-offs. What is Data Modelling?

Testing the Pipeline on a use-case and learning how to make our life easier — When building and deploying Machine Learning models, it is best practice to try to make them possibly as end to end possible, meaning try to group most of the model related data transformations into one single object. In the ML world, the easiest way to adopt pipelines is by using…

Going through the most important functions and commands of this library on a single Dataset. — Pandas is one of the most popular python libraries used for data manipulation and analysis. It enables a variety of reading functions for a wide range of data formats, commands to best select the subset you want to analyze. It provides a variety of tools for data manipulation such as…

Interactivity and animation in a few lines of code using the GIF library — Animations are an engaging way of narrating a phenomenon. Any creature with intelligence has always been enchanted by animated and interactive visualizations. “A lot of the time in animation is spent getting the story right — that’s something you can’t rush.” — Jennifer Yuh Nelson This great quote from the…

All these Functions will be tested on a use case, helping you better your analysis and get better insights — What is Time Series Data? These type of data are sequentially ordered data over time and these observations are typically collected at regular intervals, this could be: Every Second/Minute/Hour Daily Monthly Quarterly/Yearly Some Real-life Examples are: Monthly Sales Data Stock Market prices

This post is geared towards aspirant data scientists and machine learning (ML) learners & practitioners. — This post is not meant for seasoned statisticians. This is geared towards data scientists and machine learning (ML) learners & practitioners, who like me, do not come from a statistical background. For a person being from a non-statistical background the most confusing aspect of statistics, are the fundamental statistical tests…

A Step by Step Analysis: From Business Understanding to Model Monitoring — In order to make a Data Science life cycle successful, it is important to understand each section well and distinguish all the different parts. …