Data Quality Scorecard
Parameter definitions are available in Appendix
| Parameter | Yes | No | Comments |
|---|---|---|---|
| Duplicate rows > 10% ? | ✓ | 100% Unique Rows | |
| Missing values > 50% for one or more columns ? | ✓ | 0 Columns have missing values > 50% | |
| Most recent updates is before 6 months ago ? | ✓ | Latest Date: 15-Nov-2019 | |
| Data Contain PII (Personally identificable information)? | ✓ |
1. Metadata Summary
1.1. Product Structure Tree
Number of Files Received: 1
1.2. Data Dimensions
Number of Columns: 5
Number of Rows: 360,961
Date Range: 08-Jan-2014 to 15-Nov-2019
1.3. Structure/Data Types
A list of columns present in the dataset, the respective datatypes and the first value present.
1.4. Data Subset
Sample 6 rows of the dataset.
1.5. Data Completeness
Discrete Columns: Columns consisting of discrete/categorical values
Continuous Columns: Columns consisting of continuous values
All Missing Columns: Columns in which all values are missing
Complete Rows: Rows with no missing values
Missing Observations: Number of missing values
1.6. Missing Data
The plot shows the percentage of missing values for each column, color-coded on a spectrum from Green (0%) to Red (100%)
2. Univariate Analysis
The following sections provide a Univariate Analysis of the dataset, i.e. the dataset is being analyzed one column at a time.
2.1. Histogram and Statistical Summary - Continuous Variable(s)
Histogram is a representation of the distibution of numerical data. The positioning values are binned and plotted on the X-axis and the corresponding frequency is plotted on the Y-axis.
2.1.1 Column - Column 2
2.1.3： Column 4
2.2. Frequency Counts - Categorical Variable(s)
2.2.1. Column 1
Below are the top 10 and bottom 10 Column 1s by frequency:
| Column 1 | Frequency | Column 1 | Frequency |
|---|---|---|---|
| CABO | 2,138 | RCL | 2,132 |
| CNK | 2,138 | TGT | 2,132 |
| CPB | 2,138 | CAG | 2,133 |
| DISH | 2,138 | CHD | 2,133 |
| DRI | 2,138 | HRB | 2,133 |
| FWONK | 2,138 | KDP | 2,133 |
| GM | 2,138 | MHK | 2,133 |
| KSS | 2,138 | VIA | 2,133 |
| MIK | 2,138 | WMT | 2,133 |
| NCLH | 2,138 | BKNG | 2,134 |
3. Bivariate Analysis
The following sections provide a Bivariate Analysis of the dataset, i.e. the dataset is being analyzed two or more column at a time.
3.1. Correlation Analysis
The coefficient of correlation is a measure of the linear correlation between two variables X and Y. It has a value between +1 and -1, where 1 is total positive linear correlation, 0 is no linear correlation, and -1 is total negative linear correlation.
Following is a correlation heatmap for all numerical columns. It is colored using a Blue-White-Red spectrum in which the extreme colors represent -1 and +1 respectively.
4. Time Series Analysis
Periodicity: Daily
Below graph shows the trend of data collection over the period Jan-2014 to Nov-2019.
Appendix 1: Data Summary
1 Univariate Analysis
Univariate Analysis involves the analysis of one variable at a time.
1.1 Histogram
A Histogram visualizes the distribution of a numerical field over its continuous range of values. Each bar in a histogram represents the tabulated frequency at each interval/bin.
1.2 Density Plot
A Density Plot visualizes the distribution of data over a continuous interval or time period. This chart is a variation of a Histogram that uses kernel smoothing to plot values, allowing for smoother distributions by smoothing out the noise. The peaks of a Density Plot help display where values are concentrated over the interval.
1.3 QQ Plot
A QQ plot is a scatterplot created by plotting two sets of quantiles (theoretical and sample) against one another. The shape of the QQ plot indicates whether the data is normally distributed, skewed, or has a heavy tail.
2 Bivariate Analysis
Bivariate Analysis involves the analysis of two variables for the purpose of determining the empirical relationship between them. It explores the concept of the relationship between two variables, whether there exists an association and the strength of this association, or whether there are differences between two variables and the significance of these differences.
2.1 Correlation
A correlation matrix displays the coefficient of correlation for every pair of variables present in the dataset. This allows you to see which pairs have the highest correlation. It can be used to summarize data, as an input into a more advanced analysis, and as a diagnostic for advanced analyses.
3 Trend Charts
Trend charts are simple and efficient graphical representations of time-series data. Monthly trend charts can often reveal seasonal trends for a variable while yearly trend charts show trends over a longer period.
Appendix 2: Data Quality Scorecard
Tresvista’s Data Quality (DQ) framework is designed to assess the data quality and data health. Data quality monitoring is performed on an ongoing basis to ensure sustainable data quality.
Dimensions of Data Quality
A Data Quality Dimension is a term used to describe a data quality measure that can relate to multiple data elements including attribute, record, table, system or more abstract groupings such as business unit, company or product range. While there are multiple parameters on which a dataset can be assessed in terms of quality, we have identified the following 5 core dimensions for our assessment.
1. Availability – Sufficient availability of data points
It is ensuring that enough data is available to end users and applications, when and where they need it for further analysis. This is particularly important, as many of the machine learning algorithms require enough data samples for training and testing/validating the models.
2. Completeness – All required data is captured, and no data is missing
It is identifying the percentage of records with non-NULL values. It can also be termed as comprehensiveness. Missing or incomplete data can hamper the analysis and affect the interpretability of the insights.
3. Uniqueness – Data redundancy should be avoided
It points out that there should be no data duplicates reported. Asserting uniqueness of the entities within a data set implies that no entity exists more than once within the data set and that there is a key that can be used to uniquely access each entity (and only that specific entity) within the data set.
4. Recency – Data is recent and not outdated
It is the degree to which information is recent with the current period. It measures how “up-to-date” information is, and whether it is correct despite possible time-related changes.
5. Consistency – Uniform data types across the column
It refers to the data values in one column being consistent across the column. A strict definition of consistency specifies that two data values drawn from the same column must not conflict with each other (column level consistency).