This project focuses on analyzing the results of an A/B test conducted by Vanguard, a leading investment management company, to evaluate the impact of a new user interface (UI) design on client behavior and process completion rates. Our goal was to assess whether the new design improved client experience and identify areas for optimization. Key methods included exploratory data analysis (EDA), hypothesis testing, and experiment evaluation.
-
Data Exploration and Cleaning:
- Explored datasets to identify patterns, trends, and data quality issues.
- Cleaned data by addressing missing values, removing duplicates, and fixing inconsistencies.
-
Performance Metrics and KPIs:
- Completion Rates: Calculated the proportion of clients who completed the process (67.7% overall). Analyzed differences between Test (69.3%) and Control (65.6%) groups.
- Time Spent on Each Step: Measured the average time clients spent on each step, identifying Steps 2 and 3 as potential bottlenecks.
- Error Rates: Found that 29.95% of clients experienced at least one backward step, indicating areas of confusion in the process.
-
Behavior Analysis:
- Found that primary clients were older (average age: 46.18 years) and long-standing (average tenure: 12.05 years).
- Analyzed drop-off points, revealing a 32.3% abandonment rate and providing insights into process friction points.
-
Hypothesis Testing:
- Conducted statistical tests (e.g., Two-proportion Z-test) to confirm that the higher completion rate in the Test group was statistically significant.
- Performed ANOVA and Tukey’s HSD tests to evaluate differences in client balances across groups, identifying significant imbalances that may impact fairness.
-
Experiment Evaluation:
- Assessed experiment structure, randomization, and potential biases.
- Highlighted the importance of addressing initial imbalances in client demographics and balances for future experiments.
-
Visualization:
- Created Tableau dashboards to visualize results and support decision-making.
- Visualizations highlighted completion rates, step durations, and error rates across client groups.
- A comprehensive analysis report detailing findings, conclusions, and recommendations.
- Tableau dashboard visualizing key insights.
- Jupyter notebooks with code for data exploration, testing, and evaluation.
- A Results document gathering all the insights and a detailed step-by-step with all the analyses performed.
- A clear and concise README summarizing the project.
- A project presentation outlining main results and actionable insights.
- The Test group had a statistically significant higher completion rate than the Control group, indicating the new design's effectiveness.
- Steps 2 and 3 require optimization to reduce client effort and improve process efficiency.
- Addressing high error rates (29.95%) and imbalances in initial client demographics and balances is critical for future experiments.
- Further segmentation of clients and analysis of drop-off points can help enhance the user experience.