How Solvd built an efficient data engineering pipeline for NerdWallet
Service
Data Engineering
Client
Nerdwallet
Industry
Financial Services
Platform
Cloud
NerdWallet is a prominent financial technology company that empowers individuals to make informed financial decisions.
The company was founded in 2009 and they offer a range of online tools and resources that help users manage their money effectively. NerdWallet’s user-friendly platform provides expert advice on credit cards, mortgages, personal loans, insurance, and investing.
The company has gained widespread recognition for its commitment to consumer advocacy.
Visit the website
2:12
What NerdWallet Said
Our partnership has been marked by a high level of professionalism, technical expertise, and dedication to achieving shared objectives. Solvd’s exceptional proficiency in advanced technologies has greatly enhanced our data-driven capabilities, enabling us to extract valuable insights and optimize our operations.
Vice President, Engineering Operations at NerdWallet
$538.9
million
Annual revenue
19
million
The number of unique monthly users
770+
The number of employees
Our Client’s Goals
Build a scalable and robust data infrastructure that can support NerdWallet’s growing data needs
Establish data governance policies and procedures for ensuring data quality, security, and compliance
Develop efficient data engineering pipelines for ingesting, processing, and transforming diverse data sources
Foster a culture of analytics-based decision-making
Continuously monitor and optimize data systems for performance issues
Collaborate with cross-functional teams in order to identify and address data engineering challenges and opportunities
Duration
Ongoing since 2021
Effort
22 Engineers
Challenges and Solutions
- Getting high-quality data
It was necessary to ensure that the gathered information was accurate, consistent, complete and reliable throughout its complete lifecycle. To deal with this challenge, Solvd implemented a comprehensive approach that includes data profiling and validation techniques that identify and rectify anomalies, missing values, and discrepancies. Our comprehensive approach also included data cleansing and standardization methods, robust quality monitoring processes, and continuous data governance practices.
- Ensuring smooth data integration
The integration of data from various sources that included different formats, structures, and systems was a complex task that required data transformation and consolidation. Solvd’s team of experts developed a robust ETL pipeline that can handle data extraction from various sources and also perform the necessary data transformations.
- Ensuring successful third-party tool implementation
This task wasn’t easy, because it was necessary to ensure a seamless integration with the existing systems and workflows and address all of the potential challenges such as compatibility issues, data security concerns and customization problems. Solvd’s experts helped NerdWallet migrate to new tools: from Redshift (AWS) to Snowflake and from Google Analytics to Segment. These migrations contributed to the remodeling of their data and tables. These migrations were instrumental in helping us create new processes, the development of new DAGs, and QA approaches.
- Managing large databases
This task was challenging since it was necessary to deal with a high volume and a wide variety of data. To best manage this problem, Solvd optimized storage issues, implemented efficient indexing strategies, and designed scalable data processing pipelines. This ensured timely data retrieval and reduced latency, helping out with the maintenance of the overall system’s performance.
Technologies Used
AWS Kinesis
AWS DynamoDB
AWS EC2
AWS S3
AWS Lambda
AWS Athena
AWS Glue Crawler
AWS Firehose
AWS Redshift
PostgresSQL
Oracle
Python
Node.js
Microsoft Sql
Docker
MaxMind
RESULTS
Improved Data Accuracy and Quality
- Enhanced data accuracy and quality by implementing data validation and cleansing techniques. Also, Solvd corrected the company’s important metrics: revenue, page views, clicks, monetizable clicks, and the number of unique users
- Reduced data errors and inconsistencies which lead to more reliable and trustworthy insights
- Minimized the risk of making poor decisions that were based previously on inaccurate or incomplete information
Enhanced Decision-Making
- Enabled analytics-based decision-making by providing accurate and timely information to stakeholders
- Empowered business users with access to meaningful insights and analytics-ready datasets
- Facilitated faster and more informed decision-making processes that lead to improved business outcomes
Better Return on Investment (ROI)
- Achieved a positive return on investment through improved operational efficiency and analytics-based decision-making
- Quantified cost savings and productivity gains that resulted from optimized data engineering processes
- Demonstrated the value of data engineering services for driving business growth and profitability
Working closely with Nerdwallet, we have built robust data engineering pipelines, implemented cutting-edge algorithms, and harnessed the power of big data to drive actionable insights. We are proud to have contributed to Nerdwallet’s success story and we look forward to empowering their business further with our advanced data engineering expertise.
Data Platform Engineer
