List: Data Engineering | Curated by Nathan Hanks

Dec 16, 2023
52 stories
Data EngineeringBuilding Datalakes, data platforms, ingestion
In
Ancestry Product & Technology
by
Thomas Cardenas
Scaling Ancestry.com: How to Optimize Updates for Iceberg Tables with 100 Billion RowsOne of the most interesting datasets at Ancestry is the Hints database. This is used to alert users that potential new information is…
Feb 23, 2023
3
Feb 23, 2023
3
Ben Rogojan
Mistakes I Have Seen When Data Teams Deploy AirflowWhat You Should Consider When Working With Airflow
Oct 29, 2023
2
Oct 29, 2023
2
Jan Kammerath
Relational Database Systems Are Becoming A Problem — But What To Do About It?I love SQL and relational databases, but I know that their days are numbered after I have seen what trouble they cause after decades.
Aug 15, 2023
54
Aug 15, 2023
54
In
AWS in Plain English
by
Anna Geller
Apache Iceberg Crash Course for AWS users: Amazon S3, Athena & AWS Glue ❤️ IcebergHow to turn AWS data lake into a data lakehouse using Iceberg, the open table format
Aug 4, 2023
2
Aug 4, 2023
2
In
Geek Culture
by
Chouaieb Nemri
Transactional Data Lakes — a Comparison of Apache Iceberg, Apache Hudi and Delta LakeUltimate guide to Iceberg, Hudi and Delta Lake. ACID guarantees, Partition and Schema evolution, Time-travel and Platform Compatibility
Jan 1, 2023
2
Jan 1, 2023
2
In
Adevinta Tech Blog
by
Adevinta
Practical tricks and tips to reduce AWS EMR costsFind out how to reduce the costs of leveraging EMR as a managed big data cluster platform to run Spark workloads
Nov 15, 2022
Nov 15, 2022
In
TDS Archive
by
Prukalpa
What I Got Wrong: Looking Back at My 2022 Predictions for the Modern Data StackWhere we started vs. where we are now in the data world
Dec 22, 2022
8
Dec 22, 2022
8
In
TDS Archive
by
Arli
Simple Parquet Tutorial and Best PracticesHands-on tutorial for starting your Parquet learning
Dec 16, 2022
7
Dec 16, 2022
7
In
TDS Archive
by
Nishakanthi Angulgamuwa
Working with Hive using AWS S3 and PythonBeginner’s guide to maintaining a Hive schema with external data storage and executing Hive queries using Python
Dec 30, 2019
1
Dec 30, 2019
1
In
AWS in Plain English
by
Kirity
Let's design a serverless ETL pipeline with AWS servicesIn this article, let’s try to design a scalable, serverless ETL pipeline with AWS services.
Oct 1, 2022
8
Oct 1, 2022
8
In
AWS in Plain English
by
Adriano Nicolucci
Most Common Data Architecture Patterns For Data Engineers To Know In AWSSome architectural examples of how we can process data in AWS depending on our data needs.
Oct 13, 2022
3
Oct 13, 2022
3
In
TDS Archive
by
Khuyen Tran
Create Robust Data Pipelines with Prefect, Docker and GitHubStore Your Workflow in GitHub and Execute it in a Docker Container
Sep 8, 2022
3
Sep 8, 2022
3
In
CodeX
by
incident.io
Modern data stack for startupsThis is how we built our data stack. If you’re an org that care about data access for all, follow this guide and we guarantee great result.
Apr 21, 2022
2
Apr 21, 2022
2
In
Databand, an IBM Company
by
Eitan Chazbani @ Databand
The Top Data Quality Metrics You Need to Know (With Examples)Data quality metrics can be a touchy subject, especially within the focus of data observability.
May 12, 2022
May 12, 2022
In
Provectus
by
German Osin
Finding the Right Data Catalog SolutionWhen organizations begin to adopt artificial intelligence, machine learning, and big data analytics, they realize that none of these…
Apr 6, 2022
5
Apr 6, 2022
5
In
Google Cloud - Community
by
Neil Kolban
GCP BigLake introductionBigLake is the name given by Google to an underlying data access engine used to provide access to data stored in either BigQuery or in…
Apr 15, 2022
1
Apr 15, 2022
1
 This story is no longer available
LEON DE SILVA
Step by Step Guide To Writing A Simple AWS Glue Job in PythonDescribes how to write an AWS Glue Job using python to load data from a S3 bucket to a postgreSQL database
Jul 3, 2021
4
Jul 3, 2021
4
In
TDS Archive
by
Prukalpa
The Metadata Foundation that Your Data Mesh NeedsHow activating your metadata can power the Data Mesh dream
Apr 22, 2022
Apr 22, 2022
In
NVIDIA Merlin
by
Even Oldridge
Recommender Systems, Not Just Recommender Modelsby Even Oldridge and Karl Byleen-Higley
Apr 14, 2022
5
Apr 14, 2022
5