Intro
This project began as an attempt to put three separate interests in the same place: a hobbyist’s enthusiasm for fitness and wearables, a professional background in data engineering and data science, and a wish to spend more time in cloud infrastructure and web development. The result was a year of Extract, Transform and Load (ETL) operations run over millions of rows of smartwatch data drawn from three Garmin devices and an Apple Watch Series 8.
Three goals were set at the outset. The first was simply to build the thing end to end, since a pipeline that spans device, cloud and website exercises a wider range of skills than any one of those does alone. The second was analytical: the Garmin and Apple devices represent the best each manufacturer currently offers, and the differences in what they record seemed worth measuring rather than assuming. The third was to gain deeper practical experience of Amazon Web Services (AWS) and of serving the results from a Django site hosted on Google Cloud.
Background
The Apple Watch has been reported as carrying the most accurate optical heart rate sensor on the market (1, 2, 3), hence my interest in how closely the Garmin data would track it. The comparison is a fair one on paper: the Garmin Forerunner 945 LTE uses the fourth generation of Garmin’s Elevate optical heart rate monitor, while the Apple Watch Series 8 carries Apple’s third generation optical sensor (see here). Garmin has since released a fifth generation Elevate sensor on the Fenix 7 Pro in June 2023. Both generations are claimed to approach the non-invasive gold standard of a chest strap, which measures the heart’s electrical signal directly rather than inferring rate from optical absorption.
Over the last six years I have collected data from three Garmin watches: a Vivoactive 3 from 2017 to 2019, a Vivoactive 4 from 2019 to 2021, and a Forerunner 945 LTE from 2021 to the present. That amounts to six years of Garmin data against one year of Apple Watch data. With each device recording every few seconds, 24 hours a day, the totals accumulate quickly; for reference there are roughly 500,000 minutes in a year.
Since September 2022 I have worn both the Apple Watch and the Forerunner simultaneously, so that the two devices can be compared directly on the same wrist-days rather than across different periods. I did not opt for the Apple Watch Ultra, since the sensors that most people actually care about are identical between the Ultra and the Series 8.
Obtaining the data
Each manufacturer has its own quirks, and they differ most in where your data lives by default.
Garmin is not privacy-first in this respect; all watch data is uploaded to Garmin’s cloud infrastructure for analysis and then served back to the watch and to Garmin Connect for viewing. To retrieve my own six years of records I used GarminDB (thanks to Tom Goetz), which pulls the full history using Garmin Connect credentials and parses the FIT format into SQLite, from which CSV export is straightforward.
Apple takes the opposite approach, storing health data on the iPhone rather than on company servers, which makes extraction easier. The Health app exports the entire record as a single XML file, which can be AirDropped to a MacBook. I wrote a Python script to parse that file into one CSV per category, such as HeartRate.csv, Sleep.csv and Steps.csv. I subsequently wrote a custom iOS Shortcut that posts the health data as JSON to an API endpoint over HTTP; this worked well and would be the more viable option for daily or weekly dumps were the project ever to become something larger.
With the CSVs prepared, the data was ready for the cloud. I created two AWS S3 buckets via the CLI and uploaded roughly 100 MB.
Extract, transform, load
An AWS Lambda fires whenever new files land in the raw S3 buckets, triggering an AWS Glue job that performs the transformation in PySpark and writes the result to a set of clean buckets.
A second Lambda then starts a Glue Crawler to index the CSV structure, at which point the data is queryable through SQL in AWS Athena and can be dashboarded in AWS QuickSight. A parallel path exists for the analysis I wanted to publish rather than merely explore: a Lambda POSTs the data to my Django site on Google Cloud through a Django-Ninja endpoint, which parses it into Cloud SQL where the site picks it up and renders it via Django-Plotly-Dash.
The two-path design was deliberate. Athena and QuickSight are the right tools for ad hoc interrogation of six years of records, whereas the Django path serves the much smaller subset that belongs on a public page and needs to load quickly.
The insights
The pipeline is built and data flows through both paths, so the comparison that motivated the project now has a page of its own: Health Data Comparison plots the Apple and Garmin records against one another for a chosen metric, defaulting to average heart rate. It merges the two devices on date, hence it reports only the days on which both were actually worn, which is the fair basis for the comparison.