Teaching my Apple Watch where my hand is

The question

Can one determine where a person is holding their wrist using nothing but the accelerometer and gyroscope in an Apple Watch Series 8? The candidate positions are a coat pocket, a jeans pocket, an arm hanging by the side, a forearm raised for texting, and running.

The answer is yes, and comfortably so: 0.995 window macro-F1 and 57 of 57 walks classified correctly under honest validation. However the most valuable finding of this project had nothing to do with modelling. It was a single mislabelled row in a spreadsheet.

This follows on from my earlier post on my year of Garmin versus Apple data. Same wrist, same watch, a considerably narrower question.

The data

Recordings were made with SensorLog on an Apple Watch Series 8, each session being one walk or run with my hand held in a fixed position throughout and labelled afterwards in a spreadsheet. Of 69 label rows, 3 were blank and 4 recorded as “don’t know”, leaving 60 usable sessions and 57 once the upstairs label was excluded. Quality was unusually good: a single consistent 58-column schema, zero missing values, no gaps exceeding 0.5 s, and a steady 10 Hz throughout except for two files at approximately 29.6 Hz.

Every session was linearly interpolated onto a uniform 10 Hz grid, which absorbs both the anomalous files and all timestamp jitter through a single code path, then cut into 5-second windows of 50 samples with 50% overlap. This yields 19,050 windows across 12 channels: raw accelerometer, user acceleration, gravity and rotation rate, each in three axes.

Why this is solvable

The gravity vector, expressed in the watch’s own reference frame, indicates the direction in which the wrist is pointing, and averaged over a session it functions as something close to a fingerprint. Texting sits alone at a gravity X of approximately zero, being the only position in which the forearm is horizontal and the screen faces upwards.

Animating the vector makes the point rather better than any static plot; trail length below is a direct measure of how much the wrist actually turns.

Texting is essentially a stationary dot and a coat pocket barely twitches, whereas an arm by the side sweeps a long arc once per stride. A freely swinging arm rotates the wrist; a pocketed one cannot. Cadence, by contrast, is of no use here. All four walking positions share a 1.8 Hz step rate and only by side carries a strong 1.0 Hz gravity component, corresponding to that one wrist rotation per stride.

Three traps

These mattered considerably more than the choice of model, and each is capable of producing an excellent number and a useless classifier.

Split by session, never by row. Consecutive rows are 0.1 s apart and near-identical, and each session carries its own tight orientation signature, hence a random row-level split permits the model to identify the session and simply look up its label. Holding everything else constant, a random row-level 5-fold scores 0.9934 against 0.9345 for an honest session-grouped one. The inflation is the lesser problem; the row-level split reports 0.993 for a model that gets 3 of 57 walks wrong, and would therefore have concealed the mislabel discussed below. All results here use StratifiedGroupKFold grouped by session, so the effective sample size is 57, not 19,050.

Drop the leaky columns. SensorLog supplies 58 columns and several are actively harmful. location* memorises the route, which in turn identifies the session; activity* is Apple’s own on-device classifier, so using it borrows a label rather than predicting one; motionYaw and motionQuaternion* are expressed in the XArbitraryZVertical frame, so heading is session-arbitrary and acts as a session identifier. The subtle one is pedometerCurrentCadence, which is exactly 0.00 for every August and September session and roughly 1.7 to 1.9 for every January and February one. That is a recording-era fingerprint rather than a gait measurement, and era correlates with class, since walk_coat is 100% Jan/Feb and run is 8 of 9 Aug/Sep. A tree splits on it immediately and scores well for entirely the wrong reason.

Some classes cannot be classes. I also recorded an upstairs label amounting to three sessions of roughly 12 seconds each, 37 seconds in total. There exists no session-grouped fold in which it is meaningfully trained and tested, hence its exclusion.

Models and results

Three families were evaluated on identical folds: 257 hand-crafted features per window fed into a HistGradientBoostingClassifier; a 1D CNN on raw windows, as both a compact 4-layer network and InceptionTime; and harnet5, a ResNet-18 encoder self-supervised on over 700,000 person-days of UK Biobank wrist accelerometer data, run frozen and fine-tuned.

The four working models are statistically indistinguishable. Exact McNemar on session-level errors returns p = 1.000 for GBT against CNN against InceptionTime; they do not merely tie, they fail on precisely the same three sessions. A feature ablation confirms why the additional capacity buys nothing, with gravity features alone scoring 0.9327 and spectral features alone 0.9368 against 0.9345 for all 257. The signal is sufficiently redundant that any reasonable subset saturates. The GBT is thus the model to use, winning on 40 seconds of laptop CPU rather than 10 to 25 minutes of GPU, interpretable feature importances, and nothing to ship but a small scikit-learn model.

The frozen harnet5 probe is the interesting failure, collapsing to 0.23 and predicting walk_jeans for very nearly everything. This is informative rather than broken. The UK Biobank encoder was pretrained on Axivity wrist sensors whose mounting orientation is arbitrary, with objectives describing motion dynamics, which likely makes its features close to orientation-invariant. Static wrist orientation is precisely the signal that distinguishes hand position, so the pretrained representation appears to discard the one thing required here. Pretraining on 700,000 person-days guarantees nothing if the objective was invariant to your signal.

The label finding

Only 3 of 57 sessions were misclassified and all three were walk_coat against walk_side.

Every other coat session is called coat by 99 to 100% of its windows. Furthermore the first row was flagged from raw gravity statistics (gX −0.90, gY 0.36, a textbook walk_side signature) before any model was trained, and all five models fail on it.

Retraining with that one label corrected takes the GBT from 0.9345 to 0.9950 and from 54/57 to 57/57 sessions, the CNN behaving near-identically at 0.9947. Critically, the other two sessions were left untouched and flipped from confidently wrong to confidently right of their own accord. One walk_side session sitting inside the walk_coat class was contaminating the model’s notion of a coat and dragging two genuine walk_side sessions across the boundary. Flipping all three is worse, at 0.9658 and 56/57, because two of those flips teach the model something false. One caveat is worth stating: StratifiedGroupKFold stratifies on labels, so altering them reshuffles fold membership and a small part of the improvement is reshuffling rather than signal.

The lesson I take is not simply that labels should be clean, which everyone already says. It is that a class holding only seven sessions has no defence against one bad one, and that the honest validation scheme is what allowed the problem to surface at all. The row-level split would have reported 0.993 and the mislabel would still be sitting there.

Onto the watch

A model that lives only in a notebook is a hobby, so the classifier was exported to Core ML and wrapped in a small watchOS app. The shipped model is the CNN rather than the GBT, since coremltools provides no converter for HistGradientBoostingClassifier and reimplementing 257 features including FFTs in Swift is a large and error-prone undertaking. It takes 9 channels rather than 12, because raw accelerometer arrives on a separate watchOS callback with separate timestamps and is not reconstructable from device motion; dropping it costs nothing measurable, the 9-channel model scoring 0.9949 macro-F1 across 57/57 sessions, and the watch then requires one time-aligned callback instead of two unsynchronised ones.

The app also starts a HealthKit workout session. Three of the five classes have the wrist down or pocketed, so the screen turns off, and a plain foreground watchOS app suspends and ceases to receive motion data. Without the workout session the demonstration would only ever work for texting. This is not the sort of problem one finds before attempting to run the thing on a wrist.

Conclusions

The modelling is finished. At approximately 99.5% on clean labels there is nothing further to win on this dataset; bigger architectures, longer training and additional feature engineering all chase a ceiling that label quality has already set. The remaining questions concern robustness, and of these the most pressing is generalisation across wearers, since this is one person, one watch, one wrist and one orientation, and the gravity signature carrying most of the signal will rotate or flip for a left-wrist wearer. Beyond that, all seven walk_coat sessions fall in January and February, so cross-validation cannot detect a seasonal gap when there is no summer coat data to hold out.

The headline figure is 0.995. The finding I actually value is that one incorrect cell in a spreadsheet cost more accuracy than the entire gap between a gradient-boosted tree and a self-supervised ResNet trained on 700,000 person-days of wrist data.