The Mathematics Behind the Mirror

Crop tight to a face and you throw away half the story. Bad light, an odd angle, a low-res frame: any of them can fake a few years in either direction.

First the model finds you. It draws a box around each face and the body it belongs to, matches the two up, and ignores everyone else in the shot. Each crop gets resized without squashing and normalised (rescaled to the range the network expects). Then it is chopped into a fine grid of patches, dense enough to catch the things that actually age a face: fine lines, the way light scatters in skin. Two streams, face and body, trade notes through attention (the network deciding which patches matter most). When your face is blurry, the body quietly carries more weight. Out the end comes one number: your age.

The heartbeat signal is tiny. A fraction of a percent change in skin colour, sitting under camera noise, auto-brightness drift, and every little wobble of your head.

So we go hunting for it. A 478-point face mesh (a grid the model lays over your face) tracks your forehead and cheeks frame by frame. Inside those patches we keep only skin, then average thousands of pixels at once. That averaging is the whole trick: noise is random, so it cancels out, while the pulse adds up. We also watch a quiet patch of background and subtract its lighting changes, so a flickering lamp never gets mistaken for your heart. What is left is a clean colour trace per channel, the raw material for everything that follows.

Red, green and blue all carry the pulse, but they also carry glare and motion. Pick any single channel and you get the heartbeat tangled up with junk.

The fix is to stop trusting any single colour. We tilt the signal onto a plane that is lined up with skin tone (the POS projection, short for plane-orthogonal-to-skin). On that plane the brightness junk from lights and movement mostly cancels, and the pulse survives. The clever part: the exact tilt is set on the fly from the signal itself, so it keeps adjusting as your light and your movement change.

Here is a trap. A normal filter cleans up the signal but nudges every peak slightly later in time. And peak timing is the one thing we cannot move, because heart-rate variability lives in the gaps between beats.

So we filter twice. We run a band-pass (it keeps the heart-rate frequencies and throws away the rest) forwards through the signal, then run it again backwards. The two passes cancel each other's time shift, so the peaks land exactly where your heart put them. This forwards-and-backwards trick is called zero-phase filtering. We also strip out the slow drift first, so a room that gradually brightens does not look like a real trend.

So what is your heart rate, really? A few seconds of noisy signal can hide the true beat under echoes: harmonics at double the rate, ghosts at half the rate, and a smear of camera noise.

To find it we look at the signal in the frequency domain (which frequencies are strongest) using Welch's method, which averages several overlapping windows so one noisy patch cannot fool us. We flatten the camera's background hiss, double-check we have not locked onto a harmonic or a half-rate echo, and then sharpen the winning peak to below the spacing of the bins. We nudge gently toward rates that make biological sense, but we never hard-block anything.

Read the heart rate one frame at a time and it jumps around. The loudest frequency keeps flickering between a few candidates, so a naive reading looks like static.

So we stop reading frame by frame and read the whole story at once. We treat the rate as something that moves smoothly and is free to drift, but pays a steep price to jump. Then an algorithm called Viterbi finds the single most likely path through all the frames together (the same idea that decodes a noisy phone signal). It leans on a short delay so it can glance slightly ahead before committing. The result is a heart rate that tracks you, instead of twitching.

The interesting part of HRV is measured in milliseconds. That is finer than the gap between two camera frames, so straight off the webcam we cannot see it yet.

We get there by upsampling. A smooth curve (a cubic spline) is drawn between the frames, so we can time a beat to within a few milliseconds. We rescale the wave so a fading exposure cannot bias things, then find each beat by matching it against an average beat built from your own recording. Intervals that look off get flagged, gaps stay honest, and a clearly missed beat is rebuilt before we read the timing. Only then do we measure the gaps between beats.

Your heart never beats like a metronome, and that is good news. The tiny wobble in the gaps between beats is your nervous system at work, and that is what HRV captures.

From the cleaned beat-to-beat intervals we work out the usual HRV family. RMSSD and its partner SD1 track the calming, parasympathetic side (rest and recovery). SDNN sums up the overall swing. pNN50 counts the bigger jumps, and a stress index borrowed from Russian space medicine (the Baevsky index) reads how rigid your rhythm is. We stabilise each number across several sub-windows, so a noisy scan tells you it is unsure instead of handing you a confident wrong answer.

Stiff arteries do not change how fast you beat. They change the shape of each pulse wave, in ways far too subtle to spot by eye.

To see shape, we take the second derivative of the pulse (basically its acceleration, which exaggerates the bends in the wave). That turns each beat into a row of little landmark bumps, and the sizes of those bumps shift as arteries stiffen. We average many beats together for a clean read, then pull out a vascular ageing index, a reflection index, and how wide the pulse runs. These are relative scores, not a clinical stiffness number.

Your breathing hides in the pulse too, in three faint ways at once. On its own, each one is shaky.

Breathing slowly drifts the baseline up and down, it speeds up and slows your heart (that is respiratory sinus arrhythmia, a normal healthy thing), and it nudges the height of each beat. We estimate all three separately and only report a breathing rate when they agree. Because the beats are unevenly spaced in time, we use a method built for ragged data (the Lomb-Scargle periodogram) instead of a plain Fourier transform.

Any one of these methods can be confidently, completely wrong. So no single number gets to speak for itself.

Every value you see has been cross-checked. We work out the heart rate four different ways: from the spectrum, from the beat intervals, from the Viterbi track, and from chopped-up sub-windows. A number only ships when those agree and the reading has settled. When they argue, the scan keeps going, trusts the weak sources less, or simply declines to answer. Knowing when to stay quiet is the most important thing the whole pipeline does.