Photo by Simon Migaj on Unsplash

In this article, I’ll explain how you can develop a simple app with Flutter that measures heart rate variability and displays it in a chart using only the phone’s camera and flash.

Screenshot of the application.

. . .

Concept

You’ve probably seen or know of devices that people clip to their fingers in hospitals that measure their heart rate, or smartwatches capable of measuring your heart rate. They all have one thing in common: They measure the heart rate with a technique called photoplethysmography.

A photoplethysmogram (PPG) is an optically obtained plethysmogram that can be used to detect blood volume changes in the microvascular bed of tissue. — Wikipedia

Shining a light into a blood irrigated tissue, we can measure the variability of reflected light and extract the variation of blood flow. As we all know, the blood flow is dependent on the heart rate, so we can calculate the heart rate using the blood flow variation.

Vandenberk, et. al. (2017). Clinical Validation of Heart Rate Apps: Mixed-Methods Evaluation Study. JMIR Mhealth Uhealth. 5. e129. 10.2196/mhealth.7254.

So, in our application, we’ll shine the camera’s flash and measure the intensity reflected using the phone’s camera. More specifically, we’ll measure the average value of all the pixel’s intensity of the camera image. Then, if we cover the camera and flash with our finger, the intensity measured will vary with the blood flow.

. . .

Code

Dependencies

First, we need to install the dependencies:

  • charts_flutter — Material Design data visualization library written natively in Dart.
  • wakelock — This Flutter plugin allows you to enable and toggle the screen wakelock on Android and iOS, which prevents the screen from turning off automatically.
  • camera — A Flutter plugin for iOS and Android allowing access to the device cameras.
...

dependencies:
  flutter:
    sdk: flutter
  cupertino_icons: ^0.1.2
  charts_flutter: ^0.9.0
  wakelock: ^0.1.4+1
  camera: ^0.7.0+4
  
...

Application

Our application’s interface is divided into three files: main.dart, homePage.dart, and chart.dart.

main.dart - Here we only need to set the HomePage widget as our home widget, so it displays when the application runs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
import 'package:flutter/material.dart';
import 'package:flutter/services.dart';
import 'homePage.dart';

void main() => runApp(MyApp());

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      title: 'PPG',
      theme: ThemeData(
        brightness: Brightness.light,
      ),
      home: HomePage(),
    );
  }
}

HomePage - The core of the application is written on the homePage.dart file, which is our HomePage widget.

First, we need to create a Scaffold and, in its body, insert a centered IconButton, that will activate or deactivate the camera for the reading process.

import 'package:camera/camera.dart';
import 'package:flutter/material.dart';
import 'package:wakelock/wakelock.dart';
import 'chart.dart';

class HomePage extends StatefulWidget {
  @override
  HomePageView createState() {
    return HomePageView();
  }
}

class HomePageView extends State<HomePage> {
  bool _toggled = false;
  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: Colors.white,
      body: SafeArea(
        child: Center(
          child: IconButton(
            icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
            color: Colors.red,
            iconSize: 128,
            onPressed: () {
              if (_toggled) {
                _untoggle();
              } else {
                _toggle();
              }
            },
          ),
        ),
      ),
    );
  }
}

As one can see, there is a boolean _toggled which stores the state of the IconButton. Now, we only need to define the functions _toggle and _untoggle.

For now, these functions will only change the value of the _toggle variable:

_toggle() {
  setState(() {
    _toggled = true;
  });
}

_untoggle() {
  setState(() {
    _toggled = false;
  });
}


Screenshot of the untoggled and toggled iconButton, respectively.

The next step, is dividing the screen into three equal parts, using three Expanded widgets inside a Column:

@override
Widget build(BuildContext context) {
  return Scaffold(
    backgroundColor: Colors.white,
    body: SafeArea(
      child: Column(
        children: <Widget>[
          Expanded(
            child: Container(
              color: Colors.red,
            ),
          ),
          Expanded(
            child: Center(
              child: IconButton(
                icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
                color: Colors.red,
                iconSize: 128,
                onPressed: () {
                  if (_toggled) {
                    _untoggle();
                  } else {
                    _toggle();
                  }
                },
              ),
            ),
          ),
          Expanded(
            child: Container(
              color: Colors.black,
            ),
          ),
        ],
      ),
    ),
  );
}
Screen divided into 3 equal parts.

On the bottom Container we display the real-time chart, where the camera’s data will be displayed. A margin and round corners were also added to this Container.

import 'package:camera/camera.dart';
import 'package:flutter/material.dart';
import 'package:wakelock/wakelock.dart';
import 'chart.dart';

class HomePage extends StatefulWidget {
  @override
  HomePageView createState() {
    return HomePageView();
  }
}

class HomePageView extends State<HomePage> {
  bool _toggled = false;
  List<SensorValue> _data = [];

  ...

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: Colors.white,
      body: SafeArea(
        child: Column(
          children: <Widget>[
            Expanded(
              child: Container(
                color: Colors.red,
              ),
            ),
            Expanded(
              child: Center(
                child: IconButton(
                  icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
                  color: Colors.red,
                  iconSize: 128,
                  onPressed: () {
                    if (_toggled) {
                      _untoggle();
                    } else {
                      _toggle();
                    }
                  },
                ),
              ),
            ),
            Expanded(
              child: Container(
                margin: EdgeInsets.all(12),
                decoration: BoxDecoration(
                    borderRadius: BorderRadius.all(
                      Radius.circular(18),
                    ),
                    color: Colors.black),
                child: Chart(_data),
              ),
            ),
          ],
        ),
      ),
    );
  }
}
Screenshot of the chart container.

For the chart widget, we will use the package charts_flutter. The file chart.dart contains a StatelessWidget that displays in a chart points provided in a list. Each point is constituted by a DateTime value, which indicates the x value, and a double value that represents the y value.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
import 'package:charts_flutter/flutter.dart' as charts;
import 'package:flutter/material.dart';

class Chart extends StatelessWidget {
  final List<SensorValue> _data;

  Chart(this._data);

  @override
  Widget build(BuildContext context) {
    return new charts.TimeSeriesChart([
      charts.Series<SensorValue, DateTime>(
        id: 'Values',
        colorFn: (_, __) => charts.MaterialPalette.green.shadeDefault,
        domainFn: (SensorValue values, _) => values.time,
        measureFn: (SensorValue values, _) => values.value,
        data: _data,
      )
    ],
        animate: false,
        primaryMeasureAxis: charts.NumericAxisSpec(
          tickProviderSpec:
              charts.BasicNumericTickProviderSpec(zeroBound: false),
          renderSpec: charts.NoneRenderSpec(),
        ),
        domainAxis: new charts.DateTimeAxisSpec(
            renderSpec: new charts.NoneRenderSpec()));
  }
}

class SensorValue {
  final DateTime time;
  final double value;

  SensorValue(this.time, this.value);
}

Regarding the upper Container, we wish to divide it into two halves. On the left one, will display the CameraPreview and, on the right half a Text containing the Beats Per Minute (BPM).

class HomePageView extends State<HomePage> {
  bool _toggled = false;
  List<SensorValue> _data = [];
  CameraController _controller;

  ...

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: Colors.white,
      body: SafeArea(
        child: Column(
          children: <Widget>[
            Expanded(
              child: Row(
                children: <Widget>[
                  Expanded(
                    child: Center(
                      child: _controller == null
                          ? Container()
                          : CameraPreview(_controller),
                    ),
                  ),
                  Expanded(
                    child: Center(
                      child: Text("BPM"),
                    ),
                  ),
                ],
              ),
            ),
            Expanded(
              child: Center(
                child: IconButton(
                  icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
                  color: Colors.red,
                  iconSize: 128,
                  onPressed: () {
                    if (_toggled) {
                      _untoggle();
                    } else {
                      _toggle();
                    }
                  },
                ),
              ),
            ),
            Expanded(
              child: Container(
                margin: EdgeInsets.all(12),
                decoration: BoxDecoration(
                    borderRadius: BorderRadius.all(
                      Radius.circular(18),
                    ),
                    color: Colors.black),
                child: Chart(_data),
              ),
            ),
          ],
        ),
      ),
    );
  }
}

The CameraPreview widget requires a valid CameraController. So we need to initialize the controller once we press the heart button and it’s toggled. Once the button is untoggled, we must also dispose of the controller:

Future<void> _initController() async {
  try {
    List _cameras = await availableCameras();
    _controller = CameraController(_cameras.first, ResolutionPreset.low);
    await _controller.initialize();
  } catch (Exception) {
    print(Exception);
  }
}

_disposeController() {
  _controller.dispose();
  _controller = null;
}

Now, we only need to integrate these functions in the functions that toggle the button.

_toggle() {
  _initController().then((onValue) {
    setState(() {
      _toggled = true;
    });
  });
}

_untoggle() {
  _disposeController();
  setState(() {
    _toggled = false;
  });
}

We should also override the dispose method and dispose of the CameraController once the application is disposed.

@override
void dispose() { 
  _disposeController();
  super.dispose();
}

We must not forget to activate the camera’s flash and then start the ImageStream, which will provide the images that we will process. The function _scanImage will take care of it.

Future<void> _initController() async {
  try {
    List _cameras = await availableCameras();
    _controller = CameraController(_cameras.first, ResolutionPreset.low);
    await _controller.initialize();
    Future.delayed(Duration(milliseconds: 100)).then((onValue) {
        _controller.setFlashMode(FlashMode.torch);
    });
    _controller.startImageStream((CameraImage image) {
      _scanImage(image);
    });
  } catch (Exception) {
    print(Exception);
  }
}

The function _scanImage calculates the average of the camera image’s red channel and adds the value to the data list, which is displayed on the chart explained above. Notice that we limit the number of points of the data list to 50 values.

_scanImage(CameraImage image) {
  double _avg =
      image.planes.first.bytes.reduce((value, element) => value + element) /
          image.planes.first.bytes.length;
  if (_data.length >= 50) {
    _data.removeAt(0);
  }
  setState(() {
    _data.add(SensorValue(DateTime.now(), _avg));
  });
}

We do not need to process every frame, therefore, we can select a sampling rate. In this example, a sampling rate of 30 samples/seconds is used. For that purpose, we have a boolean, _processing, which becomes true once the _scanImage function is called and stays that way for 1/30 seconds, then returns to false. The _scanImage function will only be executed if the boolean _processing is false.

Future<void> _initController() async {
  try {
    List _cameras = await availableCameras();
    _controller = CameraController(_cameras.first, ResolutionPreset.low);
    await _controller.initialize();
    Future.delayed(Duration(milliseconds: 100)).then((onValue) {
        _controller.setFlashMode(FlashMode.torch);
    });
    _controller.startImageStream((CameraImage image) {
      if (!_processing) {
        setState(() {
          _processing = true;
        });
        _scanImage(image);
      }
    });
  } catch (Exception) {
    print(Exception);
  }
}

_scanImage(CameraImage image) {
  double _avg =
      image.planes.first.bytes.reduce((value, element) => value + element) /
          image.planes.first.bytes.length;
  if (_data.length >= 50) {
    _data.removeAt(0);
  }
  setState(() {
    _data.add(SensorValue(DateTime.now(), _avg));
  });
  Future.delayed(Duration(milliseconds: 1000 ~/ 30)).then((onValue) {
    setState(() {
      _processing = false;
    });
  });
}

For good practice, we should also set the _processing value to false every time we change the heart button’s toggled state.

_toggle() {
  _initController().then((onValue) {
    setState(() {
      _toggled = true;
      _processing = false;
    });
  });
}

_untoggle() {
  _disposeController();
  setState(() {
    _toggled = false;
    _processing = false;
  });
}

We now have an application that measures the blood flow volume variability and displays it in a chart. Now we only need to calculate the heart rate, which is the frequency of the plotted signal. I used a simple algorithm that measures the average and the max along our window data, sets the threshold to the mean of those values, and detects the peaks above that threshold. It then updates the BPM value with an attenuation coefficient so we don’t have abrupt changes.

class HomePageView extends State<HomePage> {
  ...
    
  double _alpha = 0.3;

  _toggle() {
    _initController().then((onValue) {
      setState(() {
        _toggled = true;
        _processing = false;
      });
      _updateBPM();
    });
  }
  
  ...

  _updateBPM() async {
    List<SensorValue> _values;
    double _avg;
    int _n;
    double _m;
    double _threshold;
    double _bpm;
    int _counter;
    int _previous;
    while (_toggled) {
      _values = List.from(_data);
      _avg = 0;
      _n = _values.length;
      _m = 0;
      _values.forEach((SensorValue value) {
        _avg += value.value / _n;
        if (value.value > _m) _m = value.value;
      });
      _threshold = (_m + _avg) / 2;
      _bpm = 0;
      _counter = 0;
      _previous = 0;
      for (int i = 1; i < _n; i++) {
        if (_values[i - 1].value < _threshold &&
            _values[i].value > _threshold) {
          if (_previous != 0) {
            _counter++;
            _bpm +=
                60000 / (_values[i].time.millisecondsSinceEpoch - _previous);
          }
          _previous = _values[i].time.millisecondsSinceEpoch;
        }
      }
      if (_counter > 0) {
        _bpm = _bpm / _counter;
        setState(() {
          _bpm = (1 - _alpha) * _bpm + _alpha * _bpm;
        });
      }
      await Future.delayed(Duration(milliseconds: (1000 * 50 / 30).round()));
    }
  }

  ...
    
}

The last thing we need to do is avoid the screen from turning off while the :

_toggle() {
  _initController().then((onValue) {
    Wakelock.enable();
    setState(() {
      _toggled = true;
      _processing = false;
    });
    _updateBPM();
  });
}

_untoggle() {
  _disposeController();
  Wakelock.disable();
  setState(() {
    _toggled = false;
    _processing = false;
  });
}

At last, we have an application that measures the blood volume variability and estimates the BPM.

The full code:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
import 'package:camera/camera.dart';
import 'package:flutter/material.dart';
import 'package:wakelock/wakelock.dart';
import 'chart.dart';

class HomePage extends StatefulWidget {
  @override
  HomePageView createState() {
    return HomePageView();
  }
}

class HomePageView extends State<HomePage> {
  bool _toggled = false;
  bool _processing = false;
  List<SensorValue> _data = [];
  CameraController _controller;
  double _alpha = 0.3;
  int _bpm = 0;

  _toggle() {
    _initController().then((onValue) {
      Wakelock.enable();
      setState(() {
        _toggled = true;
        _processing = false;
      });
      _updateBPM();
    });
  }

  _untoggle() {
    _disposeController();
    Wakelock.disable();
    setState(() {
      _toggled = false;
      _processing = false;
    });
  }

  Future<void> _initController() async {
    try {
      List _cameras = await availableCameras();
      _controller = CameraController(_cameras.first, ResolutionPreset.low);
      await _controller.initialize();
      Future.delayed(Duration(milliseconds: 100)).then((onValue) {
        _controller.setFlashMode(FlashMode.torch);
      });
      _controller.startImageStream((CameraImage image) {
        if (!_processing) {
          setState(() {
            _processing = true;
          });
          _scanImage(image);
        }
      });
    } catch (Exception) {
      print(Exception);
    }
  }

  _updateBPM() async {
    List<SensorValue> _values;
    double _avg;
    int _n;
    double _m;
    double _threshold;
    double _bpm;
    int _counter;
    int _previous;
    while (_toggled) {
      _values = List.from(_data);
      _avg = 0;
      _n = _values.length;
      _m = 0;
      _values.forEach((SensorValue value) {
        _avg += value.value / _n;
        if (value.value > _m) _m = value.value;
      });
      _threshold = (_m + _avg) / 2;
      _bpm = 0;
      _counter = 0;
      _previous = 0;
      for (int i = 1; i < _n; i++) {
        if (_values[i - 1].value < _threshold &&
            _values[i].value > _threshold) {
          if (_previous != 0) {
            _counter++;
            _bpm +=
                60000 / (_values[i].time.millisecondsSinceEpoch - _previous);
          }
          _previous = _values[i].time.millisecondsSinceEpoch;
        }
      }
      if (_counter > 0) {
        _bpm = _bpm / _counter;
        setState(() {
          _bpm = (1 - _alpha) * _bpm + _alpha * _bpm;
        });
      }
      await Future.delayed(Duration(milliseconds: (1000 * 50 / 30).round()));
    }
  }

  _scanImage(CameraImage image) {
    double _avg =
        image.planes.first.bytes.reduce((value, element) => value + element) /
            image.planes.first.bytes.length;
    if (_data.length >= 50) {
      _data.removeAt(0);
    }
    setState(() {
      _data.add(SensorValue(DateTime.now(), _avg));
    });
    Future.delayed(Duration(milliseconds: 1000 ~/ 30)).then((onValue) {
      setState(() {
        _processing = false;
      });
    });
  }

  _disposeController() {
    _controller.dispose();
    _controller = null;
  }

  @override
  void dispose() {
    _disposeController();
    super.dispose();
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      backgroundColor: Colors.white,
      body: SafeArea(
        child: Column(
          children: <Widget>[
            Expanded(
              child: Row(
                children: <Widget>[
                  Expanded(
                    child: Center(
                      child: _controller == null
                          ? Container()
                          : CameraPreview(_controller),
                    ),
                  ),
                  Expanded(
                    child: Center(
                      child: Text(
                        (_bpm > 30 && _bpm < 150 ? _bpm.round().toString() : "--"),
                        style: TextStyle(fontSize: 32, fontWeight: FontWeight.bold),
                      ),
                    ),
                  ),
                ],
              ),
            ),
            Expanded(
              child: Center(
                child: IconButton(
                  icon: Icon(_toggled ? Icons.favorite : Icons.favorite_border),
                  color: Colors.red,
                  iconSize: 128,
                  onPressed: () {
                    if (_toggled) {
                      _untoggle();
                    } else {
                      _toggle();
                    }
                  },
                ),
              ),
            ),
            Expanded(
              child: Container(
                margin: EdgeInsets.all(12),
                decoration: BoxDecoration(
                    borderRadius: BorderRadius.all(
                      Radius.circular(18),
                    ),
                    color: Colors.black),
                child: Chart(_data),
              ),
            ),
          ],
        ),
      ),
    );
  }
}

Here’s the working application:

Application measuring the variation of blood flow on the index finger.

The GitHub repository for this project can be found here

On my GitHub repository, you’ll notice that I updated the application to using a Timer.periodic, which presented better results. I also added more customization and an animation the heart-shaped button that simulates a beating heart. I didn’t include it in this tutorial since its not necessary.