Investigating the Reliability of an Empatica E4 for In-the-Field Experiments

E4 Empatica watches are a helpful tool that allows researchers to capture several data points with real-time streaming and visualization graphs. These devices are currently validated under non-movement conditions, however, few studies note that there is little data on how accurate an E4 watch is when participants are not sitting still or are in ambulatory settings. As a result, our team decided to test the accuracy of Empatica (E4) watches over long periods, and when performing hard physical activity. Participants wore the watch for 8 – 12 hours daily, for 5 days. During these 5 days, participants followed their routine and performed tasks such as carrying groceries, eating, and exercising. Once five days had passed, researchers pulled HRV data points consisting of 5-minute sessions from each day-to-day trial. By looking at the data points in Microsoft Excel, and comparing them to the premade graphs the E4 Connect website offered, researchers were able to quantitatively count the number of times the E4 did not accurately read the HRV data. Out of 300 data points, the E4 stopped working approximately 146 times. The results of this experiment ultimately indicate that the E4 is not a practical tool to use underneath motion conditions.

Hi everyone, welcome to our STIP project for summer 2022. This summer, Susan Pham and I worked together to determine the reliability of an E4 for in-field experiments. We hope you enjoy our presentation.

So our project centered around determining the accuracy of an E4 watch under strong motion conditions. We know that an E4 is quite reliable when a participant is staying still, however when we were conducting our literature review for other projects, we noted that many studies suggest that there is not enough research to prove that the E4 is reliable for experiments where the participants move around, or perform strenuous activities. So we asked the question, how reliable is it when a participant is in motion? This summer, our to see if the E4 still maintains its gold standard status for experiments that involve strong movements, such as exercising or simply wearing an E4 in day-to-day activities.

So to start off, what is an E4? An E4 is a wristband that measures several physiological indicators of stress in real-time, including electrodermal activity, which is related to sweat gland activity. Blood volume pulse, which helps the watch derive the heart rate variability (which is basically just a measure in variation between each heartbeat), and your external skin temperature. An increasement of any of these signals would give an indication that a person is stressed, or emotionally aroused. Here’s a quick demonstration on what the E4 looks like, and how it works.

This is the E4. As you can see, it’s quite a bulky watch. To use it, first, you remove the back charging panel. When you want to turn it on, you press the small circular button that’s on the front portion of the watch. This should cause the watch to emit blue blinking light. This means that the E4 is in discovery mode, and it’s ready to be paired to the streaming app. To put it on, you just wrap the band around your arm and make sure the sliver nodes press against your wrist firmly. Let’s see exactly how the watch connects to the streaming app.

Let’s check out the streaming app for the E4. As you can SEE, the app is called E4 realtime. To connect to the watch, we click on connect e3 and start streaming. Once we do that, we confirm which watch we want to use. The app will then show its acquisition of the physiological signals. As you can see, it takes a few seconds to get the app properly connected to the watch, and for it to begin working. It takes a bit longer, around 7 seconds or so, for the watch to accurately determine the Heart rate variability. The e4 can be a little finicky in determining this, which is why it does take a few seconds. However, if we press on the back button, we’ll be able to see other physiological indicators that we talked about previously, such as eda, temperature. We can even see the duration of the session, and how much battery life the watch itself has. If we want to stop streaming, we simply click on the stop streaming button. We click on yes, and as you can see in the sync section, the app states that the data is uploading. Once it reaches 100 percent and states the data was uploaded, this means that we’ll be able to see our data on the E4 Connect website later.

So now that you have a good idea of what the E4 looks like, and how it works, it’s important to know the mythology we used for our experiment. we decided to see if we could quantitatively determine the accuracy for the E4, when a participant is in movement. To do this, we compared the data gathered from an experiment where participants are sitting still and watch a video while wearing the e4, to an experiment where the participants are actively moving in day-to-day life and are wearing the e4 for a period of 8 to 12 hours .

Here’s a data set from both of our experiments. On the right is the experiment where the participants were sitting sill, whereas on the left is when participants are in motion. As you can see, there is a clear difference. The data graph from when the participant is sitting still, which is on the right, is consistent the heart rate is clear and steady.
In contrast, the heart rate when the participant is going about their day-to-day life is shaky and inconsistent. The long flatline indicates that the E4 was not able to read the HRV correctly, and the sudden decreases in heart rate are the result of the E4 not being able to connect to the streaming app, most likely due to motion. Just by looking at this, it’s clear, that the E4 is not very reliable when the participant is moving.

To quantitatively determine how accurate the E4 was for our motion experiments, we looked at the raw HRV data for the day-to-day experiments. The total session lasted an hour, but we randomly chose a section of 5 minutes from the total hours. From those 5 minutes, we looked at the raw HRV, and the E4 graph, and counted how many flat lines and zeroes there were in that duration of 5 minutes. We counted these flatline and zeroes as these would have given indications that the E4 had stopped accurately recording. We then added the total number of flatlines and zeroes and divided that by the total number of data points, which was 300. Performing these calculations gave us an accuracy rate of 46%, which means that this isn’t a device you want to use if you’re going to perform any experiments with hard physical activity.

So undeniably, E4 watches are useful research tools that allow researchers to capture physiological signals accurately. However, based on our test results, the watch gives too many unreliable readings when the participant is in motion. This has allowed us to conclude that the E4 is not the ideal equipment to use in experiments that involve long durations of wear, or that involve hard movement.

Here is the bibliography for this project on the reliability on empatica E4 watches for in-the-field experiment

Lastly, Thank you to George Mason’s OSCAR office for allowing us to work in this research opportunity. Thank you to Dr. Chaplin for supervising us. And Thank you to Venkat and Nathalia for supporting and guiding us throughout this summer’s program.