The problem

Privacy protection of facial images has attained prominence over the past decade owing to the digitalization of medical records and concerns about data breaches. Facial images are useful for identifying signs of disease; however, these inevitably record patient biometric identifiers. Thus, the first challenge is to separate biometric identity from medical information that can be derived from facial images.

Additionally, concerns about image breaches have hampered the development of digital health technology (for example, deep learning assistance) that depends on broad participation in medical data collection. The second challenge is to improve the willingness of healthcare users to share their facial images and to reduce concerns about the misuse of facial-recognition technology.

The solution

We set out to develop an alternative procedure for sharing and recording facial images, and to provide an additional data format for privacy protection.

As periocular biometrics are one of the most distinctive subsets of individual biometric information1, we focused on four pathological ocular manifestations that involve more than ten abnormal phenotypes. We developed the digital mask (DM), which inputs an original video of a patient’s face and outputs a de-identified three-dimensional (3D) reconstructed video based on the complementary use of deep learning and 3D reconstruction. Deep learning achieves feature extraction from various facial parts, while 3D reconstruction automatically digitizes the shapes and motions of 3D faces, eyelids and eyeballs based on the extracted facial features2,3,4 (Fig. 1). Converting DM-reconstructed videos back to the original videos is extremely difficult because most of the necessary information is no longer retained in the set of digital representations that constitute this mask.

The DM uses RGB (red-green-blue) images (individual frames from a video) as input and outputs 3D reconstructed meshes. For a particular frame, the algorithm first extracts 2D facial landmarks from the RGB image and fits a set of face-model weights for 3D face reconstruction. The algorithm then extracts 2D eyelid landmarks and 2D semantic lines, and fits eyelid model weights for 3D eyelid reconstruction. Finally, the algorithm extracts 2D iris landmarks and solves eyeball rotation for 3D eyeball reconstruction. SGAEM, simplified geometry and appearance eyeball model. © 2022, Yang, Y. et al., CCBY 4.0.

Experiments were then conducted to validate the efficiency of the DM. First, to assess reconstruction performance of the DM, we quantitatively evaluated the error between original videos and DM videos at the pixel level. Error was extremely low, ranging from 0.81% to 1.61%. Second, we evaluated the DM in clinical practice. The independent diagnoses from the original videos and the diagnosis from the DM-reconstructed videos were highly consistent (k > 0.8). We also compared the DM with the traditional de-identification method of cropping and found that the risk of being identified was decreased in the masked patients. Third, we evaluated the willingness of patients to share videos processed anonymously by the DM. Over 80% of patients believed that the DM can alleviate privacy concerns and expressed an increased willingness to share their personal information. Finally, we confirmed that the DM can also evade artificial intelligence–powered facial-recognition algorithms.

The implications

‘Protecting privacy’ does not equate to ‘absolute removal of identity characteristics’5. One of the most important principles of privacy protection is balancing disclosure risk against data utility. Therefore, the purpose of the DM is to provide an approach to health-information disclosure that de-identifies protected health information as much as possible, without compromising the ability of clinicians to reach a diagnosis.

In addition to its potential utilization in research and routine clinical practice, the DM could be applied to telemedicine, including online automatic diagnosis and patient triage for more-efficient healthcare delivery. Furthermore, the DM can obtain quantitative parameters (such as the degree of eyeball rotation, eyelid shape parameters, and rotation frequency), which might help diagnosis in the future. Additionally, many other non-ocular disorders involve facial manifestations, and we propose that with further development, the DM has the potential to be applied in, for example, otorhinolaryngology, neurology, and oral and maxillofacial surgery.

One limitation of our study is that the reconstruction of conjunctival hyperemia, eyelid edema and abnormal tissue growth, such as ocular tumors, remains challenging owing to insufficient model capacity. We intend to improve the DM by including a sufficiently large sample of abnormal cases for detailed analysis, or by constructing an extra sub-model on top of the existing model. In addition, the risk that the DM might be compromised still remains, and we hope to formulate relevant rules of technology security in the future.