The Usage of Robotic Process Automation and Artificial Intelligence for Handwritten Medical Document Digitalization

Abstract

This study presents two automatic approaches that can be applied to digitize medical handwritten forms. Both approaches utilize an RPA (Robotic Process Automation)-based software robot that converts the handwritten form into a digital entry. The RPA robot uses a taxonomy specific to medical form and associates the extracted data with the taxonomy. The first approach uses an OCR (optical character recognition provided by Google Cloud Vision/Azure Vision to create a DOM (Document Object Model) model. The second approach integrates the OpenAI ChatGPT4o model to extract handwritten medical data and transform them into typed data. In both approaches, the digitalized form is sent to a data extraction activity that uses UiPath’s machine learning API to extract the data from the form. Because the medical form can not be found in the UiPaths standard templates, a data extraction template must be applied. After the data extraction process the saved data can be sent to a database, spreadsheet. The aim of the study was to determine which data extraction methods offer a better result. Access to this medical data is restricted to physicians and nurses employed at the specific medical facility. By creating new non-standard form templates and associated taxonomies, the system can be scaled as desired. It must be noted that the quality and readability of handwriting has a massive impact on the digitalization outcome. Using messy handwriting with Google Vision, the results were approximatively 65% of correct recognition after five runs on the same document, and using capitalized print letters, the result was approximatively 99%. Using the ChatGPT4o model the transcription had to be run at least three times to achieve 99% transcription on messy handwriting. The version with the capitalized print letter offered the best results using ChatGPT4o model.