Human in the loop information extraction increases efficiency and trust

Abstract

Automation is often focused on data-centred measures of success, such as accuracy of the automation or efficiency gain of individual automated steps. This case study shows how a human-assisted information extraction system, that keeps the human in the loop throughout the creation of information extraction rules and their application, can outperform less transparent information extraction systems in terms of overall end-to-end time-on-task and perceived trust. We argue that the time gained through automation can be wiped out by the perceived need of end users to review and comprehend results, where the systems seem obscure to them.