Documents come in a variety of shapes and sizes, encompassing structured, semi-structured, and unstructured content. They may feature images as backgrounds or foregrounds, formatting elements like checkboxes, and various combinations aiming to enhance legibility. From printed and handwritten texts to scanned images and complex tables, the possibilities are endless. The challenge of processing such documents and converting their content into a digital format can be associated to gold mining in the digital realm. You might wonder how this process works.
Once a document is converted into a digital format, it becomes possible to establish data relationships, contextualize information, and operate more efficiently with minimal human intervention. This is particularly beneficial in business contexts, where processing bills and payments can be significantly expedited. For instance, imagine automatically extracting the "Total Amount Due" from an invoice and processing the payment if it falls below a certain threshold; if it exceeds that amount, it can be flagged for human review. It's clear that the value of Document Processing is substantial.
Let's explore the "Intelligence" component of document processing. The first step is to identify the type of document, followed by extracting content based on its classification. This process requires embedding intelligence within the system. Just as humans gradually develop an understanding of a document—classifying it and honing their processing skills—computer systems are engineered to enhance their comprehension and content extraction capabilities over time. This field is referred to as Intelligent Document Processing, and IBM Automation Document Processing (ADP) is a standout product that excels in this area.
To comprehend how such systems are constructed, let's break it down from the top. Documents are classified based on their basic structure: they can be categorized as unstructured text, semi-structured (such as invoices or phone bills containing tables), or structured (like paragraphs in an article). The classification process involves making intelligent decisions based on the quantity of information elements present in the document. Once classified, Optical Character Recognition (OCR) activates to convert all content into digital attributes. Next, the system identifies the specific document type or class and extracts the corresponding key-value pairs (such as customer names, addresses, account numbers, etc.), Tables. This extraction utilizes a combination of AI and traditional techniques, resulting in a JSON output. In a comprehensive setup, this JSON can be parsed and linked to a workflow for further processing. For instance, the JSON might include the "Total Amount Due" along with its value, which can be utilized to trigger automated payment processing or further review of the invoice.
IBM ADP is an incredibly flexible product that allows supervisors to utilize a Designer environment for configuring the system. This includes setting up document classes, defining keys, and customizing extraction methods. Additionally, users can train models, test extraction accuracy, and once they are satisfied with the results, deploy the setup into the Runtime environment for live, automated intelligent document processing.
ADP 24.0.1 released Dec 2024 is the latest stable version, please refer to the links in the reference section below for more information. To make it more interesting here are a few technologies it leverages
- OCR for document scanning
- AI/ ML for document & page classification, KVP extraction, fixing scanning issues
- Natural Language Extraction using IBM SystemT a declarative information extraction for defining custom extraction handlers
Reference links
- ADP Product page https://www.ibm.com/products/document-processing
- ADP Product Documentation https://www.ibm.com/docs/en/cloud-paks/cp-biz-automation/24.0.1?topic=automation-document-processing
- Multi-part series of blogs on Table Extraction https://medium.com/@praveenmidde/table-extraction-in-content-capture-domain-why-what-and-how-8a3c90c79874
