Data quality is getting renewed attention in the age of AI. Large language models (LLMs) are trained on vast amounts of text and images, but to be useful in a corporate context they need enterprise-specific information. For example, a customer service chatbot will need to know about your particular account, the order history, the corporate policies around returns and warranties, and perhaps also have access to instruction manuals and troubleshooting guides. Unstructured data like policy manuals will be accessed via retrieval augmented generation (RAG), but a customer service chatbot will also need access to structured data sources such as databases containing customer orders and delivery information as well as invoicing and payment information. This is usually done via APIs or direct database queries. Normally, an LLM will access corporate data indirectly, such as via an API, database stored procedure or a microservice, rather than being allowed to run SQL queries directly against production databases.
Given all this, it is clear that the quality of the data within those corporate databases is going to be crucial if the customer service chatbot is to be effective. Data about a customer order, delivery and payment needs to be up to date, complete and accurate. In reality, the quality of data in large enterprises is rarely pristine. Customer and product data is often scattered amongst numerous systems, and some versions of this data may be out of date or incomplete, and possibly just wrong. I recall working on a project dealing with B2B customer data when I was in the oil industry, and we discovered that much of the customer database for one particular business was duplicated: for every genuine customer record there were, on average, four duplicates. This was an extreme case, but duplication rates of 10-30% are common. Sustained data quality efforts can reduce this to 1%, but barely a fifth of organisations manage this. Moreover, business data decays over time: people move house, companies merge, product lines change. Business data typically decays at around 30% annually.
An industry of data quality tools exists to help with this issue. Such tools can profile data, spot outlier values, detect probably duplicated records, detect incomplete data records, merge duplicates into single trusted records, and fill gaps in data e.g. by completing postal codes for address records missing them. Data quality rules can be entered manually, for example specifying uniqueness for customer codes, or that prices are within a specific range, or that emails must follow a valid format (having an @ symbol) or that postal codes or social security numbers follow a set pattern. A person’s age might be specified as having to be within a range (perhaps 0 to 125; the oldest ever human lived to over 122 years old), while delivery ranges may also be set. However, it can be seen that manually coding such rules for every single column of data in an enterprise is a big job, so data quality vendors have used various automation techniques to help with this.
One important automation technique is machine learning. A machine learning model can observe the flow of data and learn what is normal. A purchase that is vastly larger than previous ones from a particular customer, or a person with an age of 230, or deliveries to a branch suddenly dropping to zero, are all examples of abnormal data, and the machine learning model can flag this as a potential data error. Perhaps there was a good reason why that branch delivery stopped, but a human can be dispatched to look into it. Machine learning models are also good at detecting potential duplicate records. “Andy Hayler”, “A. Hayler” or “Andrew Hayler”, all living at the same address may indicate likely duplicate records. Various algorithms can be used, from the “random forest“ algorithm (which combines the output of multiple decision trees to reach a single result) to neural networks. Machine learning can classify data into categories such as addresses, product codes or personally identifiable information. It can infer information rather than having explicit rules, so for example customers under the age of 18 rarely have mortgages. After observing historical data and corrections, a machine learning algorithm can recommend new data quality rules. Machine learning is excellent for detecting outlying values, spotting patterns and anomalies, predicting missing data and finding duplicates. Unlike manually written matching rules, machine learning models can learn subtle patterns from historical matching decisions.
LLMs can also play a role in data quality, but they have different strengths and weaknesses compared to machine learning. LLMs approach the problem very differently. LLMs are good at understanding and generating language, so are good at spotting contextual issues in unstructured text. They can describe why two records may be a match in plain English or explain anomalies to a business user. LLMs can interpret free-text customer complaints, detect inconsistent product descriptions, explain why a record was flagged, generate suggested data-quality rules and help stewards investigate issues. LLMs can dramatically reduce the effort required to create rules. Instead of a rule like:
Age < 18
Mortgage = FALSE
someone can simply write
“Find customers who appear too young to hold a mortgage.”
Each AI technique has its own strengths and weaknesses. For example, decision trees and random forests are good at classifying records are easy to understand and are accurate. However, they struggle with very complex relationships. Neural networks can model complex relationships and are excellent at matching but require a lot of training data and are black boxes, so they cannot explain their decisions. A technique called isolation forest is good at detecting anomalies and scales well, but cannot explain their reasoning, and may flag some false positives. Autoencoders are great at sophisticated anomaly detection but are computationally expensive. LLMs are excellent at understanding text and interacting with users in natural language. However, they remain less reliable than specialised machine-learning models for large-scale numerical analysis and statistical anomaly detection. They are also relatively expensive to run, can hallucinate, and are not well suited to scanning millions of structured records.
For this reason, an optimal way forward for data quality tools is to use a blend of techniques, applying the appropriate technique to the particular data under consideration. Data with a lot of unstructured text might be routed to an LLM, whereas data matching might be routed to a neural network, or outlier detection to an Isolation Forest or an autoencoder… Rather than a one-size-fits-all approach, data quality vendors should embrace a range of available techniques and apply the best tool for the particular job at hand.
The future of data quality is therefore unlikely to be dominated by any single AI technique. Instead, successful platforms will orchestrate traditional rules, machine learning and LLMs together, applying each where it delivers the greatest value.
