Identity Data Quality Before Automation: Correlation, Unmatched Accounts, Duplicates, and Drift

Before enabling IAM automation, four data problems need to be understood and handled: matching accounts to their owners (correlation), classifying accounts that match nobody (unmatched accounts and orphans), resolving people who exist twice (duplicates), and keeping records aligned with reality (drift). Automation does not fail loudly on bad data. It runs, provisions, and deprovisions exactly as instructed, and distributes every data error to the connected systems. This article describes how we evaluate identity data before turning automation on, following the same Connect, Cleanup, Automate progression that Evolveum’s midPoint methodology uses.

Why automation exposes bad data

Manual identity administration tolerates bad data surprisingly well. A helpdesk operator who sees two similar records picks the right one from context. An admin who receives a deprovisioning ticket for the wrong person notices the mistake and asks. Humans apply judgment at every step, which hides the underlying quality problems.

Automation removes the judgment and keeps the data. A typical scenario: HR marks a contractor as departed, the platform disables the linked accounts, and it turns out those accounts belong to a different person whose records were incorrectly merged years earlier. The mistake had existed the whole time. Automation simply executed it.

A pattern we see on nearly every assessment: applications carry technical accounts that accumulated over the years. Each was added for a reason, an integration, a data load, a monitoring tool, and the reason was rarely written down. When correlation flags them, an internal investigation usually starts: who created this account, what does it do, is anything still using it. That investigation is not a failure of the assessment. It is the point of it.

What bad identity data actually means

Identity data quality is the degree to which you can answer four questions with confidence:

1. Who does this account belong to? Every account in every system should trace to a current person or a documented technical owner.
2. Is this person represented exactly once? One person, one canonical governed identity, even if that person has multiple employments, contracts, or accounts.
3. Are the attributes correct? Department, manager, location, employment status. The fields your access policies will run on.
4. Does reality match the records? The access people actually have should match what your systems say they have.

There is a subtler dimension behind question three that gets missed in most data quality discussions. An attribute can be technically valid and still be unusable for governance. Every employee may have a department value, so completeness looks perfect. But if HR fills that field inconsistently, or if department has no relationship to how access should actually be assigned, the data is still poor for IAM purposes. When we evaluate identity data we look at four levels: completeness (is the field filled), correctness (is the value true), consistency (is it used the same way everywhere), and suitability (can an access policy safely depend on it). Many automation projects stall on the fourth level after passing the first three.

The four problems

Correlation: matching accounts to people

Correlation means determining which account belongs to which person, across every connected system. On paper this is a join operation. On real data it is one of the hardest tasks of the project.

The HR system says Jana Kováčová. Active Directory says jkovacova. The ERP system says KOVACJ01. An old application says jana.kovacova@formercompanyname.com because it was set up before a rebrand. Same person, four representations, no shared identifier.

Start with a stable person-level business identifier where one exists, and verify what it actually identifies: an employee number can represent an employment rather than a person. Then use reliable system-specific identifiers and additional attributes as supporting correlation signals. Email addresses, UPNs, logins, and names can all be useful, but their reliability depends entirely on how the organization manages them. The principle: prefer stable, immutable business identifiers whenever they exist. Human-readable attributes such as names and email addresses are useful signals but poor identity keys. Names change, emails get reissued, and two people named Novák will eventually work in the same department.

midPoint supports this with correlation rules that combine multiple attributes with confidence levels. Matches above a configured confidence link automatically. Ambiguous cases go to a human for resolution. That human step is not a limitation of the tooling, it is the design: an engine that automatically resolves every ambiguous candidate can eventually link two different people, and recovering from a wrong merge costs far more than working a review queue.

The hardest cases in real integrations are rarely the obvious ones. They are identities that are technically valid in every source system but mean something different in each one. HR may model a person. Another source may model employments or contracts. The target system only knows accounts. Before automating anything, those concepts have to be reconciled, and no correlation rule can do that reasoning for you. The data can therefore be correct in every individual system while the identity model created by combining those systems is still wrong.

Unmatched accounts and orphans: classification before cleanup

After correlation runs, some accounts will match nobody. It is tempting to call that remainder the orphan list and schedule it for deletion. That shortcut causes outages.

The unmatched remainder is your investigation queue, not your deletion queue.

Some of those accounts are genuine orphans: a former employee’s account that outlived the offboarding, a vendor account from a finished project. Others are legitimate service accounts, technical identities, pre-created accounts waiting for a hire, protected administrative accounts, or valid users whose attributes are simply too poor to correlate automatically. Evolveum’s methodology makes the same point: unmatched accounts get classified first, and legitimate exceptions get marked and protected before any automated cleanup runs.

The distinction that matters for governance is not orphan versus non-orphan. It is explained versus unexplained. An unexplained account is a governance problem regardless of its origin. An intentionally unmanaged technical account with a documented purpose, owner, protection policy, and review cycle is a different thing entirely, and it can be governed and reviewed differently.

On scale, the problem is well documented. Orchid Security’s Identity Gap 2026 snapshot, based on telemetry from enterprise applications across North America and Europe collected between April 2025 and March 2026, found that 40 percent of accounts across the analyzed applications belonged to users who could no longer be found in the HR system, and were classified as orphan accounts. Earlier, Varonis’s 2021 assessment of 50 manufacturing companies found that 44 percent of them carried more than 1,000 enabled ghost user accounts. Different samples, different years, same direction: unowned access can accumulate when nobody is explicitly responsible for removing it.

This matches what we find in practice. The legitimate exceptions that turn up during classification are mostly non-human identities: integration accounts, service accounts, and other technical identities added to applications over time. They are valid, often still in use, and almost never documented, which is exactly why classification has to come before cleanup.

Duplicates: one person, several identities

Duplicates are the mirror image of orphans: one person, several identity records. Rehires are the classic source. Someone leaves, returns two years later, and HR creates a fresh record because the old one was archived. Each record can accumulate its own accounts, and neither shows the person’s full access.

The consequences are quiet but serious. Access reviews show a reviewer half of someone’s access. Segregation-of-duties checks can miss conflicts when the conflicting permissions sit on two different records. Offboarding disables one identity and leaves the other active.

A good rehire design preserves or rediscovers the original identity rather than creating another one. In midPoint this can be implemented with stable identifiers, correlation rules, and an appropriate lifecycle model, so that a returning person is matched to their existing identity instead of getting a new one. Existing duplicates can be merged, midPoint provides interactive identity merge for exactly this, but merging is corrective surgery. The design goal is preventing the duplicate in the first place, and that depends on decisions about identifiers, retention, and HR process made well before the rehire happens.

Drift: when reality and records diverge

Correlation, classification, and deduplication are project activities. Drift management is permanent. Drift is the gap that opens between what your IAM platform believes and what actually exists in the connected systems.

It accumulates through ordinary work. An admin creates an account directly in an application because a request was urgent. A group membership granted during an incident never gets removed. A SaaS administrator adds a user through the vendor console. None of it passes through the IAM platform, so the platform’s picture of reality degrades quietly.

Reconciliation is the countermeasure: a scheduled comparison between the IAM platform’s state and each connected system, with every discrepancy flagged and either corrected automatically or routed to a human, depending on policy. An account that appears in a target system with no counterpart in the platform is exactly how new orphans are born, and scheduled reconciliation can detect it on the next cycle instead of leaving it undiscovered until an audit or incident.

The sequence: Connect, Cleanup, Automate, Keep It Clean

Evolveum’s midPoint methodology describes iterative cycles of connecting systems, cleaning data, and then automating processes. Our project experience supports the ordering strongly, so we use it as the backbone of the data quality phase.

Connect: see reality before changing it. Connect the HR and other authoritative sources, together with the most important target systems, without writing changes back to those connected systems. Let correlation run and produce the honest picture: match results, the unmatched queue, disputed correlations, evidence of possible duplicates, attribute gaps.

Cleanup: classify and resolve. Classify the unmatched accounts into orphans, exceptions, and data errors. Resolve disputed correlations. Merge confirmed duplicates. Correct the attributes your planned policies depend on, with special attention to consistency and suitability, not just completeness.

Automate: progressively, with simulation. Enable lifecycle automation progressively, starting with the populations and processes whose data and policies are best understood, and expand from there. Movers are often the most complex, because they require decisions about which existing access should be retained, changed, or removed. For changes where simulation is applicable, we treat it as a standard safety step before execution against production data: midPoint’s simulation shows what a policy would do before it does it, and the first cleanup pass is exactly the situation it exists for.

Keep it clean: reconciliation as routine. Schedule reconciliation against every connected system and triage discrepancies as operational work, not annual archaeology. A cleaned environment without reconciliation can start drifting again immediately.

How to assess where you stand

You do not need an IAM platform for a first honest look. You need exports and a few hours.

1. Map your identity sources and creation paths. Which systems are authoritative for employees, contractors, and other identity populations? Which systems can create accounts independently? If the answer includes “the helpdesk sometimes creates them manually,” write that down.
2. Measure the match rate on your three most important systems. Export accounts and records from the relevant authoritative source(s), and join them on whatever identifier you have. The percentage that fails to match cleanly is your correlation problem, quantified.
3. Count accounts with no match in the expected authoritative identity source. That is your investigation queue.
4. Check the fields your future policies need. Completeness first, then spot-check correctness and consistency. A filled field is not automatically a usable one.
5. Ask when each system was last compared against its authoritative identity source. If the answer is “at the last audit,” drift is accumulating now.

What acceptance criteria look like

There is no universal correlation percentage at which automation suddenly becomes safe. What matters is that ambiguous and unmatched cases are understood, classified, and handled by policy before destructive automation is enabled. That said, projects need concrete acceptance criteria, so here is the shape they usually take. Treat these as examples to adapt, not industry thresholds:

• Every account in every connected system is either matched to an owner, classified as a documented exception, or sitting in a triaged investigation queue with a decision pending
• Automated correlation handles the clear majority of accounts on core systems, and the manual remainder is small enough that a named person actually works it
• No unexplained duplicates in the identity store
• The attributes your first wave of policies depends on have been checked for completeness, correctness, consistency, and suitability
• Reconciliation runs on schedule against every connected system, and discrepancy triage is someone’s defined job

Frequently asked questions

How long does an identity data cleanup take?

It depends less on data volume than on how many cases need a human decision: ambiguous matches, unmatched accounts to classify, duplicates to confirm, and how quickly data owners respond. Connecting systems in read mode early is the fastest way to replace guesses about duration with a real queue you can size.

Can we automate first and clean up as we go?

Automating on unresolved data propagates the errors to every connected system and makes cleanup harder, because you then have to separate legitimate automated changes from amplified mistakes. Progressive automation is the workable middle ground: enable low-risk flows on well-understood populations first, and expand as classification completes.

Is there a match rate that makes automation safe?

No universal number exists, and be skeptical of anyone quoting one. Safety comes from the unmatched and ambiguous cases being classified and policy-handled, not from the matched percentage crossing a threshold. A 99 percent match rate with an ignored 1 percent is riskier than a lower rate with a worked investigation queue.

Are all orphan accounts a security risk?

An unexplained account is always a governance problem. But not every account without a human owner is bad: service accounts, technical identities, and pre-created accounts can be legitimate if they have a documented purpose, a responsible owner, protection policies, and a review cycle. The goal is zero unexplained accounts, not zero exceptions.

Does midPoint require clean data before connecting?

The opposite. Connecting midPoint in read mode is how you measure the problem: its correlation produces the match results, unmatched accounts, disputed correlations, and the evidence needed to investigate possible duplicates. The rule is not clean before connecting. It is clean before automating changes.

Start with the honest picture

Before automating lifecycle processes, we can assess the identity data underneath them. We connect the authoritative and target systems in read mode, evaluate correlation, classify unmatched accounts and duplicates, and turn the findings into a concrete cleanup and automation plan.

Discuss an identity data assessment

Related Resources

• Joiner-Mover-Leaver Automation Done Right
• SAP IDM to midPoint Migration
• Microsoft MIM to midPoint Migration
• Our Implementation Methodology
• midPoint Consulting and Implementation Services

Sources

• Orchid Security: Identity Gap 2026 Snapshot. 40% of accounts across analyzed enterprise applications classified as orphaned; North America and Europe, April 2025 to March 2026 telemetry.
• Varonis 2021 Manufacturing Data Risk Report. 44% of 50 assessed manufacturing companies had 1,000+ enabled ghost user accounts.
• Evolveum midPoint Documentation: Correlation. Multi-attribute, confidence-based correlation with human resolution.
• Evolveum First Steps Methodology. Connect, Cleanup, Automate; classification of unmatched accounts.
• Evolveum midPoint Documentation: Simulations. What-if evaluation before execution.
• Evolveum midPoint Documentation: Identity Merge. Interactive ex-post merge of duplicated identities.