First cousin once removed to KRB_AP_ERR_MODIFIED
Most anyone who would be interested in reading an article like this has very likely encountered the error, KRB_AP_ERR_MODIFIED. This error tells us one thing: The account secret (aka password hash) that is being used to decipher the ticket cannot decipher the ticket.
The most common reasons are:
- The computer upon which the decipher occurs has a broken Secure Channel.
(In short, the secret (aka computer's password hash) is not the same between the computer and the Domain and/or the DC that issued the ticket.) - The Service Principal Name is on the wrong account.
- Misconfiguration of a service.
- Wrong account being used.
- Clustering incorrectly configured.
- Name resolution routes connections to the wrong server (CNAMEs are often the culprit).
- Something on the network has mangled the packet.
- Malicious activity.
So what does KRB_AP_ERR_BAD_INTEGRITY tell us?
KRB_AP_ERR_BAD_INTEGRITY tells us one thing: a failure to decipher a Kerberos referral ticket.
When an account (the client), be it a user or a machine, wants to access resources in another trusting domain, the client must first get a referral ticket (Inter-Realm TGT) from a Key Distribution Center (KDC, aka Domain Controller) in its own domain. The client can then present the referral ticket to a KDC in the Trusting Domain.
The referral ticket is enciphered with a secret shared between the two domains. This common secret is stored on the Trusted Domain Object (TDO) in the domain partition of Active Directory. Since the KDCs in the different domains share a common secret, they can both encipher and decipher tickets with that secret.
If the receiving KDC cannot decipher the referral ticket, using the secret on its copy of the TDO in its domain, then the resulting error is KRB_AP_ERR_BAD_INTEGRITY.
Let’s Repro!
So, this is all well and good, a bunch of words, now let’s break it and then fix it!
We’re going to examine Kerberos traffic in a network trace when all is working, compared to what we see when we get the error. And then lastly, we’ll try to fix our lab environment.
The Lab
Here’s our lab:
We have a root domain (contoso.com) with two child domains, na.contoco.com and eu.contoso.com with the NetBIOS names, CONTOSO, NA and EU. Each domain has two domain controllers.
For our test scenario, the user jondoe, in the NA domain, wants to access files shared from a server in the EU domain. Our user, na\jondoe, is interactively logged on to the file server NAFS in the NA domain. To get to the file share, jondoe is going to have to get two referral tickets, one for CONTOSO and one for EU.
To maintain the focus of this document, we will only examine the Kerberos traffic in the network traces. We will not explore DNS or network problems. Note however, that network and name resolution problems could be why the secret on the TDO is wrong.
When everything is working as expected …
In our good scenario, na\jondoe has interactively logged on to the file server NAFS and then, using file explorer, connected to \\eufs.eu.contoso.com\eu_share.
In the frames below, we see the Kerberos conversations that the client had with the DCs in the different domains.
legend:
Bright green with black text == Our client
Dark green with white text == NA domain DCs
Black with white Text == CONTOSO Domain DCs
Red with white text == EU Domain DCs
In frames, 1535, 1536, 1543 and 1544, we see our standard AS request and eventual successful response with a TGT from the domain in which the user resides, na.contoso.com. We’ll skip looking into this well-known pattern. Let’s get on to the referrals.
Next, in frame 1553, we see the client attempt to get a service ticket for cifs/eufs.eu.contoso.com. Note the Realm and the Sname. As we can see, they are two different domains.
So, in frame 1556, we get a referral ticket from a NA DC for the CONTSO domain. Why? Because DCs in NA do not have any accounts with a Service Principal Name (SPN) that matches “cifs/eu.contoso.com”. So, the DC gives the client a referral ticket to another Domain, along the trust path, that may know more about this resource, in this case, since it is a child domain, the referral is for the parent domain.
Let’s take a closer look at frame 1556 …
This is our referral ticket from a NA DC. Note the Crealm and Cname: references to the domain in which the user resides. (Cname and Crealm can also be read as 'Client Realm' and 'Client Name'.) The Sname (Service Name), the service for whom this ticket is intended, is in a different domain, in this case krbtgt/CONTOSO.COM. Note the bit in red at the bottom. This cipher is derived from the secret stored on the contoso.com trust object, the TDO, stored in the domain partition on DCs in the NA domain.
Next, frame 1565 we see the client make a request to a root DC. One of the things we can see in this TGS request is the ticket we received from a DC in the NA domain. We can see it here, in blue:
Note the bit in green. This is what the client is requesting, a service ticket for krbtgt/EU.CONTOSO.COM.
The root DC, in frame 1567, gives the client yet another referral ticket, and you guessed it, this time using the secret on the TDO for the EU domain that is stored in the domain partition of the CONTOSO domain.
In frame 1587, we see the client present the ticket given to it by the root domain to a DC in the EU domain, and asking for a ticket for the service, cifs/eufs.eu.contoso.com.
We can see this frame is nearly identical to 1565. But in our next frame, 1589, instead of yet another referral, we get our ticket for the file server EUFS. DCs in EU have an SPN that matches cifs/eufs.eu.contoso.com, and can get on with the business of granting the user the ticket, which happens in frame 1589, below.
At this point the client will make an SMB connection to the file server and present the ticket in the SMB session setup.
In summary, the client wants to get to a resource in another domain. Since the client’s domain does not have this resource, it gets a referral to another domain in the trust-path that might host this resource or know where it might be. This process continues along the trust-path until a KDC can find the SPN for the service the client is requesting.
The term 'trust-path' used a few times, refers to the relationship of the domains and/or forests to one another. In this scenario, in a forest, the path goes from a child to the parent to another child. This is no direct trust between the two child domains. In a multi-forest scenario, the trust path can go from root to root, or perhaps even so called "short-cut" trusts. Keeping the trust-path in mind is a good idea.
Looking at versions of secrets
Before we break this lab, let’s take a look at something else very important and is at the heart of what we are discussing, the attributes, trustAuthIncoming and trustAuthOutgoing. These are attributes on TDOs; they store our shared secrets.
These attributes have attributes of their own, such as their version, the time they were changed and upon with DC the change was made. These 'extra attributes', the metadata, is not exposed in tools such as ADSI Edit, Active Directory Users and Computers (ADUC) snap-in. (It is possible to see the values with ldp.exe, but on a DC by DC basis, which makes it not very convenient.) We cannot directly query these values with the common tools conveniently. But we can query the metadata about the attributes with repadmin /showmetadata. Also, this command allows us to query every DC in a domain with one command.
Logon to EUDC01 and open an elevated command prompt, and then issue the following command:
repadmin /showobjmeta * "CN=contoso.com,CN=System,DC=eu,DC=contoso,DC=com"
For large environments with many DCs, redirect the output to a text file like so:
repadmin /showobjmeta * "CN=contoso.com,CN=System,DC=eu,DC=contoso,DC=com" > c:\MyData.txt
This is going to give us a lot of information, but we are only interested in the attributes trustAuthIncoming and trustAuthOutgoing on the DCs in the EU domain. For purposes of brevity and formatting, I’m only going to display the portion of interest.
/showobjmeta against DC eudc01.eu.contoso.com
Org.Time/Date Ver Attribute
============= === =========
2023-12-14 15:49:39 13 trustAuthIncoming
2023-12-14 15:35:25 13 trustAuthOutgoing
/showobjmeta against DC eudc02.eu.contoso.com
Org.Time/Date Ver Attribute
============= === =========
2023-12-14 15:49:39 13 trustAuthIncoming
2023-12-14 15:35:25 13 trustAuthOutgoing
Notice the versions (ver), they are consistent on all the DCs in the EU domain. The current version is 13.
Now let’s take a look at the TDO for the EU domain on the DCs in the CONTOSO domain:
repadmin /showobjmeta * "CN=eu.contoso.com,CN=System,DC=contoso,DC=com"
… yields the following:
/showobjmeta against DC rootdc01.contoso.com
Org.Time/Date Ver Attribute
============ === =========
2023-12-14 15:35:25 13 trustAuthIncoming
2023-12-14 15:49:39 13 trustAuthOutgoing
/showobjmeta against DC rootdc02.contoso.com
Org.Time/Date Ver Attribute
============= === =========
2023-12-14 15:35:25 13 trustAuthIncoming
2023-12-14 15:49:39 13 trustAuthOutgoing
Again, we see consistent values on all the DCs in the CONTOSO domain for these attributes.
Keep in mind that these attributes contain our secret. The fact that the version is 13 means that secret for the trust has been changed 12 times since it was created. Domain Controllers take care of this for us. You can read more about that here: TDO password changes.
Take note that, although this lab environment shows the same version in each domain, and the same times, they may not always be the same. If the versions are not the same between the two domains, this is not a cause for concern. Time stamps between the domains will be nearly identical when DCs change the secret as described in the article linked in the previous paragraph. If done manually, discussed later in this blog, the time stamp differences will be larger.
You can read more about where the secrets are stored here, 6.1.6.9.1 trustAuthInfo Attributes. These values AuthenticationInformation and PreviousAuthenticationInformation are not exposed anywhere in the GUI or with any tools. NO HUMANS ALLOWED!
Queue Terminator music
Breaking the Lab
How are we going to break this environment and generate the error?
We will shut down EUDC01and then change the trust secret. We’re going to change it twice to be sure that the value stored in PreviousAuthenticationInformation is new and unknown to EUDC01. After the trust secret has been changed, we’ll shut down EUDC02, and bring EUDC01 back up. EUDC01 will have the old secret(s) and DCs in the parent domain will have the new secret(s).
Change the Secret
To change the secret on the trust, we’ll going to use the netdom command.
Logged on as a domain admin in the parent, run the following command:
netdom trust contoso.com /domain:eu.contoso.com /resetOneSide /passwordT:RedBoat51 /userO:administrator /passwordO:*
And now, on a child DC, logged on as a domain admin in EU:
netdom trust eu.contoso.com /domain:contoso.com /resetOneSide /passwordT:RedBoat51 /userO:administrator /passwordO:*
Notice that the parameter ‘PasswordT’ has the same value in both domains … the shared secret. We’ll give a moment for replication to occur and then do these commands again, with a different ‘passwordT’ value.
Prove we changed it twice
Now that we’ve changed the secret a couple of times, let’s check the version of trustAuthIncoming and trustAuthOutgoing. We’ll use repadmin /showobjmeta like we did before.
Here’s for the parent:
/showobjmeta against DC rootdc01.contoso.com
Org.Time/Date Ver Attribute
============= === =========
2024-01-03 18:20:35 15 trustAuthIncoming
2024-01-03 18:20:35 15 trustAuthOutgoing
/showobjmeta against DC rootdc02.contoso.com
Org.Time/Date Ver Attribute
============= === =========
2024-01-03 18:20:35 15 trustAuthIncoming
2024-01-03 18:20:35 15 trustAuthOutgoing
And for the child domain, EU:
/showobjmeta against DC eudc02.eu.contoso.com
Org.Time/Date Ver Attribute
============= === =========
2024-01-03 18:20:53 15 trustAuthIncoming
2024-01-03 18:20:53 15 trustAuthOutgoing
Notice that we only have one DC in EU, remember, EUDC01 is currently down.
And last, we will shut down EUDC02 and bring EUDC01 back up. We must make sure that EUDC02 is all the way down before we start EUDC01. If we don’t, EUDC01 will replicate from EUDC02 and get the new secrets.
Bad Scenario
After all that, the domain trust is now broken, and we should be able to generate a KRB_AP_ERR_BAD_INTEGRITY when we have our user jondoe attempt to get to the file share \\eufs.eu.contoso.com\eu_share. Let’s take a look …
We have an entire crop of KRB_AP_ERR_BAD_INTEGRITY! Enough to make Kerberos salad!
Kerberos Salad by AI – Looks crunchy.
We can ignore frames, 46, 48, 1068 and 1071. These are the result of background group policy processing.
Using our good scenario as a guide, we can see in frames 1048, 1051, 1058 and 1059 our typical pattern for getting a TGT in our domain.
Frames 1291 and 1293, we’re trying to get a TGS for the server eufs.eu.contoso.com from an NA DC, and we get the expected referral ticket:
Then, in frames 1302 and 1307, we ask a DC in the root domain for a ticket for the EU domain and get the expected response:
Note the bit in red here. This is ciphered with a secret that is newer than the secret on EUDC01. In our next frames, 1382 and 1384, we get the error:
How to fix it?
Fixing this lab environment is very easy, we know exactly how we got here. The simplest method would be to boot up EUDC02 and allow the EU DCs to replicate – but that would be cheating, right?
Not Cheating …
Ok – Let’s see if we can fix this lab without cheating by not booting up EUDC02 and allowing replication in the EU domain to do its thing. Instead, we’ll use the netdom command, as we had done before, setting a new secret between CONTOSO and EU while only EUDC01 is up and running.
So, what’s going to happen when we change the TDO secret on EUDC01 and then we eventually bring EUDC02 backup, which version will win? Hmmm, who knows, let’s find out!
Here’s the status of the TDOs in each domain before booting EUDC02:
contoso.com
/showobjmeta against DC rootdc01.contoso.com
Org.Time/Date Ver Attribute
============= === =========
2024-01-04 12:52:35 18 trustAuthIncoming
2024-01-04 12:52:35 18 trustAuthOutgoing
/showobjmeta against DC rootdc02.contoso.com
Org.Time/Date Ver Attribute
============= === =========
2024-01-04 12:52:35 18 trustAuthIncoming
2024-01-04 12:52:35 18 trustAuthOutgoing
eu.contoso.com
/showobjmeta against DC eudc01.eu.contoso.com
Org.Time/Date Ver Attribute
============= === =========
2024-01-04 12:35:33 19 trustAuthIncoming
2024-01-04 12:35:33 19 trustAuthOutgoing
Note that EUDC02 is down right now, so we can’t see what the version of the attributes are – When I broke the lab, the version was at 21 on EUDC02 before it was shut down.
Those readers paying close attention will note that earlier in this doc, the values were 15 … how did we get to 19 and 21? Reason is, during the writing of the doc, I also use this lab to do customer support, and had to fix it so I could use it … interrupting my experiments, then I had to break it again :p
Next step is to change the secret on the TDOs for CONTOSO and EU.
For CONTOSO we’ll run this command on ROOTDC01:
netdom trust contoso.com /domain:eu.contoso.com /resetOneSide /passwordT:GoldBird78 /userO:ghost /passwordO:*
For EU we’ll run this command on EUDC01:
netdom trust eu.contoso.com /domain:contoso.com /resetOneSide /passwordT:GoldBird78 /userO:ghost /passwordO:*
Now when we look at the version of our attributes trustAuthIncoming and trustAuthOutgoing we have version 20 on EUDC01 and version 19 on the DCs in the root domain, contoso.com.
At this point, when we test na\jondoe going to the share in EU, we get all the expected Kerberos referrals and tickets. Since we’ve already examined that network traffic, we’ll not cover it again.
So … we should be all fixed, right? Let’s boot up EUDC02 and see what happens …
Tick, Tock, Tick, Tock …
… uh … broken!?
The version of our attributes are now 21, on both EU DCs! But I didn’t do anything (innocent halo)! Something else must have changed it!
That something was AD replication. Recall, when we used netdom to change the TDO secrets on EUDC02, when we broke the lab, we did it twice. So EUDC02 had version 21, and the downed EUDC01 had version 19. Then, when we fixed the trust with netdom and changed the secret on the on EUDC01 we got version 20. … 20 is less than 21. EUDC02 won replication.
So even though we got the lab to work again, and fixed our trust problem, soon as we booted EUDC02, the problem came back! It had a higher version for the attribute, so the TDO secret on EUDC02 won replication and broke the trust again!
Using the Domains and Trust snap-in, on the PDC in EU, (EUDC01) I went and validated the Outgoing trust to CONTOSO, and when prompted, entered domain admin credentials to validate the other side of the trust. I was then prompted to reset the password because the trust could not be validated. I chose yes, and then everything started to work again. Yay!
Using the Domain and Trusts snap-in is no different than had I issued the netdom command on both sides of the trust a couple more times before restarting EUDC02. Had I done it again, that also would have worked as EUDC01 would have had version 22 and won replication when EUDC02 was brought back with version 21. (In a case where both DCs have the same version, the one with the most recent time stamp would win replication. If by chance they had the same time stamp, the InvocationID would have been used to determine the winner. See the section titled Multi-Master Replication for more on that.)
The lesson here is, when dealing with this and trying to change trust secrets, you may need to change the secret more than once if DCs are being bounced up and down and/or replication is acting up – Best bet is, change it twice as it won't hurt anything. Just allow enough time for domain wide replication to occur between each change.
How does any of this help me, you ask.
What this document can’t tell you is how to fix this in your environment, it is likely more complex than this lab environment. But we know one thing for sure – this error is generated because the secret used to encipher the ticket is not the same secret being used to decipher the ticket. Who has the out of sync secrets and why will need to be investigated and corrected.
When trouble shooting, get a network trace from the client and look for the KRB_AP_ERR_BAD_INTEGRITY error. Keep in mind who the client is. For things like ADFS, IIS, SQL, Exchange and other server applications, the client may be the server-side application and not the end user’s computer.
Once in this state, there are some common symptoms you may notice:
- Users are being prompted for passwords over and over.
- Logon failures.
- Trust validation failures.
- Active Directory Replication problems, particularly between Global Catalog servers in different domains.
- Clearing the client Kerberos cache fixes the problem, perhaps temporarily or permanently, (klist purge).
Note you can have these exact same symptoms for reasons other than the TDOs not sharing the same secret. If you’re not seeing the KRB_AP_ERR_BAD_INTEGRITY error, the problem is likely something else.
If you do find it …
Run the repadmin /showobjmeta command as we did earlier in this document. Check and see if the version values (the metadata) for the attributes trustAuthIncoming and trustAuthOutgoing on the TDO are consistent on all DCs in the domain. Do this for both the domain that created the referral ticket and the domain receiving the referral ticket. Do not worry if the versions are different in the different domains. Remember, the TDO is stored on the domain partition of each domain and are different objects in their respective domains. They can have different version values in different domains. What’s important is that all the DCs in each domain have the same version for trustAuthIncoming and trustAuthOutgoing and that both domains have the same secret.
When you run repadmin /showobjmeta, you may discover that some DCs in the domain cannot be contacted. If that is the case, go to that DC and run the command. You may very well find that the version is different. It could be that a network connectivity failure (or something else) is at the root of the problem. If DCs can’t replicate properly, you may have one or more DCs with a different secret.
If all the versions are correct across DCs in the respective domains, then we can conclude that the secret is not the same in both domains. If this is the case, use the netdom command, as done previously in this doc, or the Domains and Trusts snap-in, to reset the secret. You may need to do it more than once. If you do it more than once, give a moment or two between changes to allow AD replication to do its thing. You could remove and recreate the trusts, but that incurs more risk and expense compared to using netdmon or the Domains and Trusts snap-in.
Conclusion
In my experience, this error most often occurs in complex environments that are undergoing significant change. Such as DCs being replaced, upgraded or moved around. Also, authoritative restores from old backups or snapshots. Network changes such as topology, new equipment, security software, etc, can also lead to this, as these changes can break replication. And lastly and most unfortunate, large complex environments that are not being properly maintained.
Since we know that the secrets on TDOs are changed automatically once every 30 days (ref: TDO password changes) and we know we keep the previous secret, a change that happened 60 days ago could be why the TDO secrets are out of sync.
References
How trust relationships work for forests in Active Directory
https://learn.microsoft.com/en-us/entra/identity/domain-services/concepts-forest-trust
TDO password changes
Active Directory Forest Recovery - Reset a trust password on one side of the trust
3.3.5.7.5 Cross-Domain Trust and Referrals
TrustedDomain Object
https://learn.microsoft.com/en-us/windows/win32/secmgmt/trusteddomain-object
6.1.6.7 Essential Attributes of a Trusted Domain Object
6.1.6.9.1 trustAuthInfo Attributes
Detailed Concepts: Secure Channel Explained
Service principal names
https://learn.microsoft.com/en-us/windows/win32/ad/service-principal-names
Machine Account Password Process
4768(S, F): A Kerberos authentication ticket (TGT) was requested
https://learn.microsoft.com/en-us/windows/security/threat-protection/auditing/event-4768
How Active Directory Replication Works (Section: Multi-Master Replication)
Active Directory replication troubleshooting guidance
Internet Engineering Task Force (IETF) Link:
Provides a good walk through of how Kerberos Referrals work – See section 8, “Server Referrals”.
Kerberos Principal Name Canonicalization and KDC-Generated Cross-Realm Referrals
https://datatracker.ietf.org/doc/id/draft-ietf-krb-wg-kerberos-referrals-12.html