Microsoft has given us countless different products and services over the last few decades. Undoubtedly, we know the important role they have played for individuals and businesses on the journey towards utilizing technology to better our lives. When looking at the Microsoft ecosystem, one of the best examples of undeniable excellence is the Microsoft Intune family of products.

With this product, Microsoft gives you an integrated solution that enables you to seamlessly manage all your devices. Central to this solution is the single console that we know as the Microsoft Intune admin center which brings together Intune and Configuration Manager.

The capabilities of this platform can offer your organization a premium device-management solution like no other. The area that we want to focus on today will have to do with the troubleshooting of tenant attach and device actions.

Comparing Tenant Attach to Co-management

For a lot of people, it may be difficult to distinguish between these two. So, let’s start by looking at where these two options differ. Both of these make up two of the three primary cloud attach features. For a Configuration Manager environment to be cloud-attached, it has to use at least one of the three primary cloud attach features.

Tenant Attach gives you the advantage of having your device records in the cloud as well as the ability to take actions on these devices from the cloud-based console. IT admins can perform remote actions on tenant-attached devices such as sync machine policy, sync user policy, run scripts on clients, deploy applications, and much more.

Furthermore, users will be able to manage endpoint security for the attached devices from the Intune admin center for both Windows Servers and Client devices.

On the other hand, Co-management is going to modify your on-premises Configuration Manager environment without the need for a significant migration effort.

One of the more attractive features of Co-management is how it allows you to easily switch workloads such as compliance policies from SCCM to Intune. By enabling you to manage workloads from the cloud, your organization gets to benefit from a simpler device management experience.

Tenant Attach prerequisites

To make use of Tenant Attach, you will need to meet the following requirements:

• When you decide to apply this onboarding change, you’ll need to sign in using an account that is a Global Administrator.
• An Azure cloud environment.
• With version 2107, United States Government customers will be able to use tenant attach features in the US Government cloud such as account onboarding, tenant sync to Intune, device sync to Intune, and device actions in the Microsoft Intune admin center.
• The Azure tenant and the service connection point must have the same geographic location.
• To access the Microsoft Intune admin center, there needs to be at least one Intune license for the administrator.
• The administration service in Configuration Manager needs to be functional.
• If your central administration site has a remote provider, you need to follow the instructions for the CAS has a remote provider scenario in the CMPivot article.

PERMISSIONS

In addition to the above, there will also be a few requirements for the user accounts performing device actions and these include:

• The user account should be a synced user object in Microsoft Entra ID (hybrid identity).
• The Initiate Configuration Manager action permission under Remote tasks in the Microsoft Intune admin center.

The troubleshooting process

Now and again, all of us will encounter issues with the products and services we use. That’s why it’s important to work with providers that offer great support services so that we avoid downtime. Microsoft puts a lot of effort into ensuring that customers get as much support as they need for the various products in its ecosystem. Understandably, tenant attach is no different.

At this point, most admins should be aware that you can sync Configuration Manager clients to the Microsoft Intune admin center. And from that admin center, some client actions can run on the synchronized clients.

These available actions include the sync user policy, sync machine policy, and app evaluation cycle. After an administrator runs an action from the Microsoft Intune admin center, the notification request will forward to the Configuration Manager site. And from there, it will forward to the client.

LOG FILES

The logs you need to use will be found on the service connection point and these are:

• CMGatewaySyncUploadWorker.log
• CMGatewayNotificationWorker.log

You should also use the logs located on the management point:

• BgbServer.log

Lastly, there are other logs that will be found on the client:

• CcmNotificationAgent.log

Review your upload

You’ll need to follow the steps given below:

• Open CMGatewaySyncUploadWorker.log from <ConfigMgr install directory>\Logs.
• You will see the next sync time recorded in a log entry similar to this format Next run time will be at approximately: 02/28/2024 10:15:30.
• The log entries that you should look at for device uploads should be something like this Batching N Records. In this instance, N represents the number of changed devices uploaded since the last upload.
• Admins should expect uploads every 15 minutes for changes. However, once that is done, they will probably need to wait up to 10 minutes to view the client changes appearing in the Microsoft Intune admin center.

Configuration Manager components and log flow

SMS_SERVICE_CONNECTOR: Will utilize the Gateway Notification Worker to process the notification from the Microsoft Intune admin center.

SMS_NOTIFICATION_SERVER: Receives the notification and subsequently creates a client notification.

BgbAgent: The client gets the task and runs the requested action.

SMS SERVICE CONNECTOR

Following the initiation of an action from the Microsoft Intune admin center, the CMGatewayNotificationWorker.log will process the request.

Received new notification. Validating basic notification details…

Validating device action message content…

Authorized to perform client action. TemplateID: RequestMachinePolicy TenantId: a1b2c3a1-b2c3-d4a1-b2c3-d4a1b2c3a1b2 AADUserID:     a1b2c3a1-b2c3-d4a1-b2c3-d4a1b2c3a1b2

Forwarded BGB remote task. TemplateID: 1 TaskGuid: a43dd1b3-a006-4604-b012-5529380b3b6f TaskParam: TargetDeviceIDs: 1

A notification is received from the Microsoft Intune admin center.

Received new notification. Validating basic notification details..

Validation of user and device actions is carried out.

Validating device action message content…

Authorized to perform client action. TemplateID: RequestMachinePolicy TenantId: a1b2c3a1-b2c3-d4a1-b2c3-d4a1b2c3a1b2 AADUserID:     a1b2c3a1-b2c3-d4a1-b2c3-d4a1b2c3a1b2

Forwarding of the remote task to the SMS NOTIFICATION SERVER.

Forwarded BGB remote task. TemplateID: 1 TaskGuid: a43dd1b3-a006-4604-b012-5529380b3b6f TaskParam: TargetDeviceIDs: 1

SMS NOTIFICATION SERVER

At this point, you should be expecting a task to be sent from the management point to the corresponding client as soon as the message has been forwarded to the SMS NOTIFICATION SERVER. In the BgbServer.log, which is on the management point, you should see the following:

Get one push message from database.

Starting to send push task (PushID: 7 TaskID: 8 TaskGUID: A43DD1B3-A006-4604-B012-5529380B3B6F TaskType: 1 TaskParam: ) to 1 clients  with throttling (strategy: 1 param: 42)

BgbAgent

The last step will occur on the client and you can view it in the CcmNotificationAgent.log. As soon as the task has been received, it will then request the scheduler to carry out the action. And once the action has been carried out, a confirmation message will appear:

Receive task from server with pushid=7, taskid=8, taskguid=A43DD1B3-A006-4604-B012-5529380B3B6F, tasktype=1 and taskParam=

Send Task response message <BgbResponseMessage TimeStamp=”2020-01-21T15:43:43Z”><PushID>8</PushID><TaskID>9</TaskID><ReturnCode>1</ReturnCode></BgbResponseMessage> successfully.

Common issues

In this section, we’ll take a look at some of the issues that admins may often encounter.

Unauthorized to perform client action

For whatever reason, there may be situations where administrators may not have the required permissions in Configuration Manager. If that happens, you’ll see an Unauthorized response in the CMGatewayNotificationWorker.log.

Received new notification. Validating basic notification details..

Validating device action message content…

Unauthorized to perform client action. TemplateID: RequestMachinePolicy TenantId: a1b2c3a1-b2c3-d4a1-b2c3-d4a1b2c3a1b2 AADUserID: 3a1e89e6-e190-4615-9d38-a208b0eb1c78

Administrators should verify that the user running the action from the Microsoft Intune admin center has all the necessary permissions on the Configuration Manager site.

Known issues

Data synchronization failures

When there are issues with the hierarchy onboarding configuration, you may end up facing challenges with viewing the tenant attach details in the Microsoft Intune admin center. This could potentially happen in situations where onboarding a hierarchy that has already been onboarded occurs. However, you may also detect this issue from entries in the GenericUploadWorker.log and CMGatewayNotificationWorker.log files.

Workaround for data synchronization failures

Resetting the tenant attach configuration will require you to follow the steps below:

• Offboard the hierarchy. If the tenant attach is already enabled, you need to disable device upload and offboard by editing the co-management properties.
• In the Configuration Manager admin console, go to Administration > Overview > Cloud Services > Cloud Attach. (Select the Co-management node if it is version 2103 and earlier)
• In the ribbon, select Properties for your co-management production policy.
• Next, you need to remove the Upload to Microsoft Endpoint Manager admin center selection from the Configure upload tab.
• Once everything’s completed, select Apply.

You need to give the service about 2 hours to clean up the existing record. Once the above has been completed, you can onboard the hierarchy again.

Example errors in log files that require resetting the tenant attach configuration

Errors for AccountOnboardingInfo and DevicePost requests in GenericUploadWorker.log

[GetNotifications] Response from https://us.gateway.configmgr.manage.microsoft.com/api/gateway/Notification is: 401 (Unauthorized)

Response status code: 401 (Unauthorized) Activity ID: 4c536a72-fd7f-4d08-948a-3e65d2129e44

Web exception when getting new notification

Exception details:

[Warning][CMGatewayNotificationWorker][0][System.Net.WebException][0x80131509]

The remote server returned an error: (401) Unauthorized.    at Microsoft.ConfigurationManager.ServiceConnector.ExtensionMethods.<GetResponseAsync>d__13.MoveNext()

Response in the web exception: {“Message”:”An error has occurred.”}

Errors for device actions in CMGatewayNotificationWorker.log

[GetNotifications] Response from https://us.gateway.configmgr.manage.microsoft.com/api/gateway/Notification is: 401 (Unauthorized)

Response status code: 401 (Unauthorized) Activity ID: 4c536a72-fd7f-4d08-948a-3e65d2129e44

Web exception when getting new notification

Exception details:

[Warning][CMGatewayNotificationWorker][0][System.Net.WebException][0x80131509]

The remote server returned an error: (401) Unauthorized.    at Microsoft.ConfigurationManager.ServiceConnector.ExtensionMethods.<GetResponseAsync>d__13.MoveNext()

Response in the web exception: {“Message”:”An error has occurred.”}

Specific devices don’t synchronize

Another issue that you may need to deal with has to do with specific devices, which also happen to be Configuration Manager clients, not being uploaded to the service. So, what devices should you expect to be potentially affected by this issue?

In scenarios where a device is a distribution point that uses the same PKI certificate for both the distribution point functionality and its client agent, then the device won’t be included in the tenant attach device sync.

Furthermore, administrators should look out for the behavior that is typical in such instances. During the on-boarding phase as you are going through the tenant attach process, a full sync will be carried out the first time.

You should be aware that any other sync cycles after this one will be delta synchronizations. If the impacted devices are updated in any way, that update will result in the device being removed from the sync.

When the Configuration Manager site is configured to require multi-factor authentication, most tenant attach features don’t work

More troubleshooting

If you find yourself in a situation where the SMS provider machine that communicates with the service connection point is configured to use multi-factor authentication, you won’t be able to install applications, run CMPivot queries, and perform other actions from the admin console.

Instead, you will get a 403 error code, forbidden. What you would normally do to address this is to configure the on-premises hierarchy to the default authentication level of Windows authentication.

The platform is designed to enable you to determine what the minimum authentication level should be for admins to access Configuration Manager sites. You should view this as a great feature for enhancing security because it means that admins have to sign in to Windows with the appropriate level before they can access Configuration Manager.

Authentication

Furthermore, this applies to all components that access the SMS provider. Configuration Manager supports a handful of authentication levels and these are as follows:

• Windows authentication: Authentication with Active Directory domain credentials is necessary. Note that this setting represents the previous behavior, as well as the current default setting.
• Certificate authentication. Authentication with a valid certificate that has been issued by a trusted PKI certificate authority is necessary. You also need to know that you don’t configure this certificate in Configuration Manager. Configuration Manager requires the admin to be signed into Windows using PKI.
• Windows Hello for Business authentication: In this case, you need a strong two-factor authentication that’s linked to a device and also uses a PIN or biometrics. Before choosing this particular setting, you need to note that the SMS Provider and administration service will require the user’s authentication token to contain a multi-factor authentication (MFA) claim from Windows Hello for Business. In simple terms, all this means is that users of the console, SDK, PowerShell, or administration service are required to authenticate to Windows with their Windows Hello for Business PIN or biometric. If not done this way, the site rejects the user’s action. Another key thing to also remember is that this behavior is specific to Windows Hello for Business and does not apply to Windows Hello.

What to do when Configuration Manager components for tenant attach fail to connect to the backend cloud service

Another issue that some have encountered concerns the failure of Configuration Manager components for tenant attach to connect to the backend cloud service when you run a client action from the Microsoft Intune admin center. In those instances, you may see the error given below:

Failed to check and load service signing certificate. System.ArgumentException: Mismatch certificate subject name

According to the information available from Microsoft, this problem occurs in versions earlier than the Configuration Manager version 2203 hotfix rollup after a change in public certificates on July 27, 2022. The reason for this has to do with the change that came about in public certificates on July 27, 2022, where OU=Microsoft Corporation was removed from the public certificate.

Even though this change was carried out, the configuration manager database still retained the old subject name and this then caused the load check failure. Below are some example entries in the CMGatewayNotificationWorker.log file in the top-level site in the hierarchy:

Error occured when process notification with notification Id <notification Id>. Ignore the notification. SMS_SERVICE_CONNECTOR_CMGatewayNotificationWorker

Exception details: SMS_SERVICE_CONNECTOR_CMGatewayNotificationWorker

[Warning][CMGatewayNotificationWorker][0][System.IO.InvalidDataException][0x80131501]

Failed to check and load service signing certificate. System.ArgumentException: Mismatch certificate subject name at Microsoft.ConfigurationManager.ManagedBase.CertificateUtility.ServiceCertificateUtility.VerifyCertificate(X509Certificate2 certificate, Boolean crlCheck, X509Chain& certificateChain, X509Certificate2Collection extraStore)

and at Microsoft.ConfigurationManager.ManagedBase.CertificateUtility.ServiceCertificateUtility.Reload()

at Microsoft.ConfigurationManager.ManagedBase.CertificateUtility.ServiceCertificateUtility.Exists(String thumbprint)

and at Microsoft.ConfigurationManager.ServiceConnector.AccountOnboardingWorker.\<RefreshServiceSigningCertificateIfNotExistsAsync>d__19.MoveNext()

ADDRESSING THE ISSUE

To address this challenge, there are a couple of methods that you can employ. The first thing you can do is to install the Configuration Manager version 2203 hotfix rollup if you happen to be running Configuration Manager version 2203.

However, if your version of Configuration Manager is a previously supported one, you’ll first need to upgrade to Configuration Manager version 2203. After doing that, you can proceed with the installation of the Configuration Manager version 2203 hotfix rollup.

In the update rollup for Microsoft Endpoint Configuration Manager current branch, version 2203, the following issues have been addressed:

• Application approvals through email not working with a cloud management gateway due to a missing Microsoft Entra token.
• Metadata revisions to previously published metadata-only updates not being synchronized to Windows Server Update Services (WSUS) as expected.
• If a window happens to be left open for a few minutes, the task Sequence Editor running on Windows Server 2022 would fail to apply changes to a task sequence. After this happens, you would see the following message:

Error connecting to provider, smsprov.log may show more details.

• In situations where the Client checking status frequency (minutes) value has been set below 60, the BitLocker compliance status will be temporarily inaccurate.
• Admins have also experienced the incorrect removal of some users and their group memberships by the SMS_AZUREAD_DISCOVERY_AGENT thread of the SMA_Executive service in cases when the site server is configured with a non-US English locale. You’ll have have noticed the removals occurring when the discovery cycle runs after the 12th day of the month. Errors will be recorded in the SMS_AZUREAD_DISCOVERY_AGENT.log file at various times during the discovery cycle and they will be similar to the following:

1. ERROR: Encountered SqlException The conversion of a nvarchar data type to a datetime data type resulted in an out-of-range value.

2. ERROR: Exception message: [The conversion of a nvarchar data type to a datetime data type resulted in an out-of-range value.]

3. ERROR: Group full sync request failed. Exception: System.NullReferenceException: Object reference not set to an instance of an object.

More troubleshooting

• When adding a CMPivot query as a favorite, it is split into two lines and characters are removed.
• When searching Software Update Groups in the Configuration Manager console, the Name criteria is not an option.
• Instead of the value you may have previously noticed, the Browse button for Content location in the properties for a deployment would return an empty location.
• The implicit uninstall setting won’t be respected by an application that is targeted to a device collection but deployed in the context of the user.
• Typing a Name value in the Create Orchestration Group wizard occurs at a below normal speed.
• A misleading error message (false negative) is generated on a target distribution point that is co-located with a site server. You could expect to see this during content distribution from a parent site to a child site. In the distmgr.log, the false negative will appear in this format:

~RDC:Failed to set access security on \\<SiteServerFQDN>\SMSSIG$\<PkgID>.1.tar for package <PkgID> signature file

Conclusion

In some cases, unfortunately, both options involving installation of the Configuration Manager version 2203 hotfix rollup may not work to fix the above issues. For those situations, you’ll need to open a support ticket with the Configuration Manager support team.

From there, you’ll be able to get the appropriate assistance to resolve the problem in the supported version of Configuration Manager in your environment.