Device loses domain connectivity due to UDP port exhaustion

Symptoms

A Windows Server initially reported as Group Policy failing to update. Further investigation found that the server was unable to resolve the domain name despite having no firewall blocking traffic and having the correct DNS servers configured. Common troubleshooting tools such as Netdom and NSLookup were also failing. Some network traffic was working however.

The server had last restarted 6 months before and the issue had started 2 months after that. For now let’s ignore the elephant in the room of why it had taken 4 months for the issue to be noticed.

Solution

The failing tools all related to UDP traffic.

Running “Netstat –ano –p UDP” showed that all dynamic UDP ports 49152 to 65535 were being used by one process, “Cisco Presence Server Plug-in.exe”. This was UDP port exhaustion. No other service was able to use the UDP dynamic port range which effectively blocked the server from communicating with the domain.

Restarting the server cleared the exhausted ports and the application owner was requested to raise this behaviour with the vendor.

GoToMeeting Launcher error 1392

When running the Go To Meeting test session from here I got an error code of 1392. When opening the log below I saw the following:

08:43:56 2CC8 I: created file ‘C:\Users\username\AppData\Local\Temp\26\B0560F7D-757E-4C65-A168-2364C6F4606F\G2MCoreInstExtractor.exe’

08:43:56 2CC8 E: Installer launch failed; command was: “C:\Users\username\AppData\Local\Temp\26\B0560F7D-757E-4C65-A168-2364C6F4606F\G2MCoreInstExtractor.exe” “/Action Join” “/EGWAddress 216.115.208.230” “/EGWDNS egwglobal.gotomeeting.com” “/EGWPort 80,443,8200” “/MeetingID 333699349” “/Mode Terse” “/Product g2m” “/colClientUiReadyEvent Global\BC84F869-6926-4A19-9926-7111917D85E8” “/sessionTrackingId ul-0396-0714-9b0e-beb9799efb97”
08:43:56 2CC8 E: [installEndpoint] completed: 2001 (ECError::eSomeError), Method “CreateProcess” returned Win32 error [1392]. The file or directory is corrupted and unreadable.

From what I could see, the launcher had two stages. The first was to download
G2MCoreInstExtractor.exe.

08:43:56     2CC8 I: created file 'C:\Users\username\AppData\Local\Temp\26\B0560F7D-757E-4C65-A168-2364C6F4606F\G2MCoreInstExtractor.exe'

The second would launch that to download the remaining files. This is where it appeared to be failing.

 Installer launch failed; command was: "C:\Users\username\AppData\Local\Temp\26\B0560F7D-757E-4C65-A168-2364C6F4606F\G2MCoreInstExtractor.exe"

The issue was not that it was failing to launch but that it was failing to download in the first place. The proxy server had an ‘allow .EXE file downloads’ set for GoToMeeting.com but not for the GetGo Content Distribution Network which hosts the file G2MCoreInstExtractor.exe. Adding the GetGo URLs that host the GoToMeeting file to the allow list resolved this:

getgo.com:443/builds/g2m/
getgocdn.com:443/builds/g2m/
getgo.com/builds/g2m/
getgocdn.com/builds/g2m/

Intune Windows 10 Autopilot deployment profile MENU BUG – Don’t be caught out!

Microsoft recently released two hotly anticipated Intune Preview features:

Self-Deploying Autopilot – Azure AD Joined
User-Driven Autopilot – Hybrid Azure AD Joined

Naturally Microsoft has added logic to the menu selections in the Autopilot Deployment Profiles that greys out incompatible menu options. I have found and reported to Microsoft a bug that allows you to select incompatible options. Doing so causes you Autopilots to fail.

Creating a new policy and selecting Deployment mode: User-Driven Autopilot – gives you two options:

Azure AD Join
Hybrid Azure AD Joined (Preview)

Selecting Self-Deploying (Preview) locks the ‘Join to Azure AD as’ drop down menu on to Azure AD Joined. This is desired because Hybrid Azure AD Join is not available for Self-Deploying mode (yet).

Now the bug…

Save a policy with the User-Driven – Hybrid Azure AD Joined options selected.
Go back in to the policy and select Properties.
Change the Deployment Mode to “Self-Deploying”.
Note that the ‘Join to Azure AD as’ drop down stays greyed out on the invalid option ‘Hybrid Azure AD Joined’ .

The impact…

This is not just cosmetic. By doing this the Autopilot device will fail on the Enrolment Status Page (ESP) with the error: 0x801c03ed.

Windows 10 AutoPilot Error 80004005

Update 7th Feb 2023 - This post was written for an earlier version of Autopilot. The solution in this post does not apply to any versions of Windows 10 that are currently in support.

During Windows AutoPilot in User Driven mode you may see the following error on the Network tab (after Windows has downloaded the AutoPilot profile but before the reboot). It takes about 10-20 minutes for this error to display.

Something went wrong.
Confirm that you are using the correct sign-in information and that your 
organisation uses this feature. You can try to do this again or contact 
your system administrator with the error code 80004005

Check that the device has a clear line of sight to to a Domain Controller. This is a requirement of User-Driven Autopilot.

List all Windows Servers in a Domain using Windows Powershell

Get-ADComputer -Filter * -Properties * | ?{$_.OperatingSystem -like "*Server*"}

What’s happening here?

Get all computers in the domain (filter -*) and all properties (the property we want is not in the filtered properties that you get if you just run Get-ADComputer without -Properties *)
Send those computers through a Where-Object (?{}) filter block.
Filter by the attribute OperatingSystem containing the word ‘Server’

You can pipe this again and select some filtered attributes. For instance, if you just want the name and the location in AD then you run it like this:

Get-ADComputer -Filter * -Properties * | ?{$_.OperatingSystem -like "*Server*"} | select Name, DistinguishedName

Windows Deployment Services in Azure = No

Can SCCM be truly in Azure only? That depends on whether you consider an on premise PXE service as a part of the SCCM infrastructure or not.

WDS is not supported in Azure: https://support.microsoft.com/en-us/help/2721672/microsoft-server-software-support-for-microsoft-azure-virtual-machines

But we all know that ‘not supported’ doesn’t always mean ‘doesn’t work’. There are many things in the Microsoft world that start off as not supported but end up being fully fledged features.

Let’s put it to the test;

In Azure, an SCCM primary site server running a PXE enabled DP with a boot image and Windows 10 image task sequence deployed to to the collection ‘All Unknown Computers’.
On premise, a DHCP server with options 66 and 67 configured. An RRAS running a IKEv2 VPN to Azure.
Any/Any is open on the firewalls between the Azure and on prem subnets.

What we get is…

In the screenshot below:

10.1.0.50 – PXE Boot Client

10.1.0.9 – VPN Gateway

10.0.0.4 – WDS (SCCM)

pxe

The WDSNBP.com file is downloaded using TFTP. But when it tries to get the boot image file it gets ‘No response from Windows Deployment Services server’.

After this the PXE client switches to download the pxeboot.com boot image which is the wrong image. The PXE session is therefore a failure.

There is no firewall block and clearly some traffic is being responded to by the WDS.

Using netmon we get the below trace from the WDS server

azuretrace

The first image, column 1 is the source IP, column 2 is the destination. We can see that the Request is being received by the WDS server and an ACK packet is being sent back to the PXE client. This is repeated.

Below are the ports the traffic is using. One of them is UDP 68:

pxeport

So why doesn’t the PXE client think the WDS server is responding?

Below is the netmon from the on premise RRAS server, remember this is the VPN gateway on prem.

It shows the requests being relayed to Azure, but nothing coming back. Something in Azure is blocking this.

A little googling and this comes up: https://docs.microsoft.com/en-us/azure/virtual-network/virtual-networks-faq

“What protocols can I use within VNets?

You can use TCP, UDP, and ICMP TCP/IP protocols within VNets. Unicast is supported within VNets, with the exception of Dynamic Host Configuration Protocol (DHCP) via Unicast (source port UDP/68 / destination port UDP/67).”

And so there we have it. Azure appears to be actively interfering with UDP port 68 traffic, likely preventing it from being routed down the VPN. So in this scenario, the PXE service needs to be setup on premise.

Only by saying that the PXE service is not a part of the SCCM infrastructure could we, in this scenario, say that SCCM is entirely cloud based.