The icon for Microsoft Edge is now placed by default in every user profile.
It is not placed in Public Desktop, but created for each user at logon (DOH!)

Thank god there is way to stop this behavior.

You can simple add the following registry key:
Key: HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer
Value: DisableEdgeDesktopShortcutCreation
Data: 1
Type: REG_DWORD

If your using MDT (Microsoft Deployment Toolkit) or ConfigMgr (System Center Configuration Manager)
You can add the following oneliner task sequence step, to stop the creation of the Microsoft Edge icon.
Commandline: reg.exe add HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer /v DisableEdgeDesktopShortcutCreation /t REG_DWORD /d 1

In case your wondering what i have in the steps to disable Cortana, let me share them:

Registry tweaks for Build and Capture or Windows 10 Deployment task sequences

Disable Cortana Voice:
reg.exe add HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\OOBE /v DisableVoice /t REG_DWORD /d 1

Disable Cortana Search:
reg add “HKLM\SOFTWARE\Policies\Microsoft\Windows\Windows Search” /v “AllowCortana” /t REG_DWORD /d 0 /f

Disable Cortana Search Box:
reg add “HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Search” /v “SearchboxTaskbarMode” /t REG_DWORD /d 0 /f