I maintain roughly ~90,000 total devices for a large enterprise which is also a IT provider for three other organisations. All operating across Australia and remotely around the globe. All, which I won't disclose here.
Having so many endpoints to manage can be difficult. But it shouldn't be. We use Microsoft System Center Configuration Manager (SCCM/ConfigMgr), the go to for deployments and management in the Windows ecospace for Medium and Large Enterprises.
Here are some of my opinions and learnings while working in my role.
You should be on Windows 10.
With the recent end of life for Windows 7 you run the risk of being attacked via new and unknown exploits post 14th January 2020. There is little excuse for running an unsupported OS in a production environment for day to day activities.
Windows 10 is the currently supported platform of Windows-as-a-Service. Currently this is updated via versions, normally delivered bi-annually. Each version has a 18 or 30 month (if your on Enterprise for September releases) servicing period on release.
If you're unable to move from Windows 7 then Microsoft do provide Extended Security Updates for Volume Licensed customers for an additional 3 years.
ENCRYPT ALL THE THINGS!
If you don't want anyone in the public viewing something, you should encrypt it. BitLocker which is present in Windows Pro and Enterprise, gives you the native capability of doing this. But you could run into some issues:
- Faulty Hardware
Hardware inventively fails, in many cases I've thrown this to bad hard disks. I'd really suggest confirming S.M.A.R.T. status of the disks. More than a few times I've had this down to a intermittent failing disk or a bad controller.
- Unsupported Hardware
Realistically for BitLocker you should be purchasing devices which include TPM 2.0 modules. I've also encountered some aged hardware which had driver incompatibilities that causes BitLocker to intermittently fail to read the TPM and asks for the recovery key.
Self Encrypting Drives may also be an option for encrypting your SSD's however in 2018 there was a number of manufacturers who incorrectly implemented the OPAL standard which allowed recovery of the decryption key.
While Hardware encryption is more efficient I'd advise to either avoid using it all together and stick with a Software solution like BitLocker, or use hardware and software encryption together. As of writing Microsoft currently recommends to enforce software encryption, as in previous iterations of BitLocker if the disk was able to use hardware encryption, BitLocker defaulted to using it.
Deploy your applications AS APPLICATIONS in ConfigMgr.
A downfall I see is that some folks who package applications do not package them up as applications, ConfigMgr allows you to create revisions and detection rules that allow the install and uninstallation of applications. Many applications come as MSI's which make this easy via GUIDs but even straight .exe's you can detect using files or registry entries to confirm values or simply the existence of them. Sometimes a little extra work saves more work and headaches later on.
Avoid Program Packages where possible unless your dropping content or other limitations. You severely impact the detection capabilities of applications should it be installed this way and don't assume a deployment collection will be your source of truth. I've seen plenty of applications that were dropped out of the user/computer collection, manually installed or the collection was simply deleted.
Service your WIMs!
I've personally come two issues and experiences.
Freshly deployed instances of Windows are missing crucial updates every patch cycle or out of cycle patch released. Meaning whatever WIM you have deployed, wont be patched to date.
"But I run the updates in the Task Sequence, I should be fine!" you exclaim. Wrong.
Not only with extended times to build taking time to download and install MSU packages you also also come across another problem if you also build devices from either: A) Collections with no update deployments, or, B) Build from Unknown.
They simply don't get updates when the step runs as there's no update deployment.
"Ah but I actually already service my WIMs via ConfigMgr, you're concerns are moot." You snarkily respond. Yes, but you're doing it wrong.
Based on recommendations online and even from Microsoft consultants I've worked with. ConfigMgr does not do a good job of injecting patches into a WIM, you may notice your WIMs getting bigger every single time and that's a problem...
If your organisation has Distribution Points in slow bandwidth or restricted hard disk space scenarios this then becomes a problem. At least in Australia's unfortunate broadband situation.
Two solutions can be done here:
This solution has a potential to be automated via PowerShell and you can start to look at automating the copy of WIM to the CAS source location followed up by more PowerShell to interact with ConfigMgr to update the distribution points automatically.
Scripts could be kicked off on a schedule say... 18:00 on a Friday to allow a couple days to ensure all DP's have content by Monday.
This is a GUI front end for OSDeploy. If you're more comfortable with a GUI and manually updating the ConfigMgr content to DPs.
As long as you do these options you can easily maintain your WIM that's essentially up to date.
Don't forget your Office patches!
If you deploy Office 2019/365 via the Office Deployment Toolkit you can also consider scripting this to run at a schedule to download up to date versions, copy to source location and update your DP content. That way it's ready for new builds and deployments with the latest patches.
Consider going towards Co-Managed ConfigMgr with Intune in the future.
ConfigMgr won't be around forever and it seems Microsoft is pushing Enterprises users to more cloud solutions.
The first step in Co-Management is enabling Windows Updates via Microsoft instead of WSUS.
This means you will have Windows do the updates via the CDNs rather than your own WSUS servers. Not only you get your patches quicker, you don't need to worry about other points of failure on the WSUS or ConfigMgr sides.
Did you know if you schedule an standard Automatic Deployment Rule, you're still missing multiple daily Windows Defender updates (roughly a new set of malware detections every 2 hours!).
Worst case you can still enable and use MAPS to grab delta updates for detection rules (I would recommend this for both WSUS and WUfB setups).
If possible move away from Java and Flash.
Java has been used for quite some time but it's a very common surface to exploit via Java on the web. Not to mention public updates for Java 8 (the last to support Java web applets) ended on January 2019. While a subscription can be bought to obtain security updates for Java 8, it's an added cost and long term commitment to keep secure.
I've taken the liberty to strip Java 8 in the fleet by default, whereas previous iterations of our SOE included it. It's now turned into an opt-in process for Java so those web applications that may continue to use it with a stringent domain whitelist.
Similarly Flash has had the same process applied. Stripped from the SOE. Adobe will be ceasing support for Flash by December this year, many browser vendors have ceased their own support already.
You should always seek to look to update your web applications to use HTML5, and if that's not possible it may be a sign to move to another solution that does - its been a long time coming, there's no excuse for not doing so. Both Java and Flash are legacy and end of general life, it's very much time to move on.
As my own general preference, it is to have a minimal SOE to reduce as much application attack surface as possible. This gives you less to maintain and more flexibility but more importantly your fleet is a reduced risk. Sure if you need it make it available but ensure you lock it down, wither it be in bubble virtual desktop infrastructure or whitelisting rules.
That will be all for now.
As I come across more learnings in the future or a practice to suggest I'll seek look to make updates to this post.