Windows Server Broken Update Fix

Updates can break on Windows Server for a number of reasons, so this is not an all encompassing fix. But over the years I have seen this occasionally. What happens is that the server does not update and when you attempt a manual update via the Windows Updates setting screen, the updates never download: the downloads are typically stuck at 0% complete. This is apparently due to a corruption of the files within the “C:\Windows\SoftwareDistribution” folder. Because the update process is hung, the update service also appears to be hung up so the process to repair the problem is slightly more involved that one would expect.

First, open the services control applet:

-Click the "Start" Button
-Type "services.msc"
-Set Windows Update to "disabled"
-Reboot the server

Once the server has re-started, verify that the Windows Update service is not running:

-Click the "Start" Button
-Type "services.msc"

Rename the SoftwareDistribution folder:

Rename C:\Windows\SoftwareDistribution to SoftwareDistribution.old

Reset the update service:

-Click the "Start" Button
-Type "services.msc"
-Set windows update to "manual"
-Start the service

Now manually run the updates and make sure they are downloaded and installed. Then perform the required server reboot.

Ubuntu Server: Google Authenticator Configuration

Before installing and configuring the Google Authenticator on a virtual server you need to check and do a few few things to avoid getting locked out of the server. I have seen this grub configuration issue on virtual Ubuntu 18.04, 20.04, and 22.04 server installs on both HyperV and vSphere infrastructures. Also, make sure that you have multiple ssh terminals open prior to restarting sshd in the event the configuration is incorrect.

In the examples below I am only targeting admin accounts with sudo privileges – normal user accounts can log in without being prompted for a Google Authenticator code. Users logging in at the terminal will not be prompted for the authentication code. This configuration only affects ssh logins.

Grub Recovery Issue:

Prior to google authenticator setup you should check to make sure that the grub bootup menu is displayed long enough for you to capture it in the event that you need to do an emergency recovery.  Doom on you if you don’t change this behavior in the grub configuration.

vim /etc/default/grub

GRUB_DEFAULT=0
#GRUB_TIMEOUT_STYLE=hidden
GRUB_TIMEOUT=20
update-grub

Reboot and verify the grub screen appears in the terminal window for a suitable amount of time.

Google Authenticator Install and Configuration – as root:

As a security suggestion – open multiple ssh sessions to avoid possible lockout during configuration.

A second suggestion – Users should not be added to the group (i.e. sudo) until after doing the user authenticator setup if they will be doing the setup using SSH. User setup of the authenticator from the local terminal will not matter since this configuration only affects ssh logins.

Check for users in certain group, in this case “sudo”:

getent group sudo

Install Gooogle Authenticator:

apt-get install libpam-google-authenticator

Edit and add to the bottom of the file (the order of these two lines matters):

vim /etc/pam.d/sshd

# Google authenticator - only for users in sudo group
auth [success=done default=ignore] pam_succeed_if.so user notingroup sudo
# Google authenticator - prompt for code
auth required pam_google_authenticator.so

Edit the sshd config file:

vim /etc/ssh/sshd_config

# For google authenticator
# Enable challenge-response passwords
ChallengeResponseAuthentication yes

Restart ssh daemon:

systemctl restart sshd

Client Setup – As normal user:

As a normal user in the terminal:

google-authenticator

Answer the following questions:

Do you want authentication tokens to be time-based (y/n) y

In the phone app, add key or use QR code. Then enter the code from the app:

Enter code from app (-1 to skip): <code>

Copy emergency one-time use codes to a safe file.

Do you want me to update your "/home/shawn/.google_authenticator" file? (y/n) y
Do you want to disallow multiple uses of the same authentication
token? This restricts you to one login about every 30s, but it increases
your chances to notice or even prevent man-in-the-middle attacks (y/n) y
By default, a new token is generated every 30 seconds by the mobile app.
In order to compensate for possible time-skew between the client and the server,
we allow an extra token before and after the current time. This allows for a
time skew of up to 30 seconds between authentication server and client. If you
experience problems with poor time synchronization, you can increase the window
from its default size of 3 permitted codes (one previous code, the current
code, the next code) to 17 permitted codes (the 8 previous codes, the current
code, and the 8 next codes). This will permit for a time skew of up to 4 minutes
between client and server.
Do you want to do so? (y/n) n
If the computer that you are logging into isn't hardened against brute-force
login attempts, you can enable rate-limiting for the authentication module.
By default, this limits attackers to no more than 3 login attempts every 30s.
Do you want to enable rate-limiting? (y/n) y

At this point you are done. Remember to add/re-add users back to the sudo group if you removed them.

Microsoft PKI – Decommission CA Server

Abbreviated mostly  from: https://docs.microsoft.com/en-us/troubleshoot/windows-server/windows-security/decommission-enterprise-certification-authority-and-remove-objects

Several notes:

These instructions assume that all issued certificates are expired.

If the issued certificates still exist and are active within the expire time or are revoked, you will need to go through a set of steps to revoke the remaining issued certificates and update/transfer the revocation list to the current PKI systems. Generally you can just remove the certificate templates from the old server and let the certificates expire or be re-issued by the newer issuing servers. It all depends on how quickly the old PKI server needs to be removed from inventory.

Steps are different on a single-tier PKI set up, so don't do this unless you are sure!

Get and pair the server name and CA name. This will show all info about all CA servers, so choose the server name wisely. From PowerShell or CMD terminal as admin:

certutil

Shutdown certificate services on the server being decommissioned:

certutil -shutdown

Delete CA private key using the name gathered from above (I have had issues with this step and probably is not important if you securely erase the drive immediately after server decommissioning):

certutil -delkey <"ca name">

Uninstall Certificate Services:

"Server Manager", "Manage", "Remove Roles and Features"
Click through wizard and de-select "Active Directory Certificate Services"

Restart the server.

ONLY do the following if issued objects like CRL and AIA no longer matter! DO NOT remove the Certificate Templates if you are replacing this PKI server!

Remove CA Objects from AD:

"Server Manager", "Tools", "Active Directory Sites and Services"

Select appropriate icon in left window pane, "View", "Show Services Node"

"Services", "Public Key Services"

Under the "AIA", "CDP", "Certification Authorities", and "Enrollment Services":

Delete the CA object
NOTE: should already be missing under "Enrollment Services" because it is removed during the removal of certificate services.

There are more advanced cleanup procedures in the link at the top if you feel that these deletion steps did not work properly.

There is also a method to recover the certificate templates if they are deleted.