Patching EC2 instances

We patch every EC2 instance in UF's AWS accounts every Sunday. University policy requires critical and high vulnerabilities to be fixed within 14 days, and all others within 30. Weekly patching installs updates from the operating system's own repositories. Software you installed some other way, and container images, aren't updated by it and need their own updates.

What you need to do

1. Check that your instances show up in Systems Manager. In the AWS console, open Systems Manager, then Fleet Manager. An instance that isn't listed isn't scanned or patched. See "If an instance isn't in Fleet Manager" below.
2. If two or more instances run the same service outside an Auto Scaling group, tag them so they aren't all rebooted at once. Give all of them the tag ufit:ha-group with the same value, and give half of them ufit:patch-wave = B. The B half is patched only after the other half is back and healthy.
3. If something has to happen before or after patching, such as stopping an application or checking a GPU driver, tag the instance ufit:patch-hooks = true and put executable scripts at /opt/ufit/patch-hooks/pre.sh and /opt/ufit/patch-hooks/post.sh. post.sh runs after patching and after any reboot. A script that exits with an error marks that instance's patching as failed. Hooks work on Linux instances only.
4. If an instance can't be patched every week, open an AWS ticket and tell us why. If we approve an exception, we tag the instance ufit:patch-exception with the ticket number, and patching skips it until the exception ends. Every exception has an end date.
5. For Elastic Beanstalk, turn on managed platform updates in the environment. Patching skips Beanstalk instances because Beanstalk replaces them during its own updates.
6. To see results, open Systems Manager, then Patch Manager, and look at the compliance report.

Schedule

Times follow Eastern time, including daylight saving time. An instance reboots only when an update needs it, for example after a kernel update.

Auto Scaling groups

Instances in an Auto Scaling group are patched one at a time, and the group stops at the first failure. When the group has at least two running instances and its desired capacity is above its minimum, each instance is taken out of the load balancer before it's patched. Otherwise the instance is patched in place, and a reboot interrupts the service as it would on a standalone instance. If any instance in the group's target groups is unhealthy when patching starts, the group is skipped that week.

If an instance isn't in Fleet Manager

Systems Manager needs three things on the instance. The SSM Agent must be installed and running; Amazon Linux, Ubuntu and Windows images from AWS include it. The instance needs outbound HTTPS to Systems Manager, through the internet or VPC endpoints. And instance metadata must require IMDSv2 (in Terraform, http_tokens = "required" in metadata_options). Instances with no IAM role of their own, or whose role doesn't grant Systems Manager permissions, are managed through a default role, and that only works with IMDSv2.