RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1861977 - RHSA-2020:3216 grub2 security update renders system unbootable
Summary: RHSA-2020:3216 grub2 security update renders system unbootable
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: shim
Version: 8.2
Hardware: x86_64
OS: Linux
urgent
urgent
Target Milestone: rc
: 8.0
Assignee: Bootloader engineering team
QA Contact: Release Test Team
URL:
Whiteboard:
Depends On:
Blocks: 1862230 1862231 1862232
TreeView+ depends on / blocked
 
Reported: 2020-07-30 05:15 UTC by are-support
Modified: 2023-12-15 18:38 UTC (History)
109 users (show)

Fixed In Version: shim-15-15.el8
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1862230 1862231 1862232 (view as bug list)
Environment:
Last Closed: 2020-11-04 02:06:11 UTC
Type: Bug
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
shimx64.efi (1.18 MB, application/x-ms-dos-executable)
2020-07-30 18:32 UTC, Javier Martinez Canillas
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Bugzilla 1862045 0 urgent CLOSED Grub or Shim dies since updating to grub2-2.02-0.86.el7_8 / shim-x64-15-7.el7_8 2023-10-06 21:18:00 UTC
Red Hat Knowledge Base (Solution) 5272311 0 None None None 2020-07-30 10:51:05 UTC
Red Hat Product Errata RHBA-2020:4574 0 None None None 2020-11-04 02:07:34 UTC

Description are-support 2020-07-30 05:15:35 UTC
Description of problem:
Applying the RHSA-2020:3216 grub2 security update and the RHSA-2020:3218 kernel security and bug fix update on a fresh "minimal" installation of RHEL 8.2 renders the system unbootable.

How reproducible:
Consistently

Steps to Reproduce:
1. Install RHEL 8.2 "minimal" version from Binary DVD iso downloaded on 7/29/2020 on system running in EFI mode
2. Apply current updates as of 7/29/2020 with "yum update"
3. Reboot system

Actual results:
System hangs after POST and the grub menu never loads

Expected results:
System should display kernel version selection menu and then boot to RHEL

Additional info:
Reproducible on several consecutive clean installs of RHEL 8.2 and on one existing RHEL 8.2 system.

Comment 1 Sandro Bonazzola 2020-07-30 06:38:30 UTC
Hit the same issue right now on my work laptop

Comment 3 Wade Mealing 2020-07-30 07:15:32 UTC
You may be able to work around this by disabling secure boot in the BIOS.    Not ideal.. for sure but if you can't see grub.. sounds like something is up in the secure boot chain.

Comment 6 Sandro Bonazzola 2020-07-30 08:33:03 UTC
I booted my system with the installation ISO in troubleshoot mode, chrooted to /mnt/sysimage, regenerated the grub.cfg with grub2-mkconfig and executed grub2-install.
Now I can go past grub but got stuck to emergency shell failing swtching root.

Comment 7 Sandro Bonazzola 2020-07-30 09:22:06 UTC
(In reply to Sandro Bonazzola from comment #6)
> I booted my system with the installation ISO in troubleshoot mode, chrooted
> to /mnt/sysimage, regenerated the grub.cfg with grub2-mkconfig and executed
> grub2-install.
> Now I can go past grub but got stuck to emergency shell failing swtching
> root.

solved this by actually copying the grubenv file to /boot/grub2 instead of relying on the symlink to efi.

Comment 8 Clifford Perry 2020-07-30 09:36:23 UTC
Hi, 
If you are Red Hat customers, please do open a support case with Red Hat Support. This will allow our support engineers to gather debugging data that may help to understand what is happening in these individual situations. 

Thanks,
Cliff

Comment 11 Renaud Métrich 2020-07-30 09:56:46 UTC
See also BZ 1862045 on RHEL7.

Happens on HPE hardware with secure boot disabled.

Solution is to downgrade grub2 / shim / mokutil

Comment 12 Renaud Métrich 2020-07-30 10:21:03 UTC
Seen on HPE ProLiant XL230k Gen10 with secure boot disabled.

Comment 15 Jaroslav Suchanek 2020-07-30 12:09:44 UTC
I hit it as well and resolved by

1) booting to rhel-8 (centos-8) troubleshooting mode
2) chrooted to /mnt/sysimage
3) setup networking
4) yum downgrade shim-x64 grub2\*
5) reboot

Switching boot mode to legacy in bios does not help.

Comment 16 Javier Martinez Canillas 2020-07-30 12:14:02 UTC
(In reply to Jaroslav Suchanek from comment #15)
> I hit it as well and resolved by
> 
> 1) booting to rhel-8 (centos-8) troubleshooting mode
> 2) chrooted to /mnt/sysimage
> 3) setup networking
> 4) yum downgrade shim-x64 grub2\*
> 5) reboot
> 
> Switching boot mode to legacy in bios does not help.

Thanks a lot for the information. Could you please try to downgrade them individually
to confirm which package is causing the issue? Since if I understood correctly the
problem happens when Secure Boot is disabled, that should be possible.

Comment 17 Rohan Maity 2020-07-30 12:42:04 UTC
Solution tells to protect yum from upgrading
I got new laptop from RHEL8.2 have not yum upgrade system. But I might need to run `yum update` in order to install some dependencies

would this solution of protecting yum from upgrading is enough for me ?



Protect yum from upgrading the packages by adding the following entry in /etc/yum.conf
Raw
exclude=grub2* shim* mokutil

Comment 21 Christof Efkemann 2020-07-30 14:24:29 UTC
The problem is in grub2-efi-x64, specifically in the grubx64.efi. You can have everything else up-to-date but use grubx64.efi from the previous RPM, and the system can boot again (assuming that SecureBoot is turned off, of course).

Comment 23 Peter Jones 2020-07-30 14:50:47 UTC
(In reply to Christof Efkemann from comment #21)
> The problem is in grub2-efi-x64, specifically in the grubx64.efi. You can
> have everything else up-to-date but use grubx64.efi from the previous RPM,
> and the system can boot again (assuming that SecureBoot is turned off, of
> course).

Just so I've got clear data - which version is "the previous RPM" for you?

Comment 24 Christof Efkemann 2020-07-30 14:55:53 UTC
(In reply to Peter Jones from comment #23)
> Just so I've got clear data - which version is "the previous RPM" for you?

The previously installed RPM was grub2-efi-x64-1:2.02-81.el8.x86_64

Comment 28 Christof Efkemann 2020-07-30 15:23:31 UTC
One additional observation:
Like I said, we were able to boot the new kernel (4.18.0-193.14.2.el8_2.x86_64) with the grubx64.efi from the 2.02-81.el8.x86_64 RPM.
However, this seems to work only when bypassing shim altogether.

I. e.:
UEFI -> shim -> grubx64.efi (2.02-87) -> *crash*
UEFI -> shim -> grubx64.efi (2.02-81) -> *crash*
UEFI -> grubx64.efi (2.02-87) -> *crash*
UEFI -> grubx64.efi (2.02-81) -> kernel -> *success*

Comment 29 Javier Martinez Canillas 2020-07-30 15:45:38 UTC
(In reply to Christof Efkemann from comment #28)
> One additional observation:
> Like I said, we were able to boot the new kernel
> (4.18.0-193.14.2.el8_2.x86_64) with the grubx64.efi from the
> 2.02-81.el8.x86_64 RPM.
> However, this seems to work only when bypassing shim altogether.
> 
> I. e.:
> UEFI -> shim -> grubx64.efi (2.02-87) -> *crash*
> UEFI -> shim -> grubx64.efi (2.02-81) -> *crash*
> UEFI -> grubx64.efi (2.02-87) -> *crash*
> UEFI -> grubx64.efi (2.02-81) -> kernel -> *success*

This is with shim-x64-15-14.el8_2.x86_64 right? What happens if you downgrade the shim package?

Comment 31 Christof Efkemann 2020-07-30 16:42:01 UTC
(In reply to Javier Martinez Canillas from comment #29)
> (In reply to Christof Efkemann from comment #28)
> > One additional observation:
> > Like I said, we were able to boot the new kernel
> > (4.18.0-193.14.2.el8_2.x86_64) with the grubx64.efi from the
> > 2.02-81.el8.x86_64 RPM.
> > However, this seems to work only when bypassing shim altogether.
> > 
> > I. e.:
> > UEFI -> shim -> grubx64.efi (2.02-87) -> *crash*
> > UEFI -> shim -> grubx64.efi (2.02-81) -> *crash*
> > UEFI -> grubx64.efi (2.02-87) -> *crash*
> > UEFI -> grubx64.efi (2.02-81) -> kernel -> *success*
> 
> This is with shim-x64-15-14.el8_2.x86_64 right? What happens if you
> downgrade the shim package?

No, that's with shim-x64-15-13.el8.x86_64.

However, after writing the last comment, I got some doubts about my previous observations -- being unable to boot grubx64.efi version 2.02-87 without shim resulting in a crash. Turns out, when I tried to reproduce this crash just now, I wasn't able to. So the revised list looks like this:

UEFI -> shim -> grubx64.efi (2.02-87) -> *crash*
UEFI -> shim -> grubx64.efi (2.02-81) -> *crash*
UEFI -> grubx64.efi (2.02-87) -> kernel -> *success*
UEFI -> grubx64.efi (2.02-81) -> kernel -> *success*

Therefore, my initial assessment (grubx64.efi is the culprit) clearly seems wrong -- the problem seems to lie with shim-x64-15-13.el8.x86_64.

(NB: shim-x64-15-14.el8_2.x86_64 isn't available on my system -- is that from AppStream?)

Comment 32 Javier Martinez Canillas 2020-07-30 18:32:23 UTC
Created attachment 1702984 [details]
shimx64.efi

Please test with the shim EFI binary attached.

It has to be copied to /boot/efi/EFI/redhat/

Comment 36 Ludwig 2020-07-30 19:52:53 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/

This worked for me on CentOS 8.

Comment 37 Jaroslav Suchanek 2020-07-30 20:25:01 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/

Works well on lenovo t470s with rhel-8.2.

Comment 38 Christof Efkemann 2020-07-30 20:27:11 UTC
(In reply to Ludwig from comment #36)
> (In reply to Javier Martinez Canillas from comment #32)
> > Created attachment 1702984 [details]
> > shimx64.efi
> > 
> > Please test with the shim EFI binary attached.
> > 
> > It has to be copied to /boot/efi/EFI/redhat/
> 
> This worked for me on CentOS 8.

Same here, works on CentOS 8.2 (of course, I didn't try SecureBoot - I assume that the signature check would fail in this case (RHEL shim/CentOS GRUB)).

Comment 40 BlairS 2020-07-31 02:27:41 UTC
Thank you Sandro Bonazzola for that tip on grubenv.  Also tried to fix this from the troubleshooter (centos 8) and had the same switch root fail/emergency console.

Banging my head all day but at least I'll sleep good tonight.  Tomorrow I'll see if there are fixed packages.  Turning off updates for now.

Cheers

Comment 42 Nerijus 2020-07-31 06:54:32 UTC
(In reply to Sandro Bonazzola from comment #6)
> I booted my system with the installation ISO in troubleshoot mode, chrooted
> to /mnt/sysimage, regenerated the grub.cfg with grub2-mkconfig and executed
> grub2-install.
> Now I can go past grub but got stuck to emergency shell failing swtching
> root.

Did the same, but disabled secure boot in BIOS. Ended up with a system that boots to `grub>` prompt. Entering `normal` at it resumes usual OS boot with image menu, etc.

Comment 44 Robert Scheck 2020-07-31 11:43:04 UTC
For us (non-secure-boot environment), replacing /boot/efi/EFI/redhat/shimx64.efi (from shim-x64-15-14.el8_2.x86_64) by /boot/efi/EFI/redhat/grubx64.efi (from grub2-efi-x64-2.02-87.el8_2.x86_64) lead us back to a bootable system. Opened case 02717116 at the Red Hat customer portal anyway, but I guess it's related to this RHBZ.

Comment 45 Scott Marshall 2020-07-31 12:04:38 UTC
(In reply to Wade Mealing from comment #3)
> You may be able to work around this by disabling secure boot in the BIOS.   
> Not ideal.. for sure but if you can't see grub.. sounds like something is up
> in the secure boot chain.

My UEFI configuration already has SecureBoot disabled.
The CentOS 7 server (CentOS 7.8.2003) locked up at the point it was loading the shimx64.efi module; grub never even got a chance to display its menu.

Fortunately, I had another box with the prior GRUB2 EFI modules as I'd not updated it yet, mainly because I was distrustful of new shim modules!

After booting the dead host from USB/ISO rescue and mounting the boot/efi file system, I copied all the old efi modules to their correct locations.
No change to grubenv.

Now I'm running with the older grub2 efi modules and the latest kernel (3.10.0-1128.18.2.el7)

Comment 47 Jan "Yenya" Kasprzak 2020-07-31 12:36:05 UTC
I can confirm that CentOS 7 is also broken by the latest grub/shim update. With
grub2-2.02-0.86.el7.centos.x86_64 and shim-x64-15-7.el7_9.x86_64 it does not boot for me,
grub2-2.02-0.81.el7.centos.x86_64 and shim-x64-15-2.el7.centos.x86_64 the system is booting OK.

Comment 48 Bertus 2020-07-31 12:53:01 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/

Yesterday I spend several hours ending up booting in recovery mode and perform a roll back to previous version as described in https://access.redhat.com/solutions/3486741, today I tried the grub update again to find out the systems did fail again to boot.

On CentOS 8.2 copied the EFI file into /boot/efi/EFI/centos/ and the server is back to normal. Thanks for this fix.

Comment 49 crxssi 2020-07-31 14:54:37 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/

I am on an HP ML350G10 & CENTOS8.2.  Same problem as everyone else- after the update and reboot, the server froze on the "HP Enterprise" logo.  I booted rescue media and was unable to rollback the affected packages as suggested by https://access.redhat.com/solutions/5272311  I just get a bunch of "lowest version already installed, cannot downgrade it."  And I was unable to locate a previous version (or even name of) shimx64-15-13.el8.x86_64.

That is when I decided to just give your shimx64.efi a try. I renamed the broken one, then copied the supplied shimx64.efi to /boot/efi/EFI/centos/ and rebooted the system.  Grub came up normally.  Choose newest kernel, late in the boot it complained about "Warning -- SELinux targeted policy relabel is required", it worked on that a while, then it rebooted itself again.  After that reboot, I have 4.18.0-193.14.2.el8_2.x86_64 up and running again, apparently normal.

Not sure where you got the file, but it works great. Thank you!  Now we just wait for updated packages (once they figure out what went wrong).  For sure I will be more prepared for the next attempt...

Comment 50 Peter Ajamian 2020-07-31 15:22:16 UTC
This issue just hit mainstream:
https://linux.slashdot.org/story/20/07/31/158212/red-hat-security-update-renders-systems-unbootable

Comment 51 godzillante 2020-07-31 21:34:12 UTC
Can confirm on a Centos 7 server.
I had to boot with a recovery USB stick and then downgraded grub, shim and also mokutil as yum dependency.

Then the system rebooted without problems

Comment 52 Steevithak 2020-08-01 02:24:10 UTC
I did a "yum update" this afternoon on our CentOS 8.2 server and it seems to be bricked now. I get the Dell logo and then a black screen, no grub no CentOS. I see some people above saying they've fixed it by replacing a "shim" file - can anyone point me to detailed instructions on how to do this with a non-bootable server? Is there a bootable CD or flash stick image that has the fix on it?

Comment 53 James K 2020-08-01 02:49:20 UTC
(In reply to Nerijus from comment #42)
> (In reply to Sandro Bonazzola from comment #6)
> > I booted my system with the installation ISO in troubleshoot mode, chrooted
> > to /mnt/sysimage, regenerated the grub.cfg with grub2-mkconfig and executed
> > grub2-install.
> > Now I can go past grub but got stuck to emergency shell failing swtching
> > root.
> 
> Did the same, but disabled secure boot in BIOS. Ended up with a system that
> boots to `grub>` prompt. Entering `normal` at it resumes usual OS boot with
> image menu, etc.

I am seeing the same thing.  How did you fix it?

Comment 54 Steevithak 2020-08-01 03:14:43 UTC
Update: I found this page at RedHat that is supposed to solve the problem:

https://access.redhat.com/solutions/5272311

I tried applying to my CentOS 8.2 but ran into a couple of problems:

1. All the steps are in clear text except Step 2, which is critical to the solution. Step 2 is pay-walled and you have pay a support fee to access it which seems like bit of a jerk move for a critical issue like this. But I am using CentOS and don't pay for support so I guess I can't complain too much. Without step 2, you can't easily start networking after booting to the troubleshooting mode of the install DVD. I googled and found enough info bring up the ethernet interface, give it an ip, and set a route, so I got minimal networking. So far so good.

2. The solution as described in the above document says to fix the problem by using "yum downgrade shim\* grub2\* mokutil" but that doesn't work on my CentOS 8.2, I just get a yum error saying I already have the oldest version and there's nothing to downgrade to.

If anyone has successfully fixed this on a CentOS 8.2. and could post a step-by-step guide for what they did, it would be appreciated. I have a feeling it's going to be long night... :(

Comment 55 Colin.Simpson 2020-08-01 03:35:42 UTC
Centos 8.2 said earlier versions of the packages weren't available when I tried the yum downgrade from the chrooted environment.
I just rpm -qa shim-x64\* grub2\* and noted the packages I had. 

Then manually downloaded the previous versions from:


http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/

Then set an ip in the chroot and scp'd these packages onto this box. Used rpm -Uvh --force to install them. I then rebooted, and came up , it did and SELinux relabel and rebooted again.then all fine.

https://lists.centos.org/pipermail/centos-devel/2020-July/055953.html 

Other people seem to be doing the same, there maybe a neater way someone knows.

Comment 56 Wouter De Borger 2020-08-01 09:55:57 UTC
Work around that gets the system to boot (doesn't fix it, just allowed me to boot, without secure boot)

1. drop to boot menu, then to uefi shell (hold F11 during boot on my system)
2. navigate to the efi folder  (uefi shell is just like dos, for me the commands where:)

---
FS0:
cd EFI 
cd CENTOS
--- 
3. directly load grubx86.efi

---
grubx64.efi
---

Comment 57 Clifford Perry 2020-08-01 14:46:09 UTC
I just wanted to post a brief note that updated shim packages are now available and can be used in conjunction with previously released grub2, fwupd, and fwupdate packages. Both https://access.redhat.com/security/vulnerabilities/grub2bootloader and https://access.redhat.com/solutions/5272311 have been updated with links to the new shim package bugfix errata.

Comment 58 Thomas Stephen Lee 2020-08-01 18:17:36 UTC
Red Hat fixes are available.

https://access.redhat.com/errata/RHBA-2020:3265
https://access.redhat.com/errata/RHBA-2020:3262

wait for CentOS.

Comment 59 James K 2020-08-01 21:17:31 UTC
(In reply to Steevithak from comment #54)
> Update: I found this page at RedHat that is supposed to solve the problem:
> 
> https://access.redhat.com/solutions/5272311
> 
> I tried applying to my CentOS 8.2 but ran into a couple of problems:
> 
> 1. All the steps are in clear text except Step 2, which is critical to the
> solution. Step 2 is pay-walled and you have pay a support fee to access it
> which seems like bit of a jerk move for a critical issue like this. But I am
> using CentOS and don't pay for support so I guess I can't complain too much.
> Without step 2, you can't easily start networking after booting to the
> troubleshooting mode of the install DVD. I googled and found enough info
> bring up the ethernet interface, give it an ip, and set a route, so I got
> minimal networking. So far so good.
> 
> 2. The solution as described in the above document says to fix the problem
> by using "yum downgrade shim\* grub2\* mokutil" but that doesn't work on my
> CentOS 8.2, I just get a yum error saying I already have the oldest version
> and there's nothing to downgrade to.
> 
> If anyone has successfully fixed this on a CentOS 8.2. and could post a
> step-by-step guide for what they did, it would be appreciated. I have a
> feeling it's going to be long night... :(

I am not sure if you fixed the problem.  I had same issue when I tried to downgraded shim\* etc and it did not help at all.  What I did is follow the other thread that provided a shimx64.efi attachment.  I have to save it to a usb from another computer. Once I booted into the trouble shoot and recovery command prompt, I have to mount usb and copied the shim file to the /boot/efi/EFI/centos.  This method was tested by few people.  One issue I have now is instead of seeing the boot option, it started with 'Grub>' prompt and I have to type 'normal' to enter the boot option.  I am still trying to figure out how to fix this one.

Comment 60 Charlweed Hymerfan 2020-08-01 22:52:56 UTC
An alternate work-around to get bootable again is to get a recovery/live image that has a writable grub.conf, and insert only the boot entries from the broken system's grub.conf into the recovery system's grub.conf. This avoids many issues that must otherwise be resolved before one can work on a chrooted system.

For example, we were blocked by the fact that previous CentOS 7 recovery images and live USBs were too old to load some kernel modules needed to connect and downgrade, and recent CentOS 8 images could not load others. Surprisingly, the VFAT module was one of these.

Comment 65 Logan 2020-08-02 20:53:25 UTC
For CentOS 8 just install the new shim-x64-15-15.el8_2.x86_64.rpm (http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/shim-x64-15-15.el8_2.x86_64.rpm). Works like a charm.

Comment 66 Logan 2020-08-02 21:00:50 UTC
yum list grub2\* shim\*

Installed Packages
grub2-common.noarch                                                                                   1:2.02-87.el8_2                                                                         @BaseOS
grub2-efi-x64.x86_64                                                                                  1:2.02-87.el8_2                                                                         @BaseOS
grub2-pc.x86_64                                                                                       1:2.02-87.el8_2                                                                         @BaseOS
grub2-pc-modules.noarch                                                                               1:2.02-87.el8_2                                                                         @BaseOS
grub2-tools.x86_64                                                                                    1:2.02-87.el8_2                                                                         @BaseOS
grub2-tools-efi.x86_64                                                                                1:2.02-87.el8_2                                                                         @BaseOS
grub2-tools-extra.x86_64                                                                              1:2.02-87.el8_2                                                                         @BaseOS
grub2-tools-minimal.x86_64                                                                            1:2.02-87.el8_2                                                                         @BaseOS
shim-x64.x86_64                                                                                       15-15.el8_2                                                                             @@commandline

Comment 67 Farid Ghoreyshi 2020-08-03 12:55:32 UTC
(In reply to Javier Martinez Canillas from comment #32)
> Created attachment 1702984 [details]
> shimx64.efi
> 
> Please test with the shim EFI binary attached.
> 
> It has to be copied to /boot/efi/EFI/redhat/


Hi
I encountered this bug just a few days ago trying a clean install of CentOS 8.2 and did a yum update. I was like (WTF :/ What just happened) [Kidding !] and tried rescuing grub and yet no luck but thanks to this attached file I was able to successfully boot the OSs I have installed. Thanks again. Hope this fixture gets released on the stream so anyone who is experiencing this issue can fix it without having any difficulty.


Worked On Lenovo G50-80 with bios version "b0cna0ww"

Comment 68 crxssi 2020-08-03 21:29:30 UTC
(In reply to Logan from comment #65)
> For CentOS 8 just install the new shim-x64-15-15.el8_2.x86_64.rpm
> (http://mirror.centos.org/centos-8/8.2.2004/BaseOS/x86_64/os/Packages/shim-
> x64-15-15.el8_2.x86_64.rpm). Works like a charm.

Confirmed, I was able to do nothing but a "yum update" and got the package on my dead, then hacked-to-work system.  It replaced the shimx64.efi file I downloaded from this bug report.  I rebooted and it is fine and up-to-date.  According to the time stamp, the file was updated on the mirrors on Saturday (Aug 01, 2020).  Fast!

I assume the same was/is true for RHEL repos.  If so, I propose the bug report can now be resolved/closed.

Comment 69 are-support 2020-08-03 22:50:33 UTC
I can confirm that a RHEL 8.2 system with shim-x64-15-15.el8_2.x86_64.rpm installed and a RHEL 7.8 system with shim-x64-15-8.el7_8.x86_64.rpm installed both successfully reboot now.

Comment 70 R. Boza 2020-08-04 17:02:06 UTC
Hi,
I noticed that after updating to:
grub2 1:2.02-87
shim 15-15 


mokutil --list-enrolled would output:
MokListRT is empty

This prevents kmod-drivers such as kmod-nvidia from loading in 8.2 and 7.8 and boot up doesn't complete.

This is 100% reproducible for me at least. Fresh system install with either centos 7.8 or centos 8.2 with secure boot enabled. Using kmod-nvidia drivers from ELRepo on a Quadro K4200. It worked fine before this bug, I can see the keys before applying the update.

(In reply to Clifford Perry from comment #57)
> I just wanted to post a brief note that updated shim packages are now
> available and can be used in conjunction with previously released grub2,
> fwupd, and fwupdate packages. Both
> https://access.redhat.com/security/vulnerabilities/grub2bootloader and
> https://access.redhat.com/solutions/5272311 have been updated with links to
> the new shim package bugfix errata.

Comment 71 Akemi Yagi 2020-08-04 18:43:30 UTC
I wonder what happens if you try to install the ELRepo's SB key again.

Comment 72 R. Boza 2020-08-04 18:56:29 UTC
(In reply to Akemi Yagi from comment #71)
> I wonder what happens if you try to install the ELRepo's SB key again.

I tried that, and what happens is that it lets me enroll the key again even though it actually already exists but mokutil --list-enrolled continues to output MokListRT is empty. However upon trying on a fresh 8.2 before the bug, I see the key was duplicated. So not only the original key was always there, it duplicated the key without acknowledging it already existed.

Comment 73 Akemi Yagi 2020-08-04 19:05:19 UTC
(In reply to R. Boza from comment #72)
> (In reply to Akemi Yagi from comment #71)
> > I wonder what happens if you try to install the ELRepo's SB key again.
> 
> I tried that, and what happens is that it lets me enroll the key again even
> though it actually already exists but mokutil --list-enrolled continues to
> output MokListRT is empty. However upon trying on a fresh 8.2 before the
> bug, I see the key was duplicated. So not only the original key was always
> there, it duplicated the key without acknowledging it already existed.

Strange indeed. What do you see with a "keyctl list %:.system_keyring" ?

Comment 74 R. Boza 2020-08-04 19:40:12 UTC
(In reply to Akemi Yagi from comment #73)
> (In reply to R. Boza from comment #72)
> > (In reply to Akemi Yagi from comment #71)
> > > I wonder what happens if you try to install the ELRepo's SB key again.
> > 
> > I tried that, and what happens is that it lets me enroll the key again even
> > though it actually already exists but mokutil --list-enrolled continues to
> > output MokListRT is empty. However upon trying on a fresh 8.2 before the
> > bug, I see the key was duplicated. So not only the original key was always
> > there, it duplicated the key without acknowledging it already existed.
> 
> Strange indeed. What do you see with a "keyctl list %:.system_keyring" ?

I was only able to test from 8.2: (it looks like keyctl list %:.system_keyring is now keyctl list %:.builtin_trusted_keys. https://bugzilla.redhat.com/show_bug.cgi?id=1509714 documentation error reported for Fedora 26+ fixed now)
3 keys in keyring:
Centos Linux kernel
Centos Linux kpatch
Centos Linux Driver update

So it is missing ELRepo, and a few other keys that would normally be there. I think something in the update is preventing those keys from being read.

Comment 75 R. Boza 2020-08-04 20:03:08 UTC
(In reply to R. Boza from comment #74)
> (In reply to Akemi Yagi from comment #73)
> > (In reply to R. Boza from comment #72)
> > > (In reply to Akemi Yagi from comment #71)
> > > > I wonder what happens if you try to install the ELRepo's SB key again.
> > > 
> > > I tried that, and what happens is that it lets me enroll the key again even
> > > though it actually already exists but mokutil --list-enrolled continues to
> > > output MokListRT is empty. However upon trying on a fresh 8.2 before the
> > > bug, I see the key was duplicated. So not only the original key was always
> > > there, it duplicated the key without acknowledging it already existed.
> > 
> > Strange indeed. What do you see with a "keyctl list %:.system_keyring" ?
> 
> I was only able to test from 8.2: (it looks like keyctl list
> %:.system_keyring is now keyctl list %:.builtin_trusted_keys.
> https://bugzilla.redhat.com/show_bug.cgi?id=1509714 documentation error
> reported for Fedora 26+ fixed now)
> 3 keys in keyring:
> Centos Linux kernel
> Centos Linux kpatch
> Centos Linux Driver update
> 
> So it is missing ELRepo, and a few other keys that would normally be there.
> I think something in the update is preventing those keys from being read.

The output matches the one of a system with UEFI and Secure Boot disabled. However secureboot is enabled and I have doubled checked via BIOS settings and mokutil --sb-state.

it should be something like:
...asymmetric: Red Hat Enterprise Linux Driver Update Program (key 3): bf57f3e87...
...asymmetric: Red Hat Secure Boot (CA key 1): 4016841644ce3a810408050766e8f8a29...
...asymmetric: Microsoft Corporation UEFI CA 2011: 13adbf4309bd82709c8cd54f316ed...
...asymmetric: Microsoft Windows Production PCA 2011: a92902398e16c49778cd90f99e...
...asymmetric: Red Hat Enterprise Linux kernel signing key: 4249689eefc77e95880b...
...asymmetric: Red Hat Enterprise Linux kpatch signing key: 4d38fd864ebe18c5f0b7...
...ElRepo signing key...

Comment 76 Akemi Yagi 2020-08-04 20:18:17 UTC
I believe it is supposed to display:

asymmetric: The ELRepo Project (http://elrepo.org): ELRepo.org Secure Boot Key: f365ad3481a7b20e3427b61b2a26635b83fe427b

This is something that needs investigation. Thank you for your input.

Comment 77 Kyle Walker 2020-08-04 21:26:44 UTC
Thank you for the above report! I have opened the following bug to address the MOK list failure.

    1866107 – Following 1861977, the MOK list is inaccessible with "Couldn't get UEFI MokListRT" visible in the logs
    https://bugzilla.redhat.com/show_bug.cgi?id=1866107

Comment 78 Stephen Steward 2020-08-05 18:28:44 UTC
This bug has also affected my RHEL 8.2 system. After POST, the GRUB menu is not displayed. I have SecureBoot disabled by default in my UEFI BIOS settings.

Comment 79 Adam Williamson 2020-08-05 20:22:58 UTC
The bug can affect you whether or not SB is enabled. Please follow the steps in https://access.redhat.com/solutions/5272311 to recover now.

Comment 80 Nerijus 2020-08-06 06:13:02 UTC
(In reply to James K from comment #53)
> (In reply to Nerijus from comment #42)
> > (In reply to Sandro Bonazzola from comment #6)
> > > I booted my system with the installation ISO in troubleshoot mode, chrooted
> > > to /mnt/sysimage, regenerated the grub.cfg with grub2-mkconfig and executed
> > > grub2-install.
> > > Now I can go past grub but got stuck to emergency shell failing swtching
> > > root.
> > 
> > Did the same, but disabled secure boot in BIOS. Ended up with a system that
> > boots to `grub>` prompt. Entering `normal` at it resumes usual OS boot with
> > image menu, etc.
> 
> I am seeing the same thing.  How did you fix it?

Haven't found how to fix it. Still living with `grub>` prompt at boot time.

Comment 85 Siarhei Farbotka 2020-09-04 08:50:20 UTC
The issue is still reproducible on my Macmini3,1 (Late 2009) machine with CentOS 8.2 installed.

The system boots normally with shim-x64-15-11.el8, but doesn't boot with shim-x64-15-13.el8 or shim-x64-15-15.el8_2.
In all cases grub2-efi-x64-2.02-87.el8_2 is installed.

Comment 86 oktaya 2020-09-10 17:03:25 UTC
Still having this issue on a virtual machine running Centos 7.8.2003  running under a libvirt/kvm machine with the same centos version.  I have not rebooted the server itself but the virtual machine does not boot (black screen after grub) with the latest kernel even though it already had the fixed shim package.

Comment 87 Oliver Falk 2020-09-11 10:56:44 UTC
(In reply to oktaya from comment #86)
> Still having this issue on a virtual machine running Centos 7.8.2003 
> running under a libvirt/kvm machine with the same centos version.  I have
> not rebooted the server itself but the virtual machine does not boot (black
> screen after grub) with the latest kernel even though it already had the
> fixed shim package.

Hi!

Yes, there seems to be still some issue. I'd suggest to watch the following solution article, which we'll keep updated:

    https://access.redhat.com/solutions/5385031

Consider this *not* solved, until this RHBZ has been closed!

I hope this information helps!

Regards,
 Oliver

Comment 88 Petr Zatko 2020-09-14 11:29:12 UTC
Reproduced the issue in VM using shim-x64-15-14.el8
Verified using shim-x64-15-15.el8 (and grub2-efi-x64-2.02-90) as I managed to reboot without any problems and can't reproduce the issue anymore, moving to Verified.

@Siarhei since we don't have a Macmini to try to test the issue in your exact environment and we can't reproduce it, please submit a new bug if you continue to have problems the with the issue 

@oktaya your issue (black screen after grub) seems to be different than the one described: 'System hangs after POST and the grub menu never loads'. Please submit a new bug if needed.

Comment 93 errata-xmlrpc 2020-11-04 02:06:11 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (shim bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4574


Note You need to log in before you can comment on or make changes to this bug.