Lost an EC2 PEM Key? Regain SSH Access

Losing the .pem file for an EC2 Linux instance blocks SSH because sshd only accepts public keys listed in the login user's authorized_keys. AWS retains the launch key-pair metadata, while the corresponding public key was written into the instance's authorized_keys during first boot. AWS does not retain a recoverable copy of the private key. Recovery means adding a new public key to the instance without rebuilding the server or enabling password login.

If Session Manager is already working, use it first. When SSM is unavailable but the AMI supports EC2 Instance Connect, that API can provide temporary SSH access on supported distributions. On RHEL 10.2, the tested paths are Session Manager and offline EBS volume rescue. The examples use ec2-user on an EBS-backed root volume; instance IDs, IP addresses, and key names are sanitized. Workflows were tested in ap-south-1.

  Lost .pem on workstation          EC2 still has launch public key
       |                                      |
       |  ssh -i recovery.pem  FAIL           |
       X------------------------------------->| authorized_keys
       |                                      | missing new key
       |                                      |
       +--- SSM / ResetAccess / rescue volume -> append new public key
       |                                      |
       |  ssh -i recovery.pem  OK             |
       +------------------------------------->|

Lost EC2 key pair?

1. Confirm the symptom is a missing private key, not a wrong username or PEM path (permission denied publickey covers that case).
2. Create a new EC2 key pair and keep the .pem in a safe place.
3. Try Session Manager when the SSM agent and IAM instance profile are already configured.
4. Consider AWSSupport-ResetAccess when you want AWS Automation to inject a replacement key.
5. Try EC2 Instance Connect only on an AMI AWS currently lists as supported (not RHEL 10).
6. Fall back to offline EBS root-volume repair when no in-guest path works and the root volume is EBS-backed.
7. Append the new public key with correct ownership (700 on .ssh, 600 on authorized_keys).
8. Verify SSH with the new private key and remove obsolete key lines when finished.

What AWS Can and Cannot Recover

At launch, EC2 copies the selected key pair's public material into ~/.ssh/authorized_keys for the AMI default user. The EC2 console shows Key pair assigned at launch as metadata; that name does not rotate keys on a running instance.

AWS cannot:

• Re-download or email the original private .pem
• Derive the private key from the public key stored on the instance
• Change which key authorized_keys contains by selecting a different key pair in the console later

AWS can:

• Create a new key pair and deliver a new private .pem for future logins
• Let an administrator with SSM, Instance Connect, or offline volume access append a new public key line to authorized_keys

Stopping the instance does not delete EBS root data. A stop/start may change the public IPv4 address; update the SSH target after the instance returns to running.

Offline rescue requires an EBS-backed root volume. An instance-store root volume cannot be detached and mounted on another instance.

AWS also documents a cloud-init user-data recovery method that injects a replacement key on the next boot. It depends on cloud-init behavior and requires stopping, editing, and restarting the instance, so this guide prefers SSM, AWSSupport-ResetAccess, supported Instance Connect, or offline rescue where possible. See the AWS re:Post lost-key answer linked in References.

Do not enable PasswordAuthentication yes in sshd_config as the primary fix. Password login is off by default on current Linux AMIs, weakens the attack surface, and is unnecessary when any of the paths above can inject a public key.

Create a Replacement Key Pair

Before touching the instance, generate a new key pair and record its fingerprint so later SSH tests prove the correct material is in use.

Create the key pair in EC2 and save the private file locally:

aws ec2 create-key-pair --region ap-south-1 --key-name glc-lab-recovery-key --query 'KeyMaterial' --output text > glc-lab-recovery-key.pem

Restrict workstation permissions so OpenSSH will read the file:

chmod 400 glc-lab-recovery-key.pem

create-key-pair prints the PEM once. Store it in a secrets manager or encrypted backup; AWS will not show it again.

Derive a .pub file once on the workstation and reuse it in every recovery path below:

ssh-keygen -y -f glc-lab-recovery-key.pem > glc-lab-recovery-key.pub

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF9D4Ni4pR+SKxPx0a3HgBNEpTKGs9XiPSP/pf4yiqt5ifE3MDVF2u1MDRW5kMy+uvEdbokfXs+6xJSC9f00sNN0E3CL5Mf5RgJnJnfPxpv5ZGVrKDbGMUFvM4Unl0UHNkhDhUlepimw39moZ++16lygTcYgGsnmg3waFlDTPDoPUh/F4ryVa1XKcAVGIDYND4Aex6aiAFaVBCSqon6c2+36sC1AfisgRCxfo+rBscCcM997DRqoNSivtO3tph2GUXP6NWbfj9b6Igf7q9hD29sE2g8j4OdJqrejmKeGiArBUiuD8R9v8MAkd4JiPYmOQ7i4Bf7l/oRJ1RHiQJElT/

Record the fingerprint from that public file; the same value should appear in authorized_keys after recovery:

ssh-keygen -lf glc-lab-recovery-key.pub

2048 SHA256:dgZXUGqulhRfAafgDZKXXPakHdzF4CU85EDncC1K7eo no comment (RSA)

With only the new .pem on the workstation and the launch key still the sole line in authorized_keys, SSH fails before any recovery step:

ssh -i glc-lab-recovery-key.pem -o BatchMode=yes ec2-user@203.0.113.42 hostname

ec2-user@203.0.113.42: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

That publickey response confirms the network path and sshd are up; the missing fix is server-side key material, not security group rules.

Fastest Recovery if Session Manager Works

Systems Manager is the preferred path when the instance is already a managed node: SSM agent running, IAM instance profile with AmazonSSMManagedInstanceCore (or equivalent), and the instance can reach SSM endpoints. No inbound port 22 is required.

Confirm the agent reports Online before spending time on SSH-only paths:

aws ssm describe-instance-information --region ap-south-1 --filters "Key=InstanceIds,Values=i-0123456789abcdef0" --query 'InstanceInformationList[0].PingStatus' --output text

Online

From the console, open Connect → Session Manager → Start session and edit authorized_keys interactively, or push the change with Run Command. The CLI path below matches what was tested on RHEL 10.2.

Run a shell document that creates .ssh with correct ownership, appends the new public key as ec2-user, fixes modes, and prints fingerprints for verification. Paste the single line from glc-lab-recovery-key.pub into the echo command:

aws ssm send-command --region ap-south-1 --instance-ids i-0123456789abcdef0 --document-name AWS-RunShellScript --parameters 'commands=["sudo install -d -m 700 -o ec2-user -g ec2-user /home/ec2-user/.ssh","echo ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF9D4Ni4pR+SKxPx0a3HgBNEpTKGs9XiPSP/pf4yiqt5ifE3MDVF2u1MDRW5kMy+uvEdbokfXs+6xJSC9f00sNN0E3CL5Mf5RgJnJnfPxpv5ZGVrKDbGMUFvM4Unl0UHNkhDhUlepimw39moZ++16lygTcYgGsnmg3waFlDTPDoPUh/F4ryVa1XKcAVGIDYND4Aex6aiAFaVBCSqon6c2+36sC1AfisgRCxfo+rBscCcM997DRqoNSivtO3tph2GUXP6NWbfj9b6Igf7q9hD29sE2g8j4OdJqrejmKeGiArBUiuD8R9v8MAkd4JiPYmOQ7i4Bf7l/oRJ1RHiQJElT/ | sudo -u ec2-user tee -a /home/ec2-user/.ssh/authorized_keys >/dev/null","sudo chmod 600 /home/ec2-user/.ssh/authorized_keys","sudo ssh-keygen -lf /home/ec2-user/.ssh/authorized_keys"]' --query 'Command.CommandId' --output text

Poll the command until status is Success, then read stdout. The fingerprint list should include both the original launch key and the new recovery key:

2048 SHA256:4XQZSF4K6uQ/SL1Y91qit6wtKZtuKHg7nw5uD4faTFM glc-lab-old-key (RSA)
2048 SHA256:dgZXUGqulhRfAafgDZKXXPakHdzF4CU85EDncC1K7eo no comment (RSA)

Confirm ownership and modes on the home directory tree:

aws ssm send-command --region ap-south-1 --instance-ids i-0123456789abcdef0 --document-name AWS-RunShellScript --parameters 'commands=["sudo ls -ld /home /home/ec2-user /home/ec2-user/.ssh /home/ec2-user/.ssh/authorized_keys"]' --query 'Command.CommandId' --output text

drwxr-xr-x. 3 root     root      22 Sep 26 15:47 /home
drwx------. 3 ec2-user ec2-user  74 Sep 26 15:47 /home/ec2-user
drwx------. 2 ec2-user ec2-user  29 Sep 26 15:47 /home/ec2-user/.ssh
-rw-------. 1 ec2-user ec2-user 784 Sep 26 15:51 /home/ec2-user/.ssh/authorized_keys

Retest SSH from the workstation with the new private key:

ssh -i glc-lab-recovery-key.pem ec2-user@203.0.113.42 hostname

ip-172-31-18-195.ap-south-1.compute.internal

When SSM stays Offline, repair the agent only if another management path already exists (Serial Console, a co-admin's access, or a prior SSM session). Otherwise move to AWSSupport-ResetAccess, supported EC2 Instance Connect, or offline EBS rescue.

AWS Automation option: AWSSupport-ResetAccess

AWSSupport-ResetAccess automates an offline EC2Rescue-style workflow. On Linux it creates and injects a new SSH key and stores the generated private key encrypted in Systems Manager Parameter Store. AWS documents this runbook specifically for lost EC2 keys.

The automation stops the target instance and creates a backup before modifying access. Review that operational impact before you start. A stop/start cycle can also change a non-Elastic public IPv4 address.

Start the automation from the Systems Manager console under Documents → AWSSupport-ResetAccess, or invoke it through the API when your account policy allows SSM Automation on the instance. When it completes, retrieve the new private key from the Parameter Store path shown in the execution output, then verify SSH with that key before removing older authorized_keys lines.

On the RHEL 10.2 lab, Session Manager and offline rescue were tested directly; use AWSSupport-ResetAccess when you prefer AWS-managed key injection over manual volume editing.

Use EC2 Instance Connect on a Supported AMI

EC2 Instance Connect pushes a temporary SSH public key that expires in about 60 seconds. It helps when SSH port 22 is reachable and the caller has ec2-instance-connect:SendSSHPublicKey, but it does not replace adding a permanent line to authorized_keys unless you follow up with a shell session.

EC2 Instance Connect is useful on supported AMIs, but AWS's current installation documentation lists RHEL 8 and RHEL 9 rather than RHEL 10. For the RHEL 10.2 lab in this article, use Session Manager or offline EBS rescue as the supported recovery paths. Use the steps below only on an AMI that AWS currently lists as supported (for example Amazon Linux 2023, Ubuntu, or RHEL 9).

Amazon Linux and Ubuntu AMIs often ship the agent. On RHEL 8 and RHEL 9, install the ec2-instance-connect and ec2-instance-connect-selinux RPMs from the AWS distribution URL documented for your release.

Push glc-lab-recovery-key.pub for the target OS user (Availability Zone must match the instance):

aws ec2-instance-connect send-ssh-public-key --region ap-south-1 --instance-id i-0123456789abcdef0 --availability-zone ap-south-1c --instance-os-user ec2-user --ssh-public-key file://glc-lab-recovery-key.pub

{
    "RequestId": "00e32b86-4192-41a8-bd03-22264601e8fb",
    "Success": true
}

Success: true only means the key was accepted; connect within about 60 seconds while the push is valid, then append the same public key to authorized_keys using the live-instance steps in the next section.

Offline Recovery with a Rescue Instance

When SSM is offline, AWSSupport-ResetAccess is not an option, and Instance Connect does not apply, repair the root filesystem offline. This section only shows the key-repair part; the separate rescue-instance guide covers the full EBS detach/reattach workflow.

This path requires an EBS-backed root volume in the same Availability Zone as the rescue host. Launch or reuse a rescue instance using a key pair you still possess (here, glc-lab-recovery-key). Stop the locked instance so the root volume can detach cleanly:

aws ec2 stop-instances --region ap-south-1 --instance-ids i-0123456789abcdef0

Wait until the instance state is stopped, then snapshot the root volume before any offline edit:

aws ec2 create-snapshot --region ap-south-1 --volume-id vol-0abc1234567890def --description "Before EC2 SSH key recovery"

create-snapshot returns a snapshot ID immediately while the snapshot completes in the background. Detach the root volume only after the instance is fully stopped (note the volume ID and original device name, often /dev/sda1):

aws ec2 detach-volume --region ap-south-1 --volume-id vol-0abc1234567890def

Attach the volume to the rescue instance as a secondary device:

aws ec2 attach-volume --region ap-south-1 --volume-id vol-0abc1234567890def --instance-id i-0fedcba9876543210 --device /dev/sdf

On the rescue host, list block devices and pick the XFS root partition (commonly the third partition on Nitro instances):

lsblk -f

NAME        FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
nvme0n1
├─nvme0n1p1
├─nvme0n1p2 vfat   FAT16 ESP   97D2-64D3                             190.8M     4% /boot/efi
└─nvme0n1p3 xfs          root  c78c1f15-7ec8-4bc4-b1b6-0e557f828ec1      8G    18% /
nvme1n1
├─nvme1n1p1
├─nvme1n1p2 vfat   FAT16 ESP   97D2-64D3
└─nvme1n1p3 xfs          root  c78c1f15-7ec8-4bc4-b1b6-0e557f828ec1

Mount the foreign root partition under /mnt/rescue-root. Use nouuid on XFS when the rescue host and detached volume share the same filesystem UUID:

sudo mount -o nouuid /dev/nvme1n1p3 /mnt/rescue-root

mount exits silently when the filesystem attaches; append the recovery public key using the rescue-mounted steps in the next section before unmounting.

Add the New Public Key Correctly

sshd ignores authorized_keys that are writable by group or others, or owned by the wrong numeric UID. The safe command differs depending on whether you are on the original instance or on a rescue host with the root volume mounted elsewhere.

On the live instance (Session Manager or Instance Connect)

Use this path when the shell runs on the locked instance itself (SSM Session Manager, SSM Run Command, or an Instance Connect session on a supported AMI).

Create .ssh owned by the login user when the directory is missing:

sudo install -d -m 700 -o ec2-user -g ec2-user /home/ec2-user/.ssh

Append the public key as ec2-user so the file stays owned by the account that will log in:

cat glc-lab-recovery-key.pub | sudo -u ec2-user tee -a /home/ec2-user/.ssh/authorized_keys >/dev/null

Set the mode sshd expects on RHEL:

sudo chmod 600 /home/ec2-user/.ssh/authorized_keys

Verify ownership on the live instance:

sudo ls -ld /home/ec2-user/.ssh /home/ec2-user/.ssh/authorized_keys

drwx------. 2 ec2-user ec2-user  56 Sep 26 15:51 /home/ec2-user/.ssh
-rw-------. 1 ec2-user ec2-user 784 Sep 26 15:57 /home/ec2-user/.ssh/authorized_keys

On a rescue-mounted root volume

Create .ssh under the mounted path without forcing the rescue host's user ID:

sudo install -d -m 700 /mnt/rescue-root/home/ec2-user/.ssh

Append the key with tee, then copy ownership from the target home directory so numeric UID/GID match the original instance:

cat glc-lab-recovery-key.pub | sudo tee -a /mnt/rescue-root/home/ec2-user/.ssh/authorized_keys >/dev/null

chown --reference applies the home directory's UID/GID to the whole .ssh tree:

sudo chown -R --reference=/mnt/rescue-root/home/ec2-user /mnt/rescue-root/home/ec2-user/.ssh

Set the modes sshd expects on the mounted copy, starting with the directory:

sudo chmod 700 /mnt/rescue-root/home/ec2-user/.ssh

Then restrict the key file itself to owner-read/write:

sudo chmod 600 /mnt/rescue-root/home/ec2-user/.ssh/authorized_keys

Confirm modes on the mounted tree before unmounting:

sudo ls -ld /mnt/rescue-root/home/ec2-user/.ssh /mnt/rescue-root/home/ec2-user/.ssh/authorized_keys

drwx------. 2 ec2-user ec2-user  56 Sep 26 15:51 /mnt/rescue-root/home/ec2-user/.ssh
-rw-------. 1 ec2-user ec2-user 784 Sep 26 15:57 /mnt/rescue-root/home/ec2-user/.ssh/authorized_keys

Use chmod values 700 on .ssh and 600 on authorized_keys. Avoid chmod 777 or root-owned key files; both produce the same Permission denied (publickey) symptom documented in the publickey troubleshooting guide.

Restore the Root Volume

After offline edits, unmount the rescue copy of the root filesystem:

sudo umount /mnt/rescue-root

Detach the volume from the rescue instance:

aws ec2 detach-volume --region ap-south-1 --volume-id vol-0abc1234567890def

Reattach it to the original instance at the same device name recorded before detach (/dev/sda1 in the tested layout):

aws ec2 attach-volume --region ap-south-1 --volume-id vol-0abc1234567890def --instance-id i-0123456789abcdef0 --device /dev/sda1

Start the instance and wait for status checks to return ok:

aws ec2 start-instances --region ap-south-1 --instance-ids i-0123456789abcdef0

A stop/start cycle can assign a new public IPv4 address. Refresh the SSH target from the console or API before testing.

Verify SSH and Remove Temporary Access

Confirm the new private key authenticates and returns the expected hostname:

ssh -i glc-lab-recovery-key.pem ec2-user@203.0.113.42 hostname

ip-172-31-18-195.ap-south-1.compute.internal

On the instance, list fingerprints in authorized_keys and confirm the recovery key is present:

sudo ssh-keygen -lf /home/ec2-user/.ssh/authorized_keys

2048 SHA256:4XQZSF4K6uQ/SL1Y91qit6wtKZtuKHg7nw5uD4faTFM glc-lab-old-key (RSA)
2048 SHA256:dgZXUGqulhRfAafgDZKXXPakHdzF4CU85EDncC1K7eo no comment (RSA)

Remove keys that should no longer authenticate (for example the lost launch key line) only after confirming the new key works in a second session. Keep at least one working key line to avoid locking the instance again.

Terminate the rescue instance if it was created only for this repair. Instance Connect temporary keys expire on their own; no file cleanup is required for the push API.

Record which replacement key now has access so another administrator can identify it later. The console Key pair assigned at launch field still shows the original name until the instance is replaced; that metadata does not control which keys sshd accepts.

Troubleshooting

Summary

A lost EC2 .pem does not destroy instance data, but AWS cannot recreate the private key. To regain SSH access, add a public key for a private key you still control into the correct user's authorized_keys with strict ownership and modes.

When Systems Manager already manages the host, SSM Run Command or an interactive Session Manager shell is the fastest fix on the RHEL 10.2 lab and avoids opening port 22. AWSSupport-ResetAccess automates key injection when you prefer AWS-managed offline repair. EC2 Instance Connect can add a short-lived key on AMIs AWS currently supports, but not RHEL 10 per current installation docs. When no in-guest path works and the root volume is EBS-backed, snapshot the volume, detach it to a rescue host in the same Availability Zone, edit authorized_keys offline with chown --reference so numeric UID/GID match the original home directory, use mount -o nouuid when XFS UUIDs collide, then reattach and start.

After SSH succeeds with the new .pem, remove obsolete key lines, retire single-purpose rescue instances, and store the replacement private key outside the original workstation. Password authentication is not required for any step in this workflow.

References

• Amazon EC2 key pairs
• I've lost my private key. How can I connect to my instance?
• AWSSupport-ResetAccess
• AWS Systems Manager Session Manager
• Connect to your Linux instance using EC2 Instance Connect
• Install EC2 Instance Connect on RHEL
• SendSSHPublicKey API
• Attach an Amazon EBS volume to an instance
• Troubleshoot connecting to your instance
• OpenSSH authorized_keys manual