I broke SSH by using chmod on an EC2 Instance — How do I correct?

Question

I have been running an Amazon EC2 Instance for a while, and recently accidentally changed permissions/ownership on folders recursively. I can no longer SSH into the instance. I do not recall what the exact command was. I then regained basic access using the technique provided in the answer to this question: https://serverfault.com/questions/234061/re-gaining-root-access-to-an-ec2-ebs-boot-image

I am at a loss for what to do with Step 3 ("Modify it."), though. I do not know what to modify, and in my attempts to set the correct permissions, I have lost access to a number of recovery EC2 instances as well!

For reference, I am using Win7 and Putty/WinSCP to connect to the instance. Putty displays the following 2 errors when I attempt to log on to the instance through SSH:

Server Refused Our Key

No supported Authentication methods available (server sent: public key)

I am confident I am using the correct username, IP address, and private key for my instance.

Any help would be much appreciated.

Answer 1

The solution, as the Server Fault post describes, is to mount your EBS to another (new) instance which you can connect to. The EBS will be just a drive there, and you will be able to remote connect to the instance since it is new. You can then sudo chown the directories you broke in the first place.

1. Stop (NOT terminate) your first instance from AWS console

2. Create a new linux instance from AWS console

3. Detach the EBS from the first instance, and attach it to the new instance, again from AWS console. Give it the name: /dev/sdm (On the EBS console, you'll see that the volume attached is named sdm but mapped to xvdm on Linux, so it matches the commands below. But feel free to use another letter and adjust the next commands.)

4. SSH to the new instance and execute:

   sudo mkfs -t ext4 /dev/xvdm
   sudo mkdir /old-ebs
   # optional, in case this is a bit more permanent
   sudo echo "/dev/xvdm        /old-ebs     auto    noatime,noexec,nodiratime 0 0" >> /etc/fstab
   sudo mount /dev/xvdm /old-ebs
   

5. Now you have (sudo) access to your old EBS, and you can chown the directory you need. A tip here, if you're lost, would be to copy the rights from the new instance.

6. Once you are done, you can stop the new instance, detach your EBS from it and re-attach it to the first instance, then start it.

The 3rd console command above, to add the mounted volume in the /etc/fstab so it gets mounted automatically at each reboot, is optional.