How does an operating system prevent an unprivileged process from executing a privileged instruction?

Question

In Unix,

if you are a guest user and did:

chmod 777 /

It would fail.

But how does this internally happen at the hardware level?

So far, I think this is what happens:

• The OS tries to execute that instruction.
• Information about permissions is perhaps somewhere in secondary memory. So it would issue a write instruction.
• Before 2, It would check if the user is privileged to do this. If he isn't it would just issue an error message.

Is this how it happens, or is an interrupt raised when such a situation arises? Is there a routine in the ISR table in main memory corresponding to unprivileged instructions?

Answer 1

chmod is a filesystem operation, not a privileged instruction. Filesystem permissions are not handled at the hardware level. The software (specifically the OS) sees that the process invoking the system call does not have sufficient permissions to perform the operation on the filesystem object and the system call returns with a permission error.

Answer 2

Actually, you are asking two different questions:

1. How does an operating system prevent an unprivileged process from executing a privileged instruction?
2. Why can an unprivileged process not invoke chmod 777 /?

Answer for 2:

chmod internally invokes a function from the libc (conveniently also called chmod()). This function checks whether the caller has sufficient privileges for the operation - if not, it returns error EPERM.

The answer for 1 is more interesting:

The exact mechanism depends on OS and hardware platform, but basically it's like this: All modern processors have built-in security features. This allows the OS to tell the processor: "run this program, but do not let it execute these privileged instructions". So the processor itself will enforce the restriction on allowed instructions. If the programm tries to execute a privileged instruction, the processor will pass control back to the OS, which will usually terminate the misbehaving program. For details, see e.g. https://en.wikipedia.org/wiki/Ring_%28computer_security%29