I am trying to understand how user permissions work in Linux. The kernel boots and starts init as root, right? Init then runs startup scripts and runs getty (agetty), again as root. Agetty just reads user name and runs login, still as root, I think. Nothing interesting yet. But what does login do? I wasn't able to find anything better than "it attempts to log in". Suppose login finds that password matches (and we trying to log in as usual user), how does it change user id? I thought that there should be system call for that but I wasn't able to find it (maybe I'm just blind?)

Also, about su. su has the 'setuid' bit set so when we run it, it always runs as root. But when we tell it to log in as usual user, it again needs to change user id. Do I understand correctly that the same "magic" happens in su and login when they need to change user? If so, why have two different programs? Is there any additional sorts of serious business happening when running login?

The first part is authentication: the program reads some input from the user and decides whether the user is authorized to log in. The traditional method is to read a user name and password, and check that the user is mentioned in the system's user database and that the password that the user typed is the one in the database. But there are many other possibilities (one-time passwords, biometric authentication, authorization transfer, …).

Once it has been established that the user is authorized to log in and in what account, the login program establishes the user's authorization, for example what groups the user will belong to in this session.

The login program may also check account restrictions. For example, it may enforce a login time, or a maximum number of logged-in users, or refuse certain users on certain connections.

Finally the login program sets up the user's session. There are several substeps:

Set the process permissions to what was decided in the authorization: user, groups, limits, … You can see a simple example of this substep here (it only handles user and groups). The basic idea is that the login program is still running as root at this point, so it has maximum privileges; it first removes all privileges other than being the root user, and finally calls setuid to drop that last but not least privilege.

Possibly mount the user's home directory, display a “you have mail” message, etc.

Invoke some program as the user, typically the user's shell (for login and su, or sshd if no command was specified; an X display manager invokes an X session manager or window manager).

Most unices nowadays use PAM (Pluggable Authentication Modules) to provide a uniform way of managing login services. PAM divides its functionality into 4 parts: “auth” encompasses both authentication (1 above) and authorization (2 above); “account” and “session” are as 3 and 4 above; and there's also “password”, which is not used for logins but to update authentication tokens (e.g. passwords).

login will drop root privileges when necessary. Many programs that need root privileges only initially will start as root, do what they need to do, and then drop down to a normal user account so they don't need to worry about someone using a bug in the binary to get access to a root shell. login naturally holds the privileges longer, but the principle is the same.

Actually dropping root privileges is fairly trivial. POSIX defines setuid() and setgid() functions, which change your user and group IDs, respectively (real and effective, if you're starting as root). login calls both of these, as well as initgroups() to setup any supplementary groups you may have (since setgid is just for setting your primary group ID)

The system calls you are looking for are called things like setuid and seteuid although there is actually a whole family of hem depending on exactly which variants of user identity you are trying to change.

There are also parallel calls like setgid for changing the group that a process runs as.