I created my own custom app, now I need to set this app as the default landing app. In Salesforce Classic I went to Profile -> App -> Custom App but when I logged in after I switched in lightning, I ended up in the Setup Tab. Why? How I can set the custom app as the default app also in lightning?

In lightning you would have to go to: Setup> Apps> App Manager, and then create a Lightning app. In the wizard you'll see you can configure mostly everything, including which profiles will have access to it.

Default app configuration comes from Profile. But if a profile have access to multiple App Salesforce remembers which one the last app being used by the user.

For example "Sales" is the default app for Sysadmin profile and it has also access to custom app "APP ONE" now if you are in sales app in classic view whenever you login in classic you will see sales app landing page after login.

Now you switched to lightning and gone to "App ONE" and log out when you log back in you will go to "App One" landing page.

Now you switched to classic you will go to Sales landing page - Salesforce remembered your choice.

Now you switch your view to lightning you will go to "App One" home page

So the landing page will be same if you have access to only one app in classic and lightning.

Salesforce remembers in which APP you left your classic app as well as your Lightning app.