Things to consider...

These instructions are for building a modded Minecraft Java Edition Server. This site will not help you install mods locally on your PC, although that is a future article I am considering.

These instructions won't work for PE or the Better Together version. The modding scene is almost entirely on the Java edition and you will need to be running Minecraft Java on PC (Windows, Linux or Mac) to connect to this server.
You will not be able to connect to this with your Xbox or your Ipad or the Microsoft Windows 10 Edition.

Be careful when downloading mods. There are so many shady mod sites that use AdFly links that lead to porn ads, that rehost mods without permission from
the authors and even host ads that lead to malware sites. It's a minefield, and for this reason, I highly recommend that you get your mods from CurseForge. While Curse have a tonne of mods, they don't have everything.
Minecraftforum.net seems fine too to get individual mods. I am sure there a other reputable places, but you are taking your chances.

You will also need the Forge Minecraft launcher to play mods. If you are new to mods, getting an account with Twitch/Curse and using their launcher and managing your mods there is much easier. You can't play on a modded server
with the stock Mojang Launcher. You need to install Forge for the version you want to play.

Finally, these instructions are for the latest version of Minecraft, which at the time of this writing was 1.12.2. When it comes to mods version matters. There are many great mods at version 1.7.10 for example. No worries though,
these instructions should work the same for any version you need. You just need to download the right version of Forge and the Minecraft server.jar which I will get into later.

Why Linux and Not a Fully Managed Solution?

This is a valid question. A fully managed solution like one offered by MCProhosting offers some benefits. There is no need to secure the underlying Linux OS, this is all done for you.
You get a nice graphical front end to manage your server, automated backups, DNS management to access your server, FTP access. They provide a GUI for managing your mods.

However, if you decide to go with a dedicated server you can get a $10 credit with DigitalOcean if you sign up through this referral link. A good starter setup for a small modded server is the $10 month server with 2GB of RAM and 50GB of SSD Disk.

Here are some of the reasons you may want to build your own;

1 - Maximum control right down to the OS level.
2 - Prefer working from the command line than from a Web browser to manage your server.
3 - High memory and cpu needs for your server and a VPS or dedicated server is cheaper than a higher end fully hosted solution.
4 - Have a server available which isn't costing you anything and a good internet link.
5 - Want to learn how to do it yourself from start to finish.
6 - Finally, the most important reason of all.....Because You Can!

Which Linux Distribution?

Recommending a specific Linux
distribution as "the best option" is like recommending a religion as the "best
one to follow". You will either be preaching to the converted or alienating
the person you are having the discussion with.

For my purposes I like using Debian and its derivitaves. Specifically I like
Ubuntu Server and XUbuntu Desktop LTS versions. However I have used other
distributions and have liked them as well. This guide will assume a debian
based distro using the apt package manager. When a package manager is called for, just
substitute the relevant commands for your favorite distro. They are all good.

Root Access to Your Server

The instructions assume that you have root (Administrator) access to your server. For simplicity I assume you are logging in as the root user. All of these commands will also work with sudo. If running sudo from your user account then make sure to add it when necessary. I will not be using sudo in front of these command line arguments throughout the document.

You can also download Oracle Java directly from the command line on your server using wget. Oracle requires that you accept the license agreement,
so the direct download command requires a little trickery in order to bypass this.
The following three commands will change the directory to /opt, download directly from Oracle and then extract the contents
of the file to the opt folder. Copy/paste and run each command one at a time.

a) The next step will be to setup the Java environment so that your system recognizes how to access both the java binary
and java compiler. The compiler is optional but I always install it as a personal preference as I also host spigot servers which need it for build.

b) Make sure you set this version of java (and the compiler) as your default versions. These commands will give
you a list of versions to select, just select the right version option. If this is the only version on your server then it will tell you so and select that one automatically.
Do the same for the java compiler (javac).

update-alternatives --config java
update-alternatives --config javac

c) Finally check your version to make sure you are using the right one

Optional - Removing Old Versions of Java

Old versions of java won't affect anything if you have plenty of disk space. To save a bit of space on your disk you can remove old versions that are no longer used. Simply remove the links to the binaries with update-alternatives and delete the java folder. To remove version 1.8.0_121 for example;

Before installing Forge you will need to install and run the stock minecraft-server.jar file provided by Mojang.

a) create the folder

cd /opt
mkdir minecraft
cd minecraft

b) download the minecraft server.jar and forge jar installer file to the /opt/minecraft folder. You can get the latest directly from Mojang's website but all versions are also available at mcversions.net.
Go to mcversions.net
Download the server version you require and copy it to your /opt/minecraft folder. In this case we are going with version 1.12.2.

Now download the version 1.12.2 Forge Installer. Pick the Latest Installer file (not the Universal file) and copy it to your /opt/minecraft folder.

You should now have two jar files in your /opt/minecraft folder only.

Tip: If connecting to your server from a Windows based pc, use Filezilla or WinSCP to transfer the files to your Linux server

*Note: -Xms1024M and -Xmx2000M are parameters telling java to start with 1GB of ram as a minimum for the server jar file and let it grow to just under 2GB max as needed.
If your server has more ram to dedicate then you can adjust these numbers. More players and more mods mean you need more. These are not constructs unique to Minecraft. Do a google search on (xms xmx jvm) and you
will get some good information on how to manage a JVM's heap. From there you can make a better determination on what you should set this to for your servers resources and how you will be using your server.

For example, on a VPS with 1GB of ram I have set xms and xmx to 512M and 1008M respectively. So start the JVM with 512MB of ram and let it grow to just under 1GB max.

b) accept the EULA. The first time you try to run your minecraft server you will be required to accept the EULA.

nano eula.txt

c) Change the following line to true and save the file

eula=true

d) Run the server a second time to generate your world. World generation will take a little longer the first time. Subsequent server starts
will be much quicker.
Before doing this, you can download the following server.properties file and replace
the one that was created automatically. (or copy/paste directly to your existing server.properties file from the console).
This server.properties file will give you a standard survival world on normal difficulty.

Edit the file with your preferred seed if you have one and it will generate the world that you specify.Minecraft Atlas is great site to find new interesting seeds. Although the site
says Minecraft 1.8, the seeds are compatible with 1.12.2. This site will not likely work for versions earlier than 1.8.View the Wiki for all server.properties configuration options.

nano server.properties
change the following line with your preferred seed directly after the = sign
level-seed=
If you leave it blank Minecraft will generate a random world on first launch

On a low power system you can lower the view distance. Start with 10 for your modded server,
and adjust it downward if you get some lag. If you have plenty of CPU and RAM you can increase it as well.

e) Stop the server and run the forge installer file. In the minecraft-server console type 'stop' and hit enter.
Run the file with --installServer flag.

Right before the last line 'exit 0' add the following command, save and exit the file. This will allow the minecraft server to start in a
detached screen session when the server boots up.

screen -dm -S minecraft /opt/scripts/minecraft.sh

b) To access your console after bootup use the following command;

screen -r minecraft
To exit the screen session use the following command
CTRL AD

Note: Some people may question why I am using rc.local instead of an init script to start the minecraft server. I don't want to run the server in the background, I want
a full screen session accessible to view and interact with the minecraft server console. If I want to stop the server I want to issue a stop command there, not from a script in init.d. Also rc.local is one line
and is super easy. It just works well for this situation. I use init scripts for other things and they are great, but this is just a preference I have for a minecraft server. Do what
works for you.

If you want friends to access the server from outside your home network you will have to forward external requests to port 25565 on
your router to your server. Check the documentation for your router.

a) If you are installing this on a VPS or dedicated server you will most likely want to enable the firewall with netfilter/iptables. You should be blocking ports that are not used or
that you do not want to expose to the internet. An example rule for a minecraft server would be as follows;

Note:If you are not comfortable doing this then a VPS or dedicated server solution may not be a good one for you. If you manage a full Linux server connected directly to the internet with a public IP address, it is your
responsibility to ensure you are hardening and securing it properly. If this is not something you want to do then a managed solution with a minecraft server hosting company like MCProhosting may be a better option for you. They take care of all the dirty work like this.

b) Activate whitelisting. It is extremely important to activate whitelisting if your server is visible on the internet. Port 25565
is a popular port and is actively scanned by all kinds of people using automated scripts.
They are looking for open servers so they can login and cause havoc on your world. Whitelisting will stop
unauthorized users from joining your world.
In your server.properties set the following value to true;

white-list=true
Restart your minecraft server and run the following command at the console for each user you would like to allow.
whitelist add minecraft_user1
whitelist add minecraft_user2
To remove a user
whitelist remove minecraft_user2

When an unauthorized user tries to connect to your server they will simply get a message that they are not
whitelisted and will be turned away.

Copy the minecraft_backup.tar.gz to a safe location, preferably another computer. This file can be used to restore your world if needed. Get into the habit of doing this regularly. You can even use the cron scheduler to automate the process at a specified time each day as explained in Step 12.

The importance of regular backups cannot be overstated. It happens, you try to log into your awesome minecraft
world and you see exception errors only. Or you enter the world and all kinds
of things have disappeared, your sweet enchanted sword is gone, your diamonds
chest is empty. Hours of work and play are now destroyed. Your world may have
corrupted for any number of issues such as software problems or disk errors.

If you did your backups like we discussed in Step 9 then you are laughing.
To restore your previous backup do the following;

You should see the backup happening. Once completed open the file it created
which should be in the location that you specified. Once you have confirmed
that your backup works, create a scheduled task to automate the backups.

f) Create a scheduled task with the cron scheduler
Make sure you are logged in as root so that it writes to your root user crontab.

crontab -e

Enter this line at the end of your root crontab and then save it.

02 2 * * * /opt/scripts/mcbackup.sh &> /dev/null

This will create your backup every day at 2:02 am. That's it, in your folder
where you specified your backups to be created you will have the following
after one week;