Backstage work to enable interactive web terminals on shared runners

This issue contains part one (back-end items) to implement customer-facing interactive web terminals, which will happen in https://gitlab.com/gitlab-org/gitlab-ce/issues/56073. Please see that issue for details other than the first part of the implementation plan.

Implementation Plan

1. Create a new runner manager

Create a new private runner manager private-runners-manager-5.gitlab.com either under gitlab-runner-prm-gce-us-east1-c or gitlab-runner-prm-gce-us-east1-d which will be used for initial testing for the web terminals, with this we will not interupt the current runner managers if we find performance/securirty issues and we can isolate any problems we might encounter.

The runner will the same executor docker+machince the only difference between private-runners-manager-5.gitlab.com & private-runners-manager-4/3.gitlab.com would be the [session_server] section.

concurrent = 300check_interval = 4metrics_server = "0.0.0.0:9402"sentry_dsn = "***"+ [session_server]+ listen_address = "0.0.0.0:8093" # <- This is where the runner will bind the server+ advertise_address = IP:PORT # <- This the address that will be sent to GitLab.com that will be used to contact the runner and create the websocket tunnel. Ideally this is just the IP of the machine with the same port as the `listen_address`

Questions:

Will the runner logs are automatically picked up or is there any configuration needed there. (infrastructure)

For the value of advertise_address is it simply taking the IP of the machine? Is there some load balancing that needs to be done? (infrastructure)

Requirements/Stakeholders:

~Verify Someone with access to create a new runner

SRE Someone with GCP access to create a new box and expose the IP adress

2. Register a new runner with GitLab.com

A new runner will be registered for the gitlab-org group same as the other private runner managers. The runner should be tagged and only pick up jobs with tags. This runner can be used for both the web terminal inside of the web IDE to start off with.

The runner should have the following tags:

interactive-web-terminal

gitlab-org

Requirements/Stakeholders:

SRE/~Verify Need help from a GitLab.com admin to add the runner.

3. Start using it for web ide on https://gitlab.com/gitlab-org/gitlab-ee

Configure the web ide to use private-runners-manager-5.gitlab.com by specifying tags the tags mentioned in step 2. An example of the web ide configuration can be found below

Create something similar to /debug/jobs/list?v=2 or append to /metrics to add the following data:

Available terminals

Active terminals

With this data we can create a new graph which specifies how many available terminals are available and how much are active
Note:This has not been implemented yet and needs to be discussed if we need something like this

Requirements/Stakeholders:

SRE&~Verify: Help/Mentoring from the infrastructure team is needed to creating the needed monitoring @dawsmith would appreciate the help 🙏

5. Security concerns

We should consider checking if there are any security concerns since this is exposing the gitlab-runner component to the outside world.

Main Concerns:

Exposing gitlab-runner IP to GitLab.com

The user will have access to the docker container, on a root level. Since it's using docker+machince executor and the user escapes from the container he will be in the provisioned machine and not in the runner manager VM.

Requirements/Stakeholders:

Security&~Verify Help/Mentoring from the security team is needed to evaluate this @kathyw would appreciate the help 🙏