GitLab 9.4 released with Related Issues and Web Application Monitoring

GitLab 9.4 Released with Related Issues and Web Application Monitoring, New Navigation, Group Milestones and much more!

Surprise is harder to achieve when you do everything in the open. But working in the open gives us the power to tell you why we're shipping what we're releasing today and how this release is setting up GitLab for something even better in the future.

GitLab 9.4 lays the foundation of much that is to come, while still giving you some new powers today. You can now formally relate issues to each other, our out-of-the-box-magic monitoring now collects many more metrics without any configuration and we've quadrupled the things you can do with variables in CI.

On top of this, we're giving you an actual glimpse into the future with a opt-in Beta of our new navigation. We hope that we can work with you to make it an improvement everyone loves.

We are also thrilled to announce that we are shipping a GitLab PowerUp for Trello, making it easy for you to integrate your Trello boards with GitLab! 🎉

Additionally, to empower our integrations set, we're keen to make your life easier with our new Slack App for GitLab.com!

And if one glimpse doesn't suffice, we're aiming to fully automate the configuration of your DevOps toolchain with the vision of Auto DevOps, which will analyze your application and automatically configure your CI/CD pipeline to build, test, and deploy to Kubernetes. To see where we’re heading, check out our vision for Auto DevOps!

Matt added support for EC2 instance profile credentials when using AWS hosted elasticsearch clusters, while we could only use static IAM credentials to access them before. This is a complex contribution and a significant improvement to GitLab’s integration with elasticsearch, as it allows some aspects of our Advanced Global Search functionality to be automatically configured when GitLab is run on AWS.

Key features released in GitLab 9.4

Related Issues

Whenever you share a link from one issue to another, GitLab shortens it and crosslinks it automatically. But when issues get longer and projects more complex, it becomes hard to manage links and quickly find related issues.

To solve this problem, we’re introducing Related issues. With Related issues, you can formally declare another issue as related. A link to the other issue, its status and name will be shown in each issue.

Simply paste a link to the issue you want to link or search for it by typing # (as you were able to do already) to link it. In the future, we will introduce different types of relationships through this mechanism.

New Navigation

To make it easier and faster to get around GitLab, we’re working on updating our navigation. Because a new navigation can be a big disruption, we’re releasing the first step as an opt-in configuration with GitLab 9.4.

To enable the new navigation, click on your profile image in the top right corner and choose Turn on new navigation.

We have made adjustments to the global top navigation and introduced contextual navigation in the left menu depending on what page you are currently viewing. The new UI is still a work in progress and will replace the existing navigation in the next few months, please see our blog post about our process and what work still needs to be done.

Group-level Secret Variables

Secret variables are really useful when you need a safe place to store sensitive information. Until now, secret variables were stored at the project level. However, we know its common for different projects in the same group to share information on deployment or credentials for accessing external services.

Group-level Secret Variables remove the need to duplicate variables from one project to the next: now you can enter these values once, and each project or subgroup in the group will access them automatically. It’s also really simple to update these values. You just change them in one place and they’ll be automatically modified for all of the projects.

Variables in Pipeline Schedules

In GitLab 9.2 we introduced Pipeline Schedules to automatically run pipelines at a specific interval of time, but most teams also want to specify different values for specific variables when running the schedule.

In GitLab 9.4, we’ve added the ability to define variables when creating or modifying a pipeline schedule: these values will be added to all the other variables already defined. Using this feature, you can also redefine existing variables to have a different value only for that specific run, for example if you want to have a “daily” pipeline running some tests in a different way.

Environment-specific Secret Variables

Variables are often the right solution to define values that are then used during deployments to specific environments. Since different environments (e.g.: staging and production) may require different values for the same task, such as the app name, it is important to create a direct binding between some variables and the related environment.

With GitLab 9.4, Environment-specific Variables are introduced to solve this issue, as developers can now define which environments will receive a variable, even using wildcards to include dynamic environments, like `review/*. It is now easy to deploy to different environments with a minimal effort!

GitLab Power-Up for Trello

Using both Trello and GitLab? Now you can make that experience even better with the new GitLab Power-Up! In Trello, when viewing one of your boards, simply go to Power-Ups and scroll to the GitLab Power-Up. After setup, you will be able to attach merge requests to Trello cards.

In Trello, you will need to configure your domain, such as gitlab.com/api/v4 for GitLab.com, and add your personal token.

GitLab Slack App for GitLab.com

🚀 GitLab already integrated deeply with Slack (and Mattermost, Microsoft Teams, and HipChat), but we didn’t have an app in the Slack App Directory yet. Today we do 🎉! That means setting up Slack integration with your projects on GitLab.com is now much easier.

You can set it up from your project settings in GitLab (Settings > Integrations). Soon it will be available from the Slack App directory as well. We’re working together with Slack on making sure private instances will be able to use the same Slack App in the near future. Of course, private instances are able to integrate with Slack using the manual steps outlined in the documentation.

Other improvements in GitLab 9.4

Improved Internationalization

We are accelerating our efforts with Internationalization. A big thank you to members of our community who are already contributing additional languages including Chinese, French, Japanese and Brazillian Portuguese. A huge thanks to Huang Tao for continuously contributing to the cause!

In GitLab 9.4 we have added Internationalization support for the Commits page.

Unified Slack Interface

With this release, we are unifying the issue information shown in Slack in response to a link being pasted, a Slack slash command being executed, or a Slack notification. This provides a coherent experience regardless of how an issue is referenced in Slack.

Group Milestones

Milestones are fundamental to issue tracking. They are often used to designate a sprint (week 35), a release (9.4) or a category (Backlog) of issues and merge requests. Often milestones span multiple projects: you were already able to quickly create milestones in multiple projects at once in GitLab. To make this experience better, we’ve now added the ability to create Group milestones.

Group milestones behave exactly like their counterpart project milestones, but are created in the group and from there available to all projects directly belonging to that project.

To create a group milestone, visit your group and go to Issues and then Milestones.

PostgreSQL High Availability (Beta)

GitLab 9.4 arrives in Beta for deploying the bundled PostgreSQL server in a highly available manner, with a simple manual action to recover in the event a node goes down. This allows for a more robust deployment of GitLab, reducing the duration of an outage and potential for data loss. We are working to automate the failover process in a subsequent release.

Additional GitLab Service Metrics

With GitLab 9.4 we have added additional instrumentation to our Ruby on Rails code, providing insight into the HTTP requests, latency, and errors at the Rack middleware layer. We will continue to instrument additional areas of GitLab in subsequent releases.

Mini-Graph for Multi-Project Pipelines

We recently introduced Multi-Project Pipeline Graphs to easily follow dependencies between related projects. With GitLab 9.4 we extend this also to mini-graphs in the Merge Request widget, where you can now see pipelines for downstream projects with their current status.

Customizable Path for CI/CD Configuration

GitLab defines CI/CD configuration in a YAML file named .gitlab-ci.yml that is located in the root of the repository. There are cases where you need to specify a different location for the definition of your pipelines, for example when you mirror a SVN repository and cannot have files in the root of the project.

Starting with GitLab 9.4, you can now specify in Settings > Pipelines a custom path that will be used to read CI/CD configuration instead of the default .gitlab-ci.yml. A variable named $CI_CONFIG_PATH is available to jobs in order to access the current config location.

Object Storage for CI Artifacts

As companies continue to embrace CI/CD across the organization, their artifact storage needs naturally increase as well. With GitLab 9.4 you can move existing CI artifacts to Amazon S3, in order to free local space and enable artifacts to be saved cost effectively, reliably, and with nearly infinite scalability. For now this operation has to be run every time you want to move your local artifacts to S3, but in a future iteration it will be automatically done so all the new artifacts will be saved on object storage as soon as they’re created, without the need of a manual migration.

New Cache Policy for CI/CD Configuration

The default behaviour of a caching job is to download the files at the start of execution, and to re-upload them at the end. This allows any changes made by the job to be persisted for future runs, and is known as the pull-push cache policy.

The default behaviour of a caching step is to restore and archive dependencies of your jobs, allowing speed-up of subsequent runs. If any change is made to the cached content, it is pushed to the cache server by default, and this behavior is known as the pull-push cache policy.

If you’re not interested in updating the cached files for a specifig job, you can skip the upload step by setting policy: pull in the job specification. Alternatively, if you have a job that unconditionally recreates the cache without reference to its previous contents, you can use policy: push to save a lot of unnecessary load on the cache server. This feature requires GitLab Runner 9.4 or above.

Extended Docker Configuration for CI/CD

In GitLab 9.4 we introduce new advanced configuration options for .gitlab-ci.yml that allow you more flexibility when defining Docker images that you want to use for your pipelines. These improvements require also GitLab Runner 9.4 or above.

You can now specify a custom entrypoint for your Docker image, in order to override the default one: here is an example to set the entrypoint to /bin/sh, making the image suitable for GitLab CI jobs without further modifications:

image:name:super/sql:experimentalentrypoint:["/bin/sh"]

You can also specify aliases for services to run multiple concurrent instances of the same Docker image, and specify commands directly in the configuration file.

Improved Prometheus Monitoring of Kubernetes Deployments

Since the release of GitLab 9.0, the Prometheus server bundled with Omnibus GitLab has supported monitoring Kubernetes nodes for CPU and Memory metrics. With the release of 9.4, the Prometheus server will also automatically monitor any specifically annotated Pod metrics as well.

Security - Add LDAP SSL Certificate Verification

We added support for certificate verification for LDAP over SSL. This option will be disabled by default for backwards-compatibility until GitLab 10.0. Additionally, to aid in configuring a secure connection, you can now specify a CA certificate file and SSL version. Encryption method names ssl and tls have been deprecated in favor or simple_tls and start_tls, respectively.

Upcoming Omnibus Package Signing

Starting with our 9.5 release on August 22nd, we will be signing all new packages going forward. Along with the 9.5.0 package, we will also be providing signed versions of the latest 9.4 and 9.3 releases.

Package signing provides additional confidence that the .deb and .rpm files used to install GitLab have not been surreptitiously modified.

Omnibus Improvements

When using a container registry that is external to the Omnibus GitLab package, there is additional configuration required starting with the 9.4.0 release.

Previously, only the path to the registry key used for communication between GitLab and the registry (gitlab_rails['registry_key_path']) was necessary. Now along with the path, it is also required to specify the content of the key by setting registry['internal_key'] inside of /etc/gitlab/gitlab.rb. Documentation for utilizing an external registry will be available soon.

Performance Improvements

We are continuing to make great strides improving the performance of GitLab in every release. We’re committed to not only making individual instances of GitLab even faster, but also to greatly improving the performance of GitLab.com, an instance that has over 1 million users!

In GitLab 9.4 we are shipping performance improvements for issues, projects, milestones, and a lot more!

Unified Slack Interface

With this release, we are unifying the issue information shown in Slack in response to a link being pasted, a Slack slash command being executed, or a Slack notification. This provides a coherent experience regardless of how an issue is referenced in Slack.

PostgreSQL High Availability (Beta)

GitLab 9.4 arrives in Beta for deploying the bundled PostgreSQL server in a highly available manner, with a simple manual action to recover in the event a node goes down. This allows for a more robust deployment of GitLab, reducing the duration of an outage and potential for data loss. We are working to automate the failover process in a subsequent release.

Mini-Graph for Multi-Project Pipelines

We recently introduced Multi-Project Pipeline Graphs to easily follow dependencies between related projects. With GitLab 9.4 we extend this also to mini-graphs in the Merge Request widget, where you can now see pipelines for downstream projects with their current status.

Customizable Path for CI/CD Configuration

GitLab defines CI/CD configuration in a YAML file named .gitlab-ci.yml that is located in the root of the repository. There are cases where you need to specify a different location for the definition of your pipelines, for example when you mirror a SVN repository and cannot have files in the root of the project.

Starting with GitLab 9.4, you can now specify in Settings > Pipelines a custom path that will be used to read CI/CD configuration instead of the default .gitlab-ci.yml. A variable named $CI_CONFIG_PATH is available to jobs in order to access the current config location.

New Cache Policy for CI/CD Configuration

The default behaviour of a caching job is to download the files at the start of execution, and to re-upload them at the end. This allows any changes made by the job to be persisted for future runs, and is known as the pull-push cache policy.

The default behaviour of a caching step is to restore and archive dependencies of your jobs, allowing speed-up of subsequent runs. If any change is made to the cached content, it is pushed to the cache server by default, and this behavior is known as the pull-push cache policy.

If you’re not interested in updating the cached files for a specifig job, you can skip the upload step by setting policy: pull in the job specification. Alternatively, if you have a job that unconditionally recreates the cache without reference to its previous contents, you can use policy: push to save a lot of unnecessary load on the cache server. This feature requires GitLab Runner 9.4 or above.

Improved Prometheus Monitoring of Kubernetes Deployments

Since the release of GitLab 9.0, the Prometheus server bundled with Omnibus GitLab has supported monitoring Kubernetes nodes for CPU and Memory metrics. With the release of 9.4, the Prometheus server will also automatically monitor any specifically annotated Pod metrics as well.

Upcoming Omnibus Package Signing

Starting with our 9.5 release on August 22nd, we will be signing all new packages going forward. Along with the 9.5.0 package, we will also be providing signed versions of the latest 9.4 and 9.3 releases.

Package signing provides additional confidence that the .deb and .rpm files used to install GitLab have not been surreptitiously modified.

Performance Improvements

We are continuing to make great strides improving the performance of GitLab in every release. We’re committed to not only making individual instances of GitLab even faster, but also to greatly improving the performance of GitLab.com, an instance that has over 1 million users!

In GitLab 9.4 we are shipping performance improvements for issues, projects, milestones, and a lot more!

Deprecations

openSUSE 42.1

As the openSUSE community has ended support for version 42.1, GitLab has ended support as well as previously announced. Please upgrade to OpenSUSE 42.2 which is officially supported.

Removal date: July 22nd, 2017.

GitLab CI API v1, GitLab Runner 1.11.x

In 9.0 we released a new version of GitLab Runner that is based on the new API v4 instead of the old CI API v1. We are still supporting the old version of the API in GitLab, so users that are still using GitLab Runners 1.11.x can take their time for the migration process. With GitLab 9.6, planned to be shipped on September 22nd, we are going to remove the old CI API from GitLab, making GitLab Runner 1.11.x unable to communicate with the system. If you are using old GitLab Runner (<9.0), or any tools that are using GitLab CI API v1, an upgrade will be required.

Upgrade barometer

For this release we have migrations, post-deploy migrations, and to help with the larger migrations we have introduced background migrations.

GitLab.com migrations took approximately 24 minutes and post-deploy migrations amounted for a total of around 54 minutes. Background migrations on the other hand are sidekiq jobs that will run synchronously, for this release these migrations took around two days to complete, no side effects were expected nor present, these target older pipeline builds so newer executed pipelines are not affected.