This thread should be a place you can ask questions or suggest new features for the Continuous Integration we use with FreeOrion. It isn't meant as a complete documentation but rather a rough outline of what CI is and can do for us.

But let start with a short overview of what is Continuous Integration, how it can help us making a better game and how the current infrastructure looks like:

What is Continuous Integration?

Continuous Integration (CI) is a term for the permanent and complete rebuild of a project from source in a clean build environment which is started after every source code change. In its most basic form this can be done by the developer himself before committing his code changes any may be limited to only rebuilding the application. However this is a very tedious task for the developer: Cleaning the build environment and rebuilding the whole application may be be complex and take some time, which can be better utilized for actual designing/development/bug hunting. Because of that the task of CI is usually done by a build server. The build server is some computer that does nothing else than listening to source code changes in a repository, rebuilding the application in a clean build environment and reporting the build status back to the developers.

CI isn't just limited to rebuilding the project and reporting compile errors. As long as you can write some test or check that can work without human intervention it can be used in CI. This can include, but is not limited to:

Unit testing (Does a small and isolated piece of code conform with some assertions written by developer)

Integration testing (Does a bigger and interacting piece of code conform with some assertions written by developer ;) )

Code Linting (Does the code follow a formatting style?)

Static code analysis (Is there some dubious code use, which may is a certain indicator for a hidden bug? E.g for C++: iterator use after erase, double free, null pointer dereferencing)

Code test coverage (Do the tests actually cover relevant pieces of application code, are there untested branches? If so, how much is not covered?)

Documentation coverage (Does documentation exist for a function, are all parameter documented? Are special code cases (e.g. Exception thrown) documented? Are the cross references valid?)

When adding all of those checks to the CI they are done for every single code change. This can aid the developer by pointing out flaws in his change early and helps code reviewers by doing tedious reviewing tasks so that the reviewer can focus on high level concepts like correct algorithm implementation/usage and code design rather than wrongly indented code.

On top of that CI may help with the software delivery (release). As a CI build is done from a clean build environment there it is way harder to create broken releases. Also a CI build may trigger not only on code changes, but on time events or code tagging too. This does allow regular releases on a scheduled time (nightly builds/weekly builds) or on tagging (release tags) without much human intervention.

What CI tools does FreeOrion use?

Currently we rely on several free services and tools to implement the CI. Those are:

GitHub - The place were we host the code on and which triggers via a webhook (HTTP requests send by GitHub on certain events) other services whenever a commit is done or a pull request is submitted.

TravisCI - A CI hoster, which provides virtual machines for the event of building source code. Travis provides Ubuntu and MacOSX based virtual machines to build the code and a web interface to both review the build reports and stop/rerun a build from the same commit. The configuration is done via a file called `.travis.yml` file in the root of the source repository, which is a conglomerate of configuration values and bash shell code snipplets.

AppVeyor - Another CI hoster, which does essentially the same as Travis, but provides virtual machines for Windows. The configuration is done via a file called `appveyor.yml` file in the root of the source repository, which is a conglomerate of configuration values and powershell code snipplets.

Docker - This is a virtualization software for applications, which does allows users to describe Linux virtual machine image by writing installation instructions for said image. It also consists of a hosting service to store those precomposed VM images and make them available for download. We use this tool to compose a virtual machine for Linux, which contains all build dependencies for FreeOrion. TravisCI by itself provides a rather slim and conservative designed virtual machine image of Ubuntu, which doesn't allow a somewhat recent C++11 build environment + all the special tools we would need for more complex CI tasks.

freeorion-bot - This is a GitHub/Docker account, solely created for the purpose of interacting with the services mentioned above. Currently when a build happens it is done on behalf of the freeorion-bot, not a certain developer. This will become more useful when weekly builds are a thing or then the CI will do semi-automatic code reviews.

What CI tasks are already in place for FreeOrion?

Currently we have the bare essentials in place.

For the `freeorion/freeorion` repository we do:

Build every commit/commit series and pull requests on Windows, Ubuntu 16.04 and MacOSX 10.10.

Report build issues on commits/commit series and pull requests on the GitHub web page and via developer eMail.

Build checks itself are limited to whether a build successfully compiles or not.

For the `freeorion/freeorion-sdk` repository we do:

Build every commit/commit series and pull requests on Windows and MacOSX 10.10, create a Docker image for `freeorion/freeorion` and update it if necessary.

Report build issues on commits/commit series and pull requests on the GitHub web page and via developer eMail.

Build checks itself are limited to whether a build successfully compiles or not.

Create release draft of the FreeOrionSDK from the build artifacts whenever a developer tags a release (in case you ever wondered why `freeorion-bot` is the one releasing the SDK and not me).

What are the limitations of the FreeOrion CI

As open source project we don't have any other investment than our interest, the knowledge and our time. This means we're pretty much limited to whatever we can get free of charge. All of GitHub, TravisCI and AppVeyor provides their services free of charge for open source projects but limit the use of said service. For example TravisCI limits the time to build to 50 minutes. If a build exceeds this time limit it is killed off. AppVeyor limits this to 60 minutes. When adding CI tools we need to keep this in mind. Also I would consider it a sign of good will to reduce the build time whenever possible, may it be by optimizing the code, the build or by other means and also keep in mind that we share these resources with other open source projects.

Another point are the capabilities of the CI in general. As already mentioned CI is limited to checks, which work non-interactive. When considering a new great tool for CI keep always in mind that the tool MUST work without a user typing commands into a shell or a user clicking away error messages because the virtual machines don't expose any user interface.

Also (but this is a personal interprecation of me) CI should never modify the project. That means the CI service should NEVER commit to the source or modify history. A CI should report issues, but not try to fix them. Maybe the CI does a dumb decision when trying to fix a bug, maybe the CI is buggy itself. The CI is a helper for tiresome tasks and a constant reminder for things that may cause problems, but the final decision is up to the developer.

adrian_broher, thanks for setting up the CI and explaining wit and offering to answer questions.

Thanks also for fixing the unit tests. They have been broken for a long time.

I have questions.

1. Do you know of specific techniques that improve build times?

2. Do you know how much the PIMPL idiom improves build times?

I use it to improve re-compilation times. I have not checked its impact on a clean build's duration.

3. You recently added a job to check PEP8 conformance. I've tried to reproduce freeorion's C++ style with some style checkers, clang and astyle. I was unsuccessful.

Could we tweak freeorion's C++ style so that it works with one of these style checkers? Adopting a style that can be checked with the CI would make that aspect of code reviews disappear.

If the answer to this question is yes, then it might deserve its own thread to answer what style checker we should use and what style conventions we are willing to change in order to facilitate automation.

AAH!! I really need some warning like The post you're editing is not your own, continue?. Thankfully I still had the original tab open.

Thanks also for fixing the unit tests. They have been broken for a long time

Unfortunately the current state is still mostly broken and still needs repair and tweaking.

1. Do you know of specific techniques that improve build times?

In general I experienced the biggest improvements (in different projects unrelated to FO) when the amount of included headers was reduced. The problem with headers used is that included files may include 5 other files, which may in turn include around 2 to 10 files, which in turn … you get the idea. Suddenly you have thousands of file accesses during the compilation of a single file because someone used and include instead of a forward declaration. On a system with HDDs this can slow down the compile significantly. The other problem are templates. Whenever a template instantiated this happens in the context of the compilation unit. When using the sample template type ever compilation unit compiles the functions separately and the linker finally discard all duplicated instantiations.

2. Do you know how much the PIMPL idiom improves build times?

I use it to improve re-compilation times. I have not checked its impact on a clean build's duration.

Idoms like PIMPLs may help with the includes, but I don't feel like they should by used everywhere because they increase the code complexity in all dimensions (harder to understand, to write, a bit of runtime overhead. A more simple solution is the strict separation of independent classes into different headers. This cuts down the inclusion of unneeded headers significantly. Where meaning of independent may differ between cases. For example a lot of code need to know the declaration of ValueRef::ValueRefBase but only a small subset of code needs to know the declaration of subclasses of ValueRef::ValueRefBase. However the subclasses themselve have dependencies, which are completely irrelevant for most of the client code. Cutting down those dependencies should help a lot in terms of total and incremental compile time as code changes don't cascade through the code.

Regarding the templates the use of C++11 extern templates may help here. However I haven't collected enough experience with those yet.

3. You recently added a job to check PEP8 conformance.

No I haven't. The task is still open. The commits you're referring to are intended for IDE/editor use (and to gain experience with the tools). I'm still not sure how integrate this into the CI build. So far I'm reading up about the various external services like codeclimate or codecov and try to understand if they are a good fit into the current CI.

I've tried to reproduce freeorion's C++ style with some style checkers, clang and astyle. I was unsuccessful.

The code FO style conventions are IMO very special in the bad kind of way. There are various exceptions and conditions but lax rules in other places. I tried those tools too and failed the same way. They are not flexible enough to represent the formatting (which says enough about the formatting to be honest).

Could we tweak freeorion's C++ style so that it works with one of these style checkers? Adopting a style that can be checked with the CI would make that aspect of code reviews disappear.

If the answer to this question is yes, then it might deserve its own thread to answer what style checker we should use and what style conventions we are willing to change in order to facilitate automation.

Well, from the technical view it is of course it is possible. But is the team willing to change habits? Is there consensus on the bikeshed color? Also a bulk change of the existing code will cause a disruption in code history which I'm not very fond of. On the other hand reformatting code only when touched could lead to cases where the code will never change, because it works the way it exists.

In the end I consider the style enforcement not as important as running test, deployment and static code analysis which I would prefer to working on first.

For travis, at least looking at adding [skip travis], similar to integral [skip appveyor] (supposedly, I've not had a circumstance where it would have been useful yet to test).
Travis is a little more complex for file exclusion (e.g. facebook/react PR grepping git diff).

I'm going to add long test for the game. What if split current BUILD_UNITTEST=true test on two separate jobs: one for UI and GG only and the second one with BUILD_HEADLESS=On enable for parser and game only?

o01eg wrote:I'm going to add long test for the game. What if split current BUILD_UNITTEST=true test on two separate jobs: one for UI and GG only and the second one with BUILD_HEADLESS=On enable for parser and game only?

I don't like this idea, because the whole HEADLESS build is dubious for me. Separating by regular build (server+human client+ai client) and a unit test only build would be more logical to me. Also I need to investigate if we can share build artifacts between the stages so the test stage can use the artifacts generated by the build stage somehow.

Cjkjvfnby wrote:All commits from the PR creation or from the last CI build execution should have an[must have such a skip] entry [for the test(s) to be skipped]. (edited by Dilvish for clarity)

Could you edit that again / rephrase? It's not clear to me what this means.

Edit: Also, is it possible to disable parts of the CI build? eg. Something in the UI code changed, so I don't need a rebuild of the parsers, or in the server, so I don't need the client, AI, parsers, or GG code rebuilt.

Cjkjvfnby wrote:All commits from the PR creation or from the last CI build execution should have an[must have such a skip] entry [for the test(s) to be skipped]. (edited by Dilvish for clarity)

Could you edit that again / rephrase? It's not clear to me what this means.

My understanding of the situation, which I had tried describing better, was that the CI checks would only be skipped if all of the new commits said to skip them. Would it clarify things more to say "in order for the tests to be skipped" rather than just "for the tests to be skipped"? (edit, I'll answer my own question and say 'yes, it would clarify more', and you can answer the implied question about whether that is clarified enough)

If I provided any code, scripts or other content here, it's released under GPL 2.0 and CC-BY-SA 3.0

Geoff the Medio wrote:Also, is it possible to disable parts of the CI build? eg. Something in the UI code changed, so I don't need a rebuild of the parsers, or in the server, so I don't need the client, AI, parsers, or GG code rebuilt.

Not quite sure what you're asking for.

TravisCI already does incremental builds for Linux and MacOS and ccache is smarter than the average developer on the decision what to and what not to compile bringing the compile times down to a few minutes total.