execute TTCN3 test suites against "latest" feeds

It would be good to run our test suites not only against current master builds, but also against the "latest" packages. This way we can nicely show the amount of bugs fixed (and/or regressions pending) between the latest tagged version[s] and the current master.

In I6a564206dd81743deb1eb27eca7081bc333d7434 of docker-playground.git I already introduced Dockerfiles:

generalization of the "jenkins.sh" scripts to either test master or latets

migration of ttcn3 jenkins jobs to JJB, so we can template them and duplicate them

Potential problems: The tests use hard-coded IP addresses, and docker cannot have two networks with overlapping addresses, even if there's no routing between them :( We would then have to make sure that no ttcn3-bts-test-master and ttcn3-bts-test-latest jobs are running in parallel on the same build slave. A possible work-around would be to have separate build slaves for master/latest. Using different IP addresses (172.18.9.0/24 for ttcn3-bts-test-master and 172.17.9.0/24 for ttcn3-bts-test-latest) is of course desirable, but this would require different osmo-*.cfg and different TTCN-3 testsuite config files, etc :(((

Potential problems: The tests use hard-coded IP addresses, and docker cannot have two networks with overlapping addresses, even if there's no routing between them :(

We could do it like gitlab CI: each docker instance is isolated inside a virtual machine. Then we don't need different IPs when running "latest" and "master". And as additional benefit, the config files could be used without any changes when running the TTCN3 testsuite without Docker (when we use 127.0.0.1 as the IPs).

laforge: do you want that? If so, I could make a follow-up issue for that.

A possible work-around would be to have separate build slaves for master/latest

Potential problems: The tests use hard-coded IP addresses, and docker cannot have two networks with overlapping addresses, even if there's no routing between them :(

We could do it like gitlab CI: each docker instance is isolated inside a virtual machine. Then we don't need different IPs when running "latest" and "master".

how is this done exactly? do you hav a pointe explaining the setup?

"Projects hosted in GitLab can have CI tasks defined in their .gitlab-ci.yml files. These tasks are performed by runners which are essentially virtual machines which run your builds in Docker containers. These machines can run any of your builds that are compatible with Docker."

"Shared runners" are the ones that can be used for free on gitlab.com (vs. running your own gitlab instance). These runners can have different "executors" (environments to run the docker container in, SSH, shell, VirtualBox, ...), and from the config linked in the article above, they are using the "docker+machine" executors.

"The Docker Machine is a special version of the Docker executor with support for auto-scaling. It works like the normal Docker executor but with build hosts created on demand by Docker Machine."

With that being said, using "Docker Machine" might be a bit over-engineered for our use case. We could also run a Qemu VM with an image we created before, connect to it via SSH, execute the test, and kill it afterwards. I think this can be done in a ~100 line python script.

To make writing the new JJB YML files easier, I've dumped the existing Jenkins jobs with jenkins-job-wrecker and converted them to YML. I will still need to rewrite them from scratch, but that should make it easier to see the what is common/different between the ttcn3-* jobs.

I've introduced a new CONTAINER_SUFFIX variable in all jenkins.sh files of the testsuites. By default it is set to "master", and the Jenkins job can override it to "latest".

Besides that, there is a new docker_images_require() function in jenkins-common.sh (docker-playground.git). It accepts docker image names (ttcn3-sip-test, debian-jessie-build, ...) and builds all of them which do not exist. That function gets called in all jenkins.sh files now. This has the following advantages:

all images will be present before the testsuite starts (or the script will abort there, before running network_create; right now it just runs the testsuite even if images are missing)

we can use the $CONTAINER_SUFFIX variable in the required image names

this list can easily be maintained (vs. the top level makefile, which is not really maintained and it's easy to make mistakes there like depending on an image name that is not a target in the makefile, which will then always appear to be built because the folder exists)

it is not necessary to run an additional command (make ...) before running jenkins.sh, as it will build the images. this simplifies the jenkins jobs (right now each jenkins job is calling make on the images that need to be built for its job before running jenkins.sh)

So far one short testsuite.yml seems to be able to generate all the testsuite-* jenkins jobs :)

On a related note, I've had a discussion in the local hackerspace about the overlapping network addresses problem. Someone said, it should be possible to use overlapping network addresses if we configure the network bridges ourselves and do not let docker manage them, just tell docker to use them. Something like that, I have not looked into it more. Maybe this is helpful.

From the online help:Block build if certain jobs are runningHelp for feature: Block build if certain jobs are runningEnable the build blocker to prevent this job to run while one of the here configured other jobs are running. The blocked jobs stays in the queue until all blocking jobs are not running.

On a related note, I've had a discussion in the local hackerspace about the overlapping network addresses problem. Someone said, it should be possible to use overlapping network addresses if we configure the network bridges ourselves and do not let docker manage them, just tell docker to use them. Something like that, I have not looked into it more. Maybe this is helpful.

Thanks, I'd actually prefer to get upstream fix their issues. Linuxnetwork namespaces exist to provide isolation, and hence there's nothingthat prevents you from having the same internal addresses/subnets indifferent namespaces. The restriction docker imposes is an undueconstraint on the capabilities of the Linux kernel network stack.

It may be as simple as disabling a single line inside docker source codeitself, I haven't yet had the time to look at it in more detail. Mightbe worth to at least report it as an issue with them.

Please don't rename the existing jobs, even if it ends up being inconsistent.

Okay. I'll delete the new ones, and change the naming scheme of the JJB config to match the existing ones.It would be possible to overwrite the existing (manually created) jobs that way, but I will comment that out until the "latest" jobs have been tested properly.

Also, regarding the "latest" build jobs: It seems their test results analyzer is broken.The build are successful as per 'console output', but the junit XML is not imported

I did not have the test result analyzer configured in the jenkins job builder config at that point. Later I did, and the test result analyzer became functional (I ran other jobs for that though).

ttcn3-bts-test-latest: changed to depend on osmocom-bb-host-master, even if we are testing against latest. Because there is no "latest" tag for osmocom-bb (so we can't build a osmocom-bb-host-latest docker container).

ttcn3-bsc-test-sccplite-latest: figuring out why this is stuck and runs forever. Unless assumed earlier, this is not related to sudo - all processes are running as root inside the Docker container anyway. I can reproduce this on my laptop, and it only happens when running -latest, not for -master. I am certain that it is not related to ttcn3-tcpdump-start.sh now, because it says that this script was executed successfully before it hangs:

Right now I'm installing a few debugging utils (pstree, strace, ...) in my local debian-stretch-titan container, so I can enter the docker container while it hangs and figure out what is hanging exactly.

ttcn3-bts-test-latest: changed to depend on osmocom-bb-host-master, even if we are testing against latest. Because there is no "latest" tag for osmocom-bb (so we can't build a osmocom-bb-host-latest docker container).

Not nice, but ok as an interim solution for now.

ttcn3-bsc-test-sccplite-latest: figuring out why this is stuck and runs forever. Unless assumed earlier, this is not related to sudo - all processes are running as root inside the Docker container anyway. I can reproduce this on my laptop, and it only happens when running -latest, not for -master. I am certain that it is not related to ttcn3-tcpdump-start.sh now, because it says that this script was executed successfully before it hangs:

please simply disable this job for the time being and work on other topics meanwhile.

Right now I'm installing a few debugging utils (pstree, strace, ...) in my local debian-stretch-titan container, so I can enter the docker container while it hangs and figure out what is hanging exactly.

The problem is likely that one of the tests is executing without a proper guard timer, i.e. it's waiting indefinitely for some event to occur which then never happens. The log files and possibly pcap file should indicate where exactly. I guess it's ok to put this aside until somebody with more experience can hav ea look and meanwhile work on other stuff.