Dockerizing OpenQA

If you checked the Installation Matrices in the last two months or so, or read any of the HappyAssassin’s blogposts, you know that we started using OpenQA to automate some of the Installation Testcases.

The OpenQA project is heavily developed, which means both quick fixes and responses to feature requests, but also a bit of a pain for “production use” – when the stuff is changing rapidly, only AdamW can keep the pace 🙂 Some of this is addressed by the OpenQA’s stable branch, but even that is changing a lot from update to update, and to asses the tool properly, we really wanted to be able to just use a certain version, figure stuff out, and set it up in production.
The other “issue” is, that you need to have an OpenSuse box to use OpenQA, and having OpenSuse in Fedora Infrastructure would most certainly be frowned upon. Thus Garretraziel (mostly) and I (mostly cheerleading) decided to “freeze” the software in a specific state, and what better to use than Docker !

In the end, it took us two days of what I like to call “extreme Dockerizing” – extensive swearing, getting elbows deep into the code, and some tears of joy and hate – to get it done.

The weirdest error we had to solve, was not caused by the OpenQA though, but by the content of the OpenSUSE’s base Docker image. What rendered as an “No OpenID provider found at $URL”, was in fact caused by the missing ca-certificates package. So if you ever need to use the Docker image for accessing secure http pages, do not forget to install the package in your Dockerfile.

Some other issues were tied to the fact, that the OpenQA WebUI and Worker are not really that prepared to be on separate machines – it works in the end, but sharing of some directories needed to be solved. This is the thing that we need to solve better, in some time – now we just have a shared VOLUME which contains the shared data, and some configuration. That means we need to set a+rwx on the shared dir, and figure out either UID/GID squashing or ACLs later.

But, in the end, IT WORKS! If you want to run your OpenQA instance, you can do it from the comfort of your home^W distribution of choice. It is quite simple, in the end, you just

Get the docker images:

docker pull fedoraqa/openqa_webui 
docker pull fedoraqa/openqa_worker

Pull our repo containing some bits and pieces(thanks Bernhard for pointing me to the .gitignore files for adding empty dirs):

git clone -b feature/docker https://bitbucket.org/rajcze/openqa_fedora_tools openqa_fedora_tools
cd openqa_fedora_tools/docker
git co -b docker be42c791474b02131e0f65413e90605998c420d0
git submodule init
git submodule update
cp -a data.template data
chmod -R 777 data

Next up, run the WebUI:

docker run -d -h openqa_webui -v `pwd`/data:/data --name openqa_webui -p 8080:443 fedoraqa/openqa_webui

It is important, that you are in the root of the cloned git repository, since the -v `pwd`/data:/data parameter mounts the directory into the Docker’s VOLUME.

Check whether WebUI runs by accessing https://localhost:8080 and logging in. Go to https://localhost:8080/api_keys, generate key and secret. Uncomment the [openqa_webui] and [localhost] sections in data/conf/client.conf, and fill in the key and secret.

Populate OpenQA’s database:

docker exec openqa_webui /var/lib/openqa/tests/fedora/templates

And run the worker:

docker run -h openqa_worker_1 --name openqa_worker_1 -d --link openqa_webui:openqa_webui -v `pwd`/data:/data --privileged fedoraqa/openqa_worker 1

Once again, make sure that you are in the root of the cloned git repository, we are once again using that data dir.
Running multiple workers is also easy – just substitute all the 1‘s with 2/3/4…
If nothing went wrong, then you should see the worker as connected in the WebUI.

To run a job
you need to download a Fedora Installation image (Server DVD, or Server Netinst) into the data/factory/iso directory. For Fedora-Server-netinst-x86_64-22_Beta_TC6.iso the command running a default-install check would be:

docker exec openqa_webui /var/lib/openqa/script/client isos post ISO=Fedora-Server-netinst-x86_64-22_Beta_TC6.iso DISTRI=fedora VERSION=rawhide FLAVOR=generic_boot ARCH=x86_64 BUILD=22_Beta_TC6

And that’s it. You can see, that there still are some quirks – the most painful being access control to the shared data – but hey, it’s just two days of work, and we’re not done yet 😉