In this post I am going to talk about my work for phone builds inside the Taskcluster infrastructure. Mozilla is slightly moving from Buildbot to Taskcluster. Here I am going to give a survivor guide on Firefox OS phone builds.
A task is nothing more than a json file containing the description
of the job to execute. But you don't need to handle the json directly, all tasks
are written in YAML, and it is then processed
by the mach command. The in tree tasks are located
at testing/taskcluster/tasks and the build tasks are
My favorite command to try out a task is the
mach taskcluster-build command.
It allows you to process a single task and output the json formatted task ready
for Taskcluster submission.
$ ./mach taskcluster-build \ --head-repository=http://hg.mozilla.org/mozilla-central --head-rev=tip \ --firstname.lastname@example.org \ tasks/builds/b2g_desktop_opt.yml
Although we specify a Mercurial repository, Taskcluster also accepts git repositories interchangeably.
This command will print out the task to the console output. To
run the task, you can copy the generated task and paste it in the
task creator tool. Then just
Create Task to schedule it to run. Remember that you need
Taskcluster Credentials to run Taskcluster
tasks. If you have
installed, you can the pipe the mach output to
is what we use for effectively build stuff. Mozharness
architecture, despite its code size, is quite simple. Under the
scripts directory you find the harness scripts. We are specifically
interested in the b2g_build.py script. As the script
name says, it is responsible for B2G builds. The B2G harness configuration
files are located at the b2g/config directory. Not
surprisingly, all files starting with "taskcluster" are for Taskcluster
Here are the most common configurations:
- This is the default vcs used to clone repositories when no other is given. [tc_vcs](http://tc-vcs.readthedocs.org/en/latest/) allows mozharness to clone either git or mercurial repositories transparently, with repository caching support.
- The actions to execute. They must be present and in the same order as in the build class `all_actions` attribute.
- The credentials to send update data to the OTA server.
- `True` if this is a nightly build.
- Upload info. Not used for Taskcluster.
- Maps externals repository to [mozilla domain](https://git.mozilla.org).
- Environment variables for commands executed inside mozharness.
The listed actions map to Python methods inside the build class, with
_. For example, the action
checkout-sources maps to the method
checkout_sources. That's where the mozharness simplicity comes from: everything boils
down to a sequence of method calls, just it, no secret.
For example, here is how you run mozharness to build a flame image:
python <gecko-dir>/testing/mozharness/scripts/b2g_build.py \ --config b2g/taskcluster-phone.py \ --disable-mock \ --variant=user \ --work-dir=B2G \ --gaia-languages-file locales/languages_all.json \ --log-level=debug \ --target=flame-kk \ --b2g-config-dir=flame-kk \ --repo=http://hg.mozilla.org/mozilla-central \
Remember you need your flame connected to the machine so the build system can extract the blobs.
In general you don't need to worry about mozharness command line because it is wrapped by the build scripts.
Hacking Taskcluster B2G builds
All Taskcluster tasks run inside a docker container. Desktop and emulator B2G builds
run inside the
builder docker image. Phone builds are more complex, because:
Mozilla is not allowed to publicly redistribute phone binaries.
Phone build tasks need to access the Balrog server to send OTA update data.
Phone build tasks need to upload symbols to the crash reporter.
Due to (1), only users authenticated with a @mozilla account are allowed
to download phone binaries (this works the same way as private builds). And
because of (1), (2) and (3), the
phone-builder docker image is secret,
so only authorized users can submit tasks to it.
If you need to create a build task for a new phone, most of the time you will starting from an existing task (Flame and Aries tasks are preferred) and then make your customizations. You might need to add new features to the build scripts, which currently are not the most flexible scripts around.
If you need to customize mozharness, make sure your changes are Python 2.6 compatible, because mozharness is used to run Buildbot builds too, and the Buildbot machines run Python 2.6. The best way to minimize risk of breaking stuff is to submit your patches to try with "-p all -b do" flags.
Need help? Ask at the #taskcluster channel.