Since I haven’t blogged in a while a lot of stuff has piled up on my desk and I’ll get it out in batches in this post I’ll go over a few of the bigger changes we did while I was away on the bootcamp and also give you a bit of an update on what we’ve been doing within the bootcamp itself.

The week before we launched the bootcamp our Mac build servers reached a very heavy workload, this was becoming disruptive to our general developer population as builds got queued at a high rate. After a bit of investigation it seems that this could usually be pinned to specific users with very long build times.

Normally build time is divided as:

The latter is pretty fast since our servers have a very fast connection, the first depends on your Internet speed however the queue is problematic…​

We have a limited number of servers, if the build time is too long our servers can become occupied and then people will remain in queue for quite a long time. Worse, because our queue system is biased based on grade if a paying or a few high ranking paying user send builds in succession this might create a situation where low ranking users (free or basic) might be denied access to the server as a higher rank user will constantly step ahead in line.

Slow builds hurt everyone, up until now we always assumed people would align and try to avoid slow builds but I’m guessing that was naive. The incentive to do that isn’t enough. So we installed a timeout quota on a portion of the build phase which limited some builds.

It was a bad idea to do this right before the bootcamp but I felt some urgency due to the large amount of queuing. Initially we limited the time to 15 minutes on the xcode compile phase which is normally around 1.5-2 minutes. Due to some complaints I raised it to 25 minutes and some people still couldn’t get their builds thru (this should give you a sense of how problematic this traffic jam was). Finally Dave produced an idea for improvement in the build process that we implemented and deployed this weekend and it should produce faster builds for those use cases. I don’t see a difference with my builds but if your builds were taking 50 minutes they should be much faster now.

This change shouldn’t impact most users but since it changes the way headers are included some native code that relies on headers being included by VM generated code might fail.