Approximately 10’ minutes reading time

Maxwell Anselm writing about Life in the slow lane.

I want to take this opportunity to say why I believe going back to Jenkins alongside writing and maintaining scripts is a step backwards, re-iterate what the goal of Windmill is and what has been guiding its development.

Before I go into the details, I do wholeheartedly understand where Maxwell (among others in my mind) is coming from. I have been an iOS Developer since 2009 and in the past I too have used Jenkins (with bash scripts) to automate the delivery of iOS apps. For the record, I have also used Xcode Bots and Buddybuild. I have never used fastlane; I am just familiar with it.

Building from the Command Line with Xcode

As it stands, if you want to build (test, export, archive, etc.) an Xcode project, you have to use Xcode’s command-line tools. I believe this is great!

• It demonstrates that Apple is keen on providing developer tools beyond an editor and a simulator and hopes you can help with that. It tells me it’s been listening to the voice of the community to support continuous integration systems.
• It puts the responsibility to develop and maintain the command-line tools squarely on Apple. We should cherish, embrace and encourage Apple to do this. Apple is better positioned to know what’s coming, build to specification and provide support for developers in a timely fashion.

I have witnessed the progress Apple has made which, to me, demonstrates commitment.

• xcodebuild has unified Xcode actions providing consistency and interoperability between them. Back in 2014, you had to use xcrun PackageApplication to export an IPA. Today xcodebuild -exportArchive -archivePath xcarchivepath -exportPath destinationpath -exportOptionsPlist path mirrors Xcode’s workflow and is compatible with xcodebuild archive.
• Apple strives for feature parity across Xcode and its command-line tools. e.g. You can notarize your app from the command line.
• Sessions on automation and command-line tools are part of WWDC. e.g. What’s New in Signing for Xcode and Xcode Server
• Apple writes documentation like the Technical Note TN2339 and drops hints on what the command-line tools are there for.

Note: build-for-testing and test-without-building provide support for continuous integration systems. - “Building from the Command Line with Xcode FAQ, Updated: 2017-06-19”

That’s not to say that Apple gets everything right. I too have fought with and found running tests from the command-line to be unreliable. Having to restart the CoreSimulatorService as a result. Windmill is prone to this too.

I won’t claim to know what’s at fault here. I do believe Apple knows it reflects bad on them. Xcode engineers are working on this after all. Plus, it puts every tool that relies on them in a bad light.

Dependencies

Windmill relies heavily on existing conventions as set by Apple and aims to establish a best practice on how to automate building, testing and distributing apps. At this stage, what Windmill is able to achieve and to what extent is defined by Apple. There is no way around it. This is a conscious decision. To pretend otherwise would be to rely on reverse engineering, side effects, undocumented features and observable behaviours. For bringing that basic level of automation it should be enough. - “Announcing Windmill”

With that in mind, I made the conscious decision to build Windmill with Xcode’s command-line tools and those tools alone¹.

It’s unfortunate that as of today we can’t completely rely on xcodebuild to provide automation for Xcode projects. It is precisely for this reason why we need to keep giving feedback to Apple so that it can continue developing and supporting the command-line tools.

For example, it is not possible to know whether an Xcode project has a test target (i.e. has tests to run) until you have tried to run those tests. In other words, you have to run xcodebuild test-without-building, trap the error by querying the exit code and “assume” the project has no tests.

For Windmill, I had to write a ProcessManager² that executes a Process and can recover from an error code.

Another example is when the output of xcrun simctl list devices --json changed, effectively being unable to find the available iOS simulators.

This is what Apple had to say:

While we try to not be unnecessarily disruptive, the format of that output is subject to change over time as features are added and we need to adjust it.

Which is fine! As long as changes like this are better communicated, documented, formally specified, deprecated and retired like any API.

Reliability

Which brings us to reliability and Maxwell’s strong argument against fastlane or any automation tool built on top of Xcode’s command-line tools for that matter.

Seeing this stuff makes me nervous, because I worry that fastlane is doing a bunch of extra work that

• at best, I don’t care about or
• at worst, is going to cause problems that I have to fix (don’t laugh, read on…)"

The -showBuildSettings tells you what deployment target a project is or what its product name is. It helps finding the correct simulator to use since xcodebuild requires you to specify what destination to use with build-for-testing.

Xcode reads all of that from the project file under “Build Settings” and defaults to a scheme (or a target) and gives you a list of simulators appropriate for your project. xcodebuild is a command-line tool after all and Xcode is a GUI app.

Still, why should Maxwell care about all of this? As far as he is concerned he does know what scheme to build for his project and what destination (i.e. simulator) to test against.

It is true that when you build a tool for the many, you do need to accommodate for cases that go beyond the needs of the one. This rings more true when you build an automation tool that has to be reliable, predictable, consistent across environments and configurations.

To pick the correct device to build and test for that means being able to do so across Xcode installations and upgrades, removal of simulators, changes in the deployment target of your app, etc than merely choosing what destination may be valid right now.

Establishing a process that automates all aspects of the delivery of your app for quality assurance in order to save time, money and effort demands more than simply knowing what scheme to build your project with; as you may have witnessed in an effort to ditch, in this case, fastlane.

More importantly, it is about making continuous delivery ubiquitous, accessible, understood by everyone in your team so that together you can progress further.

Collectively as an iOS community, we have spent an insane amount of time and put an immense amount of effort in automating the delivery of our apps by writing flaky scripts in bash using Jenkins, reverse engineering the Xcode project file format, writing blog posts with our findings.

Yet we are none the wiser and have barely made a dent.

Yes, some of it is not critical or essential to any one of us as individuals but working on and having a tool that has these goals in mind will lead to a better tool for everyone in the long run. It will allow us to move on to whatever the next challenge in delivering iOS apps is.

Complexity

“I felt like I had to babysit it every day, constantly diagnosing whether build or test failures were legitimate. Consequently the rest of my team never trusted it.” - Maxwell Anselm, Life in the slow lane

When Windmill first came out, it didn’t surface any errors so you had to go through the logs to find out what went wrong. In this regard, it was no better than a bare-bone Jenkins installation showing you the log of a build job. As a developer, you had to manually go through the output of xcodebuild to figure out what went wrong.

Since Windmill is a developer tool, developers were eager to debug it! “If only I could see what command Windmill is running, I can try and replicate the issue”.

You see, I knew right then³ that Windmill was failing to live up to its mission. You shouldn’t have to debug Windmill!

Since version 2.0, Windmill mirrors the Xcode log. xcodebuild does not provide a formatted output so I had to build it and actively maintain it for each release. This costs time, effort and it’s not streamlined to Xcode upgrades. Still, I am willing to pay that price for Windmill to deliver on its promise.

I have put an insane amount of effort and time to make sure Windmill mirrors Xcode’s behaviour so that when Windmill fails it does in the exact same way as Xcode.

Windmill has a long way to go before it can be the tool I envision. As an example, right now Windmill is reactive only to code changes.

If a distribution certificate expires, Windmill will surface that error the next time you commit a code change rather than at the time it expires or even better, warn you in advance.

Why use anything else but xcodebuild?

People don’t realise how much time they have spent installing, building, maintaining, communicating, debugging their way to, at the very least, automate building and testing their iOS app until they decide they’ve had enough.

In Maxwell case, his post on Distributed iOS builds with Jenkins was in Aug 3, 2018 and “Life in the slow lane” in Mar 12, 2019.

Speaking for Windmill, I have taken the strong stance to build it on the following premise.

• Have a native, macOS app with an unparalleled user experience in an effort to make continuous delivery accessible.
• Lean on and don’t stray away from Xcode’s command-line tools provided functionality.
• Don’t resort to hacks to make Apple’s command-line tools work.
• Rely on the macOS SDK and macOS toolchain. e.g. PropertyListSerialization to convert a property list to and from several serialized formats.
• Link to Apple’s documentation and encourage developers to have an understanding of how things work where necessary.
• Minimise the use of informal contracts, i.e. parsing of standard output.
• Don’t overstep any boundaries, i.e. Xcode’s ability to manage automatic signing.
• Embrace Xcode’s provided functionality, i.e. store archives in the Organiser.
• Don’t create abstractions on top of existing concepts, e.g. what an archive is on iOS.
• Mirror and match Xcode behaviour to provide consistency across, e.g. how build errors and test failures are surfaced and communicated.
• Be a good citizen on macOS as a native app.

That’s not to say that Windmill is 100% foolproof. See its version history.

Still, what Windmill offers today, even at version 2.0, is of great value, especially if you all you need is a basic level of automation while developing your app.

In the end, I have made the decision to place my faith in Apple by building a native macOS app with Xcode’s command-line tools and invest so much in the Apple ecosystem. Until Apple comes up with something better.-

PS. I am using Windmill on macOS while developing Windmill on the iPhone and I am loving it! There is a chance of bias.
PPS. If you want to be part of the Windmill on the iPhone beta, sometime in the unknown future, please send me an email at qnoid@windmill.io.