A Series Of Steps
Approximately 9’ minutes reading time
This post expects that you are familiar with object lifecycles, closures, functional programming, memory management, asynchronous execution,
Processin the Swift language and the Mac/iOS SDK.
11 May 2019, added custom operator and a reference to SE-0253.
Table of Contents
The backbone of Windmill on the Mac is a series of steps1, that execute serially.
Checkout -> Build -> Test -> Archive -> Export
A few things to note.
- The output of one step may act as the input to the next.
- A step executes asynchronously.
- Each step must be successful before the next one is taken.
- Steps only make forward progress. If one step fails, the whole series is considered to have failed and no more steps are taken.
- The implementation of each step may differ. e.g. one step might be launching a
Process, another one might start a
The problem domain may sound familiar2 to that the
NSOperationQueuesolve. Apple has an excellent WWDC talk describing how powerful and useful can be in the context of dependencies; See Advanced NSOperations.
Windmill, does not make use of
NSOperations for the following, arguably subjective reasons.
- I wanted a more refined control over object lifecycles; i.e. what dependencies/relationships a step has, what arguments it needs to run, what it expects as an input.
- I wanted the concepts of a step, success, output/input, optional next step to be prominent and part of the API design.
- I wanted steps to have low coupling. Being able to reuse a step and create a different series of steps.
- I wanted a step to allocate memory lazily as part of its execution rather than as part of its allocation.
- I wanted the relationship between steps to look like it’s “modelled” at compile time.
- I wanted less of an architecture and more of a design pattern.
- I wanted an abstraction that is “lightweight” rather than another execution layer on top of launching a process and/or making an HTTP request.
Let’s take a step back.
echo.swift You can use
swift echo.swift in the command line to run it.
The code above prints the message
"Hello World!" asynchronously.
get.swift You can use
swift get.swift in the command line to run it.
The code above makes an asynchronous HTTP GET request to decode a base64 message.
Let’s see how the code samples above would typically be used in an app. A
Base64Service which might look familiar with what you find in the “network layer”.
Base64Service can be used like so:
EchoService can be used like so:
Run-of-the-mill so far.
With both services at hand, we want to be able to use them so that given an encoded message, we can:
- Decode the message.
- On success, echo the decoded message.
Let’s see how that would look.
Now that we have a better sense of what we are working with, let’s break it down.
The output of one step may act as the input to the next.
Notice how the
message argument to the
EchoService#echo(queue:message:) travels from the successful result of the
message is the “output” of the
Base64Service and the “input” to the
EchoService. In that sense, you can think of an
InputOutput type as a key/value storage that can be used to pass data between steps. Similar to how the
NotificationCenter uses the
userInfo whenever it posts a
From that, the definition of a step is trivial.
Steps only make forward progress.
Let’s look at the implementation of the
Think of the
EchoStep as merely a type that creates a
Step that echoes a message and also defines its input/output. It has an
EchoService as a dependency but does not own it since the lifecycle of the
EchoService3 is not directly related to the step execution.
Look closely at the
EchoStep#make(queue:next:) implementation which returns a
Step and think back to its type alias.
It returns a function, that accepts a single
InputOutput type as its input and has no return statement. It expects the
io argument to have a “message” value.
Equally, notice how the
Step, passes it’s output to the
next step, which in this case is the message it echoed. It’s important to understand that the
InputOutput type is a short lived instance only meant to be used in creating the step. It is not meant to be long lived, passed throughout the whole series of steps or as a global variable where every input/output is kept there.
This is how you would create a new
EchoStep and use it
There is one more thing.
Each step must be successful before the next one is taken.
Notice how the
EchoStep#make(queue:next:) above accepts an optional next
Step type that only calls on the condition of a successful callback.
With that in mind, let’s define a new type
SuccessfulStep, which creates a “successful”
Step and optionally passes the next one.
This type will become useful later as part of building a series of steps. For now, take a look at the
Focus on the
DecodeStep#successful(queue:) -> SuccessfulStep. This function accepts every argument the
#make one does, except the next
#successful function is useful to better express a series of steps. It’s best demonstrated in auto-completion.
DecodeStep(base64Service: base64Service).successful() defines a
SuccessfulStep. It does not create one. This allows you to defer passing the next
Step until after you have created it. Alternatively, you would have to write the code backwards, starting with creating the echo
Keep in mind that this is still the creation phase. Nothing is executed until you take the
series step and call it with an
Try passing an invalid base64 encoded String and see what happens.
Now that we can chain a series of
DecodeStep, let’s use them to decode a twice encoded base64 String.
One thing you may be struggling with is how the heck does this line of code work?
Especially if you are not already familiar with functional programming. It’s ok! Let’s go through this.
decodeOnce instance is a function (a
SuccessfulStep). Once you call it, (i.e.
decodeOnce(...) it resolves to a
Step, that step is the one the
DecodeStep makes. So the
series instance is a
Step function that you can call. (I have specifically declared its type so that this is clear.)
decodeTwice instance is also a
SuccessfulStep, once you call it, it resolves to a
Step also made by the
DecodeStep. It’s the
next step in the series.
echo instance is a
Step (also a function). It’s the second
next step in the series.
You may have noticed that throughout the code, there isn’t much error handling. This was done on purpose since I wanted to focus on the design pattern of steps.
Having said that, you can easily incorporate error handling either through a delegate and/or notifications.
As an example, consider the following:
Mattt has an excellent article on Error4
As long as you remember these 3 types, you can apply this pattern across your codebase.
Windmill 3.0 also uses it to distribute an export efficiently, securely and in a performant upload that requires 3 steps.
There is also nothing preventing you from combining async with sync steps.
Source on GitHub
Would be nice if instead of brackets, it was possible to define a custom operator. However, as far as I know, that requires an explicit type instead of a typealias. If you have a suggestion, please let me know and will update this post.
Here it is using an explicit type. Effectively, you wrap a function in a struct type, rather than simply create a typealias of the function. This way, you can also define a custom operator. In our case, that would be a
I just came across Daniel Steinberg’s talk, You’re doing it wrong in which he mentions the Swift Evolution Proposal 0253 - Introduce callables which is coming.
AFAICS, effectively you will be able to call a struct type as if it is a function. Which makes the struct
SuccesfulStep retain it’s nature as if it was a typealias! Here is how a series of steps now looks inspired by Matt Gallagher release of CwlViews.
I have also updated the source code.
Would also be nice if there was some type safety for the
InputOutput arguments as well as its “type” required for each step.
- Chris Eidhof, Daniel Eggert, and Florian Kugler for the wonderful Functional Swift book. It’s been extremely helpful while trying to get my head around functional programming.
- Daniel H Steinberg for his amazing insight and sharing of knowledge. His Somewhere Between Tomorrowland and Frontierland - Daniel Steinberg talk was an eye opener.
In practice there are more steps than these. For this post, I’ll keep it simple by focusing on the user facing ones. ↩
Most likely the
echoServiceis owned by a
UIViewControlleror at the very least, it is a leaf in its memory graph. Most likely, you don’t want to take the echo step (i.e.
self.echoService?.echo) if, say, the
UIViewto display the message has been deallocated. ↩