Approximately 9’ minutes reading time

This post expects that you are familiar with object lifecycles, closures, functional programming, memory management, asynchronous execution, URLSession and Process in the Swift language and the Mac/iOS SDK.

Update

11 May 2019, added custom operator and a reference to SE-0253.

Table of Contents

Introduction

The backbone of Windmill on the Mac is a series of steps¹, that execute serially.

Checkout -> Build -> Test -> Archive -> Export

A few things to note.

• The output of one step may act as the input to the next.
• A step executes asynchronously.
• Each step must be successful before the next one is taken.
• Steps only make forward progress. If one step fails, the whole series is considered to have failed and no more steps are taken.
• The implementation of each step may differ. e.g. one step might be launching a Process, another one might start a URLSession.

The problem domain may sound familiar² to that the NSOperation and NSOperationQueue solve. Apple has an excellent WWDC talk describing how powerful and useful can be in the context of dependencies; See Advanced NSOperations.

Windmill, does not make use of NSOperations for the following, arguably subjective reasons.

• I wanted a more refined control over object lifecycles; i.e. what dependencies/relationships a step has, what arguments it needs to run, what it expects as an input.
• I wanted the concepts of a step, success, output/input, optional next step to be prominent and part of the API design.
• I wanted steps to have low coupling. Being able to reuse a step and create a different series of steps.
• I wanted a step to allocate memory lazily as part of its execution rather than as part of its allocation.
• I wanted the relationship between steps to look like it’s “modelled” at compile time.
• I wanted less of an architecture and more of a design pattern.
• I wanted an abstraction that is “lightweight” rather than another execution layer on top of launching a process and/or making an HTTP request.

Let’s take a step back.

Prefix

#!/bin/swift
import Foundation

let message = "Hello World!"

let process = Process()
process.launchPath = "/bin/echo"
process.arguments = [message]
process.launch()

The code above prints the message "Hello World!" asynchronously.

#!/bin/swift
import Foundation

let group = DispatchGroup()
group.enter()

let session = URLSession.shared

let message = "SGVsbG8gV29ybGQh"
var request = URLRequest(url: URL(string: "https://httpbin.org/base64/\(message)")!)
request.httpMethod = "GET"
request.addValue("text/html", forHTTPHeaderField: "accept")

session.dataTask(with: request) { (data, response, error) in
    guard let response = response as? HTTPURLResponse else {
        preconditionFailure()
    }
    
    switch (response.statusCode) {
    case 200:
        if let data = data, let message = String(data: data, encoding: .utf8) {
            print(message)
        }
    default:
        print("failure")
    }

    group.leave()
    
}.resume()

let result = group.wait(timeout: .now() + 10)
print(result)

The code above makes an asynchronous HTTP GET request to decode a base64 message.

Let’s see how the code samples above would typically be used in an app. A Base64Service which might look familiar with what you find in the “network layer”.

class Base64Service  {

    typealias DecodeCompletionHandler = (Result) -> Void

    let session: URLSession

    init(session: URLSession = URLSession.shared) {
        self.session = session
    }

    func decode(queue: DispatchQueue? = nil, encoded message: String, completionHandler: @escaping Base64Service.DecodeCompletionHandler) {

        var request = URLRequest(url: URL(string: "https://httpbin.org/base64/\(message)")!)
        request.httpMethod = "GET"
        request.addValue("text/html", forHTTPHeaderField: "accept")
        session.dataTask(with: request) { (data, response, error) in
            guard let response = response as? HTTPURLResponse else {
                preconditionFailure()
            }
        
            switch (response.statusCode) {
            case 200:
                if let data = data, let message = String(data: data, encoding: .utf8) {
                    (queue ?? DispatchQueue.main).async {
                        completionHandler(.success(message: message))
                    }
                }
            default:
                (queue ?? DispatchQueue.main).async {
                    completionHandler(.failure)
                }
            }
        }.resume()
    }
}

An EchoService type.

class EchoService  {

    typealias EchoCompletionHandler = (Result) -> Void

    func echo(queue: DispatchQueue? = nil, message: String, completionHandler: @escaping EchoService.EchoCompletionHandler) {
        let process = Process()
        process.launchPath = "/bin/echo"
        process.arguments = [message]
        process.terminationHandler = { process in
            
            let isSuccess = process.terminationStatus == 0

            (queue ?? DispatchQueue.main).async {
                if isSuccess {
                    completionHandler(.success(message: message))
                } else {
                    completionHandler(.failure)
                }
            }
        }

        process.launch()
    }
}

The Base64Service can be used like so:

let base64Service = Base64Service()
let encoded = "SGVsbG8gV29ybGQh"

base64Service.decode(encoded: encoded) { result in 
    
}

The EchoService can be used like so:

let echoService = EchoService()
let message = "Hello World"

echoService.echo(message: message) { result in
	
}

Run-of-the-mill so far.

With both services at hand, we want to be able to use them so that given an encoded message, we can:

1. Decode the message.
2. On success, echo the decoded message.

Let’s see how that would look.

let echoService = EchoService()
let base64Service = Base64Service()

base64Service.decode(encoded: "SGVsbG8gV29ybGQh") { result in 
    
    guard case .success(let message) = result else {
        return
    }

    echoService.echo(message: message) { _ in
    
    }
}

Baby Steps

Now that we have a better sense of what we are working with, let’s break it down.

The output of one step may act as the input to the next.

Notice how the message argument to the EchoService#echo(queue:message:) travels from the successful result of the Base64Service.

The message is the “output” of the Base64Service and the “input” to the EchoService. In that sense, you can think of an InputOutput type as a key/value storage that can be used to pass data between steps. Similar to how the NotificationCenter uses the userInfo whenever it posts a Notification.

typealias InputOutput = [AnyHashable : Any]

From that, the definition of a step is trivial.

Steps only make forward progress.

typealias Step = (_ io: InputOutput) -> Swift.Void

Let’s look at the implementation of the EchoStep.

import Foundation

struct EchoStep {

    weak var echoService: EchoService?

    func make(queue: DispatchQueue? = nil, next: Step? = nil) -> Step {
        return { io in

            guard let message = io["message"] as? String else {
                preconditionFailure()
            }

            self.echoService?.echo(queue: queue, message: message) { result in
                guard case .success(let message) = result else {
                    return
                }

                next?(["message":message])
            }
        }
    }
}

Think of the EchoStep as merely a type that creates a Step that echoes a message and also defines its input/output. It has an EchoService as a dependency but does not own it since the lifecycle of the EchoService³ is not directly related to the step execution.

Look closely at the EchoStep#make(queue:next:) implementation which returns a Step and think back to its type alias.

typealias Step = (_ io: InputOutput) -> Swift.Void

It returns a function, that accepts a single InputOutput type as its input and has no return statement. It expects the io argument to have a “message” value.

Equally, notice how the Step, passes it’s output to the next step, which in this case is the message it echoed. It’s important to understand that the InputOutput type is a short lived instance only meant to be used in creating the step. It is not meant to be long lived, passed throughout the whole series of steps or as a global variable where every input/output is kept there.

This is how you would create a new EchoStep and use it

let echoService = EchoService()
let echoStep = EchoStep(echoService: echoService)

let echo = echoStep.make()
let message = "Hello World!"

echo(["message":message])

There is one more thing.

Each step must be successful before the next one is taken.

Notice how the EchoStep#make(queue:next:) above accepts an optional next Step type that only calls on the condition of a successful callback.

With that in mind, let’s define a new type SuccessfulStep, which creates a “successful” Step and optionally passes the next one.

typealias SuccessfulStep = (_ next: Step?) -> Step

This type will become useful later as part of building a series of steps. For now, take a look at the DecodeStep.

struct DecodeStep {

    weak var base64Service: Base64Service?

    func make(queue: DispatchQueue? = nil, next: Step? = nil) -> Step {
        return { io in

            guard let encoded = io["message"] as? String else {
                preconditionFailure()
            }

            self.base64Service?.decode(queue: queue, encoded: encoded) { result in 
    
            guard case .success(let message) = result else {
                return
            }

            next?(["message": message])
            }
        }
    }

    func successful(queue: DispatchQueue? = nil) -> SuccessfulStep {
        return { next in 
            return self.make(queue: queue, next: next)
        }
    }
}

Focus on the DecodeStep#successful(queue:) -> SuccessfulStep. This function accepts every argument the #make one does, except the next Step.

The #successful function is useful to better express a series of steps. It’s best demonstrated in auto-completion.

Notice how DecodeStep(base64Service: base64Service).successful() defines a SuccessfulStep. It does not create one. This allows you to defer passing the next Step until after you have created it. Alternatively, you would have to write the code backwards, starting with creating the echo Step first.

Keep in mind that this is still the creation phase. Nothing is executed until you take the series step and call it with an InputOutput parameter.

series(["message":"SGVsbG8gV29ybGQh"])

Try passing an invalid base64 encoded String and see what happens.

Now that we can chain a series of DecodeStep, let’s use them to decode a twice encoded base64 String.

let base64Service = Base64Service()
let echoService = EchoService()

let decodeOnce = DecodeStep(base64Service: base64Service).successful()
let decodeTwice = DecodeStep(base64Service: base64Service).successful()

let echo = EchoStep(echoService: echoService).make()

let series:Step = decodeOnce(decodeTwice(echo))
series(["message":"U0dWc2JHOGdWMjl5YkdRaA=="])

One thing you may be struggling with is how the heck does this line of code work?

let series:Step = decodeOnce(decodeTwice(echo))

Especially if you are not already familiar with functional programming. It’s ok! Let’s go through this.

The decodeOnce instance is a function (a SuccessfulStep). Once you call it, (i.e. decodeOnce(...) it resolves to a Step, that step is the one the DecodeStep makes. So the series instance is a Step function that you can call. (I have specifically declared its type so that this is clear.)

The decodeTwice instance is also a SuccessfulStep, once you call it, it resolves to a Step also made by the DecodeStep. It’s the next step in the series.

Finally, the echo instance is a Step (also a function). It’s the second next step in the series.

Error handling

You may have noticed that throughout the code, there isn’t much error handling. This was done on purpose since I wanted to focus on the design pattern of steps.

Having said that, you can easily incorporate error handling either through a delegate and/or notifications.

As an example, consider the following:

guard case .success(let message) = result else {

    self.delegate?.failure(error: StepError.failed("decode"))

    return
}

Mattt has an excellent article on Error⁴

Conclusion

typealias InputOutput = [AnyHashable : Any]
typealias SuccessfulStep = (_ next: Step?) -> Step
typealias Step = (_ io: InputOutput) -> Swift.Void

As long as you remember these 3 types, you can apply this pattern across your codebase.

Windmill 3.0 also uses it to distribute an export efficiently, securely and in a performant upload that requires 3 steps.

There is also nothing preventing you from combining async with sync steps.

Sample Project

Source on GitHub

Feedback

Would be nice if instead of brackets, it was possible to define a custom operator. However, as far as I know, that requires an explicit type instead of a typealias. If you have a suggestion, please let me know and will update this post.

Custom Operator

Here it is using an explicit type. Effectively, you wrap a function in a struct type, rather than simply create a typealias of the function. This way, you can also define a custom operator. In our case, that would be a SuccesfulStep.

infix operator -->: AssignmentPrecedence

public struct SuccessfulStep {
    let success: (_ next: Step?) -> Step
    
    call(_ next: Step?) -> Step {
        return success(next)
    }
}

extension SuccessfulStep {
    static func --> (success: SuccessfulStep, next: @escaping Step) -> Step {
        return success.call(next)
    }
}

I just came across Daniel Steinberg’s talk, You’re doing it wrong in which he mentions the Swift Evolution Proposal 0253 - Introduce callables which is coming. AFAICS, effectively you will be able to call a struct type as if it is a function. Which makes the struct SuccesfulStep retain it’s nature as if it was a typealias! Here is how a series of steps now looks inspired by Matt Gallagher release of CwlViews.

let series = decode --> echo

and

let series = decodeFirst --> decondSecond --> echo

I have also updated the source code.

Would also be nice if there was some type safety for the InputOutput arguments as well as its “type” required for each step.-

Acknowledgements

• Chris Eidhof, Daniel Eggert, and Florian Kugler for the wonderful Functional Swift book. It’s been extremely helpful while trying to get my head around functional programming.
• Daniel H Steinberg for his amazing insight and sharing of knowledge. His Somewhere Between Tomorrowland and Frontierland - Daniel Steinberg talk was an eye opener.
• httpbin.org
• cryptii