Custom Processor

This guide explains how to add a custom Audio Processing component with MimiCore inside MimiSDK, providing your users with Mimified audio.

Introduction

MimiCore provides the focal point for Mimified Audio Processing via the MimiProcessingController. While this itself does not do anything much audio related, it provides a flexible interface that communicates parameters and other data between the processing and the rest of the application.

MimiProcessingController acts as a middle-man, communicating between a MimiProcessingHandler (which does the actual processing) and external objects that have an interest.

Usage

Activation

To begin receiving processing related events and data, you must first activate your MimiProcessingHandler. This will be your communication platform and allow you to react to any updates and provide the necessary processing data.

import MimiSDK
import MimiCoreKit

class CustomAudioProcessor: MimiProcessingHandler {
}

class AppDelegate: UIApplicationDelegate {

    func application(_ application: UIApplication, 
                     didFinishLaunchingWithOptions launchOptions: [UIApplication.LaunchOptionsKey: Any]?) -> Bool {

        // Activate processing handler...
        try? Mimi.core.processing.activate(handler: CustomAudioProcessor(), completion: { (success, error) in
            // Handle activation result
        })

        return true
    }
}

Communication

Once activated, communication can begin between the handler and MimiCore. You will need to implement the following MimiProcessingHandler functions:

func setParameter(_ parameter: MimiProcessingParameter.Writable,
                  result: ((MimiProcessingParameter.WriteResult) -> Void)?)

func getParameter(_ parameter: MimiProcessingParameter.Readable,
                  result: ((MimiProcessingParameter.ReadResult) -> Void)?)

The primary function of the processing handler is to achieve the delivery of MimiProcessingParameter parameters in both directions. All communication between MimiCore and the processing handler is expected to be asynchronous, and the handler is responsible for providing results to each operation. There are a few interesting things to note:

  • Not all cases MimiProcessingParameter are read / write, there are a specific subset that can be written and others that can be read.
  • For all parameters that are read, the processing handler is responsible for providing state and caching as appropriate.
  • If a request to set or get a parameter takes longer than 10 seconds, it will fail automatically with a timeout.

Setting Parameters

Processing parameters delivered through the setParameter() function are required to be set as appropriately on your audio stack to provide Mimi processing. The following parameters belong to MimiProcessingParameter.Writable:

  • .isEnabled - A boolean describing whether Mimi Processing is currently enabled or not.
  • .intensity - A float describing the current amount of processing that should be applied (range between 0.0 and 1.0).
  • .preset - MimiPersonalization.Preset model object which contains a payload that can be applied to a Mimi Processor to provide personalized audio.
var isEnabled: Bool = false
var intensity: Float = 0.0
var preset: MimiPersonalization.Preset? = nil

func setParameter(_ parameter: MimiProcessingParameter.Writable,
                  result: ((MimiProcessingParameter.WriteResult) -> Void)?) {
    switch parameter.value { // switch on the parameter with an associated value.
    case .isEnabled(let value):
        self.isEnabled = value // enable / disable processing here
    case .intensity(let value):
        self.intensity = value // apply intensity here
    case .preset(let value):
        self.preset = value // apply new preset here
    default:
        break
    }
    result?(.success) // everything is happy
}

The result of a setParameter() operation is either a success or a failure with an associated error. So say you failed to set a new isEnabled value:

enum MyCustomProcessorError: Error {
    case badThing
}


func setParameter(_ parameter: MimiProcessingParameter.Writable,
                  result: ((MimiProcessingParameter.WriteResult) -> Void)?) {
    switch parameter.value { // switch on the parameter with an associated value.
    case .isEnabled(let value):
        result?(.failure(error: MyCustomProcessorError.badThing))
    default:
        break
    }
}

Getting Parameters

As mentioned previously, there are some processing parameters that MimiCore reads from the processing handler, and for these the handler is responsible for maintaining and providing state.

For example - all Mimi UI will use the value of isEnabled provided by getParameter(). If this parameter fails to load then the UI will display an errornous state.

Parameters that can be read are part of MimiProcessingParameter.Readable:

  • .isEnabled - A boolean describing whether Mimi Processing is currently enabled or not.
  • .canEnable - A boolean describing isEnabled should be attempted to be set to true. If this is false then it is assumed something is preventing processing from being enabled.
  • .intensity - A float describing the current amount of processing that should be applied (range between 0.0 and 1.0).
  • .fitting - MimiPersonalization.FittingInfo model which provides data about the current processing environment and in turn how presets should be generated.

*Note - by default processing parameters will be delivered in a continuous manner, meaning that they will be streamed immediately with UI updates. If you would prefer to receive discrete (singular) deliveries, see Configuration. *

var isEnabled: Bool = true
var intensity: Float = 0.5
var preset: MimiPersonalization.Preset? // some

func getParameter(_ parameter: MimiProcessingParameter.Readable,
                  result: ((MimiProcessingParameter.ReadResult) -> Void)?) {
    switch parameter {
        case .isEnabled:
            result?(.success(.isEnabled(self.isEnabled)))
        case .canEnable:
            result?(.success(.canEnable(self.preset != nil))) // Only canEnable when preset is not nil
        case .intensity:
            result?(.success(.intensity(self.intensity)))
        case .fitting:
            result?(.success(.fitting(.for(techLevel: 3)))) // Mimi Tech 3
        default:
            break
    }
}

The result of a setParameter() operation is either a success with the associated requested value, or a failure with an associated error. So say you failed to get the current isEnabled value:

enum MyCustomProcessorError: Error {
    case badThing
}

func getParameter(_ parameter: MimiProcessingParameter.Readable,
                  result: ((MimiProcessingParameter.ReadResult) -> Void)?) {
    switch parameter {
    case .isEnabled:
        result?(.failure(error: MyCustomProcessorError.badThing))
    default:
        break
    }
}

Invalidation

So far the communication has all been one way, MimiCore dictating new and requesting current parameters. However there may be the case where a processing handler wants to inform MimiCore that something has changed in its state.

For example - the processor the handler was communicating with has suddenly had a catastrophic problem and is now offline, but MimiCore previously thought all was well and that isEnabled is set to true.

This is where invalidation comes in…

A MimiProcessingHandler has the ability to invalidate parameters, which will notify and cause MimiCore to automatically take appropriate actions to recover its state.

// Something terrible has happened

let error = MyProcessorError.catastrophy
invalidate(parameters: .isEnabled, .intensity, cause: error)

This would inform MimiCore that the current state of isEnabled and intensity can no longer be relied on, and it must take appropriate action to recover.

If you want to invaldate the entire state of a processing handler, this is also available:

invalidateAll(cause: nil) // no error we just wanted to refresh everything.

Configuration

Some processing behaviors are also able to be configured if the default behaviors are not suitable for your particular environment.

This is available by setting config on MimiProcessingController.

MimiCore.shared.processing.config = .default

Parameter Delivery Style

By default, processing parameters are delivered in a continuous manner, meaning they are delivered immediately when the parameter is updated from UI. This can be customized to an alternate delivery style if preferred:

  • .continuous (Default) - Deliver parameters as they arrive in a continuous stream.
  • .discrete() - Deliver parameters in a discrete manner only when the values settle and is committed. This has a default settleTime (duration to wait for the parameter to settle before committing to a delivery) of 0.2s, but can be manually set with the associated value.

Example:

MimiCore.shared.processing.config = .custom(parameterDeliveryStyle: .discrete())

Parameter Delivery Timeout

There is a timeout attached to every parameter update that is delivered to the active processing handler. If this timeout is exceeded the delivery is considered a failure. By default this timeout has a duration of 10s, but can be changed if required.

However, we would encourage trying to improve latency performance as much as possible, as a delay of 10 seconds or greater when applying a parameter will have a significant impact on percieved performance in the user interface.

Example:

MimiCore.shared.processing.config = .custom(parameterDeliveryStyle: .default,
                                            parameterDeliveryTimeout: 20.0)