Processing¶

This guide explains how to add a custom Audio Processing component with MimiCore inside MimiSDK, providing your users with Mimified audio.

Introduction¶

The Android MimiSDK 5.0.0 release brings many changes, of particular significance is the redefining of the Processing APIs.

While some names might sound familiar, quite a lot has changed, so please read this guide carefully.

Tip: As with the MimiSDK in general, the Processing APIs make heavy use of Kotlin Coroutines. We recommend having a basic understanding of their concepts.

Goals¶

We started with a number of goals to improve the experience for partners using the Processing APIs:

Provide easily accessible and transparent exposure of state, including for failures.
Reduce overall parameter API complexity.
Provide simpler high-level abstractions.
Improve testability through distinct abstraction layers.

Architecture¶

The new Processing APIs can be divided into four main components:

Processing Controller - The root entry point and context for Processing API access.
Processing Session - The provider of access to processing Processing Parameters when activated.
Processing Parameters - Individual parameter objects that provides state, data and mutation.
Processing Applicators - Responsible for applying the Processing Parameter values to external systems.

System Overview¶

The following section describes the components in the Processing APIs.

Processing System Overview

Figure: Processing System Components

ProcessingController¶

Overview¶

The ProcessingController is a controller owned and initialized by MimiCore.

It is responsible for the activation and deactivation of Processing Sessions, and therefore can be used as the source of truth for the active ProcessingSession instance.

Responsibilities¶

Centralized location for accessing Processing APIs.
Owner of active Processing Session
Provides activation and deactivation of Processing Session.

Activation¶

As the ProcessingController owns the ProcessingSession status; it is the location where processing can be "activated" or "deactivated".

Note: Activation refers to the ability for parameters values to be transferred, not whether the physical processing effect is enabled or disabled.

Before activating a ProcessingSession, you will need to create a preset parameter data source. The SDK currently provides two preset parameter data sources which are explained in Preset Parameter Data Sources.

A MimiProcessingSession can be activated as follows:

val fitting = Fitting(techLevel = 4) // An example fitting, yours will be defined by your Processing system
val defaultPresetDataSourceConfig = MimiPresetParameterDataSourceConfiguration.Default(fitting = fitting)
val defaultPresetDataSource = MimiCore.personalizationController.createPresetParameterDataSource(defaultPresetDataSourceConfig)
MimiCore.processingController.activateSession(presetDataSource = defaultPresetDataSource)

Once there is an activated ProcessingSession, it is possible to access the ProcessingParameters and configure them appropriately for use.

Lifecycle¶

The ProcessingController is responsible for handling the ProcessingSession lifecycle in terms of activation and deactivation.

Activating a session will set the activeSession property on the ProcessingController allowing the session to be easily referenced.
Only a single session can be activated at one time.
Deactivation clears the activeSession property. It does not directly affect the applied Mimi processing.
Activating and deactivating a session triggers MimiObserver notifications to subscribed observers.

Tip: Most partner integrations will only need to use a single, long-lived ProcessingSession that roughly aligns with the application lifecycle.

ProcessingSession¶

Provides access to Processing Parameters.

Overview¶

As described in the previous section, a ProcessingSession is owned by a ProcessingController, which is responsible for handling the session lifecycle in terms of activation/deactivation.

The ProcessingSession itself owns the ProcessingParameters, and therefore is the gateway to parameter access.

Responsibilities¶

Owner of all Processing Parameters, providing access to; isEnabled, intensity and preset.
Source of truth for fitting which provides required Fitting information for Preset acquisition.
Provides the ability for processing to be "interrupted" and automatically disabled due to external events (such as the hearing test launching).

Upon activation, the ProcessingSession will be initialized with all the parameters and properties it requires (such as fitting and data source objects). The previously applied parameter values are restored to their respective parameters. At this point the parameters are ready for use.

Processing Parameter¶

The focal point for state, application and data.

Overview¶

A ProcessingParameter represents an individual parameter that can be used to control Mimi processing functionality. It encompasses a value that can be mutated and applied asynchronously to subscribed components called Applicators.

Responsibilities¶

In general terms, a ProcessingParameter is responsible for:

Providing access to the current parameter value.
Applying values to a collection of subscribed Applicators using the value application sequence.
Fetching values from remote data sources asynchronously (such as preset data).
Providing state and value updates to a collection of subscribed observers.

Types of ProcessingParameters¶

To implement these behaviors on Android, there are two variants of ProcessingParameter:

The MimiProcessingParameter whose value is updated via a synchronous value.
The MimiFetchedProcessingParameter whose value is fetched from an asynchronously data source.

MimiSDK ProcessingParameter instances¶

The ProcessingSession defines the following ProcessingParameter instances to provide access and control of the Mimi processing:

isEnabled - Whether Mimi processing is enabled or disabled. It is a MimiProcessingParameter.
intensity - Intensity of Mimi processing. It is a MimiProcessingParameter.
preset - Preset data model for use by a Mimi processor. It is a MimiFetchedProcessingParameter.

Parameter value application sequence summary¶

The internal process of updating a ProcessingParameter value or transferring the existing value to Applicators follows a specific sequence as executed by the ProcessingParameter:

The request is passed through the Delivery Mode handler.
The Applicators are iterated through, asked to verify that they are capable of applying the value via their canApply function.
If any Applicator fails canApply verification (by returning false), the value application sequence is abandoned with an error and subscribers are notified of the failure.
Applicators are requested to apply the new value.
If any Applicator fails to apply the value (by throwing an Exception), the application sequence is abandoned with an error and subscribers are notified of the failure.
If successful, value is updated and Observers are notified of the new value.
A return result value is given for the request.

Note: If no Applicators are present - then value is immediately updated and the result is successful.

Important: A given ProcessingParameter executes each value application sequence sequentially - it does not perform concurrent updates.

Delivery Mode¶

The Delivery Mode of a ProcessingParameter describes how it handles requests to update the parameter value.

There are currently two Delivery Mode strategies: * Continuous performs each update sequentially, in the order they were requested. * Discrete debounces requests so it only executes the value application sequence once the request value settles for the given interval without updates.

Note: They are defined per Processing Parameter. Previously, a similar concept was defined across the entire Processing system.

Applicator¶

Arguably the most relevant Processing API component for most integrators is the Applicator.

An Applicator is an accessory of a ProcessingParameter that is capable of performing the value application logic for its associated ProcessingParameter. It is roughly equivalent to the previous ProcessingHandler concept.

In simple terms, its role is apply ProcessingParameter values to an external system. When there is a request to update a ProcessingParameter value, its Applicators are requested to apply this value.

While an Applicator appears quite similar to an Observer, the two components have two distinct roles within the Processing API:

An Applicator is responsible for the value application logic of a ProcessingParameter - the result of which influences the ProcessingParameter state.
An Observer is only informed of events reflecting committed changes to the ProcessingParameter state; it cannot influence the ProcessingParameter state.

In the Android MimiSDK, the concept of the Applicator is represented by the MimiParameterApplicator type.

Responsibilities¶

Provides a canApply function which can synchronously verify whether a requested value can be set on the Applicator.
Provides an apply function which will allow the applicator to asynchronously apply a value to an external source.
Optionally provides a value to be applied to other Applicators ("reverse synchronize").
Ability to be removed from a ProcessingParameter.
Ability to set a “Delivery Timeout” to specify a time duration which if exceeded is considered erroneous and will cause a value application sequence to fail.

Building an integration¶

This section highlights some of the key APIs used when integrating your processing system with the MimiSDK.

Initialization and Activation¶

Once you have already initialized MimiCore, you can access the Processing APIs from the ProcessingController and activate a ProcessingSession to add Applicators and modify ProcessingParameter values.

When activating a ProcessingSession, you need to provide a Fitting value.

The Fitting model provides data about the current processing environment and in turn how presets should be generated.

Example:

val processingController = MimiCore.processingController
val session = processingController.activateSession(Fitting(techLevel = 4)) // an example techLevel, yours will be defined by your Processing system

`ProcessingParameter` Operations¶

`ProcessingParameter` Value Operations¶

The following operations are provided on the ProcessingParameter and execute the value application sequence and return a ProcessingParameterResult indicating whether operation was successful.

Function	Behavior	Notes
`apply(value)`	The `apply` function attempts to update the `ProcessingParameter` with the given `value` by updating all registered, out-of-date `Applicator` instances.
`synchronize()`	The `synchronize` function attempts to update all registered, out-of-date `Applicator`s instances with the current `ProcessingParameter` `value`.
`fetch()`	The `fetch` function will attempt to refresh the value from the data source, and then automatically execute the value application sequence with that value.	Only available for `MimiFetchedProcessingParameter`.

Reading the `ProcessingParameter` state¶

Property/Function	Behavior	Notes
`value`	The `value` property holds the latest successfully applied Parameter value. It is updated at the end of the value application sequence
`observe()`	An `Observer` which provides callbacks for observing and value and state changes related to the `ProcessingParameter`. There are two variants; one for `MimiProcessingParameter` and one for `MimiFetchedProcessingParameter`. They have the following callbacks: `applying` - the request value is currently being applied to the Applicators. `failed` - the most recent value application failed. `ready` - the most recent value application was successful. `fetching` - the value is being retrieved from the data source (only available for `MimiFetchedProcessingParameter`).	*Important: You should not* use `observe` to apply values to your processing system; that is the responsibility of an `Applicator`.**

Example value:

val activeSession : ProcessingSession = requireNotNull(MimiCore.processingController.activeSession.state)
val isEnabled : Boolean = activeSession.isEnabled.value

Example observe:

// Acquire the active ProcessSession (assumes already activated!)
val activeSession : ProcessingSession = requireNotNull(MimiCore.processingController.activeSession.state)

// Observer for a MimiProcessingParameter, for example: isEnabled
activeSession.isEnabled.observe(
    applying = { current, applying -> 
        // add your code here to handle when a value application is in progress
    },
    failed = { current, failedToApply, exception -> 
        // add your code here to handle when a value application fails
    },
    ready = { current ->
        // add your code here to handle when value application succeeds
    }
)

// Observer for a MimiFetchedProcessingParameter, for example: preset
activeSession.preset.observe(
    fetching = { current ->
        // add your code here to handle when a value application is fetching from the data source
    },
    applying = { current, applying ->
        // add your code here to handle when a value application is in progress
    },
    failed = { current, failedToApply, exception ->
        // add your code here to handle when a value application fails
    },
    ready = { current ->
        // add your code here to handle when value application succeeds
    }
)

Adding an `Applicator` to a `ProcessingParameter`¶

There are two ways to define and add an Applicator. Each support different integration use cases depending on your needs.

The primary addApplicator function adds an "out-of-date" Applicator.

public fun addApplicator(
    canApply: (value: T) -> Boolean,
    apply: suspend (value: T) -> Unit,
    applyTimeoutMillis: Long
) : MimiParameterApplicator

Note: When adding an Applicator using this function, the Applicator will not automatically receive the current value and likewise; it has no influence on the current ProcessingParameter value.

The secondary addApplicator function adds an Applicator and automatically attempts to update the ProcessingParameter with the supplied value. The result of the update is returned as a ProcessingParameterResult.

public suspend fun addApplicator(
    value: T,
    canApply: (value: T) -> Boolean,
    apply: suspend (value: T) -> Unit,
    applyTimeoutMillis: Long
) : Pair<MimiParameterApplicator, ProcessingParameterResult>

In both cases, the Applicator is returned as a MimiParameterApplicator. This reference should be retained by the caller.

Defining your own Applicator¶

When creating an Applicator, we recommend delegating your canApply and apply functions to a class containing your custom processing logic. This custom processing logic depends entirely on your processing system. Generally, this approach helps make your code more modular and testable.

Note: This is simplified sample code to demonstrate the general sequence and may not reflect the best structure for your particular usecase.

Example: isEnabled Applicator

// Encapsulate your custom processing logic (example: isEnabled)
internal class CustomProcessingLogic {
    fun canApply(value: Boolean) : Boolean {
        TODO("Implement your logic to determine if your processing system is able to apply the given value")
    }

    suspend fun apply(value: Boolean) {
        TODO("Implement your logic to apply the given value in your processing system")
    }
}

// Declare an instance; depending on your usecase, you may want this to be a singleton.
val customProcessingLogic = CustomProcessingLogic()

// TODO - You should use an appropriate value for your integration.
val customApplyTimeoutMillis: Long = 10_000

// Access the relevant ProcessingParameter from the active ProcessingSession (assumes a ProcessingSession has been activated)
val isEnabledProcessingParameter =
    requireNotNull(MimiCore.processingController.activeSession.state).isEnabled

// Add the Applicator, delegating the calls to your custom processing logic instance.
val applicator = isEnabledProcessingParameter.addApplicator(
    canApply = customProcessingLogic::canApply,
    apply = customProcessingLogic::apply,
    applyTimeoutMillis = customApplyTimeoutMillis
)

// Cause the isEnabled ProcessingParameter to push its current value to the newly added Applicator
isEnabledProcessingParameter.synchronize()

Retain the applicator reference so that you can later remove it from the ProcessingParameter. Once an Applicator has been removed, it will no longer receive updates from the ProcessingParameter.

Example: Removing Applicator

// Removes the Applicator from its ProcessingParameter so it won't receive further updates.
applicator.remove()

Processing¶

Introduction¶

Goals¶

Architecture¶

System Overview¶

ProcessingController¶

Overview¶

Responsibilities¶

Activation¶

Lifecycle¶

ProcessingSession¶

Overview¶

Responsibilities¶

Processing Parameter¶

Overview¶

Responsibilities¶

Types of ProcessingParameters¶

MimiSDK ProcessingParameter instances¶

Parameter value application sequence summary¶

Delivery Mode¶

Applicator¶

Responsibilities¶

Building an integration¶

Initialization and Activation¶

ProcessingParameter Operations¶

ProcessingParameter Value Operations¶

Reading the ProcessingParameter state¶

Adding an Applicator to a ProcessingParameter¶

Defining your own Applicator¶

`ProcessingParameter` Operations¶

`ProcessingParameter` Value Operations¶

Reading the `ProcessingParameter` state¶

Adding an `Applicator` to a `ProcessingParameter`¶