(update: see https://github.com/atsushieno/android-audio-plugin-framework/issues/82#issuecomment-892459550 which updates all the statuses involved)

(1) At this state AAPMidiProcessor can process only MIDI 2.0 messages. For MIDI 1.0 messages we have to sort out how to adjust lengths in the messages along with modified timestamps per audio processing cycles. MIDI 1.0 is especially problematic because length bytes can vary. We likely have to change how MIDI 1.0 messages are structured on the existing channels.

(1.1) Maybe we should simply drop it and process everything in MIDI 2.0 UMPs. Then every subprojects need to reflect the protocol changes. aap-juce and all the plugins, plugin hosts, aap-lv2 based plugins (relatively easy as we already have UMP processing implementation)

(2) At this state we assume that an AAP application contains at most one instrument plugin. It is already wrong in aap-lv2-mda as there are handful of instruments (EPiano, DX10, JX10, Piano...) in the app. A possible solution would be to create <input> port for every instrument. Then we only have to (a) create MidiReceiver instances for each, and (b) determine how to map those input entries to MidiReceivers (stable indices suffice).

(3) there are some FIXMEs on terminating /unbinding device service.

It is a typical but intermittent problem that applying a plugin on aaphostsample results in freezes. It most likely happens after 3~4 times on different plugins with LV2 plugins, or crashes with JUCE plugins.

They may be different issues, and more likely plugin-side issues, but at this state it is looking like an overall framework or host problem. JUCE AudioPluginHost shows similar but different unstability too.

JUCE AudioPluginInstance::processBlock() tells us that it is up to the underlying audio plugin framework that interprets and processes MidiBuffer part. JUCE VST3 processor implements it in MidiEventList::toMidiBuffer() which populates Steinberg::Vst::Event instances. For AU it is MusicDeviceMIDIEvent.

AAP needs something similar, but there is a problem here - there should be backend feature that converts those abstract MIDI messages to the backend-specifics. For LV2 backend, it is LV2 Atom.

On main branch, aapinstrumentsample is implemented to work as MIDI 1 synth. Other plugins are either not updated or cause runtime crashes. They have to be either fixed or rewritten to work as V2 plugin.

Though, it kinda requires updating to the new scheme for parameter changes i.e. interpret MIDI2 inputs in realtime manner. That brings in unified event processing model with MIDI1 based inputs, and not everything is ready for that yet.

There are couple of sub-tasks:

• [x] implement MIDI2 processing part in PluginPreview (and thus samples-host-engine and aaphostsample) to work with the new scheme if (1) there is a MIDI2 in port and (2) there are "parameters" (it was only partially implemented).
• [x] make changes to aapinstrumentsample to also take parameter changes via UMPs.
• [x] Once PluginPreview gets fixed and starts working fine, then it should also support (send) unified MIDI2 inputs to the instantiated plugin.
• [x] midi-device-service should work regardless of whether the port is midi1 or midi2 (needs to insert UMP translator).
• Once it gets working, then aap-lv2 needs to be rewritten to migrate to the new parameter changes scheme.
  • [x] get aap-ayumi working for both midi1 and midi2 modes (with conditional build)
  • [x] update aap-import-lv2-metadata to support midi2 and parameters
  • [x] get midi-device-service working
  • [x] formalize port mappings (which became inconsistent in the new design and causing run-time crash)
  • [x] get aap-lv2-mda effects working
  • [x] get aap-lv2-mda instruments working
  • [x] get aap-lv2-fluidsynth working
  • [x] get aap-lv2-sfizz working as MidiDeviceService
  • [x] get aap-lv2-string-machine MIDI output correct synthesis results
  • [ ] get LV2 patch input working
  • [ ] implement Atom output reader (and push to AAP midi2 output port)
• The same has to go in aap-juce.
  • [x] get cmake support working
  • [x] get Projucer support working
  • [x] get plugins working
  • [x] get hosting working

After working on #103, I noticed that we need another bunch of breaking fundamental changes for extensibility:

• We would still need extensibility mechanism that plugins host can "request" calls e.g. "save preset 0 as "%'#$'%$ (binary)", which is not doable just with data pointers. We need non-local function that host can call into plugin.
• We could implement Presets extension as such, plugin service and client interacting together through binder, but that means we need another channel, which will bring in a lot of future API stability breakage. Extensibility should be realized as a generic channel.
• An extension developer (most of the extensions would be implemented by ourselves) should be able to "message" each other (plugin to host, or host to plugin), beyond C API.
• The most likely viable option for "general" purpose messaging is a pair of MIDI controller channels, as detailed at 21edbb9. Then an extension would not even need any extra messaging mechanism.
• However, for operations like sending and receiving state blob that's going to be too massive and involves lots of extraneous work (splitting large binary chunk into MDS, which is what MDS is for but still we don't really need it). Therefore it would be better if an extension can make use of some shared pointer between the plugin and the client host.

Therefore, I would tweak the existing AIDL structure and make some incompatible changes:

• Introduce plugin service extensibility API that every extension developer needs to implement for both host client and plugin service.
• Client hosts can invoke plugin service extension functions.
• Plugins can only reference host extension properties through shared memory. Host cannot get notified if a plugin referenced them.
  • An extension can provide inquiry functions ("do/did you (plugin) reference my (host) property XXX?")
• ExtensionRegistry collects the extension definitions.
• Revamp getState() and setState() AIDL messaging: implement them along with the extensibility framework.

Example Client that makes use of "AAP Presets" extension:

	AndroidAudioPluginHost* host;
	auto presetsExt =
		(aap::PresetsPluginServiceExtension*) host->get_extension(AAP_PRESETS_EXTENSION_URI);
	auto preset = presetsExt->get_preset(0);

Example Foo Plugin that implements "AAP Presets" extension:

	aap_preset_t foo_presets[];
	
	void foo_plugin_factory_initialize() {
		// initialize foo_presets from resources here
	}

	// The actual Presets extension implementation for Foo plugin
	aap_preset_t* foo_presets_get_preset(int32_t index) { return foo_presets[index]; }

	aap_presets_extension_t foo_presets_extension { foo_presets_get_preset };

	void* foo_get_extension(AndroidAudioPlugin *plugin, const char *extensionURI) {
		switch (uri_to_id(extensionURI)) {
		case AAP_PRESETS_EXTENSION_URI_ID:
			return &foo_presets_extension;
		}
	}

	AndroidAudioPlugin foo_plugin{
		nullptr,
		foo_prepare,
		...
		foo_get_extension
	};
	
	AndroidAudioPlugin* foo_plugin_instantiate (...) { return &foo_plugin; };
	
	AndroidAudioPluginFactory foo_factory {
		foo_plugin_instantiate, foo_plugin_release, nullptr
	};
	
	bool foo_factory_initialized{false};
	
	AndroidAudioPluginFactory* GetAndroidAudioPluginFactory () {
		if (!foo_factory_initialized) {
			foo_factory_initialized = true;
			foo_plugin_factory_initialize();
		}
		return &foo_factory;
	}

PluginService API (probably C, might be C++):

	// C
	void (*aap_service_extension_on_invoked_t) (
		AndroidAudioPluginHost *service,
		AndroidAudioPluginServiceExtension* serviceExtension,
		AndroidAudioPluginExtension* data);

	typedef struct {
		const char *uri;
		aap_service_extension_on_invoked_t on_invoked;
	} aap_service_extension_t;

	// C++
	class AndroidAudioPluginServiceExtension {
		virtual const char* getURI() = 0;
		void clientInvokePluginExtension(AndroidAudioPluginExtension* ext) {
			// transmit data via Binder
		}
		virtual void onServicePluginExtensionInvoked(AndroidAudioPluginExtension* ext) = 0;
	};

Presets PluginService extension implementation (probably C, might be C++):

	// It is already in aap-core.h
	typedef struct {
		const char *uri;
		int32_t transmit_size;
		void *data;
	} AndroidAudioPluginExtension;

	class MemoryBlock {
		void* buffer{nullptr};
		int32_t size{0};
		
		~MemoryBlock() {
			if (buffer)
				free(buffer);
		}
		
		void copyFrom(int32_t size, void *src) {
			if (buffer == nullptr || this->size < size) {
				if (buffer != nullptr)
					free(buffer);
				this->size = size;
				buffer = calloc(1, size);
			}
			memcpy(result->buffer, *((int32_t*) extensionInstance->data, size);
		}
	}

	// C
	void presets_plugin_service_extension_on_invoked (
		AndroidAudioPluginHost *service,
		AndroidAudioPluginServiceExtension* serviceExtension,
		AndroidAudioPluginExtension* data) {
		switch (opcode) {
		case OPCODE_GET_PRESET:
			int32_t index = *((int32_t*) ext->data);
			auto presetExtension = (aap_preset_extension_t*) service->get_extension(getURI());
			auto preset = (aap_preset_t*) presetExtension->get_preset(index);
			// write size, then data
			*((int32_t*) ext->data) = preset->data_size;
			memcpy(ext->data + sizeof(int32_t), preset->data, preset->data_size);
			return;
		}
		assert(0); // should not reach here			
	}
	
	aap_service_extension_t presets_plugin_service_extension{
		AAP_PRESETS_EXTENSION_URI, presets_plugin_service_extension_on_invoked };
	
	// C++
	class PresetsPluginServiceExtension : public AndroidAudioPluginServiceExtension {
	public:
		PresetsPluginServiceExtension(AndroidAudioPluginHost* host, AndroidAudioPluginExtension *extensionInstance)
			: AndroidAudioPluginServiceExtension(host, extensionInstance) { ... }
	
		inline const char* getURI() override { return AAP_PRESETS_EXTENSION_URI; }
		
		const int32_t OPCODE_GET_PRESET = 0;
		
		// function for client host to implement
		void get_preset(int32_t index, MemoryBlock *result) {
			*((int32_t*) extensionInstance->data) = index;
			clientInvokePluginExtension(data, OPCODE_GET_PRESET);
			int32_t size = *((int32_t*) extensionInstance->data);
			result->copyFrom(size, *((int32_t*) extensionInstance->data + 1);
		}
		
		// function for plugin service to implement, called by AudioPluginService impl.
		void onServicePluginExtensionInvoked(AndroidAudioPluginExtension* ext, int opcode) override {
			switch (opcode) {
			case OPCODE_GET_PRESET:
				int32_t index = *((int32_t*) ext->data);
				auto presetExtension = (aap_preset_extension_t*) host->get_extension(getURI());
				auto preset = (aap_preset_t*) presetExtension->get_preset(index);
				// write size, then data
				*((int32_t*) ext->data) = preset->data_size;
				memcpy(ext->data + sizeof(int32_t), preset->data, preset->data_size);
				return;
			}
			assert(0); // should not reach here			
		}
	};

ExtensionRegistry:

	class ExtensionRegistry {
		virtual AndroidAudioPluginServiceExtension* create(const char * uri, AndroidAudioPluginHost* host, AndroidAudioPluginExtension *extensionInstance) = 0;
	};

Extension service implementation needs to be compiled in the host. For plugins, only the extension header is required (the API has to be "implemented" by the plugin developer anyways).

A host application has to prepare its own ExtensionRegistry instance, and manage whatever extensions it wants to support. StandardExtensionRegistry should come up with all those standard extensions enabled.

There has been compose-samples branch for a while, but it has never been worked on for months. I started reworking on it these days.

The idea was now that there is Jetpack Compose for Desktop, it could be reusable on desktop too. It turned out that to make org.androidaudioplugin in Kotlin MPP common package for a Compose app, it must not be an aar library package, which (at least currently) isn't.

Should we isolate Android platform dependencies (Android platform and JNI based code), jniLibs, and prefab contents from the Kotlin core bits? So far I'm negative because It complicates the build.

If we don't go with MPP approach, it is still possible to apply Jetpack Compose, and it is indeed desirable for future maintainability, except for current concern that Compose is supported only on Android Studio 4.2 or later.

Ports should have some properties.

For example, it is not predictable for hosts if a port is supposed to receive a lot of values continuously (e.g. filter gain, pan, pitchbend) or rarely changed (e.g. switching legato, glissando, etc.).

LV2 seems to have such parameter kinds e.g. expensive so we might want to have something similar for better host implementation performance.

There are two blockers that seems to prevent things:

• [x] quite unsure if ASharedMemory FDs passed from the client to the plugin are valid. ASharedMemory_getSize() returns 0. And writing results in SEGV_ACCERR or SEGV_MAPPER.
• [ ] AIBinder_Class created at plugin (at AAPClientContext) side is problematic. Basically it is dummy now and there is no way to retrieve AIBinder_Class for them from scratch.

Currently we don't really have "parameters" or "properties" on a plugin. Parameters are implicitly represented as in ports, and they have broad data width just like audio inputs/outputs. "Properties" (probably implying that they are not limited to floats) are also supposed to be transmitted via those ports.

A problem here is that we shouldn't really have to require / assign ports for each single parameter. If there are 100 parameters (which is likely to happen) there will be 100 ports, 100 x 4096 bytes if the buffer size is 1024, just for one cycle. And that's too much.

What we would like to have instead is to have one single input port to send a set of parameter changes, and output port based on user inquiry of parameter values. sfizz-lv2 has this kind of ports (manifest for input port and notification port), which would be ideal for most of us. mda-lv2, which I took to learn how plugin ports would be designed, is not designed like that.

"Properties" would work in quite similar manner, but I would make it work like MIDI 2.0 Property Exchange (maybe without forcing JSON syntax). If it works like standard way, then plugins would be made of MIDI standard constructs. Requests would be sent to the control input port, and the results would be sent to the notification port. As of v0.7.4 there will be no property exchange support; parameter changes should be all sent via MIDI2 output notifications.

We wouldn't need different names for parameters and properties, they can be both "property", with an optional "type" attribute (float by default). There will be no type support. Non-float properties should be set via sysex, a port, or an extension.

Control inputs are not necessarily real-time safe. In fact most of them wouldn't be. But the inputs will be processed in real-time loop anyways, and there should be some remedy to assure that a control input does not overwrite existing control inputs. An appropriate queue without locking is required.

Since prefab packaging issue has been stuck for a while and it impacts other development direction, I'm rather thinking it's time to improve desktop development experience.

As of v0.4 (where we have JUCE processor integration) the API became somewhat inconsistent and nasty. Especially, the plugin API now exposes shared memory FDs which should be internals. There should be some API cleanup for better code quality and hackability.

On x86_64 emulator it fails with these logcat lines (no visible outputs on test results HTML):

14:26:04.377  4598-4620  AudioPluginHost org...arebonepluginsample  D  bindAudioPluginService: org.androidaudioplugin.aapbarebonepluginsample | org.androidaudioplugin.AudioPluginService
14:26:04.393  4598-4598  AudioP...ervice org...arebonepluginsample  D  onBind done
14:26:04.394  4598-4598  AAP             org...arebonepluginsample  D  AudioPluginServiceConnector: onServiceConnected
14:26:04.463  4598-4620  AAP.proxy       org...arebonepluginsample  E  beginPrepare() failed: (FIXME: due to Android SDK/NDK issue 219987524 we cannot retrieve failure details here)
14:26:04.463  4598-4620  AAP.proxy       org...arebonepluginsample  E  prepareMemory() failed: (FIXME: due to Android SDK/NDK issue 219987524 we cannot retrieve failure details here)

arm64 fails with somewhat different message but result are not realiable yet. To be added.

There seems some confusion around management. binder and proxy are now one per client-app to service-app. Service should be unbound when all connections are gone, but it seems unbinding is not appropriately handled.

It's been named as Android Audio Plugin Framework, but it will be problematic when it became really popular and causes possible Google TM issues. Can be done at any time, but I would probably do it when I create a dedicated organization.

Currently whenever we need extensions we have to use AndroidAudioPluginExtensionTarget which is not very intuitive. Especially I don't like the fact that it exposes aapxs_context to extension users.

There were handful of complaints I had when I was dealing with extensions. I will list them when I'm back to this stack.

Current AndroidAudioPluginBuffer structure is like this:

typedef struct AndroidAudioPluginBuffer {
	size_t num_buffers;
	void **buffers;
	size_t num_frames;
} AndroidAudioPluginBuffer;

It is strongly typed, but the structure is probably too straightforward. There is no way to provide safe access to those pointers. It is not ideal.

I have been aware of this, but the primary reason I didn't touch it was because it is part of the public API that affects everywhere. But it should be changed, probably sooner.

I have been wondering how we should configure MidiDeviceService per plugin e.g. (1) treat program changes as preset changes, or (2) treat Assignable Controllers as plugin parameter changes, or even (3) treat CCs as plugin parameter changes (then it works like MIDI mappings).

And I was wondering if they should be configured by the plugin developer or not. At this state I think: maybe not. If it is configurable by user then that is the best.

Those 3 options are simple checkboxes and easy to implement.