Using libpd with Unity3D on mobile devices (part 1)

Using libpd with Unity3D on mobile devices (part 1)                                       
This article by our lead developer Sebastian describes the process to get vanilla libpd (Pd’s core) running in Unity (edit mode + iOS/Android exports). If this sounds like gibberish to you, then fear not - it’s a very technical blog-post mainly targeted at fellow developers. Developers: In the article he will not explain Unity or Pd in detail. So you should have at least a vague idea how Pd and Unity work.

To learn more about PD and why we use it read this blog post by our composer Filippo.

“Pure Data (Pd) is a visual programming language developed by Miller Puckette in the 1990s for creating interactive computer music and multimedia works.”
http://en.wikipedia.org/wiki/Pure_Data

Since Tridek will use PD (Pure Data) for audio generation someone needs to glue together the existing pd code and the Unity engine. That someone is me, Sebastian (aka hagish), software developer at the Bit Barons.

Here’s what I have implemented so far. If you’re interested in the full source code: We plan to release the whole Unity-libpd package as an open source Unity plugin (called Project Kalimba) at a later point.

Content of this post

  1. API to communicate with Pd
  2. Using Pd in Unity in edit mode
  3. Libpd on iOS with API bindings
  4. Libpd on Android with API bindings
  5. Problems and future prospects

API to communicate with Pd
Using Pd the game core can focus on sending game centric events and data (like players health) to the audio engine (Pd) and Pd transforms this into audio.

That is the API (C#) we created for this communication direction (canonical mapping to libpd API):

// triggering stuff
void SendBangToReceiver(string receiverName);

// sending data
void SendFloat(float val, string receiverName);
void SendSymbol(string symbol, string receiverName);

// obviously there need to be a way to initialise everything
void Init();

// loading and closing patch files within pd
void CloseFile(int patchId)
int OpenFile(string baseName, string pathName);

Pd itself is also capable of sending events back but we do not need this at the moment. Therefore we skipped the “back direction”.

Using Pd in Unity in edit mode
For testing Pd in edit mode we decided to connect Unity via tcp to a local vanilla Pd instance. So there is no internal libpd in the edit mode. This way one can adjust the patches in the Pd gui at runtime without restarting the game. Also debugging the patches is a lot easier. The downside is that OpenFile/CloseFile does not work automatically. One needs to manually load the requested patch files into Pd.

Unity Edit Mode libpdUnity play screen (left) with some test buttons that trigger some output (buttom) in a running pd patch (right).

In our example Unity connects via localhost to port 32000 and sends a bang to the receiver “test2”. To receive these messages in Pd you need to add a “netreceive 32000” object with a linked empty message box (see (2) in the screenshot). In Unity the message “test2 bang” gets transformed into

set;
addsemi;
add test2 bang;
bang;

to trigger a “; test2 bang” (sends a bang to the object test2) message in the connected message box. This way one can address objects in Pd via receiver name without extra objects/links/messages in the pd patch. Just one netreceive + message.

Here is a C# snippet from the implementation.

public override void SendBangToReceiver(string receiverName)
{
    setup();
    constructAndSendMessagesToSendMessage(receiverName + ” bang”);
}

private void constructAndSendMessagesToSendMessage(string message)
{
    sendPdMessage(“set;”);
    sendPdMessage(“addsemi;”);
    sendPdMessage(“add ” + message);
    sendPdMessage(“bang;”);
}

// no need adding a closing ;
private void sendPdMessage(string message)
{
    if (client != null && client.Connected && stream != null)
    {
        byte[] ba = asciiEncoding.GetBytes(message.Trim().TrimEnd(new char[]{‘;’}).Trim() + “;”);
   
        stream.Write(ba, 0, ba.Length);
    }
    else
    {
        Debug.LogError(“could not send message ” + message + ” to ” + client);
    }
}

Sending data and other stuff works analog.

Libpd on iOS with API bindings
The libpd implementation works straightforward with pd’s iOS binding. Only some small adjustments to the xcode project generated by Unity were necessary:

* additional build flags

-mno-thumb -DHAVE_UNISTD_H -DHAVE_LIBDL -DPD

* in AppController.h

#import “PdAudioController.h”
#import “PdBase.h”
@property (nonatomic, retain) PdAudioController *audioController;

* in AppController.mm after startUnity

self.audioController = [[[PdAudioController alloc] init] autorelease];
[self.audioController configurePlaybackWithSampleRate:44100 numberChannels:2 inputEnabled:YES mixingEnabled:NO];

[self.audioController configureTicksPerBuffer:128];

[PdBase openFile:@”main.pd” path:[[NSBundle mainBundle] resourcePath]];
[self.audioController setActive:YES];
[self.audioController print];

* in AppController.mm at dealloc

self.audioController = nil;

* in AppController.mm after @implementation AppController

@synthesize audioController = audioController_;

The workflow for building patches and necessary resources into the IPA is currently quite provisional. One needs to manually add all necessary files into the xcode project as resources.

Libpd on Android with API bindings
On Android we use Pd’s Android binding with a custom Unity player activity (for setup and teardown):

…
public class CustomUnityPlayerActivity extends UnityPlayerActivity {

    private static final int SAMPLE_RATE = 44100;
…   
    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        try {
            initPd();
        } catch (IOException e) {
            toast(e.toString());
            finish();
        }
    }

    @Override
    protected void onStart() {
        super.onStart();
        PdAudio.startAudio(this);
    }

    @Override
    protected void onStop() {
        PdAudio.stopAudio();
        super.onStop();
    }
   
    @Override
    protected void onDestroy() {
        cleanup();
        super.onDestroy();
    }
   
    private void initPd() throws IOException {
        if (AudioParameters.suggestSampleRate() < SAMPLE_RATE) {
            throw new IOException(“required sample rate not available”);
        }
        int nOut = Math.min(AudioParameters.suggestOutputChannels(), 2);
        if (nOut == 0) {
            throw new IOException(“audio output not available”);
        }
        PdAudio.initAudio(SAMPLE_RATE, 0, nOut, 128, true);
    }

    private void cleanup() {
        // make sure to release all resources
        PdAudio.release();
        PdBase.release();
    }
}

The binding (SendBangToReceiver, …) is straightforward calling pd Android from C#:

private void _sendBangToReceiver (string receiverName)
{
    AndroidJavaClass jc = new AndroidJavaClass(“org.puredata.core.PdBase”);
    jc.CallStatic<int>(“sendBang”, receiverName);
}
   
private void _sendFloat (float val, string receiverName)
{
    AndroidJavaClass jc = new AndroidJavaClass(“org.puredata.core.PdBase”);
    jc.CallStatic<int>(“sendFloat”, receiverName, val);
}

private int _openFile (string baseName, string pathName)
{
    AndroidJavaClass jc = new AndroidJavaClass(“org.puredata.core.PdBase”);
    // TODO this is just provisional
    return jc.CallStatic<int>(“openPatch”, “/mnt/sdcard/kalimba/” + baseName);
}

As you can see in the _openFile implementation there is a hardcoded path to the sd card. Building pd patches and resources into a apk is currently not implemented. One must copy them manually onto the sd card.

Future prospects

As you can see, this is a very short introduction on how to integrate libpd with Unity on mobile devices. It represents exactly the work we’ve already done so far. Over the course of Tridek’s development, I’ll have to fix a few more problems and expand the current implementation. This is why this post is only the first part of the whole series.

Here’s what’s still to-do:

  • The build process is completely provisional. On both platforms one need to manually copy patches and resources. We plan to improve this in the future ;)
  • Performance-wise Pd works better on iOS than on Android. But this is based on non-scientific observations only. In our small test programs both platforms work good enough.
  • The pd -> game communication direction is missing. We are not sure if we will need it. So currently there is no plan to implement it.
  • We plan to add the OGG and MIDI file reader objects. Currently there is only plain vanilla pd.
  • We plan to release Kalimba as an open source Unity plugin.

More information / links

Source: schattenkind.net

1 Notes/ Hide

  1. tridek posted this

Recent comments

Blog comments powered by Disqus