Data Stream API¶

Streams are the basic building blocks of any PhysioLab^XR paradigm. Streams are connected to data sources that produces data over time. PhysioLab^XR has the following hooks to connect to data sources and create streams:

1. Network: connects to data streams broadcasted through the local area network. PhysioLab^XR supports two popular network APIs for streaming experiment data: LSL and ZMQ

2. Python API: for devices that has Python API, such as brainflow for OpenBCI devices, users can write their own data source script and connect it to PhysioLab^XR through the scripting interface. More information here.

3. Video and audio devices: for devices recognized by the OS as webcams or microphones, they will automatically appear on the list of devices available to stream in PhysioLab^XR.　More information here.

Introduction¶

Lab Streaming Layer (LSL)¶

LabStreamingLayer (LSL) is a network protocol for streaming time series data, mostly used for physiological experiments. LSL has the following characteristics:

• Native support for many popular devices (more information here on how to connect supported device with LSL).

• LSL is a cross-platform protocol, and has bindings for many programming languages. You can create your own data source script in your platform and PhysioLab^XR will take care of the receiving end.

• LSL outlets can carry meta information of your stream such as sampling rate and channel names.

LSL has API in many programming languages. For a more comprehensive guide, check out the examples in LSL documentation. Here, we will show two simple example of how to create an LSL outlet in Python and Unity.

ZeroMQ (ZMQ)¶

ZMQ is a messaging library for exchanging data between applications. ZMQ has the following characteristics:

• ZMQ is a lightweight messaging library, and it is suited for streaming data with a large number of channels (e.g., video).

• Like LSL, ZMQ is a cross-platform protocol, and has bindings for many programming languages.

• ZMQ supports many different socket patterns. PhysioLab^XR supports the subscriber-publisher socket pattern, because this pattern is more scalable when the cardinality of data is high and less error-prone in case of either the publisher or subscriber process crashes. You can find a more detailed explanation of the socket patterns here.

Write Your Own Data Source In Python¶

Using LSL¶

To run the example, make sure you have Python installed. Then, install the physiolabxr package by running the following command in your terminal:

pip install physiolabxr

Alternatively, you can just install the pylsl package by running the following command in your terminal:

pip install pylsl

Now we are ready to run the example.

1. Create a new Python file and copy the following code into it:

import time
from pylsl import StreamInfo, StreamOutlet, local_clock
from random import random as rand

# create a new stream info and outlet

stream_name = 'python_lsl_my_stream_name'

n_channels = 8
# using the local_clock() to track elapsed time
start_time = local_clock()
# track how many samples we have sent
sent_samples = 0
# set the sampling rate to 100 Hz
nominal_sampling_rate = 100

info = StreamInfo('stream_name', 'my_stream_type', n_channels, nominal_sampling_rate, 'float32',
                  'my_stream_id')
outlet = StreamOutlet(info)

# send data
while True:
    # calculate how many samples we need to send since the last call
    elapsed_time = local_clock() - start_time
    required_samples = int(nominal_sampling_rate * elapsed_time) - sent_samples
    for sample_ix in range(required_samples):
        mysample = [rand() * 10 for _ in range(n_channels)]
        # now send it
        outlet.push_sample(mysample)
    sent_samples += required_samples
    # now send it and wait for a bit before trying again.
    time.sleep(0.01)

2. Run the above script with:

python <your-file-name>.py

You can find this script in PhysioLab^XR’s GitHub repository examples-WriteYourOwnDataSourceExamples.

Using ZMQ¶

Write Your Own Data Source In Unity (C#)¶

You can Download

Using LSL¶

Using ZMQ¶

Visualize Your Data¶

Now that you have a data source running, you can visualize your data in PhysioLab^XR by following the steps below:

Visualize LSL Data¶

1. Open PhysioLab^XR (download the App here if you haven’t already).

2. Type in the name of the stream you created in the script (e.g. python_lsl_my_stream_name or unity_lsl_my_stream_name in the example above) in Add Stream Widget.

3. Select LSL as the stream type in the dropdown just right where you put in the stream name.

4. Click on the Add button, a stream widget will be added to the main window. The widget will have icon on the bottom. This means that the stream is available on the network. But we are not receiving and plotting any data yet.

5. Click on the button to start streaming the data.

6. You will be prompted to auto-set the number of channels to what’s being streamed from your script above. This is because the default number of channels is 1 for newly added stream widget. Click on Yes to auto-set the number of channels.

7. The icon will change to icon. You will see the data being plotted in line chart (explore other visualization method).

Using LSL¶

LabStreamingLayer (LSL) is a network protocol for streaming time series data, mostly used for physiological experiments. LSL has the following characteristics:

• Native support for many popular devices (more information here on how to connect supported device with LSL).

• LSL is a cross-platform protocol, and has bindings for many programming languages. You can create your own data source script in your platform and PhysioLab^XR will take care of the receiving end.

• LSL outlets can carry meta information of your stream such as sampling rate and channel names.

Write your LSL data source¶

LSL has API in many programming languages. For a more comprehensive guide, check out the examples in LSL documentation. Here, we will show two simple example of how to create an LSL outlet in Python and Unity.

Python¶

To run the example, make sure you have Python installed. Then, install the physiolabxr package by running the following command in your terminal:

pip install physiolabxr

Alternatively, you can just install the pylsl package by running the following command in your terminal:

pip install pylsl

Now, create a new Python file and copy the following code:

import time
import numpy as np
from pylsl import StreamInfo, StreamOutlet

# create a new stream info and outlet
n_channels = 8
info = StreamInfo('my_stream_name', 'my_stream_type', n_channels, 100, 'float32', 'my_stream_id')
outlet = StreamOutlet(info)

# send data
while True:
    # make a new random 8-channel sample; this is converted into a pylsl.vectorf (the data type that is expected by push_sample)
    mysample = np.random.randn(n_channels).tolist()
    # now send it and wait for a bit
    outlet.push_sample(mysample)
    time.sleep(0.01)

Run the above script with:

python <your-file-name>.py

Unity¶

Make sure you have Unity installed. You can clone this `example project <

Similar to the previous Python example, the following script will create an LSL Outlet in Unity and broadcast the data to the local area network.

Visualize the stream in PhysioLab^XR¶

Open PhysioLab^XR (download the App here if you haven’t already).

Type in the name of the stream you created in the script (my_stream_name in the example above) in Add Stream Widget.

Select LSL as the stream type in the dropdown just right where you put in the stream name.

Click on the Add button, a stream widget will be added to the main window. The widget will have icon on the bottom. This means that the stream is available on the network. But we are not receiving and plotting any data yet.

Click on the button to start streaming the data.

You will be prompted to auto-set the number of channels to what’s being streamed from your script above. This is because the default number of channels is 1 for newly added stream widget. Click on Yes to auto-set the number of channels.

The icon will change to icon. You will see the data being plotted in line chart (explore other visualization method).

Using ZMQ¶

ZMQ is a messaging library for exchanging data between applications. ZMQ has the following characteristics:

• ZMQ is a lightweight messaging library, and it is suited for streaming data with a large number of channels (e.g., video).

• Like LSL, ZMQ is a cross-platform protocol, and has bindings for many programming languages.

• ZMQ supports many different socket patterns. PhysioLab^XR supports the subscriber-publisher socket pattern, because this pattern is more scalable when the cardinality of data is high and less error-prone in case of either the publisher or subscriber process crashes. You can find a more detailed explanation of the socket patterns here.

Write your ZMQ data source¶

Similar to LSL, ZMQ can be used in many programming languages. Here, we will show a simple example of how to create a ZMQ publisher to send data in Python and Unity (C#). The following two section will show how to create ZMQ data source in Python and C# for your reference. For this example, you can choose your preferred language to create the data source.

The following code will send image stream with random data to PhysioLab^XR through ZMQ.

To create the ZMQ data source in Python, first make sure you have Python installed. Then, install the pyzmq package by running the following command in your terminal:

pip install pyzmq

Now, create a new Python file and copy the following code:

import time
from collections import deque

import numpy as np
import zmq

topic = "my_stream_name"  # name of the publisher's topic / stream name
srate = 15  # we will send 15 frames per second
port = "5557"  # ZMQ port number

# we will send a random image of size 400x400 with 3 color channels
c_channels = 3
width = 400
height = 400
n_channels = c_channels * width * height

context = zmq.Context()
socket = context.socket(zmq.PUB)
socket.bind("tcp://*:%s" % port)

# next make an outlet
print("now sending data...")
send_times = deque(maxlen=srate * 10)
start_time = time.time()
sent_samples = 0
while True:
    elapsed_time = time.time() - start_time
    required_samples = int(srate * elapsed_time) - sent_samples
    if required_samples > 0:
        samples = np.random.rand(required_samples * n_channels).reshape((required_samples, -1))
        samples = (samples * 255).astype(np.uint8)
        for sample_ix in range(required_samples):
            mysample = samples[sample_ix]
            socket.send_multipart([bytes(topic, "utf-8"), np.array(time.time()), mysample])  # send frame in the order: topic, timestamp, data
            send_times.append(time.time())
        sent_samples += required_samples
    # now send it and wait for a bit before trying again.
    time.sleep(0.01)
    if len(send_times) > 0:
        fps = len(send_times) / (np.max(send_times) - np.min(send_times))
        print("Send FPS is {0}".format(fps), end='\r')
    print(f'current timestamp is {time.time()}', end='\r', flush=True)

Run the above script with:

python <your-file-name>.py

The following code will send image stream from a Unity camera to PhysioLab^XR through ZMQ.

To make the C# script work, you need install NetMQ, the C# binding for ZMQ. NetMQ is distributed with NuGet, which is a package manager for .NET. Download the NuGetForUnity package from this page. Download the file named NuGetForUnity.<some-version>.unitypackage and import it to your Unity project.

Once you have NuGetForUnity installed, you will have a new option in your Unity Editor’s context menu called NuGet. Go to NuGet > Manage NuGet Packages. In the NuGet window, search for NetMQ and install it. This will install NetMQ and its dependencies.

Attach the following script to a game object in your scene. Set the script’s public field captureCamera to a camera in the scene whose image you want to send.

using System.Collections;
using UnityEngine;
using AsyncIO;
using NetMQ;
using NetMQ.Sockets;
using System;

public class TestZMQScript : MonoBehaviour
{
    [Header("In-game Objects")]
    public Camera captureCamera;  // in your editor, set this to the camera you want to capture

    [Header("Camera Capture Image Size")]
    public int imageWidth = 400;
    public int imageHeight = 400;

    [Header("Runtime Parameters")]
    public float srate = 15f;

    // objects to hold the image data;
    RenderTexture tempRenderColorTexture;
    Texture2D colorImage;

    [Header("Networking Fields")]
    public string tcpAddress = "tcp://localhost:5557";
    public string topicName = "CameraCapture";
    PublisherSocket socket;

    [Header("Networking Information (View-only)")]
    public long imageCounter = 0;
    public float fps = 0;

    private void Start()
    {
        // check if capture camera has been set
        if (captureCamera == null)
        {
            Debug.LogError("CameraCaptureServer: captureCamera is not set. Please set it in the editor.");
            return;
        }

        tempRenderColorTexture = new RenderTexture(imageWidth, imageHeight, 24, RenderTextureFormat.ARGB32)
        {
            antiAliasing = 4
        };

        colorImage = new Texture2D(imageWidth, imageHeight, TextureFormat.RGB24, false, true);

        ForceDotNet.Force();
        socket = new PublisherSocket(tcpAddress);
        StartCoroutine(UploadCapture(1f / srate));
    }

    IEnumerator UploadCapture(float waitTime)
    {
        while (true)
        {
            yield return new WaitForSeconds(waitTime);

            float frameStartTime = Time.realtimeSinceStartup;
            byte[] imageBytes = encodeColorCamera();

            double timestamp = Time.unscaledTime;
            socket.SendMoreFrame(topicName).SendMoreFrame(BitConverter.GetBytes(timestamp)).SendFrame(imageBytes);

            imageCounter++;
            float frameEndTime = Time.realtimeSinceStartup;
            fps = 1f / (frameEndTime - frameStartTime);
        }
    }

    public byte[] encodeColorCamera()
    {
        //render color camera and save bytes
        captureCamera.targetTexture = tempRenderColorTexture;
        RenderTexture.active = tempRenderColorTexture;
        captureCamera.Render();

        colorImage.ReadPixels(new Rect(0, 0, colorImage.width, colorImage.height), 0, 0);
        colorImage.Apply();

        return colorImage.GetRawTextureData();
    }

    private void OnDestroy()
    {
        socket.Dispose();
        NetMQConfig.Cleanup();
    }

}

In both Python and C# scripts, we created an ZMQ publisher socket that sends a random frames with 400 × 400 × 3 (480000) channels, as if it is a video stream that has a height of 400, width of 400, and 3 color channels. The publisher sends 15 frames per second. The Python script sends random images while the Unity script sends images from a Unity camera.

Now return to PhysioLab^XR (download the App here if you haven’t already). In the Add Stream line edit, type in the name of the stream you created in the script (my_stream_name in the example above).

Select ZMQ as the stream type in the dropdown just right where you put in the stream name.

After ZMQ is selected, the port number line edit will show up. Type in the port number you used in the script (5557 in the example above).

Click on the Add button. A stream widget will be added to the main window. Unlike LSL stream, ZMQ stream will not have icon on the bottom. This is because ZMQ does not have a mechanism to check if the stream is available on the network.

Now click on the button.

You will be prompted to auto-set the number of channels to what’s being streamed from your script above. This is because the default number of channels is 1 for newly added stream widget. Click on Yes to auto-set the number of channels.

Because the number of channels we set is 12288, greater than the maximum number of channels that can be plotted in a line chart. It will automatically switch to image plot (explore other visualization method). You need to set the height and width and other image meta info in its to see the image.

Click on ... button to open the plot settings. Set the width and height to 64, and Image to “rgb”.

Return to the plot widget, move your cursor to the lower left of the plot, click the [A] button that shows up to have the image auto-scale to fit the window.

You can create your own data source using the Scripting Interface in Physiolab^XR. This section in Scripting gives detailed examples on how to write your own data source using the third party device driver API and the Physiolab^XR scripting interface.

Video and audio input devices recognize by your OS are automatically detected and can be used as data sources.

Video Devices¶

PhysioLab^XR automatically detects the video input devices connected to your computer. Their name will be listed in the Add Stream dropdown as Camera <camera id>. To add an video input stream:

1. Click on the drop down of Add Stream and select the video device you want to add.

2. Click on Add button.

Audio Input Devices¶

Similar to the the video input devices, PhysioLab^XR automatically detects the audio input devices connected to your computer. Their name will be listed in the Add Stream dropdown. To add an audio input stream:

1. Click on the drop down of Add Stream and select the audio device you want to add.

2. Once selected, a few new options settings will show up on the right of the dropdown.

3. Set the sampling rate (default: 8192), frame/buffer (default: 128), and data type (default: paInt16) of the audio device.

4. Click on Add button.

Screen Capture¶

Similar to the the video input devices, PhysioLab^XR supports streaming screen capture. The name of the screen capture stream will be monitor 0 as default. To add an video input stream:

1. Click on the drop down of Add Stream and select monitor 0.

2. Click on Add button.

Refresh the list of devices¶

If you have connected a new device, you can refresh the list of devices by going to File > Settings > Video Device, and click Reload Video Devices. Similarly, you can refresh the list of audio devices by going to File > Settings > Audio Device, and click Reload Audio Devices.

If your device is recognized by the OS, but not by PhysioLab^XR and the data is not streamed correctly. Please submit an issue here.