Externally Synced Stream App Note

The T8 offers a unique feature called “Externally Synced Stream” or ESS.
ESS is a feature that allows synchronization of analog channels across multiple T8 devices.

ESS is currently in Alpha.

This means that it still under development and API might change.
There might be bugs and issues, so all feedback would be appreciated during this development phase.

ESS is only available through Ethernet connection type at this time.

Why Externally Synced Stream?

We offer multiple synchronization options for some of our T series devices:

Externally Triggered Stream starts streams for multiple devices at the same time, where streams are started within one scan period of each other.
Externally Clocked Stream allows for variable stream rates where a scan list is performed for every digital pulse, within one scan period.
The T8 has 8 analog channels that are hardware synced to truly sample simultaneously.

Externally Synced Stream combines the ideas of Externally Clocked Stream and the performance of the T8’s 8 analog channels too hardware synchronize multiple T8 analog system across multiple devices, resulting in synchronization down to 2 micro seconds.

Setting up Externally Synced Stream?

The core of how ESS works is that it is sharing a clock signal between multiple devices, with a primary device exporting its analog hardware clock signal and secondary devices using that signal to drive their analog hardware. For all this clock sharing to work, we need to wire up the devices to accept clock sources.

Primary:
- FIO0 - Clock out
- FIO2 - Timer in
- GND - Common Ground
Secondary:
- FIO1 - Clock in
- FIO2 - Timer in
- GND - Common Ground

Above are the pins needed for ESS depending on which device you are setting up. For starters, you want all devices to share a common ground, so wire all devices GND together. Next, you need wire the Clock out of a primary to the Clock in of every secondary, this can be made easier with daisy chaining. Finally, every device needs a wire from the clock line to Timer in, which is necessary for Stream mode to work.

You can wire any device for either primary or secondary by wiring FIO0-2 together on one device, and then connect the wired cluster to another cluster with a second wire for GND. This wiring setup will work for both primary and secondary but will take up one extra pin with the benefit of not worrying about which is primary or secondary.

Programming for ESS

The following section will go over how to setup an ESS system with your T8s.

To begin, you will need the correct software.

ESS is currently in alpha, and only supports Ethernet connections at this time

Download and run this LJM installer: LabJack_2024-07-16.exe

Then using Kipling 3, update all of your T8s to this firmware: T8firmware_010036_2024-07-15.bin

We provide a C# example bellow that you can use as a guide to programming your devices for ESS:

C#

using System;
using System.Diagnostics;
using LabJack;

class ESS {
    static void Main(string[] args) {
        ESS ess = new ESS();
        ess.run();
    }

    public void run() {
        // Example Parameters
        int MAX_DEVICES = 10;
        int CONNECTION_TYPE = LJM.CONSTANTS.ctETHERNET;
        double MAX_VOLTAGE = 2.0;
        int RESOLUTION_INDEX = 16;
        double DESIRED_RATE = 100.0;
        int SCANS_PER_READ = 10;
        double STREAM_DURATION_SECONDS = 1.0;

        try {
            // Check for avaiable devices on USB
            int found = 0;
            int[] dev_type = new int[MAX_DEVICES];
            int[] con_type = new int[MAX_DEVICES];
            int[] sn       = new int[MAX_DEVICES];
            int[] ip       = new int[MAX_DEVICES];
            LJM.ListAll(LJM.CONSTANTS.dtT8, LJM.CONSTANTS.ctUSB, ref found, dev_type, con_type, sn, ip);
            if (found < 2 || found > MAX_DEVICES) {
                string error = $"Could not find the correct number of T8's on USB: {found} found";
                Console.WriteLine(error);
                throw new Exception(error);
            } else {
                Console.WriteLine($"Number of devices found on T8: {found}");
            }

            // Open found T8's
            int[] handles = new int[found];
            for (int h = 0; h < found; h++) {
                LJM.Open(LJM.CONSTANTS.dtT8, CONNECTION_TYPE, sn[h].ToString(), ref handles[h]);
                Console.WriteLine($"Opened T8 {h} {sn[h]}");
            }

            // Library Configuration
            LJM.WriteLibraryConfigS("LJM_SEND_RECEIVE_TIMEOUT_MS", 0.0);
            LJM.WriteLibraryConfigS("LJM_STREAM_RECEIVE_TIMEOUT_MODE", 2.0);
            LJM.WriteLibraryConfigS("LJM_STREAM_RECEIVE_TIMEOUT_MS", 0.0);

            // Settings to apply to all devices
            for (int h = 0; h < found; h++) {
                LJM.eWriteName(handles[h], "AIN_ALL_RANGE", MAX_VOLTAGE);
                LJM.eWriteName(handles[h], "STREAM_RESOLUTION_INDEX", RESOLUTION_INDEX);
            }

            // Stream Configuration
            string[] scan_list_names = {"AIN0", "AIN1", "AIN2", "AIN3", "AIN4", "AIN5", "AIN6", "AIN7", "AIN_HEALTH"};
            int total_channels = scan_list_names.Length;
            int[] scan_list = new int[total_channels];
            int[] scan_list_types = new int[total_channels];
            LJM.NamesToAddresses(total_channels, scan_list_names, scan_list, scan_list_types);

            Stopwatch stopwatch = new Stopwatch();

            // Commence streaming
            try {
                double actual_rate;
                // Configure all but first device as secondary device
                for (int h = 1; h < found; h++) {
                    LJM.eWriteName(handles[h], "STREAM_CLOCK_SOURCE", 8);
                    actual_rate = DESIRED_RATE;
                    LJM.eStreamStart(handles[h], SCANS_PER_READ, total_channels, scan_list, ref actual_rate);
                    Console.WriteLine($"Device {h} has Started Stream as Secondary at {actual_rate} HZ");
                }

                // Configure first device as primary device
                LJM.eWriteName(handles[0], "STREAM_CLOCK_SOURCE", 4);
                actual_rate = DESIRED_RATE;
                LJM.eStreamStart(handles[0], SCANS_PER_READ, total_channels, scan_list, ref actual_rate);
                Console.WriteLine($"Device 0 has Started Stream as Primary at {actual_rate} HZ");

                int channels_per_read = total_channels * SCANS_PER_READ;

                // Stream for STREAM_DIRATION_SECONDS
                stopwatch.Start();
                while (stopwatch.Elapsed.TotalSeconds < STREAM_DURATION_SECONDS) {
                    // Read Stream for each device
                    for (int h = 0; h < found; h++) {
                        // Read Stream
                        double[] data = new double[channels_per_read];
                        int device_log = -1;
                        int driver_log = -1;
                        try {
                            LJM.eStreamRead(handles[h], data, ref device_log, ref driver_log);
                        } catch (Exception e) {
                            Console.WriteLine($"Device {h} did not stream read: {e.Message}");
                            continue;
                        }
                        Console.WriteLine($"Device {h} Stream Read:\t{device_log}\t{driver_log}");

                        // Process each scan in the read
                        for (int scan = 0; scan < SCANS_PER_READ; scan++) {
                            int index_start = scan * total_channels;

                            // Check for auto recovery
                            if (data[index_start] == -9999.0) {
                                Console.WriteLine($"Device {h} scan {scan} was an auto recovery scan");
                                continue;
                            }

                            // Check channel health
                            if (data[index_start + 8] != 255.0) {
                                string bin_str = Convert.ToString((uint)data[index_start + 8], 2).PadLeft(8, '0');
                                Console.WriteLine($"Device {h} experienced an ESD event, some AIN channels will not work: {bin_str}");
                            }

                            // Print channel data
                            Console.WriteLine($"Device {h}:{scan} " +
                                                $"{data[index_start + 0]:F2}, " +
                                                $"{data[index_start + 1]:F2}, " +
                                                $"{data[index_start + 2]:F2}, " +
                                                $"{data[index_start + 3]:F2}, " +
                                                $"{data[index_start + 4]:F2}, " +
                                                $"{data[index_start + 5]:F2}, " +
                                                $"{data[index_start + 6]:F2}, " +
                                                $"{data[index_start + 7]:F2}");
                        }
                    }
                }
            } catch (Exception e) {
                Console.WriteLine("Exception: " + e.Message);
                Console.WriteLine(e.StackTrace);
            }
            stopwatch.Stop();

            // Stop Stream with primary being last
            for (int h = found - 1; h >= 0; h--) {
                try {
                    LJM.eStreamStop(handles[h]);
                } catch (Exception e) {
                    Console.WriteLine($"Device {h} did not stop stream: {e.Message}");
                }
            }
            Console.WriteLine("All Devices have stopped stream");

        } catch (Exception e) {
            Console.WriteLine("Exception: " + e.Message);
            Console.WriteLine(e.StackTrace);
        }

        LJM.CloseAll();  //Close all handles

        Console.WriteLine("\nDone.\nPress the enter key to exit.");
        //Console.ReadLine();  //Pause for user
    }
}

We can break down this example step by step…

C#

// Example Parameters
int MAX_DEVICES = 10;
int CONNECTION_TYPE = LJM.CONSTANTS.ctUSB;
double MAX_VOLTAGE = 2.0;
int RESOLUTION_INDEX = 16;
double DESIRED_RATE = 100.0;
int SCANS_PER_READ = 10;
double STREAM_DURATION_SECONDS = 1.0;

These are parameters that you are encouraged to adjust for your experimental setup.
It is worth noting that not all combination of RESOLUTION_INDEX and DESIRED_RATE are acceptable, please reference the chart bellow for valid ranges.

Just like normal T8 streaming, the desired rate is not guaranteed, but will be as close as possible.
If exact sampling rate is required, then make sure to check the value after eStreamStart is called.

Resolution Index	Min Rate (Hz)	Max Rate (Hz)
1	892	19531
2	595	22163
3	446	19531
4	297	13020
5	223	9765
6	150	6510
7	142	6250
8	111	4883
9	75	3255
10	71	3125
11	56	2441
12	38	1627
13	36	1562
14	30	1220
15	17	610
16	10	301

C#

// Check for avaiable devices on USB
int found = 0;
int[] dev_type = new int[MAX_DEVICES];
int[] con_type = new int[MAX_DEVICES];
int[] sn       = new int[MAX_DEVICES];
int[] ip       = new int[MAX_DEVICES];
LJM.ListAll(LJM.CONSTANTS.dtT8, LJM.CONSTANTS.ctUSB, ref found, dev_type, con_type, sn, ip);
if (found < 2 || found > MAX_DEVICES) {
    string error = $"Could not find the correct number of T8's on USB: {found} found";
    Console.WriteLine(error);
    throw new Exception(error);
} else {
    Console.WriteLine($"Number of devices found on T8: {found}");
}

// Open found T8's
int[] handles = new int[found];
for (int h = 0; h < found; h++) {
    LJM.Open(LJM.CONSTANTS.dtT8, CONNECTION_TYPE, sn[h].ToString(), ref handles[h]);
    Console.WriteLine($"Opened T8 {h} {sn[h]}");
}

This code is for opening the devices. The example assumes all T8’s connected to the computer over USB are part of the ESS system. So it starts by finding the T8’s on USB, then opening them with the requested connection type. This is not necessary for your implementation.

C#

// Library Configuration
LJM.WriteLibraryConfigS("LJM_SEND_RECEIVE_TIMEOUT_MS", 0.0);
LJM.WriteLibraryConfigS("LJM_STREAM_RECEIVE_TIMEOUT_MODE", 2.0);
LJM.WriteLibraryConfigS("LJM_STREAM_RECEIVE_TIMEOUT_MS", 0.0);

These are configuration that apply to the LJM driver on your computer and not the devices. Normally when a stream is started, the driver expects data soon after, but due to how ESS must be setup across multiple devices, this configures LJM to have an infinite timeout on stream data to allow all the time you need to setup devices for ESS.

C#

// Stream Configuration
string[] scan_list_names = {"AIN0", "AIN1", "AIN2", "AIN3", "AIN4", "AIN5", "AIN6", "AIN7", "AIN_HEALTH"};
int total_channels = scan_list_names.Length;
int[] scan_list = new int[total_channels];
int[] scan_list_types = new int[total_channels];
LJM.NamesToAddresses(total_channels, scan_list_names, scan_list, scan_list_types);

This is simple stream configuration code, but it is important that each device in the ESS system has the same stream configuration. This includes the scan list, packets per read, stream scan rate, and stream resolution index.

It is also worth pointing out the AIN_HEALTH register in the scan list. This will show what AIN channels are working and is meant to monitor ESD events, which will be discussed later in this document.

C#

double actual_rate;
// Configure all but first device as secondary device
for (int h = 1; h < found; h++) {
    LJM.eWriteName(handles[h], "STREAM_CLOCK_SOURCE", 8);
    actual_rate = DESIRED_RATE;
    LJM.eStreamStart(handles[h], SCANS_PER_READ, total_channels, scan_list, ref actual_rate);
    Console.WriteLine($"Device {h} has Started Stream as Secondary at {actual_rate} HZ");
}

// Configure first device as primary device
LJM.eWriteName(handles[0], "STREAM_CLOCK_SOURCE", 4);
actual_rate = DESIRED_RATE;
LJM.eStreamStart(handles[0], SCANS_PER_READ, total_channels, scan_list, ref actual_rate);
Console.WriteLine($"Device 0 has Started Stream as Primary at {actual_rate} HZ");

This code starts stream for the secondary devices first, then the primary. It is important that the primary is started last, since starting the primary stream will start outputting the clock signal that starts the secondaries analog hardware system. Once that signal starts, all devices will start streaming.

This is also assigns which device is a secondary or a primary by setting the STREAM_CLOCK_SOURCE register.

C#

for (int h = 0; h < found; h++) {
    // Read Stream
    double[] data = new double[channels_per_read];
    int device_log = -1;
    int driver_log = -1;
    try {
        LJM.eStreamRead(handles[h], data, ref device_log, ref driver_log);
    } catch (Exception e) {
        Console.WriteLine($"Device {h} did not stream read: {e.Message}");
        continue;
    }
    Console.WriteLine($"Device {h} Stream Read:\t{device_log}\t{driver_log}");

    // Process each scan in the read
    for (int scan = 0; scan < SCANS_PER_READ; scan++) {
        int index_start = scan * total_channels;

        // Check for auto recovery
        if (data[index_start] == -9999.0) {
            Console.WriteLine($"Device {h} scan {scan} was an auto recovery scan");
            continue;
        }

        // Check channel health
        if (data[index_start + 8] != 255.0) {
            string bin_str = Convert.ToString((uint)data[index_start + 8], 2).PadLeft(8, '0');
            Console.WriteLine($"Device {h} experienced an ESD event, some AIN channels will not work: {bin_str}");
        }

        // Print channel data
        Console.WriteLine($"Device {h}:{scan} " +
                            $"{data[index_start + 0]:F2}, " +
                            $"{data[index_start + 1]:F2}, " +
                            $"{data[index_start + 2]:F2}, " +
                            $"{data[index_start + 3]:F2}, " +
                            $"{data[index_start + 4]:F2}, " +
                            $"{data[index_start + 5]:F2}, " +
                            $"{data[index_start + 6]:F2}, " +
                            $"{data[index_start + 7]:F2}");
    }
}

This is the main stream read loop of the program. Here we are looping through each device, and checking the data. At this point you can treat the data and stream read operation as a normal stream and do not need to care about primary or secondary since all the analog hardware is synced up. Normal stream data check is still necessary like checking for auto recoveries, but an extra check you should do would be checking AIN_HEALTH.

As stated earlier, AIN_HEALTH monitors which AIN channels are active on your T8. Normally the only reason an AIN channel would go down would be an ESD event, which would then turn off the AIN channels for some time as the analog system re configures. For ESS, one channel on one device should not stop the test, so the system will continue running and the bad channel will return -9999.0. By checking the AIN_HEALTH register, you can make a decision if you want to continue the experiment or not if an AIN channel goes down.

C#

// Stop Stream with primary being last
for (int h = found - 1; h >= 0; h--) {
    try {
        LJM.eStreamStop(handles[h]);
    } catch (Exception e) {
        Console.WriteLine($"Device {h} did not stop stream: {e.Message}");
    }
}
Console.WriteLine("All Devices have stopped stream");

Once you are done, you should stop all secondaries stream first then stop the primary. This is because of the clock signal that the primary is generating. The secondaries are reliant on that signal to function till the end, so if the primary shutdown first, then the secondaries could become stalled.

Additional Information and FAQ

Since the T8 is driving a signal anywhere from 0.5MHz to 5MHz to each of the devices, care should be made to monitor the signal line and not expose it to EMI or stretch it over long distances.

There are alot of wires coming in and out of a T8 for an ESS experiment, the number one problem during testing was a loose wire. Make sure every wire is secure for the experiment.