Streaming: LJM_eStreamRead gives error 1301 (LJME_LJM_BUFFER_FULL) or many -9999 values in aData. What can I try?

When the program running LJM code is unable to keep up with transferring stream data from the device to the computer, LJM_eStreamRead will return error LJME_LJM_BUFFER_FULL (1301) and/or stream data will contain many LJM_DUMMY_VALUE (-9999) values. When device sample collection greatly outpaces sample transfer from device, STREAM_AUTO_RECOVER_END_OVERFLOW can occur as well. Here's how to address these issues.

Summary

Use a ScansPerRead that is relatively large (one half of the ScanRate is a good place to start)
Try an Ethernet connection if possible
Call LJM_eStreamRead as quickly as possible
Use an additional thread or process
Increase the buffer sizes
Remove slow USB hubs
Remove / disable unused connection types
Free up processing power by ending unneeded processes
Use plain C/C++
Increase your program's priority level
Disable the LJM debug log

ScansPerRead

The larger the number of ScansPerRead you set up in LJM_eStreamStart, the less overhead there will be. Increasing ScansPerRead will see larger improvements up until the maximum samples per packet is reached, and smaller improvements after that. For a T-series USB connection, the maximum samples per packet is 24. Over Ethernet, the maximum samples per packet is 512. For example, if you have 2 channels, a ScansPerRead of 12 will utilize the maximum samples per packet over USB.

24 samples per packet ÷ 2 samples per scan = 12 scans per packet

Ethernet

Ethernet streaming usually ends up being more efficient for this reason, so if you are having problems with USB streaming, try switching to Ethernet streaming if you can. Please note that WiFi streaming is not as fast as Ethernet streaming. Direct Ethernet is even faster than Ethernet through a network.

Call LJM_eStreamRead quickly

Sometimes there can be unexpectedly long delays between LJM_eStreamRead calls, such as during intense calculations. Make sure your program is calling LJM_eStreamRead as quickly as possible. It may be worth putting some timing code in your program to calculate whether LJM_eStreamRead is called at the right frequency. For example, if your scanRate is 100,000 Hz and your ScansPerRead is 50,000, you'll want LJM_eStreamRead to be called at 2 Hz. You could also try removing all parts of the stream read loop except for the call to LJM_eStreamRead, then work your way back towards full functionality.

Use an additional thread or process

Related to the need to the previous point to call LJM_eStreamRead as quickly possible, lengthy operations should be performed in a separate thread or process from the thread that calls LJM_eStreamRead. If work being done in the loop that calls LJM_eStreamRead is slow, such as in the case of displaying graphics, then LJM_eStreamRead will fall behind reading from stream. This may cause LJMScanBacklog to increase and eventually cause a LJME_LJM_BUFFER_FULL error. To prevent this, the data from aData may be passed to an alternate thread or process which will perform the lengthy operations.

Increase the buffer sizes

Both LJM and the streaming device have buffers that can run out of space.

To increase the device buffer: write a larger value to the device's STREAM_BUFFER_SIZE_BYTES register. See the T-series stream documentation for details about STREAM_BUFFER_SIZE_BYTES.
To increase LJM's stream buffer: write a larger value to "LJM_STREAM_BUFFER_MAX_NUM_SECONDS". LJM uses this config to calculate the buffer size according to the sample rate. As of this writing, the default is 20 seconds and the maximum is determined by the amount of memory your system has available.

Remove slow USB hubs

Some USB hubs can reduce the maximum throughput of stream:

Try bypassing any USB hubs and plugging the LabJack's connection directly into the host computer's USB highest-speed port
Try a high-speed USB hub

Remove / disable unused connection types

Communication can incur non-trivial overhead in microprocessor-based devices. For example, when streaming at maximum rates via USB on the T7, plugging in an Ethernet cable can suddenly cause skipped scans to occur.

If you're streaming via USB, unplug the LabJack's Ethernet cable, then power-cycle the device before re-starting stream.

Plain C/C++

Using C or C++ is the fastest way to use LJM. You may want to try running the stream_basicC example to see how it performs and adjusting stream parameters as needed.

Priority level

LJM 1.2100 and later automatically sets a higher priority. It sets higher thread and process priority on Windows, and sets higher thread priority only on Linux and macOS (see chrt below to set process priority on Linux). LJM's priority configuration is controlled by the following:

Other methods to control priority

Windows: priority can be set by using the start command (/realtime is the highest priority)
Linux and macOS: priority can be set by using the nice command (-20 is the highest priority)
Linux: The chrt command sets the process policy and priority. chrt --rr 1 ./my_test runs my_test with the round-robin real-time policy and priority 1. Unlike nice, chrt uses larger numbers to represent higher priority.
Linux: For processes that access the disk, ionice assigns policy and priority. LJM does not typically access the disk, but ionice can be appropriate for custom programs that log to disk, for example.

Disable the LJM debug log

Though the LJM debug log is disabled by default, you can ensure it's disabled by setting LJM_DEBUG_LOG_MODE to LJM_DEBUG_LOG_MODE_NEVER(1) by using LJM_WriteLibraryConfigS—i.e. call LJM_WriteLibraryConfigS(LJM_DEBUG_LOG_MODE, LJM_DEBUG_LOG_MODE_NEVER);