Power & Source of Big Ideas

Poor "cyclictest" results from package "rt-tests"

Moderators: chensy, FATechsupport

This test tool, cyclictest, is yielding abysmal results. The latency response times for a couple of the CPUs out of the 4 we know about in the H5 are over 4.7 msecs. These numbers should be under 100 usecs as a maximum.

So far I have tried both the friendlyarm kernel and one I built froma 5.1 mainline. Both yield similar results. And there is large amounts of jitter also. Something is terribly wrong, either with the hardware or with one or more of the drivers.

Next I will try Armbian, but the problem I have with Armbian is finding the H5 kernel configuration options they use.

root@NanoPi-NEO-Plus2:/etc# cyclictest --mlockall --smp --priority=80 --interval=200 --distance=0
# /dev/cpu_dma_latency set to 0us
policy: fifo: loadavg: 0.01 0.05 0.01 1/104 1366

T: 0 ( 1363) P:80 I:200 C: 153777 Min: 5 Act: 59 Avg: 21 Max: 4799
T: 1 ( 1364) P:80 I:200 C: 153874 Min: 5 Act: 21 Avg: 21 Max: 1627
T: 2 ( 1365) P:80 I:200 C: 153686 Min: 4 Act: 20 Avg: 20 Max: 4790
T: 3 ( 1366) P:80 I:200 C: 153703 Min: 4 Act: 27 Avg: 20 Max: 1257

The above is for the stock image that comes on the eMMC out of the box.
The latest released Armbian kernel did only slightly better. Its MAX latency value was still over 3.6 msecs, however it took much longer to manifest itself. Experts on cyclic test suggest that you have to run this tool for long runs to be sure you flushed out any problems with jitter, I have seen 100 hours suggested. With Armbian the 3.6 msec delay materialized in about 4 minutes, whereas the other two kernels show the problem in less than 2 seconds.

I compared kernel configuration options, but nothing was obvious there, it happens with PREEMPT, with PREEMT_NONE, and PREEMPT_VOLUNTARY. There seems to be no silver bullet to solve this problem using that path.

I think there is some misbehaved code (that could be H5 specific) in the linux kernel. And this would encompass versions 4.14 through 5.1 of the linux kernel.

The latency following these timer interrupts in cyclictest are similar to what I have observed in a kernel driver that I wrote for a GPIO based interrupt at intr number 152. So I am thinking if it happens in the same way with cyclictest, then it takes my code out from under suspicion.

If nobody comes to my rescue soon, I will have to become a practitioner of the kernel's ftrace soon. I need to get the latency below 400 usecs consistently for my application. Its fascinating to see that the AVE latency in cyclictest is near 20 usecs, which makes it comparable to an x86 box. But this worst case latency of 4.8 msecs is is a no go for me, and probably anyone else that is doing complementary boards that use interrupts.

Who is online

In total there are 5 users online :: 0 registered, 0 hidden and 5 guests (based on users active over the past 5 minutes)
Most users ever online was 5185 on Wed Jan 22, 2020 1:44 pm

Users browsing this forum: No registered users and 5 guests