<div dir="ltr"><div dir="ltr"><div><br></div></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Sep 22, 2023 at 5:39 AM Christian Marangi <<a href="mailto:ansuelsmth@gmail.com" target="_blank">ansuelsmth@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Fri, Sep 22, 2023 at 02:28:06PM +0200, Andrew Lunn wrote:<br>
> On Fri, Sep 22, 2023 at 01:12:47PM +0200, Christian Marangi wrote:<br>
> > Commit 8fce33317023 ("net: stmmac: Rework coalesce timer and fix<br>
> > multi-queue races") decreased the TX coalesce timer from 40ms to 1ms.<br>
> > <br>
> > This caused some performance regression on some target (regression was<br>
> > reported at least on ipq806x) in the order of 600mbps dropping from<br>
> > gigabit handling to only 200mbps.<br>
> > <br>
> > The problem was identified in the TX timer getting armed too much time.<br>
> > While this was fixed and improved in another commit, performance can be<br>
> > improved even further by increasing the timer delay a bit moving from<br>
> > 1ms to 5ms.<br></blockquote><div><br></div>I am always looking for finding ways to improve interrupt service time, rather than paper over the problem by increasing batchi-ness. <div><br></div><div><a href="http://www.taht.net/~d/broadcom_aug9_2018.pdf" target="_blank">http://www.taht.net/~d/broadcom_aug9_2018.pdf</a><div class="gmail-sendToPdfFiller" style="width:40px;display:inline-block;margin-left:10px" title="Open with pdfFiller"><img style="background: none;"></div></div><div><br></div><div>But also looking for hard data.</div><div><br></div><div><div>I have tried to question other assumptions more modern kernels are making, in particular I wish more folk would experience with decreasing</div><div>the overlarge (IMHO) NAPI default of 64 packets to, say 8 in the mq case, benefiting from multiple arm cores still equipped with limited cache, as well as looking at the impact of TLB flushes,</div><div>and other deferred multi-core processing... that is looking good on a modern xeon, but might not be so good on a more limited arm. </div><div><br></div><div>Over here there was an enormous test series recently run against a bunch of older arm64s which appears to indicate that memory bandwidth is a source of problems:</div><div><br></div><div><a href="https://docs.google.com/document/d/1HxIU_TEBI6xG9jRHlr8rzyyxFEN43zMcJXUFlRuhiUI/edit" target="_blank">https://docs.google.com/document/d/1HxIU_TEBI6xG9jRHlr8rzyyxFEN43zMcJXUFlRuhiUI/edit</a><br></div><div><br></div></div><div>We are looking to add more devices to that testbed.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
> > <br>
> > The value is a good balance between battery saving by prevending too<br>
> > much interrupt to be generated and permitting good performance for<br>
> > internet oriented devices.<br>
> <br>
> ethtool has a settings you can use for this:<br>
> <br>
> ethtool -C|--coalesce devname [adaptive-rx on|off] [adaptive-tx on|off]<br>
> [rx-usecs N] [rx-frames N] [rx-usecs-irq N] [rx-frames-irq N]<br>
> [tx-usecs N] [tx-frames N] [tx-usecs-irq N] [tx-frames-irq N]<br>
> [stats-block-usecs N] [pkt-rate-low N] [rx-usecs-low N]<br>
> [rx-frames-low N] [tx-usecs-low N] [tx-frames-low N]<br>
> [pkt-rate-high N] [rx-usecs-high N] [rx-frames-high N]<br>
> [tx-usecs-high N] [tx-frames-high N] [sample-interval N]<br>
> [cqe-mode-rx on|off] [cqe-mode-tx on|off] [tx-aggr-max-bytes N]<br>
> [tx-aggr-max-frames N] [tx-aggr-time-usecs N]<br>
> <br>
> If this is not implemented, i suggest you add support for it.<br>
> <br>
> Changing the default might cause regressions. Say there is a VoIP<br>
> application which wants this low latency? It would be safer to allow<br>
> user space to configure it as wanted.<br>
><br>
<br>
Yep stmmac already support it. Idea here was to not fallback to use<br>
ethtool and find a good value.<br>
<br>
Just for reference before one commit, the value was set to 40ms and<br>
nobody ever pointed out regression about VoIP application. Wtih some<br>
testing I found 5ms a small increase that restore original perf and<br>
should not cause any regression.<br>
<br>
(for reference keeping this to 1ms cause a lost of about 100-200mbps)</blockquote><div><br></div><div>Does this driver have BQL also?</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
(also the tx timer implementation was created before any napi poll logic<br>
and before dma interrupt handling was a thing, with the later change I<br>
expect this timer to be very little used in VoIP scenario or similar<br>
with continuous traffic as napi will take care of handling packet)<br></blockquote><div><br></div><div>I would be pretty interested in a kernel flame graph of the before vs the after. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Aside from these reason I totally get the concern and totally ok with<br>
this not getting applied, was just an idea to push for a common value.<br></blockquote><div><br></div><div>I try to get people to run much longer and more complicated tests such as the flent rrul test to see what kind of damage bigger buffers did to latency, as well as how other problems might show up. Really notable in the above test series was how badly various devices behaved over time on that workload. Extremely notable in that test series above was how badly the jetson performed:</div><div><br></div><div><a href="https://github.com/randomizedcoder/cake/blob/2023_09_02/pfifo_fast/jetson.png">https://github.com/randomizedcoder/cake/blob/2023_09_02/pfifo_fast/jetson.png</a></div><div><br></div><div>And the nanopi was weird.</div><div><br></div><div><a href="https://github.com/randomizedcoder/cake/blob/2023_09_02/pfifo_fast/nanopi-neo3.png">https://github.com/randomizedcoder/cake/blob/2023_09_02/pfifo_fast/nanopi-neo3.png</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Just preferred to handle this here instead of script+userspace :(<br>
(the important part is the previous patch)<br>
<br>
-- <br>
Ansuel<br>
<br>
</blockquote></div><br clear="all"><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div>Oct 30: <a href="https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html" target="_blank">https://netdevconf.info/0x17/news/the-maestro-and-the-music-bof.html</a></div><div>Dave Täht CSO, LibreQos<br></div></div></div></div>