<div dir="ltr">Thank you, Ken.<div>This helps a lot.</div><div>Now I am sure that my current approach fits best for me =)</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr">Thank you,<div>Kostia</div></div></div></div></div></div>
<br><div class="gmail_quote">On Wed, Mar 30, 2016 at 11:10 PM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 03/29/2016 08:22 AM, Kostiantyn Ponomarenko wrote:<br>
> Ken, thank you for the answer.<br>
><br>
> Every node in my cluster under normal conditions has "load average" of<br>
> about 420. It is mainly connected to the high disk IO on the system.<br>
> My system is designed to use almost 100% of its hardware (CPU/RAM/disks),<br>
> so the situation when the system consumes almost all HW resources is<br>
> normal.<br>
<br>
</span>420 suggests that HW resources are outstripped -- anything above the<br>
system's number of cores means processes are waiting for some resource.<br>
(Although with an I/O-bound workload like this, the number of cores<br>
isn't very important -- most will be sitting idle despite the high<br>
load.) And if that's during normal conditions, what will happen during a<br>
usage spike? It sounds like a recipe for less-than-HA.<br>
<br>
Under high load, there's a risk of negative feedback, where monitors<br>
time out, causing pacemaker to schedule recovery actions, which cause<br>
load to go higher and more monitors to time out, etc. That's why<br>
throttling is there.<br>
<span class=""><br>
> I would like to get rid of "High CPU load detected" messages in the<br>
> log, because<br>
> they flood corosync.log as well as system journal.<br>
><br>
> Maybe you can give an advice what would be the best way do to it?<br>
><br>
> So far I came up with the idea of setting "load-threshold" to 1000% ,<br>
> because of:<br>
> 420(load average) / 24 (cores) = 17.5 (adjusted_load);<br>
> 2 (THROTLE_FACTOR_HIGH) * 10 (throttle_load_target) = 20<br>
><br>
> if(adjusted_load > THROTTLE_FACTOR_HIGH * throttle_load_target) {<br>
> crm_notice("High %s detected: %f", desc, load);<br>
<br>
</span>That should work, as far as reducing the log messages, though of course<br>
it also reduces the amount of throttling pacemaker will do.<br>
<span class=""><br>
> In this case do I need to set "node-action-limit" to something less than "2<br>
> x cores" (which is default).<br>
<br>
</span>It's not necessary, but it would help compensate for the reduced<br>
throttling by imposing a maximum number of actions run at one time.<br>
<br>
I usually wouldn't recommend reducing log verbosity, because detailed<br>
logs are often necessary for troubleshooting cluster issues, but if your<br>
logs are on the same I/O controller that is overloaded, you might<br>
consider logging only to syslog and not to an additional detail file.<br>
That would cut back on the amount of I/O due to pacemaker itself. You<br>
could even drop PCMK_logpriority to warning, but then you're losing even<br>
more information.<br>
<span class=""><br>
> Because the logic is (crmd/throttle.c):<br>
><br>
> switch(r->mode) {<br>
> case throttle_extreme:<br>
> case throttle_high:<br>
> jobs = 1; /* At least one job must always be allowed */<br>
> break;<br>
> case throttle_med:<br>
> jobs = QB_MAX(1, r->max / 4);<br>
> break;<br>
> case throttle_low:<br>
> jobs = QB_MAX(1, r->max / 2);<br>
> break;<br>
> case throttle_none:<br>
> jobs = QB_MAX(1, r->max);<br>
> break;<br>
> default:<br>
> crm_err("Unknown throttle mode %.4x on %s", r->mode, node);<br>
> break;<br>
> }<br>
> return jobs;<br>
><br>
><br>
> The thing is, I know that there is "High CPU load" and this is normal<br>
> state, but I wont Pacemaker to not saying it to me and treat this state the<br>
> best it can.<br>
<br>
</span>If you can't improve your I/O performance, what you suggested is<br>
probably the best that can be done.<br>
<br>
When I/O is that critical to you, there are many tweaks that can make a<br>
big difference in performance. I'm not sure how familiar you are with<br>
them already. Options depend on what your storage is (local or network,<br>
hardware/software/no RAID, etc.) and what your I/O-bound application is<br>
(database, etc.), but I'd look closely at cache/buffer settings at all<br>
levels from hardware to application, RAID stripe alignment, filesystem<br>
choice and tuning, log verbosity, etc.<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> Thank you,<br>
> Kostia<br>
><br>
> On Mon, Mar 14, 2016 at 7:18 PM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>> wrote:<br>
><br>
>> On 02/29/2016 07:00 AM, Kostiantyn Ponomarenko wrote:<br>
>>> I am back to this question =)<br>
>>><br>
>>> I am still trying to understand the impact of "High CPU load detected"<br>
>>> messages in the log.<br>
>>> Looking in the code I figured out that setting "load-threshold" parameter<br>
>>> to something higher than 100% solves the problem.<br>
>>> And actually for 8 cores (12 with Hyper Threading) load-threshold=400%<br>
>> kind<br>
>>> of works.<br>
>>><br>
>>> Also I noticed that this parameter may have an impact on the number of<br>
>> "the<br>
>>> maximum number of jobs that can be scheduled per node". As there is a<br>
>>> formula to limit F_CRM_THROTTLE_MAX based on F_CRM_THROTTLE_MODE.<br>
>>><br>
>>> Is my understanding correct that the impact of setting "load-threshold"<br>
>>> high enough (so there is no noisy messages) will lead only to the<br>
>>> "throttle_job_max" and nothing more.<br>
>>> Also, if I got it correct, than "throttle_job_max" is a number of allowed<br>
>>> parallel actions per node in lrmd.<br>
>>> And a child of the lrmd is actually an RA process running some actions<br>
>>> (monitor, start, etc).<br>
>>><br>
>>> So there is no impact on how many RA (resources) can run on a node, but<br>
>> how<br>
>>> Pacemaker will operate with them in parallel (I am not sure I understand<br>
>>> this part correct).<br>
>><br>
>> I believe that is an accurate description. I think the job limit applies<br>
>> to fence actions as well as lrmd actions.<br>
>><br>
>> Note that if /proc/cpuinfo exists, pacemaker will figure out the number<br>
>> of cores from there, and divide the actual reported load by that number<br>
>> before comparing against load-threshold.<br>
>><br>
>>> Thank you,<br>
>>> Kostia<br>
>>><br>
>>> On Wed, Jun 3, 2015 at 12:17 AM, Andrew Beekhof <<a href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>><br>
>> wrote:<br>
>>><br>
>>>><br>
>>>>> On 27 May 2015, at 10:09 pm, Kostiantyn Ponomarenko <<br>
>>>> <a href="mailto:konstantin.ponomarenko@gmail.com">konstantin.ponomarenko@gmail.com</a>> wrote:<br>
>>>>><br>
>>>>> I think I wasn't precise in my questions.<br>
>>>>> So I will try to ask more precise questions.<br>
>>>>> 1. why the default value for "load-threshold" is 80%?<br>
>>>><br>
>>>> Experimentation showed it better to begin throttling before the node<br>
>>>> became saturated.<br>
>>>><br>
>>>>> 2. what would be the impact to the cluster in case of<br>
>>>> "load-threshold=100%”?<br>
>>>><br>
>>>> Your nodes will be busier. Will they be able to handle your load or<br>
>> will<br>
>>>> it result in additional recovery actions (creating more load and more<br>
>>>> failures)? Only you will know when you try.<br>
>>>><br>
>>>>><br>
>>>>> Thank you,<br>
>>>>> Kostya<br>
>>>>><br>
>>>>> On Mon, May 25, 2015 at 4:11 PM, Kostiantyn Ponomarenko <<br>
>>>> <a href="mailto:konstantin.ponomarenko@gmail.com">konstantin.ponomarenko@gmail.com</a>> wrote:<br>
>>>>> Guys, please, if anyone can help me to understand this parameter<br>
>> better,<br>
>>>> I would be appreciated.<br>
>>>>><br>
>>>>><br>
>>>>> Thank you,<br>
>>>>> Kostya<br>
>>>>><br>
>>>>> On Fri, May 22, 2015 at 4:15 PM, Kostiantyn Ponomarenko <<br>
>>>> <a href="mailto:konstantin.ponomarenko@gmail.com">konstantin.ponomarenko@gmail.com</a>> wrote:<br>
>>>>> Another question - is it crmd specific to measure CPU usage by "I/O<br>
>>>> wait"?<br>
>>>>> And if I need to get the most performance of the running resources in<br>
>>>> cluster, should I set "load-threshold=95%" (or even 100%)?<br>
>>>>> Will it impact the cluster behavior in any ways?<br>
>>>>> The man page for crmd says that it will "The cluster will slow down its<br>
>>>> recovery process when the amount of system resources used (currently<br>
>> CPU)<br>
>>>> approaches this limit".<br>
>>>>> Does it mean there will be delays in cluster in moving resources in<br>
>> case<br>
>>>> a node goes down, or something else?<br>
>>>>> I just want to understand in better.<br>
>>>>><br>
>>>>> That you in advance for the help =)<br>
>>>>><br>
>>>>> P.S.: The main resource does a lot of disk I/Os.<br>
>>>>><br>
>>>>><br>
>>>>> Thank you,<br>
>>>>> Kostya<br>
>>>>><br>
>>>>> On Fri, May 22, 2015 at 3:30 PM, Kostiantyn Ponomarenko <<br>
>>>> <a href="mailto:konstantin.ponomarenko@gmail.com">konstantin.ponomarenko@gmail.com</a>> wrote:<br>
>>>>> I didn't know that.<br>
>>>>> You mentioned "as opposed to other Linuxes", but I am using Debian<br>
>> Linux.<br>
>>>>> Does it also measure CPU usage by I/O waits?<br>
>>>>> You are right about "I/O waits" (a screenshot of "top" is attached).<br>
>>>>> But why it shows 50% of CPU usage for a single process (that is the<br>
>> main<br>
>>>> one) while "I/O waits" shows a bigger number?<br>
>>>>><br>
>>>>><br>
>>>>> Thank you,<br>
>>>>> Kostya<br>
>>>>><br>
>>>>> On Fri, May 22, 2015 at 9:40 AM, Ulrich Windl <<br>
>>>> <a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a>> wrote:<br>
>>>>>>>> "Ulrich Windl" <<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a>> schrieb am<br>
>>>> 22.05.2015 um<br>
>>>>> 08:36 in Nachricht <<a href="mailto:555EEA72020000A10001A71D@gwsmtp1.uni-regensburg.de">555EEA72020000A10001A71D@gwsmtp1.uni-regensburg.de</a><br>
>>> :<br>
>>>>>> Hi!<br>
>>>>>><br>
>>>>>> I Linux I/O waits are considered for load (as opposed to other<br>
>>>> Linuxes) Thus<br>
>>>>> ^^ "In"<br>
>>>> s/Linux/UNIX/<br>
>>>>><br>
>>>>> (I should have my coffee now to awake ;-) Sorry.<br>
>><br>
>> _______________________________________________<br>
>> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
>> <a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
>><br>
>> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
>> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
>> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
>><br>
><br>
<br>
</div></div></blockquote></div><br></div>