<div dir="ltr">Yes, we do have our application using shared memory which is what we see when the cluster is down.</div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 17, 2016 at 10:53 PM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 05/17/2016 12:02 PM, Nikhil Utane wrote:<br>
> OK. Will do that.<br>
><br>
> Actually I gave the /dev/shm usage when the cluster wasn't up.<br>
> When it is up, I see it occupies close to 300 MB (it's also the DC).<br>
<br>
</span>Hmmm, there should be no usage if the cluster is stopped. Any memory<br>
used by the cluster will start with "qb-", so anything else is from<br>
something else.<br>
<br>
If all executables using libqb (including corosync and pacemaker) are<br>
stopped, it's safe to remove any /dev/shm/qb-* files that remain. That<br>
should be rare, probably only after a core dump or such.<br>
<span class=""><br>
> tmpfs 500.0M 329.4M 170.6M 66% /dev/shm<br>
><br>
> On another node the same is 115 MB.<br>
><br>
> Anyways, I'll monitor the usage to know what size is needed.<br>
><br>
> Thank you Ken and Ulrich.<br>
><br>
> On Tue, May 17, 2016 at 8:23 PM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a><br>
</span><div><div class="h5">> <mailto:<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>>> wrote:<br>
><br>
> On 05/17/2016 04:07 AM, Nikhil Utane wrote:<br>
> > What I would like to understand is how much total shared memory<br>
> > (approximately) would Pacemaker need so that accordingly I can define<br>
> > the partition size. Currently it is 300 MB in our system. I recently ran<br>
> > into insufficient shared memory issue because of improper clean-up. So<br>
> > would like to understand how much Pacemaker would need for a 6-node<br>
> > cluster so that accordingly I can increase it.<br>
><br>
> I have no idea :-)<br>
><br>
> I don't think there's any way to pre-calculate it. The libqb library is<br>
> the part of the software stack that actually manages the shared memory,<br>
> but it's used by everything -- corosync (including its cpg and<br>
> votequorum components) and each pacemaker daemon.<br>
><br>
> The size depends directly on the amount of communication activity in the<br>
> cluster, which is only indirectly related to the number of<br>
> nodes/resources/etc., the size of the CIB, etc. A cluster with nodes<br>
> joining/leaving frequently and resources moving around a lot will use<br>
> more shared memory than a cluster of the same size that's quiet. Cluster<br>
> options such as cluster-recheck-interval would also matter.<br>
><br>
> Practically, I think all you can do is simulate expected cluster<br>
> configurations and loads, and see what it comes out to be.<br>
><br>
> > # df -kh<br>
> > tmpfs 300.0M 27.5M 272.5M 9% /dev/shm<br>
> ><br>
> > Thanks<br>
> > Nikhil<br>
> ><br>
> > On Tue, May 17, 2016 at 12:09 PM, Ulrich Windl<br>
> > <<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a><br>
> <mailto:<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a>><br>
</div></div>> > <mailto:<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a><br>
<span class="">> <mailto:<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a>>>> wrote:<br>
> ><br>
> > Hi!<br>
> ><br>
> > One of the main problems I identified with POSIX shared memory<br>
> > (/dev/shm) in Linux is that changes to the shared memory don't<br>
> > affect the i-node, so you cannot tell from a "ls -rtl" which<br>
> > segments are still active and which are not. You can only see the<br>
> > creation time.<br>
> ><br>
> > Maybe there should be a tool that identifies and cleans up obsolete<br>
> > shared memory.<br>
> > I don't understand the part talking about the size of /dev/shm: It's<br>
> > shared memory. See "kernel.shmmax" and "kernel.shmall" in you sysctl<br>
> > settings (/etc/sysctl.conf).<br>
> ><br>
> > Regards,<br>
> > Ulrich<br>
> ><br>
> > >>> Nikhil Utane <<a href="mailto:nikhil.subscribed@gmail.com">nikhil.subscribed@gmail.com</a> <mailto:<a href="mailto:nikhil.subscribed@gmail.com">nikhil.subscribed@gmail.com</a>><br>
</span>> > <mailto:<a href="mailto:nikhil.subscribed@gmail.com">nikhil.subscribed@gmail.com</a><br>
<span class="">> <mailto:<a href="mailto:nikhil.subscribed@gmail.com">nikhil.subscribed@gmail.com</a>>>> schrieb am 16.05.2016 um 14:31 in<br>
> > Nachricht<br>
> ><br>
> <CAGNWmJVSye5PJgkdbFAi5AzO+Qq-j=<a href="mailto:2fS1c%2B0rGnqS994vV48w@mail.gmail.com">2fS1c+0rGnqS994vV48w@mail.gmail.com</a><br>
> <mailto:<a href="mailto:2fS1c%252B0rGnqS994vV48w@mail.gmail.com">2fS1c%2B0rGnqS994vV48w@mail.gmail.com</a>><br>
</span>> > <mailto:<a href="mailto:2fS1c%252B0rGnqS994vV48w@mail.gmail.com">2fS1c%2B0rGnqS994vV48w@mail.gmail.com</a><br>
> <mailto:<a href="mailto:2fS1c%25252B0rGnqS994vV48w@mail.gmail.com">2fS1c%252B0rGnqS994vV48w@mail.gmail.com</a>>>>:<br>
<span class="im HOEnZb">> > > Thanks Ken.<br>
> > ><br>
> > > Could you also respond on the second question?<br>
> > ><br>
> > >> Also, in /dev/shm I see that it created around 300+ files of<br>
> > around<br>
> > >> 250 MB.<br>
> > >><br>
> > >> For e.g.<br>
> > >> -rw-rw---- 1 hacluste hacluste 8232 May 6 13:03<br>
> > >> qb-cib_rw-response-25035-25038-10-header<br>
> > >> -rw-rw---- 1 hacluste hacluste 540672 May 6 13:03<br>
> > >> qb-cib_rw-response-25035-25038-10-data<br>
> > >> -rw------- 1 hacluste hacluste 8232 May 6 13:03<br>
> > >> qb-cib_rw-response-25035-25036-12-header<br>
> > >> -rw------- 1 hacluste hacluste 540672 May 6 13:03<br>
> > >> qb-cib_rw-response-25035-25036-12-data<br>
> > >> And many more..<br>
> > >><br>
> > >> We have limited space in /dev/shm and all these files are<br>
> > filling it<br>
> > >> up. Are these all needed? Any way to limit? Do we need to do any<br>
> > >> clean-up if pacemaker termination was not graceful? What's the<br>
> > > recommended size for this folder for Pacemaker? Our cluster will have<br>
> > > maximum 6 nodes.<br>
> > ><br>
> > > -Regards<br>
> > > Nikhil<br>
> > ><br>
> > > On Sat, May 14, 2016 at 3:11 AM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a> <mailto:<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
</span><div class="HOEnZb"><div class="h5">> > <mailto:<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a> <mailto:<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>>>> wrote:<br>
> > ><br>
> > >> On 05/08/2016 11:19 PM, Nikhil Utane wrote:<br>
> > >> > Moving these questions to a different thread.<br>
> > >> ><br>
> > >> > Hi,<br>
> > >> ><br>
> > >> > We have limited storage capacity in our system for<br>
> > different folders.<br>
> > >> > How can I configure to use a different folder for<br>
> > /var/lib/pacemaker?<br>
> > >><br>
> > >> ./configure --localstatedir=/wherever (defaults to /var or<br>
> > ${prefix}/var)<br>
> > >><br>
> > >> That will change everything that normally is placed or<br>
> looked for<br>
> > under<br>
> > >> /var (/var/lib/pacemaker, /var/lib/heartbeat, /var/run, etc.).<br>
> > >><br>
> > >> Note that while ./configure lets you change the location of<br>
> nearly<br>
> > >> everything, /usr/lib/ocf/resource.d is an exception,<br>
> because it is<br>
> > >> specified in the OCF standard.<br>
> > >><br>
> > >> ><br>
> > >> ><br>
> > >> > Also, in /dev/shm I see that it created around 300+ files<br>
> > of around<br>
> > >> > 250 MB.<br>
> > >> ><br>
> > >> > For e.g.<br>
> > >> > -rw-rw---- 1 hacluste hacluste 8232 May 6 13:03<br>
> > >> > qb-cib_rw-response-25035-25038-10-header<br>
> > >> > -rw-rw---- 1 hacluste hacluste 540672 May 6 13:03<br>
> > >> > qb-cib_rw-response-25035-25038-10-data<br>
> > >> > -rw------- 1 hacluste hacluste 8232 May 6 13:03<br>
> > >> > qb-cib_rw-response-25035-25036-12-header<br>
> > >> > -rw------- 1 hacluste hacluste 540672 May 6 13:03<br>
> > >> > qb-cib_rw-response-25035-25036-12-data<br>
> > >> > And many more..<br>
> > >> ><br>
> > >> > We have limited space in /dev/shm and all these files are<br>
> > filling it<br>
> > >> > up. Are these all needed? Any way to limit? Do we need to<br>
> > do any<br>
> > >> > clean-up if pacemaker termination was not graceful?<br>
> > >> ><br>
> > >> > -Thanks<br>
> > >> > Nikhil<br>
</div></div></blockquote></div><br></div>