Forum Discussion

Shirish_BasantR's avatar
Shirish_BasantR
Icon for Nimbostratus rankNimbostratus
Mar 10, 2020

External Monitor brings down pool instantly

Hi,

i am trying to monitor the health of elasticsearch cluster by tracking the number of nodes active in the cluster which will raise syslog with the status.

however whenever i attach the external monitor, it is bringing down all members of the pool. Could you check what is wrong with it.

###################################
# Author: Shirish Basant Rai
# Version 1.0
# Date: 06/03/2020
# Usage:
# Change log:
###################################
 
#!/bin/sh
pidfile="/var/run/$MONITOR_NAME.$1..$2.pid"
# Send signal to the process group to kill our former self and any children
#$1 and $2 are passed by default, as well as the monitor Name
if [ -f $pidfile ]
then
    kill -9 -`cat $pidfile` > /dev/null 2>&1
fi
echo "$$" > $pidfile
 
 
 
function get_EScluster_num_from_hostname() {
	#
	#
	number=$(echo $1 | awk -F"-" '{print $4}' | sed 's/[^0-9]*//g')
	echo $number
	
}
function get_vsdcluster_num_from_hostname() {
    #********************************************************************************************
    # this function will return the VSD cluster from the VM hostname
    # 1 Parameters required for this function: LINE retrieved from SYSLOG
    #********************************************************************************************
 
	number=$(echo $1 | awk -F"-" '{print $3}' | sed 's/[^1-9]*//g')
	echo $number
	
}
 
#Declare variables
declare -a response
declare -a Statelist
declare -a Nodelist
maxup=6
es1_members=3
es2_members=3
 
#POOL=Pool-STATSES-9200
 
 
#Get number of active members of the Pool
upmembers=`tmsh show /ltm pool $POOL members |grep "Current Active Members" |awk -F" : " '{ print $2 }'`
#Get the details of the pool members
response="$(curl -sk -u admin:admin GET https://localhost/mgmt/tm/ltm/pool/Pool-STATSES-9200/?expandSubcollections=true | jq . -M)"
#Get the Name of the pool members
NodeName=$(echo ${response} | jq -M '.membersReference.items[].name' | sed -e 's/^"//' -e 's/"$//' -e 's/:9200//')
#Get the state of the pool members
Nodestate=$(echo ${response} | jq -M '.membersReference.items[].state' | sed -e 's/^"//' -e 's/"$//')
#Get priority group of Pool members to define ES cluster
NodePG=$(echo ${response} | jq -M '.membersReference.items[].priorityGroup')
 
#split string to list
Nodelist=( $NodeName )
Statelist=( $Nodestate )
PGlist=( $NodePG )
 
echo "up"
if [ $upmembers -lt $maxup ]
then
	rm -f $pidfile
	n=0
	echo "up"
#then find which node is down from which cluster; need to identify the cluster from the es number
	for i in "${!Statelist[@]}"; do
			#find the vsd cluster and escluster and log which host is down for which cluster
			esclusterID=$(get_EScluster_num_from_hostname "${Nodelist[$i]}")
			vsdclusterID=$(get_vsdcluster_num_from_hostname "${Nodelist[$i]}")
			if [ "${Statelist[$i]}" == "down" ]
			then
	#for Active ES cluster
				if [ "${PGlist[$i]}" == "100" ]
				then
				   # send the syslog Alarm
				   n+=1
						if [ $n -eq 1 ]
						then
							logger -p local0.error -t GENERIC-SCRIPT "ES-Cluster-Monit SCRIPT-RESULT 0 0 Active ES cluster from VSD cluster: \"${vsdclusterID}\" is in YELLOW state"
						elif [ $n -gt 1 ]
						then
							logger -p local0.error -t GENERIC-SCRIPT "ES-Cluster-Monit SCRIPT-RESULT 0 0 Active ES cluster from VSD cluster: \"${vsdclusterID}\" is in RED state"
						else [ $n == 0 ]
							logger -p local0.info -t GENERIC-SCRIPT "ES-Cluster-Monit SCRIPT-RESULT 0 0 Active ES cluster from VSD cluster: \"${vsdclusterID}\" is in GREEN state"							
						fi	
	#for Backup ES cluster
				else
				   n+=1
						if [ $n -eq 1 ]
						then
							logger -p local0.error -t GENERIC-SCRIPT "ES-Cluster-Monit SCRIPT-RESULT 0 0 Backup ES cluster from VSD cluster: \"${vsdclusterID}\" is in YELLOW state"
						elif [ $n -gt 1 ]
						then
							logger -p local0.error -t GENERIC-SCRIPT "ES-Cluster-Monit SCRIPT-RESULT 0 0 Backup ES cluster from VSD cluster: \"${vsdclusterID}\" is in RED state"
						else
							logger -p local0.info -t GENERIC-SCRIPT "ES-Cluster-Monit SCRIPT-RESULT 0 0 Backup ES cluster from VSD cluster: \"${vsdclusterID}\" is in GREEN state"							
						fi					
				fi
				
		   fi
 
	done
	echo "up"
fi

1 Reply

  • Hello,

    You should move line #9 to top #1 to script starts working.

    Anyway, I can't confirm your script will work properly since you comment a variable at line #49 and you have returns (echo "up") without conditional points.

     

    Regards.