CLOUDSTACK-10296: Find time different from last timestamp by yadvr · Pull Request #2458 · apache/cloudstack

yadvr · 2018-02-16T10:28:52Z

This fixes a difference issue in rVR heartbeat check script raised
recently on dev@.

@blueorangutan package

blueorangutan · 2018-02-16T10:29:12Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-02-16T11:20:12Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1725

DaanHoogland

in line with the report; positive time 👍

wido · 2018-02-21T15:57:15Z

Code LGTM

yadvr · 2018-02-25T15:55:44Z

@blueorangutan test

blueorangutan · 2018-02-25T15:55:53Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

yadvr · 2018-02-26T20:05:52Z

@blueorangutan package

blueorangutan · 2018-02-26T20:06:58Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-02-26T20:42:44Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1735

yadvr · 2018-02-26T21:03:33Z

@blueorangutan test

blueorangutan · 2018-02-26T21:03:58Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

ustcweizhou · 2018-02-27T06:21:35Z

We met the issue several times on our production.
the final change is

diff --git a/systemvm/patches/debian/config/opt/cloud/templates/check_heartbeat.sh.templ b/systemvm/patches/debian/config/opt/cloud/templates/check_heartbeat.sh.templ
index 6a0d8a9..a8ddc7d 100755
--- a/systemvm/patches/debian/config/opt/cloud/templates/check_heartbeat.sh.templ
+++ b/systemvm/patches/debian/config/opt/cloud/templates/check_heartbeat.sh.templ
@@ -19,14 +19,29 @@
 ROUTER_BIN_PATH=/ramdisk/rrouter
 ROUTER_LOG=${ROUTER_BIN_PATH}/keepalived.log
 STRIKE_FILE="$ROUTER_BIN_PATH/keepalived.strikes"
+CHECKTIME_FILE="$ROUTER_BIN_PATH/keepalived.checktime"

 if [ -e $ROUTER_BIN_PATH/keepalived.ts2 ]
 then
     thistime=$(cat $ROUTER_BIN_PATH/keepalived.ts)
     lasttime=$(cat $ROUTER_BIN_PATH/keepalived.ts2)
-    diff=$(($lasttime - $thistime))
+    diff=$(($thistime - $lasttime))
+    lastcheck=0
+    if [ -e $CHECKTIME_FILE ]
+    then
+        lastcheck=`cat $CHECKTIME_FILE 2>/dev/null`
+    fi
+    thischeck=$(date +%s)
+    checkdiff=$(($thischeck - $lastcheck))
+    echo "thischeck: $thischeck, lastcheck: $lastcheck. this heartbeat: $thistime, last heartbeat: $lasttime, time diff: $diff" >> $ROUTER_LOG
+    if [ $checkdiff -ge 0 ] && [ $checkdiff -lt 30 ]
+    then
+        echo time duration after last check is less than 30 seconds, ignored >> $ROUTER_LOG
+        exit
+    fi
+    echo $thischeck > $CHECKTIME_FILE
     s=0
-    if [ $diff -ge 10 ]
+    if [ $diff -lt 30 ]
     then
         if [ -e $STRIKE_FILE ]
         then

yadvr · 2018-02-27T08:42:35Z

Trillian env has been failing due to this change. Thanks for sharing @ustcweizhou, let me apply and test your fix.
@blueorangutan package

This fixes a difference issue in rVR heartbeat check script raised recently on dev@. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

…not running Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr · 2018-02-28T14:37:04Z

@blueorangutan package

blueorangutan · 2018-02-28T14:38:06Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-02-28T15:55:29Z

Packaging result: ✔centos6 ✔centos7 ✔debian. JID-1745

yadvr · 2018-02-28T15:58:30Z

@blueorangutan test

blueorangutan · 2018-02-28T15:59:06Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2018-03-01T02:17:50Z

Trillian test result (tid-2299)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 35592 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2458-t2299-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_privategw_acl.py
Intermitten failure detected: /marvin/tests/smoke/test_public_ip_range.py
Intermitten failure detected: /marvin/tests/smoke/test_templates.py
Intermitten failure detected: /marvin/tests/smoke/test_usage.py
Intermitten failure detected: /marvin/tests/smoke/test_volumes.py
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Smoke tests completed. 62 look OK, 5 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_03_vpc_privategw_restart_vpc_cleanup	`Failure`	151.60	test_privategw_acl.py
test_04_extract_template	`Failure`	128.27	test_templates.py
ContextSuite context=TestISOUsage>:setup	`Error`	0.00	test_usage.py
test_06_download_detached_volume	`Failure`	139.67	test_volumes.py
test_04_rvpc_network_garbage_collector_nics	`Failure`	514.94	test_vpc_redundant.py

yadvr · 2018-03-13T09:21:54Z

@blueorangutan package

blueorangutan · 2018-03-13T09:22:22Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-03-13T10:02:17Z

Packaging result: ✔centos6 ✔centos7 ✖debian. JID-1769

yadvr · 2018-03-13T10:43:11Z

@blueorangutan test

blueorangutan · 2018-03-13T10:43:22Z

@rhtyd a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2018-03-13T18:10:48Z

Trillian test result (tid-2352)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 24890 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr2458-t2352-kvm-centos7.zip
Intermitten failure detected: /marvin/tests/smoke/test_vpc_redundant.py
Intermitten failure detected: /marvin/tests/smoke/test_hostha_kvm.py
Smoke tests completed. 65 look OK, 2 have error(s)
Only failed tests results shown below:

Test	Result	Time (s)	Test File
test_04_rvpc_network_garbage_collector_nics	`Failure`	516.31	test_vpc_redundant.py
test_hostha_enable_ha_when_host_in_maintenance	`Error`	1.78	test_hostha_kvm.py

yadvr · 2018-03-15T11:01:54Z

Merging this based on test results and review LGTMs.

blueorangutan · 2018-03-15T11:02:30Z

@rhtyd a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2018-03-15T11:47:25Z

Packaging result: ✔centos6 ✔centos7 ✖debian. JID-1778

DaanHoogland · 2018-03-15T12:53:17Z

wow, blueorangutan came back with a stale package result after merge! need more oil in the machine, something got stuck

yadvr · 2018-03-15T17:24:35Z

No @DaanHoogland it's the trigger that is sometimes caused by github that kicks these pkging jobs post merge.

yadvr added the component:virtual-router label Feb 16, 2018

yadvr added this to the 4.11.1 milestone Feb 16, 2018

DaanHoogland approved these changes Feb 16, 2018

View reviewed changes

rafaelweingartner approved these changes Feb 22, 2018

View reviewed changes

apache deleted a comment from blueorangutan Feb 25, 2018

apache deleted a comment from borisstoyanov Feb 25, 2018

apache deleted a comment from blueorangutan Feb 25, 2018

yadvr added 4 commits February 28, 2018 13:30

CLOUDSTACK-10296: Find time different from last timestamp

11be353

This fixes a difference issue in rVR heartbeat check script raised recently on dev@. Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Apply Wei's patch to test failures

a0caf64

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Reduce logging to avoid logging to fill ramdisk

bd19083

Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

Simplify fix, make checkrouter return fault state when keepalived is …

4ae83cc

…not running Signed-off-by: Rohit Yadav <rohit.yadav@shapeblue.com>

yadvr force-pushed the rvr-heartbeat-fix branch from d8c7e55 to 4ae83cc Compare February 28, 2018 14:36

apache deleted a comment from blueorangutan Feb 28, 2018

yadvr merged commit ab0bce2 into apache:4.11 Mar 15, 2018

Uh oh!

Conversation

yadvr commented Feb 16, 2018

Uh oh!

blueorangutan commented Feb 16, 2018

Uh oh!

blueorangutan commented Feb 16, 2018

Uh oh!

DaanHoogland left a comment

Choose a reason for hiding this comment

Uh oh!

wido commented Feb 21, 2018

Uh oh!

yadvr commented Feb 25, 2018

Uh oh!

blueorangutan commented Feb 25, 2018

Uh oh!

yadvr commented Feb 26, 2018

Uh oh!

blueorangutan commented Feb 26, 2018

Uh oh!

blueorangutan commented Feb 26, 2018

Uh oh!

yadvr commented Feb 26, 2018

Uh oh!

blueorangutan commented Feb 26, 2018

Uh oh!

ustcweizhou commented Feb 27, 2018

Uh oh!

yadvr commented Feb 27, 2018

Uh oh!

yadvr commented Feb 28, 2018

Uh oh!

blueorangutan commented Feb 28, 2018

Uh oh!

blueorangutan commented Feb 28, 2018

Uh oh!

yadvr commented Feb 28, 2018

Uh oh!

blueorangutan commented Feb 28, 2018

Uh oh!

blueorangutan commented Mar 1, 2018

Uh oh!

yadvr commented Mar 13, 2018

Uh oh!

blueorangutan commented Mar 13, 2018

Uh oh!

blueorangutan commented Mar 13, 2018

Uh oh!

yadvr commented Mar 13, 2018

Uh oh!

blueorangutan commented Mar 13, 2018

Uh oh!

blueorangutan commented Mar 13, 2018

Uh oh!

yadvr commented Mar 15, 2018

Uh oh!

blueorangutan commented Mar 15, 2018

Uh oh!

blueorangutan commented Mar 15, 2018

Uh oh!

DaanHoogland commented Mar 15, 2018

Uh oh!

yadvr commented Mar 15, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants