Possible servers gathering bug

[manishtomar]

test_scale_over_group_max_after_metadata_removal_reduced_grp_max test in https://as-jenkins.rax.io/job/_ci-integration-convergence-trial/576/console failed because otter didn't report 12 active servers. However, otter execute-convergence log shows 12 active servers with 2 duplicates (test-server7606528405-id-7606528405 and test-server5827597650-id-5827597650) in effect 10 servers. Converger hence counted it as 12 active servers and finished converging. The group state however returned did not have the duplicates possibly because they were removed due to server_id being key in cache table. The duplicate is a bug and should be fixed. Following is the log:

{
  "servers": [
    "<NovaServer(id=u'test-server7606528405-id-7606528405', state=<ServerState=ACTIVE>, created=1450382128.395439, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server7606528405-id-7606528405', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server7606528405-id-7606528405', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.128.188', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.395439Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-as3b6afe99', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server5827597650-id-5827597650', state=<ServerState=ACTIVE>, created=1450382128.401277, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server5827597650-id-5827597650', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server5827597650-id-5827597650', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.226.54', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.401277Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-as259b85df', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server7606528405-id-7606528405', state=<ServerState=ACTIVE>, created=1450382128.395439, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server7606528405-id-7606528405', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server7606528405-id-7606528405', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.128.188', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.395439Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-as3b6afe99', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server5827597650-id-5827597650', state=<ServerState=ACTIVE>, created=1450382128.401277, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server5827597650-id-5827597650', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server5827597650-id-5827597650', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.226.54', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.401277Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-as259b85df', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server9753711825-id-9753711825', state=<ServerState=ACTIVE>, created=1450382128.544047, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server9753711825-id-9753711825', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server9753711825-id-9753711825', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.95.222', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.544047Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-as0d726f63', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server6134166780-id-6134166780', state=<ServerState=ACTIVE>, created=1450382128.550397, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server6134166780-id-6134166780', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server6134166780-id-6134166780', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.18.178', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.550397Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-asb9c1f258', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server7991011655-id-7991011655', state=<ServerState=ACTIVE>, created=1450382128.555263, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server7991011655-id-7991011655', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server7991011655-id-7991011655', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.247.108', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.555263Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-as8089b782', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server2211689359-id-2211689359', state=<ServerState=ACTIVE>, created=1450382128.560173, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server2211689359-id-2211689359', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server2211689359-id-2211689359', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.215.29', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.560173Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-asd3375a61', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server5355223157-id-5355223157', state=<ServerState=ACTIVE>, created=1450382128.565015, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server5355223157-id-5355223157', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server5355223157-id-5355223157', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.194.154', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.565015Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-asa346d4c4', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server6959249405-id-6959249405', state=<ServerState=ACTIVE>, created=1450382128.569802, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server6959249405-id-6959249405', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server6959249405-id-6959249405', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.219.169', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.569802Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-as84ef8435', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server3082942401-id-3082942401', state=<ServerState=ACTIVE>, created=1450382128.574885, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server3082942401-id-3082942401', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server3082942401-id-3082942401', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.238.200', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.574885Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-as9b6aeb62', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>", 
    "<NovaServer(id=u'test-server2198133209-id-2198133209', state=<ServerState=ACTIVE>, created=1450382128.583794, image_id=u'cca73d10-8953-4949-a2f2-1e5444a4130d', flavor_id=u'2', links=[{u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/v2%2F000001%2Fservers%2Ftest-server2198133209-id-2198133209', u'rel': u'self'}, {u'href': u'http://localhost:8900/mimicking/NovaApi-4f38c6/ORD/000001%2Fservers%2Ftest-server2198133209-id-2198133209', u'rel': u'bookmark'}], desired_lbs=set([]), servicenet_address=u'10.180.155.170', json={u'status': u'ACTIVE', u'updated': u'2015-12-17T19:55:28.583794Z', u'OS-EXT-STS:task_state': None, u'name': u'9690667d-1450382101.94-as21a5cd76', u'metadata': {u'rax:autoscale:group:id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0', u'rax:auto_scaling_group_id': u'50061657-9ac2-4fdd-abcf-1e3a8952e4c0'}})>"
  ], 
  "otter_facility": "otter", 
  "converger_run_id": "074c9e73-1aba-445f-8b37-78b61d4488ff", 
  "@version": 1, 
  "@timestamp": "2015-12-17T19:55:38.545896", 
...

According to converger run logs, list servers API has been called twice with second call without the marker with second call result containing the duplicated server possibly causing the duplicates. Even with duplicates, get_scaling_group_servers function should've filtered it out. Need to investigate that.