[Bug]: Flaky Elixir Tests

[nickva]

Version

unknown

Describe the problem you're encountering

Noticed a lot of elixir tests are flaky. Almost every CI run has at least a few failed elixir tests. They retry but sometimes even 3 retries are not enough. It's not an urgent issue, of course, as retries cover it up but it does make the tests run a bit longer

05:36:45    1) test limit (BasicFindTest)
05:36:45       test/elixir/test/mango/02_basic_find_test.exs:177
05:36:45       ** (FunctionClauseError) no function clause matching in Access.get/3
05:36:45  
05:36:45       The following arguments were given to Access.get/3:
05:36:45  
05:36:45           # 1
05:36:45           {"error", "not_found"}
05:36:45  
05:36:45           # 2
05:36:45           "id"
05:36:45  
05:36:45           # 3
05:36:45           nil
05:36:45  
05:36:45       Attempted function clauses (showing 6 out of 6):
05:36:45  
05:36:45           def get(-%module{} = container-, key, default)
05:36:45           def get(map, key, default) when -is_map(map)-
05:36:45           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
05:36:45           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
05:36:45           def get(list, key, _default) when -is_list(list)-
05:36:45           def get(-nil-, _key, default)
05:36:45  
05:36:45       stacktrace:
05:36:45         (elixir 1.18.4) lib/access.ex:320: Access.get/3
05:36:45         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
05:36:45         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
05:36:45         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
05:36:45         (couchdbtest 0.1.0) test/elixir/test/support/user_docs.ex:355: UserDocs.setup/4
05:36:45         test/elixir/test/mango/02_basic_find_test.exs:13: BasicFindTest.__ex_unit__/2
05:36:45  

DiskMonitorTest [test/elixir/test/disk_monitor_test.exs]
07:33:38    * test block interactive view indexing [L#30]
07:33:38    * test block interactive view indexing (30.9ms) [L#30]
07:33:38  
07:33:38    1) test block interactive view indexing (DiskMonitorTest)
07:33:38       test/elixir/test/disk_monitor_test.exs:30
07:33:38       Assertion with == failed
07:33:38       code:  assert resp.status_code == 507
07:33:38       left:  200
07:33:38       right: 507
07:33:38       stacktrace:
07:33:38         test/elixir/test/disk_monitor_test.exs:41: (test)

In this case all 3 retries failed in a row on two separate worker (so 6 total elixir failures): https://ci-couchdb.apache.org/job/jenkins-cm1/job/FullPlatformMatrix/job/main/1871/pipeline-console/?selected-node=374

   2) test choose index most accurate (ChooseCorrectIndexForDocs)
04:56:17       test/elixir/test/mango/12_use_correct_index_test.exs:81
04:56:17       ** (FunctionClauseError) no function clause matching in Access.get/3
04:56:17  
04:56:17       The following arguments were given to Access.get/3:
04:56:17  
04:56:17           # 1
04:56:17           {"error", "not_found"}
04:56:17  
04:56:17           # 2
04:56:17           "id"
04:56:17  
04:56:17           # 3
04:56:17           nil
04:56:17  
04:56:17       Attempted function clauses (showing 6 out of 6):
04:56:17  
04:56:17           def get(-%module{} = container-, key, default)
04:56:17           def get(map, key, default) when -is_map(map)-
04:56:17           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:56:17           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:56:17           def get(list, key, _default) w

 ChooseCorrectIndexForDocs [test/elixir/test/mango/12_use_correct_index_test.exs]
04:58:13    * test choose index with id [L#128]
04:58:13    * test choose index with id (41.1ms) [L#128]
04:58:14    * test chooses idxA [L#105]
04:58:14    * test chooses idxA (19.5ms) [L#105]
04:58:14  
04:58:14    1) test chooses idxA (ChooseCorrectIndexForDocs)
04:58:14       test/elixir/test/mango/12_use_correct_index_test.exs:105
04:58:14       ** (FunctionClauseError) no function clause matching in Access.get/3
04:58:14  
04:58:14       The following arguments were given to Access.get/3:
04:58:14  
04:58:14           # 1
04:58:14           {"error", "not_found"}
04:58:14  
04:58:14           # 2
04:58:14           "id"
04:58:14  
04:58:14           # 3
04:58:14           nil
04:58:14  
04:58:14       Attempted function clauses (showing 6 out of 6):
04:58:14  
04:58:14           def get(-%module{} = container-, key, default)
04:58:14           def get(map, key, default) when -is_map(map)-
04:58:14           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:58:14           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:58:14           def get(list, key, _default) when -is_list(list)-
04:58:14           def get(-nil-, _key, default)
04:58:14  
04:58:14       stacktrace:
04:58:14         (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:58:14         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:58:14         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:58:14         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:58:14         test/elixir/test/mango/12_use_correct_index_test.exs:54: ChooseCorrectIndexForDocs.__ex_unit_setup_1/1
04:58:14         test/elixir/test/mango/12_use_correct_index_test.exs:13: ChooseCorrectIndexForDocs.__ex_unit__/2
04:58:14  

05:01:11    1) test warn on full db scan (ChooseCorrectIndexForDocs)
05:01:11       test/elixir/test/mango/12_use_correct_index_test.exs:97
05:01:11       ** (FunctionClauseError) no function clause matching in Access.get/3
05:01:11  
05:01:11       The following arguments were given to Access.get/3:
05:01:11  
05:01:11           # 1
05:01:11           {"error", "not_found"}
05:01:11  
05:01:11           # 2
05:01:11           "id"
05:01:11  
05:01:11           # 3
05:01:11           nil
05:01:11  
05:01:11       Attempted function clauses (showing 6 out of 6):
05:01:11  
05:01:11           def get(-%module{} = container-, key, default)
05:01:11           def get(map, key, default) when -is_map(map)-
05:01:11           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
05:01:11           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
05:01:11           def get(list, key, _default) when -is_list(list)-
05:01:11           def get(-nil-, _key, default)
05:01:11  
05:01:11       stacktrace:
05:01:11         (elixir 1.18.4) lib/access.ex:320: Access.get/3
05:01:11         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
05:01:11         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
05:01:11         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
05:01:11         test/elixir/test/mango/12_use_correct_index_test.exs:54: ChooseCorrectIndexForDocs.__ex_unit_setup_1/1
05:01:11         test/elixir/test/mango/12_use_correct_index_test.exs:13: ChooseCorrectIndexForDocs.__ex_unit__/2

04:51:48    1) test json range (BeginsWithOperator)
04:51:48       test/elixir/test/mango/25_beginswith_test.exs:60
04:51:48       ** (FunctionClauseError) no function clause matching in Access.get/3
04:51:48  
04:51:48       The following arguments were given to Access.get/3:
04:51:48  
04:51:48           # 1
04:51:48           {"error", "not_found"}
04:51:48  
04:51:48           # 2
04:51:48           "id"
04:51:48  
04:51:48           # 3
04:51:48           nil
04:51:48  
04:51:48       Attempted function clauses (showing 6 out of 6):
04:51:48  
04:51:48           def get(-%module{} = container-, key, default)
04:51:48           def get(map, key, default) when -is_map(map)-
04:51:48           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:51:48           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:51:48           def get(list, key, _default) when -is_list(list)-
04:51:48           def get(-nil-, _key, default)
04:51:48  
04:51:48       stacktrace:
04:51:48         (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:51:48         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:51:48         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:51:48         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:51:48         test/elixir/test/mango/25_beginswith_test.exs:28: BeginsWithOperator.__ex_unit_setup_1/1
04:51:48         test/elixir/test/mango/25_beginswith_test.exs:13: BeginsWithOperator.__ex_unit__/2

) test use index without fallback fails for empty index (JSONIndexSelectionTest)
04:53:33       test/elixir/test/mango/05_index_selection_test.exs:284
04:53:33       ** (FunctionClauseError) no function clause matching in Access.get/3
04:53:33  
04:53:33       The following arguments were given to Access.get/3:
04:53:33  
04:53:33           # 1
04:53:33           {"error", "not_found"}
04:53:33  
04:53:33           # 2
04:53:33           "id"
04:53:33  
04:53:33           # 3
04:53:33           nil
04:53:33  
04:53:33       Attempted function clauses (showing 6 out of 6):
04:53:33  
04:53:33           def get(-%module{} = container-, key, default)
04:53:33           def get(map, key, default) when -is_map(map)-
04:53:33           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:53:33           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:53:33           def get(list, key, _default) when -is_list(list)-
04:53:33           def get(-nil-, _key, default)
04:53:33  
04:53:33       stacktrace:
04:53:33         (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:53:33         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:53:33         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:53:33         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:53:33         (couchdbtest 0.1.0) test/elixir/test/support/user_docs.ex:355: UserDocs.setup/4
04:53:33         test/elixir/test/mango/05_index_selection_test.exs:274: JSONIndexSelectionTest.__ex_unit__/2

04:57:04    1) test json range (BeginsWithOperator)
04:57:04       test/elixir/test/mango/25_beginswith_test.exs:60
04:57:04       ** (FunctionClauseError) no function clause matching in Access.get/3
04:57:04  
04:57:04       The following arguments were given to Access.get/3:
04:57:04  
04:57:04           # 1
04:57:04           {"error", "not_found"}
04:57:04  
04:57:04           # 2
04:57:04           "id"
04:57:04  
04:57:04           # 3
04:57:04           nil
04:57:04  
04:57:04       Attempted function clauses (showing 6 out of 6):
04:57:04  
04:57:04           def get(-%module{} = container-, key, default)
04:57:04           def get(map, key, default) when -is_map(map)-
04:57:04           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
04:57:04           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
04:57:04           def get(list, key, _default) when -is_list(list)-
04:57:04           def get(-nil-, _key, default)
04:57:04  
04:57:04       stacktrace:
04:57:04         (elixir 1.18.4) lib/access.ex:320: Access.get/3
04:57:04         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
04:57:04         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
04:57:04         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
04:57:04         test/elixir/test/mango/25_beginswith_test.exs:28: BeginsWithOperator.__ex_unit_setup_1/1
04:57:04         test/elixir/test/mango/25_beginswith_test.exs:13: BeginsWithOperator.__ex_unit__/2
04:57:04  
04:57:04  

I think a common theme is around {error, not_found}. Maybe setup needs a tweak or need an extra clause just for not found with an internal retry?

Expected Behaviour

Flaky failures would be a bit more rare.

Steps to Reproduce

No response

Your Environment

No response

Additional Context

No response

[nickva]

Another instance of 3 failures in a row: https://ci-couchdb.apache.org/job/jenkins-cm1/job/PullRequests/job/PR-5904/2/pipeline-console/?selected-node=391

11:37:34           "id"
11:37:34  
11:37:34           # 3
11:37:34           nil
11:37:34  
11:37:34       Attempted function clauses (showing 6 out of 6):
11:37:34  
11:37:34           def get(-%module{} = container-, key, default)
11:37:34           def get(map, key, default) when -is_map(map)-
11:37:34           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
11:37:34           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
11:37:34           def get(list, key, _default) when -is_list(list)-
11:37:34           def get(-nil-, _key, default)
11:37:34  
11:37:34       stacktrace:
11:37:34         (elixir 1.18.4) lib/access.ex:320: Access.get/3
11:37:34         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
11:37:34         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
11:37:34         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
11:37:34         (couchdbtest 0.1.0) test/elixir/test/support/user_docs.ex:355: UserDocs.setup/4
11:37:34         test/elixir/test/mango/02_basic_find_test.exs:13: BasicFindTest.__ex_unit__/2
11:37:34  
11:37:34    * test bad skip [L#48]
11:37:34    * test bad skip (148.4ms) [L#48]

11:38:25           "id"
11:38:25  
11:38:25           # 3
11:38:25           nil
11:38:25  
11:38:25       Attempted function clauses (showing 6 out of 6):
11:38:25  
11:38:25           def get(-%module{} = container-, key, default)
11:38:25           def get(map, key, default) when -is_map(map)-
11:38:25           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
11:38:25           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
11:38:25           def get(list, key, _default) when -is_list(list)-
11:38:25           def get(-nil-, _key, default)
11:38:25  
11:38:25       stacktrace:
11:38:25         (elixir 1.18.4) lib/access.ex:320: Access.get/3
11:38:25         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
11:38:25         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
11:38:25         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
11:38:25         test/elixir/test/mango/25_beginswith_test.exs:28: BeginsWithOperator.__ex_unit_setup_1/1
11:38:25         test/elixir/test/mango/25_beginswith_test.exs:13: BeginsWithOperator.__ex_unit__/2
11:38:25  
11:38:25    * test compound key [L#66]
11:38:25    * test compound key (58.7ms) [L#66]
11:38:25  

11:42:31  BeginsWithOperator [test/elixir/test/mango/25_beginswith_test.exs]
11:42:31    * test invalid operand [L#120]
11:42:31    * test invalid operand (46.1ms) [L#120]
11:42:31    * test no matches [L#131]
11:42:31    * test no matches (15.3ms) [L#131]
11:42:31  
11:42:31    2) test no matches (BeginsWithOperator)
11:42:31       test/elixir/test/mango/25_beginswith_test.exs:131

[nickva]

The next one after there were 6 failures in a row in two os flavors: https://ci-couchdb.apache.org/job/jenkins-cm1/job/PullRequests/job/PR-5904/3/pipeline-console/?selected-node=391 all in elixir tests:

12:50:35           # 2
12:50:35           "id"
12:50:35  
12:50:35           # 3
12:50:35           nil
12:50:35  
12:50:35       Attempted function clauses (showing 6 out of 6):
12:50:35  
12:50:35           def get(-%module{} = container-, key, default)
12:50:35           def get(map, key, default) when -is_map(map)-
12:50:35           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
12:50:35           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
12:50:35           def get(list, key, _default) when -is_list(list)-
12:50:35           def get(-nil-, _key, default)
12:50:35  
12:50:35       stacktrace:
12:50:35         (elixir 1.18.4) lib/access.ex:320: Access.get/3
12:50:35         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
12:50:35         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
12:50:35         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
12:50:35         test/elixir/test/mango/12_use_correct_index_test.exs:54: ChooseCorrectIndexForDocs.__ex_unit_setup_1/1
12:50:35         test/elixir/test/mango/12_use_correct_index_test.exs:13: ChooseCorrectIndexForDocs.__ex_unit__/2

12:54:15    * test no matches [L#131]
12:54:15    * test no matches (56.6ms) [L#131]
12:54:15    * test no index [L#110]
12:54:15    * test no index (56.4ms) [L#110]
12:54:15  
12:54:15  ChooseCorrectIndexForDocs [test/elixir/test/mango/12_use_correct_index_test.exs]
12:54:15    * test choose index with rev [L#134]
12:54:15    * test choose index with rev (40.9ms) [L#134]
12:54:15    * test can query with range on secondary column [L#115]
12:54:15    * test can query with range on secondary column (14.2ms) [L#115]
12:54:15  
12:54:15    2) test can query with range on secondary column (ChooseCorrectIndexForDocs)
12:54:15       test/elixir/test/mango/12_use_correct_index_test.exs:115
12:54:15       ** (FunctionClauseError) no function clause matching in Access.get/3
12:54:15  
12:54:15       The following arguments were given to Access.get/3:
12:54:15  
12:54:15           # 1
12:54:15           {"error", "not_found"}
12:54:15  
12:54:15           # 2
12:54:15           "id"
12:54:15  
12:54:15           # 3
12:54:15           nil
12:54:15  
12:54:15       Attempted function clauses (showing 6 out of 6):
12:54:15  
12:54:15           def get(-%module{} = container-, key, default)
12:54:15           def get(map, key, default) when -is_map(map)-
12:54:15           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
12:54:15           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
12:54:15           def get(list, key, _default) when -is_list(list)-
12:54:15           def get(-nil-, _key, default)
12:54:15  
12:54:15       stacktrace:
12:54:15         (elixir 1.18.4) lib/access.ex:320: Access.get/3
12:54:15         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
12:54:15         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
12:54:15         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
12:54:15         test/elixir/test/mango/12_use_correct_index_test.exs:54: ChooseCorrectIndexForDocs.__ex_unit_setup_1/1
12:54:15         test/elixir/test/mango/12_use_correct_index_test.exs:13: ChooseCorrectIndexForDocs.__ex_unit__/2

2:57:47    3) test no matches (BeginsWithOperator)
12:57:47       test/elixir/test/mango/25_beginswith_test.exs:131
12:57:47       ** (FunctionClauseError) no function clause matching in Access.get/3
12:57:47  
12:57:47       The following arguments were given to Access.get/3:
12:57:47  
12:57:47           # 1
12:57:47           {"error", "not_found"}
12:57:47  
12:57:47           # 2
12:57:47           "id"
12:57:47  
12:57:47           # 3
12:57:47           nil
12:57:47  
12:57:47       Attempted function clauses (showing 6 out of 6):
12:57:47  
12:57:47           def get(-%module{} = container-, key, default)
12:57:47           def get(map, key, default) when -is_map(map)-
12:57:47           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
12:57:47           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
12:57:47           def get(list, key, _default) when -is_list(list)-
12:57:47           def get(-nil-, _key, default)
12:57:47  
12:57:47       stacktrace:
12:57:47         (elixir 1.18.4) lib/access.ex:320: Access.get/3
12:57:47         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:99: anonymous fn/1 in MangoDatabase.save_docs/3
12:57:47         (elixir 1.18.4) lib/enum.ex:1714: Enum."-map/2-lists^map/1-1-"/2
12:57:47         (couchdbtest 0.1.0) test/elixir/test/support/mango_database.ex:97: MangoDatabase.save_docs/3
12:57:47         test/elixir/test/mango/25_beginswith_test.exs:28: BeginsWithOperator.__ex_unit_setup_1/1
12:57:47         test/elixir/test/mango/25_beginswith_test.exs:13: BeginsWithOperator.__ex_unit__/2
12:57:47  
12:57:47    * test basic [L#54]
12:57:47    * test basic (69.1ms) [L#54]

12:53:12    1) test warn on full db scan (ChooseCorrectIndexForDocs)
12:53:12       test/elixir/test/mango/12_use_correct_index_test.exs:97
12:53:12       ** (FunctionClauseError) no function clause matching in Access.get/3
12:53:12  
12:53:12       The following arguments were given to Access.get/3:
12:53:12  
12:53:12           # 1
12:53:12           {"error", "not_found"}
12:53:12  
12:53:12           # 2
12:53:12           "id"
12:53:12  
12:53:12           # 3
12:53:12           nil
12:53:12  
12:53:12       Attempted function clauses (showing 6 out of 6):
12:53:12  
12:53:12           def get(-%module{} = container-, key, default)
12:53:12           def get(map, key, default) when -is_map(map)-
12:53:12           def get(list, key, default) when -is_list(list)- and -is_atom(key)-
12:53:12           def get(list, key, _default) when -is_list(list)- and -is_integer(key)-
12:53:12           def get(list, key, _default) when -is_list(list)-
12:53:12           def get(-nil-, _key, default)
12:53:12  
12:53:12       stacktrace:

and a few more

[janl]

This PR should fix all new mango elixir test flakeyness plus another one that reloads config from file that I found:

#5905

Here is my flakiness finder script if you want to play along at home:

#!/bin/sh

MAKEOPTS=$1

LOGDIR=flakey-test-finder
mkdir -p $LOGDIR

while true; do
  TIMESTAMP=`date "+%Y%m%d%H%M%S"`
  
  RUNDIR="$LOGDIR/$TIMESTAMP"
  
  mkdir -p $RUNDIR-in-progress
  
  /bin/echo -n "running $TIMESTAMP "

  /usr/bin/time -o $RUNDIR-in-progress/time.txt make elixir > $RUNDIR-in-progress/make-elixir.log 2>&1
  if [ $? -eq 0 ]; then
    echo "passed after `cat $RUNDIR-in-progress/time.txt`"
    # if the run was fine, delete RUNDIR
    rm -rf $RUNDIR-in-progress
  else
    mv $RUNDIR-in-progress $RUNDIR
    /bin/echo "failed after `cat $RUNDIR/time.txt`"
    # otherwise copy couch logs into RUNDIR
    # so they can be parsed later
    cp -r dev/logs/node1.log $RUNDIR
  fi
done

The disk quota tests fails locally for me because I do have a full disk, I can’t debug it at the moment :)

[janl]

Running my script I see three failures come in regularly and no others:

cc @rnewson

  * test Auto-reload config (1535.3ms) [L#185]

  1) test Auto-reload config (ConfigTest)
     test/elixir/test/config_test.exs:185
     Assertion with == failed
     code:  assert resp.status_code == status_assert
     left:  404
     right: 200
     stacktrace:
       test/elixir/test/config_test.exs:54: ConfigTest.get_config/4
       test/elixir/test/config_test.exs:198: (test)

ChangesAsyncTest [test/elixir/test/changes_async_test.exs]
  * test continuous filtered changes with doc ids [L#309]
  * test continuous filtered changes with doc ids (506.9ms) [L#309]

  1) test continuous filtered changes with doc ids (ChangesAsyncTest)
     test/elixir/test/changes_async_test.exs:309
     Expected truthy, got false
     code: assert Enum.member?(changes_ids, "doc3")
     arguments:

         # 1
         ["doc1"]

         # 2
         "doc3"

     stacktrace:
       test/elixir/test/changes_async_test.exs:338: (test)

ChangesAsyncTest [test/elixir/test/changes_async_test.exs]
  * test eventsource changes with limit exact [L#133]
  * test eventsource changes with limit exact (3011.1ms) [L#133]

  1) test eventsource changes with limit exact (ChangesAsyncTest)
     test/elixir/test/changes_async_test.exs:133
     ** (ArgumentError) errors were found at the given arguments:

       * 1st argument: not a list

     code: assert length(changes) == 2
     stacktrace:
       :erlang.length(:timeout)
       test/elixir/test/changes_async_test.exs:152: (test)