0425update

Author: ChorlingLau

README.md

@@ -2,14 +2,21 @@
 ![framework](pics/Framework.png)
 Uncertainty of Thought (UoT) is a novel algorithm to augment large language models with the ability to actively seek information by asking effective questions.
 
-We tested on two categories of models: open-source (Llama-2-70B) and closed-source commercial (other models). The results showed that UoT achieves an average performance improvement of 57.8% in the rate of successful task completion across multiple LLMs compared with direct prompting, and also improves efficiency (i.e., the number of questions needed to complete the task).
+We tested on two categories of models: open-source (Llama-2-70B) and closed-source commercial (other models). The results showed that UoT achieves an average performance improvement of **57.8%** in the rate of successful task completion across multiple LLMs compared with direct prompting, and also improves efficiency (i.e., the number of questions needed to complete the task).
 
 ![result](pics/result.jpg)
 
+To increase the number of items, we focus on the open-set scenario where the possibility space is unknown rather than predefined. Within this context, the number of items can increase by considering it as an infinite space, due to the lack of constraints. 
 
+![result os](pics/results_os.png)
+
+- In medical diagnosis and troubleshooting, initial patient or customer symptom descriptions help form an initial set of possibilities. However, in the game of 20 Questions, with limited early information, setting possibilities too soon may lead to wrong directions. Thus, for the first three rounds, DPOS method is used to collect more data. Afterward, the UoT approach **updates the possibilities each round** to improve the questioning strategy.
+- For each dataset, we configure the size of the possibility set for each update round, setting them at 10, 10, 10, 5, 5 and 5, respectively.
+- Compared to DPOS, the Uo'T method significantly improves performance, with enhancements of **54.9%** for GPT-3.5 and **21.1%** for GPT-4.
 
 ## Update
 
+- \[20/04/2024\]: Supplement the code and experiment results in the open-set scenarios.
 - \[19/03/2024\]: Supplement the experiment results of `Mistral-Large`, `Gemini-1.0-pro`, and `Claude-3-Opus` models.
 - \[19/03/2024\]: Add the implementation for Gemini
 - \[15/03/2024\]: Add the implementation for Gemma
@@ -68,28 +75,45 @@ For further details, follow the [official guidance](https://ai.google.dev/gemma/
 Run experiments via `run.py`, which implements the UoT algorithm, as well as the naive prompting method. Arguments are as follows:
 
 - `--guesser_model`: The name of model used to plan and ask questions
+
 - `--temperature`: Parameter for calling guesser model.
+
 - `--examiner_model`: The name of model used to provide environment feedback. Fixed to be `gpt-4` currently.
+
 - `--task` and `--dataset`: Select the corresponding task name and dataset according to the table below.
-  
+
     | Description       | task  | dataset               |
     |-------------------|-------|-----------------------|
     | 20 Question Game  | `20q` | `bigbench` / `common` |
     | Medical Diagnosis | `md`  | `DX` / `MedDG`        |
     | Troubleshooting   | `tb`  | `FiaDial`             |
 
 - `--task_start_index` and `--task_end_index`: Conduct experiment with [start, end) targets in selected dataset. (Default: entire dataset)
+
+- `--open_set_size`: Size of the possibility set after updating for open-set setting. (Default: -1, unable open-set setting)
+
+- `--size_to_renew`: When the size of the possibility set is less than this value, update the possibility set to the size of `--open_set_size`. (Consider only when `--open_set_size` > 0) 
+
+- `--n_pre_ask`: Number of DPOS rounds at the beginning. (Consider only when `--open_set_size` > 0 and self-report disable) 
+
 - `--naive_run`: If True, run with naive prompting method, otherwise UoT.
+
 - `--inform`: If True, the guesser is given answer set. (Consider only when `--naive_run` is True) 
+
 - `--reward_lambda`: Parameter $\lambda$ in uncertainty-based reward setting.
+
 - `--n_extend_layers`: Parameter $J$ -- Number of simulation steps.
+
 - `--n_potential_actions`: Parameter $N$ -- Number of candidate actions generated.
+
 - `--n_pruned_nodes`: Max number of remaining nodes in each step.
-  
+
   - If not prun, set it to 0;
   - If prun and remain exact number of nodes, set it > 0 (e.g. `10`: Each step has a maximum of 10 nodes, $M$ or $U$, remaining);
   - If prun and remain a certain proportion of nodes, set it < 0 (e.g. `-0.5`: The remaining 50% of nodes in each step).
+
 - `--expected_action_tokens`: Max tokens not to call `gpt-3.5-turbo` model simplifying the guesser's selected action.
+
 - `--expected_target_tokens`: Max tokens for each target name. Used to predict and set the `max_tokens` when calling guesser model.
 
 ## Implement Note

pics/results_os.png

@@ -3,19 +3,20 @@
 
 def evaluate_performance(file, task):
     cnt = success = 0
-    efficiency = success_efficiency = 0
+    length = success_length = 0
     with (open(file, 'r') as f):
         data = json.load(f)
     for i in data:
         cnt += 1
         if i['state'] == 1:
             success += 1
-            success_efficiency += i['turn']
-            efficiency += i['turn']
+            success_length += i['turn']
+            length += i['turn']
         else:
-            efficiency += task.max_turn
+            length += task.max_turn
 
     print('Dialogue Count:', cnt)
     print('Success Rate:', success / cnt)
-    print('Efficiency in Successful Cases:', success_efficiency / success)
-    print('Efficiency:', efficiency / cnt)
+    print('Mean Conversation Length in Successful Cases:', success_length / success)
+    print('Mean Conversation Length:', length / cnt)
+