Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shuffling Ensemble: unshuffling ensemble variants for voting is missing in the code #11

Open
HyunggyuJang opened this issue Dec 16, 2023 · 0 comments

Comments

@HyunggyuJang
Copy link

HyunggyuJang commented Dec 16, 2023

Issue Description

The code is missing the implementation for reverting shuffling in the voting process. Currently, the shuffling ensemble is implemented in

def set_order(problem, order=default_order):

However, I couldn't find a section in
x = Counter(row['cot'])
for k in x:
x[k] /= len(row['cot'])
else:
x = {}
if 'logprobs' in row:
for e in row['logprobs']:
for k in e:
if k not in x:
x[k] = 0
if row['subject'] in subjects:
x[k] += 0.5 * e[k] / len(row['logprobs'])
else:
x[k] += 2.0 * e[k] / len(row['logprobs'])
if x:
selected_answer = max(x, key=x.get)
if row['answer'] == selected_answer:
n_correct += 1
else:
n_correct += 1/4

where the original order is recovered to account for the voting result.

Expected Behavior

The code should include the necessary logic to revert the shuffling and account for the voting result.

@HyunggyuJang HyunggyuJang changed the title Shuffling Ensemble: reverting shuffling for the voting process is missing Shuffling Ensemble: unshuffling ensemble variants for voting is missing in the code Dec 16, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant