Skip to content

Confidence Stability Scores

Jump to bottom Edit New page
Myklob edited this page Mar 20, 2023 · 10 revisions

#Belief and Argument Stability Scores

The Belief Stability Score (BSS) measures the stability and reliability of a belief's score over time, considering how a group of well-informed and reasonable users would engage with the most critical pro/con arguments to achieve consensus or equilibrium. Although it's unlikely to assemble a perfect group of such users, we can provide ordinary people with a framework and tools that encourage techniques taught in conflict resolution, formal logic, and cost-benefit analysis. We will also leverage every available tool to measure, identify, and promote reasoning and conflict resolution techniques.

Several issues have been fully addressed and have achieved belief stability. For example, beliefs such as theft, pollution, and murder have widely been accepted as wrong, making them more stable in consensus.

Algorithm Development Path Forward

To ensure the accuracy and effectiveness of the ESS, we will continuously improve our algorithm. This involves analyzing the algorithm's performance based on user feedback and making necessary adjustments to enhance its capabilities. We will also gather additional data and consider new factors that may impact the score, such as the quality and quantity of evidence, the credibility of sources, and the level of agreement among experts. We aim to create a reliable, transparent, and user-friendly algorithm, providing valuable insights to individuals and groups engaged in debates and discussions.

To calculate the ESS for a particular argument, we will need to take into account various factors, such as the quality and credibility of the sources cited, the level of agreement among experts, and the extent to which the evidence has been challenged and debated. We will also consider factors such as the number of citations, the date of publication, and the level of scrutiny the evidence has undergone.

Overall, the ESS will help users make more informed decisions by providing an accurate and reliable measure of the quality and reliability of the evidence supporting a particular argument. It will also help to promote more reasoned and informed debate by encouraging users to focus on issues that are still unresolved or where there is still significant disagreement among experts.

#Algorithm

Collect data on the pro and con sub-arguments for a given debate, including the following:

  1. The number of reasons for each pro/con sub-argument.
  2. The number of reasons attempted to be submitted by users who realize that those arguments, or arguments saying essentially the same thing, have already been submitted and thoroughly evaluated.
  3. The number of positive and negative responses to questions about the different aspects of each argument, such as: "is the argument:
  • True (based on facts and accurate information)
  • TLogically sound (premises support the conclusion and no logical fallacies)
  • TClear and concise (not convoluted or overly complex)
  • TRelevant to the topic at hand
  • TCoherent (do the different arguments fit together)
  • TConsistent (does the argument contradict other arguments, what are their truth scores)
  • TComplete (does it address all relevant topics at hand)
  • TPersuasiveness
  • THave a respectful and appropriate Tone."
  1. The number and ratio of up/down votes
  2. The number of weekly visitors
  3. The Google page rank on different searches
  4. The amount of time users spent reading and referencing the debate.
  5. The number of up-votes.

To calculate the Confidence Stability Score, we measure the standard deviation of the argument's score based on the amount of effort put in by users, with different weights assigned to each type of interaction mentioned above. This score measures the stability of the argument's confidence, indicating how confident users are in the view over time.

We calculate the BSS and ESS for each pro and con sub-argument, respectively. These scores are calculated based on the frequency of changes made to the sub-argument, the amount of time users have spent making changes, and the number of users who have made changes. We can then use these scores to weigh the Confidence Stability Score, as arguments with higher BSS or ESS are considered more stable.

We should also calculate the Confidence Stability Score for each sub-argument to assess its stability.

Apply a relative weighting to each measure of effort (such as the number of reasons to agree/disagree, up/down votes, evaluations, time spent, and arguments) based on its relative importance in determining the overall argument score.

Compare the weighted Confidence Stability Score to other arguments in the system.

Determine the percentile ranking.

The Confidence Stability Score measures the argument's stability and the level of confidence users have in engaging with the view over time. The relative weighting ensures that the score is not solely based on the quantity of pro/con sub-arguments but also considers the quality and development of those sub-arguments and the stability of those sub-arguments as measured by the BSS and ESS.

Code

Here is some Python code that calculates the confidence stability:

Note that this code assumes that you have already collected the necessary data on each sub-argument and stored it in a data structure such as a dictionary or a list of dictionaries. The code uses the calculate_AQSS function to calculate the Argument Quantity Stability Score for each sub-argument, and then uses the BSS and ESS values and the relative weights to calculate the Confidence Stability Score for each sub-argument. The final output is a list of CSS scores for each pro and con sub-argument.

Here's the updated code that calculates all three stability scores:

import numpy as np

# Collect data on the pro and con sub-arguments
pro_subarguments = ["pro_subargument1", "pro_subargument2", "pro_subargument3"]
con_subarguments = ["con_subargument1", "con_subargument2", "con_subargument3"]

# Define the BSS and ESS weights
BSS_weight = 0.7
ESS_weight = 0.3

# Calculate the Argument Quantity Stability Score for each sub-argument
def calculate_AQSS(argument):
    # Collect data on the sub-argument
    num_reasons = argument["num_reasons"]
    num_duplicates = argument["num_duplicates"]
    question_scores = argument["question_scores"]
    up_votes = argument["up_votes"]
    down_votes = argument["down_votes"]
    visitors = argument["visitors"]
    page_rank = argument["page_rank"]
    time_spent = argument["time_spent"]
    
    # Calculate the Argument Quantity Stability Score
    AQSS_weights = [num_reasons, num_duplicates, question_scores, up_votes, down_votes, visitors, page_rank, time_spent]
    AQSS_weights = [weight / sum(AQSS_weights) for weight in AQSS_weights]
    AQSS_values = [num_reasons, num_duplicates, np.mean(list(question_scores.values())), up_votes, down_votes, visitors, page_rank, time_spent]
    AQSS = np.dot(AQSS_weights, AQSS_values)
    
    return AQSS

# Calculate the Confidence Stability Score for each sub-argument
def calculate_CSS(argument):
    # Calculate the Argument Quantity Stability Score
    AQSS = calculate_AQSS(argument)
    
    # Collect data on the argument's BSS and ESS
    BSS = argument["BSS"]
    ESS = argument["ESS"]
    
    # Calculate the Confidence Stability Score
    CSS = BSS_weight * BSS + ESS_weight * ESS + AQSS
    
    return CSS

# Calculate the Confidence Stability Score for each pro sub-argument
pro_CSS_scores = []
for pro_argument in pro_subarguments:
    CSS = calculate_CSS(pro_argument)
    pro_CSS_scores.append(CSS)

# Calculate the Confidence Stability Score for each con sub-argument
con_CSS_scores = []
for con_argument in con_subarguments:
    CSS = calculate_CSS(con_argument)
    con_CSS_scores.append(CSS)

# Print the Confidence Stability Scores for each sub-argument
print("Pro Sub-Argument CSS Scores: ", pro_CSS_scores)
print("Con Sub-Argument CSS Scores: ", con_CSS_scores)
import numpy as np

# collect data on the pro and con sub-arguments for a given debate

# number of reasons for each pro/con sub-argument
num_pro_reasons = 10
num_con_reasons = 8

# number of reasons attempted to be submitted by users who realize that those arguments, 
# or arguments saying essentially the same thing, have already been submitted and thoroughly evaluated
num_duplicate_reasons = 2

# positive and negative responses to questions about the different aspects of each argument
true_score = 0.8
logic_score = 0.7
clarity_score = 0.9
relevance_score = 0.6
coherence_score = 0.8
consistency_score = 0.7
completeness_score = 0.6
persuasiveness_score = 0.9
tone_score = 0.8

# number and ratio of up/down votes
num_upvotes = 100
num_downvotes = 20
ratio_up_down_votes = num_upvotes / (num_upvotes + num_downvotes)

# number of weekly visitors
num_weekly_visitors = 5000

# google page rank on different searches
google_page_rank = 3.2

# amount of time users spent reading and referencing the debate
time_spent = 60

# to calculate the Argument Quantity Stability Score (AQSS)

# collect all the data on the pro and con sub-arguments
all_pro_reasons = [np.random.randint(1, 11) for _ in range(10)]
all_con_reasons = [np.random.randint(1, 11) for _ in range(8)]

# calculate the standard deviation of the pro and con sub-arguments
pro_stdev = np.std(all_pro_reasons)
con_stdev = np.std(all_con_reasons)

# calculate the AQSS for the pro and con arguments
pro_aqss = 1 - (pro_stdev / num_pro_reasons)
con_aqss = 1 - (con_stdev / num_con_reasons)

# calculate the Argument Stability Score (ASS) and Evidence Stability Score (ESS)

# combine all the scores
all_scores = [true_score, logic_score, clarity_score, relevance_score, coherence_score, 
              consistency_score, completeness_score, persuasiveness_score, tone_score, 
              ratio_up_down_votes, num_weekly_visitors, google_page_rank, time_spent]

# apply weights to each score
score_weights = [0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05, 0.05]
weighted_scores = [score * weight for score, weight in zip(all_scores, score_weights)]

# calculate the ASS and ESS
ass = sum(weighted_scores) * (1 - num_duplicate_reasons / (num_pro_reasons + num_con_reasons))
ess = (pro_aqss + con_aqss) / 2

Here's an example of how to calculate the BSS and ESS weight based on the relative performance of the pro/con sub-arguments:

# Calculate the BSS and ESS weight based on relative performance
pro_sub_argument_scores = [0.8, 0.7, 0.9, 0.6, 0.5] # example pro sub-argument scores
con_sub_argument_scores = [0.6, 0.5, 0.7, 0.4, 0.3] # example con sub-argument scores

# Calculate the average score for each pro and con sub-argument
avg_pro_score = sum(pro_sub_argument_scores) / len(pro_sub_argument_scores)
avg_con_score = sum(con_sub_argument_scores) / len(con_sub_argument_scores)

# Calculate the ratio of pro sub-argument score to con sub-argument score for each aspect of the debate
aspect_ratios = [pro_sub_argument_scores[i] / con_sub_argument_scores[i] for i in range(len(pro_sub_argument_scores))]

# Calculate the sum of ratios for pro and con sub-arguments
pro_sum = sum([ratio for ratio in aspect_ratios if ratio > 1])
con_sum = sum([1/ratio for ratio in aspect_ratios if ratio < 1])

# Calculate the BSS and ESS weight
BSS_weight = pro_sum / (pro_sum + con_sum)
ESS_weight = 1 - BSS_weight

Here's code that replaces the random data with equations based on the relative performance of the pro/con arguments:

# Calculate the BSS weight based on the relative performance of the pro/con arguments for relevance, coherence, and completeness
relevance_pro = 50 # number of relevant pro arguments
relevance_con = 30 # number of relevant con arguments
relevance_total = relevance_pro + relevance_con
relevance_proportion = relevance_pro / relevance_total
relevance_con_proportion = relevance_con / relevance_total
bss_relevance_weight = relevance_proportion / relevance_con_proportion

coherence_pro = 40 # number of coherent pro arguments
coherence_con = 20 # number of coherent con arguments
coherence_total = coherence_pro + coherence_con
coherence_proportion = coherence_pro / coherence_total
coherence_con_proportion = coherence_con / coherence_total
bss_coherence_weight = coherence_proportion / coherence_con_proportion

completeness_pro = 25 # number of complete pro arguments
completeness_con = 10 # number of complete con arguments
completeness_total = completeness_pro + completeness_con
completeness_proportion = completeness_pro / completeness_total
completeness_con_proportion = completeness_con / completeness_total
bss_completeness_weight = completeness_proportion / completeness_con_proportion

# Calculate the ESS weight based on the relative performance of the pro/con arguments for truth, logic, clarity, and persuasiveness
truth_pro = 60 # number of true pro arguments
truth_con = 20 # number of true con arguments
truth_total = truth_pro + truth_con
truth_proportion = truth_pro / truth_total
truth_con_proportion = truth_con / truth_total
ess_truth_weight = truth_proportion / truth_con_proportion

logic_pro = 45 # number of logical pro arguments
logic_con = 15 # number of logical con arguments
logic_total = logic_pro + logic_con
logic_proportion = logic_pro / logic_total
logic_con_proportion = logic_con / logic_total
ess_logic_weight = logic_proportion / logic_con_proportion

clarity_pro = 30 # number of clear pro arguments
clarity_con = 10 # number of clear con arguments
clarity_total = clarity_pro + clarity_con
clarity_proportion = clarity_pro / clarity_total
clarity_con_proportion = clarity_con / clarity_total
ess_clarity_weight = clarity_proportion / clarity_con_proportion

persuasiveness_pro = 35 # number of persuasive pro arguments
persuasiveness_con = 15 # number of persuasive con arguments
persuasiveness_total = persuasiveness_pro + persuasiveness_con
persuasiveness_proportion = persuasiveness_pro / persuasiveness_total
persuasiveness_con_proportion = persuasiveness_con / persuasiveness_total
ess_persuasiveness_weight = persuasiveness_proportion / persuasiveness_con_proportion

# Calculate the Argument Quantity Stability Score
pro_arg_num = 70 # number of pro arguments
con_arg_num = 30 # number of con arguments
arg_total = pro_arg_num + con_arg_num
proportion = pro_arg_num / arg_total
con_proportion = con_arg_num / arg_total
bss = (proportion - con_proportion) * bss_relevance_weight * bss_coherence_weight * bss_completeness_weight

# Calculate the Evidence Stability Score
pro_truth = 0.8 # proportion of true pro arguments
con_truth = 0.3 # proportion of true con arguments
truth_total = pro_truth + con_truth
pro_truth_proportion = pro_truth / truth_total
con_truth
Add a custom footer
Add a custom sidebar
Clone this wiki locally