Updating for new exp decay function and incentive weighting (#501)

* Updating for new exp decay function and incentive weighting * update volatility * Updating miner statistics and config * Updating minerstatistics for new penalized and augmented params * Updating docs * ruff and tests * Updating terminology and some metrics * skip ChallengesProps when not in challengeperiod * update dashboard * Updating some bugs in the calculation of the penalties and weights for the metrics * update challengeperiod scorecards * Updating for new ledger penalties * Tweaking scoring categories * update dash drawdown * add cumulative return to ledger checkpoints * fix PydanticSerializationError for PerfCheckpoint * update readme for mdd elimination * Tweaking scoring to clean up and double checking against prior calculations * Updating tests and notations on functions in metrics * Updating stats metrics for weighting on tstat * calmar ratio text * Log dash data generation time * Allow quote sources to update independently of price sources being updated * Reverting adjusted annual returns --------- Co-authored-by: sli-tao <sli@taoshi.io> Co-authored-by: Derek Awender <derekawender@gmail.com> Co-authored-by: Jordan Bonilla <jbonilla@taoshi.io>
taoshidev · Feb 24, 2025 · a3e6eef · a3e6eef
1 parent fbcc2ee
commit a3e6eef
Show file tree

Hide file tree

Showing 24 changed files with 816 additions and 833 deletions.
diff --git a/README.md b/README.md
@@ -109,7 +109,7 @@ leverage and you want to reduce it to a .25x leverage position to start taking p
 of size .25x leverage to reduce the size of the position. LONG and SHORT signals can be thought of working in opposite 
 directions in this way.
 8. Miners can explicitly close out a position by sending in a FLAT signal. 
-9. Miners are eliminated if they are detected as plagiarising other miners. (more info in  the "Eliminations" section).
+9. Miners are eliminated if they are detected as plagiarising other miners, or if they exceed 10% max drawdown (more info in  the "Eliminations" section).
 10. There is a fee for leaving positions open "carry fee". The fee is equal to 10.95/5.25/3% per year for a 1x leverage position (crypto/equities/forex) <a href="https://docs.taoshi.io/tips/p4/">More info</a>
 11. There is a slippage assessed per order. The slippage cost is is greater for orders with higher leverages, and in assets with lower liquidity. 
 12. There is a minimum registration fee of 2.5 TAO on the mainnet subnet.
@@ -121,13 +121,17 @@ With this system only the world's best traders & deep learning / quant based tra
 
 # Eliminations
 
-In the Proprietary Trading Network, Eliminations occur for miners that commit Plagiarism.
+In the Proprietary Trading Network, Eliminations occur for miners that commit Plagiarism, or exceed 10% Max Drawdown.
 
 
 ### Plagiarism Eliminations
 
 Miners who repeatedly copy another miner's trades will be eliminated. Our system analyzes the uniqueness of each submitted order. If an order is found to be a copy (plagiarized), it triggers the miner's elimination.
 
+### Max Drawdown Elimination
+
+Miners who exceed 10% max drawdowns will be eliminated. Our system continuously tracks each miner’s performance, measuring the maximum drop from peak portfolio value. If a miner’s drawdown exceeds the allowed threshold, they will be eliminated to maintain risk control.
+
 ### Post-Elimination
 
 After elimination, miners are not immediately deregistered from the network. They will undergo a waiting period, determined by registration timelines and the network's immunity policy, before official deregistration. Upon official deregistration, the miner forfeits registration fees paid.

diff --git a/docs/miner.md b/docs/miner.md
@@ -20,35 +20,31 @@ A long position is a bet that the trade pair will increase, while a short positi
 
 ## Scoring Details
 
-*Risk-Adjusted Returns* is a heavily weighted scoring mechanism in our system. We calculate this metric using a combination of average daily returns and a drawdown term, assessed over different lookback periods. This approach allows us to measure each miner’s returns while factoring in the level of risk undertaken to achieve those returns. 
+PTN relies on a number of scoring metrics to build a comprehensive measure of *Risk-Adjusted Returns*. In practice, these metrics are often highly correlated, but each offers a unique lens through which to see the miners.  
 
-While returns is a significant scoring mechanic, consistency and statistical confidence each play a substantial role in scoring our miners as we look to prioritize miners with a consistent track record of success. Additionally, we have a layer of costs and penalties baked into PTN, to simulate the real costs of trading.
+While returns is a significant scoring mechanic, we also use penalties to prioritize different aspects of trading, such as the risk undertaken by the miners or their likelihood to engage in risky strategies.
 
 We calculate daily returns for all positions and the entire portfolio, spanning from 12:00 AM UTC to 12:00 AM UTC the following day. However, if a trading day is still ongoing, we still monitor real-time performance and risks. 
 
 This daily calculation and evaluation framework closely aligns with real-world financial practices, enabling accurate, consistent, and meaningful performance measurement and comparison across strategies. This remains effective even for strategies trading different asset classes at different trading frequencies. This approach can also enhance the precision of volatility measurement for strategies.
 
-Annualization is used for the Sharpe ratio, Sortino ratio, and risk adjusted return with either volatility or returns being annualized to better evaluate the long-term value of strategies and standardize our metrics. Volatility is the standard deviation of returns and is a key factor in the Sharpe and Sortino calculations.
+Annualization is used for the Sharpe ratio, Sortino ratio, and risk adjusted return with either volatility or returns being annualized to better evaluate the long-term value of strategies and standardize our metrics. In determining the correct annualization factor, we weight more recent trading days slightly higher than older trading days. This should encourage miners to regularly update their strategies and adapt to changing market conditions, continually providing the network with the most relevant signals. The most recent daily returns have a significance of about 2.5 relative to the oldest daily returns, with a pattern that tapers exponentially over time.
 
 Additionally, normalization with annual risk-free rate of T-bills further standardizes our metrics and allows us to measure miner performance on a more consistent basis.
 
 
 ### Scoring Metrics
 
-We use six scoring metrics to evaluate miners based on daily returns:  **Long Term Risk Adjusted Returns**,  **Short Term Risk Adjusted Returns**, **Sharpe**, **Omega**, **Sortino**, and **Statistical Confidence**.
+We use five scoring metrics to evaluate miners based on daily returns:  **Calmar Ratio**, **Sharpe Ratio**, **Omega Ratio**, **Sortino Ratio**, and **Statistical Confidence (T-Statistic)**.
 
-The miner risk used in the risk adjusted returns is the miner’s average portfolio drawdown, the average of the maximum drops in value seen while we have been tracking the behavior of the miner. Once the drawdown surpasses 5%, it is no longer used directly as the denominator; instead, it is multiplied by a larger factor to amplify its effect. This emphasizes that a miner in the range between 5% and 10% average drawdown is riskier than a miner below 5% average drawdown.
+The miner risk used in the risk adjusted returns is the miner’s maximum portfolio drawdown.
 
-To find the risk adjusted return, we take the annualized daily returns as the current miner return. We then divide this by the drawdown term. If, for example, a miner has a total 90-day return of 7.5% and a mean drawdown of 2.5%, their long term risk adjusted return would be 3.0.
-
-_Long term returns_ will look at daily returns in the prior 90 days and is normalized by the drawdown term.
+_Calmar Ratio_ will look at daily returns in the prior 90 days and is normalized by the max drawdown.
 
 $$
 \text{Return / Drawdown} = \frac{(\frac{365}{n}\sum_{i=0}^n{R_i}) - R_{rf}}{\sum_i^{n}{\text{MDD}_i} / n}
 $$
 
-_Short term returns_ will look at daily returns in the prior 7 days. Besides the shorter lookback window, this calculation is the same as long term returns. 
-
 The _sharpe ratio_ will look at the annualized excess return, returns normalized with the risk-free rate, divided by the annualized volatility which is the standard deviation of the returns. To avoid gaming on the bottom, a minimum value of 1% is used for the volatility.
 
 $$
@@ -73,21 +69,19 @@ $$
 t = \frac{\bar{R} - \mu}{s / \sqrt{n}}
 $$
 
-| Metric                     | Scoring Weight                 |
-|---------------------------------------|-----------------------|
-| Long Term Risk Adjusted Returns | 20%           |
-| Short Term Realized Returns| 10%                   |
-| Sharpe Ratio                         | 17.5%                   |
-| Omega Ratio                         | 17.5%                   |
-| Sortino Ratio 		 | 17.5%
-| Statistical Confidence	 | 17.5%	                 |
+| Metric                  | Scoring Weight |
+|-------------------------|----------------|
+| Calmar Ratio            | 20%            |
+| Sharpe Ratio            | 20%            |
+| Omega Ratio             | 20%            |
+| Sortino Ratio 		        | 20%            
+| Statistical Confidence	 | 20%	           |
 
 ### Scoring Penalties
 
 There are two primary penalties in place for each miner:
 
-1. Max Drawdown: PTN penalizes miners whose maximum drawdown over the past 5 days exceeds the predefined 10% limit.
-
+1. Max Drawdown: PTN eliminates miners who exceed 10% max drawdown.
 2. Martingale: Miners are penalized for having positions that resemble a martingale strategy. Two or more orders that increase leverage beyond the maximum leverage already seen while a position has unrealized loss may result in a penalty. More details on this can be found [here](https://docs.taoshi.io/tips/p15/).
 
 The Max Drawdown penalty and Martingale penalty help us detect the absolute and relative risks of a miner's trading strategy in real time.

diff --git a/meta/meta.json b/meta/meta.json
@@ -1,3 +1,3 @@
 {
-  "subnet_version": "5.3.2"
+  "subnet_version": "5.4.0"
 }
diff --git a/miner_objects/miner_dashboard/src/components/Challenges/Challenges.tsx b/miner_objects/miner_dashboard/src/components/Challenges/Challenges.tsx
@@ -9,23 +9,24 @@ interface ChallengesProps {
 }
 
 export const Challenges = ({ statistics }: ChallengesProps) => {
-  const { challengeperiod } = statistics.data[0];
-  const { status, scores } = challengeperiod;
-  const { omega, overall, return_long, return_short, sharpe_ratio, sortino, statistical_confidence } = scores
+  const { challengeperiod, scores } = statistics.data[0];
+  const { status } = challengeperiod;
+  const { CHALLENGE_PERIOD_PERCENTILE_THRESHOLD } = statistics.constants;
+
+  // if anything is in challenge period show element
+  const isInChallenge = status === "testing";
+  if (!isInChallenge) return null;
+
+  const { omega, calmar, return:returnScore, sharpe, sortino, statistical_confidence } = scores
 
   const scoreData = [
-    { label: "Overall", score: overall },
     { label: "Omega", score: omega },
-    { label: "Return Long", score: return_long },
-    { label: "Return Short", score: return_short },
-    { label: "Sharpe Ratio", score: sharpe_ratio },
+    { label: "Sharpe Ratio", score: sharpe },
     { label: "Sortino", score: sortino },
     { label: "Statistical Confidence", score: statistical_confidence },
+    { label: "Calmar", score: calmar },
+    { label: "Return", score: returnScore },
   ];
-
-  // if anything is in challenge period show element
-  const isInChallenge = !isNil(scores)  && status === "testing";
-  const passingThreshold = overall.target_percentile
 
   return (
     <Fragment>
@@ -42,7 +43,7 @@ export const Challenges = ({ statistics }: ChallengesProps) => {
                     key={label}
                     label={label}
                     value={score.percentile}
-                    target={passingThreshold}
+                    target={CHALLENGE_PERIOD_PERCENTILE_THRESHOLD}
                   />
                 );
               }

diff --git a/miner_objects/miner_dashboard/src/components/Checkpoints/Checkpoints.tsx b/miner_objects/miner_dashboard/src/components/Checkpoints/Checkpoints.tsx
@@ -29,7 +29,7 @@ export const Checkpoints = ({ statistics }: CheckpointsProps) => {
         </Title>
         <SimpleGrid mb="lg" cols={4}>
           <StatCard
-            title="Calmar Ratio (%)"
+            title="Calmar Ratio"
             item={scores.calmar}
             isPercentage={true}
             sigFigs={2}
@@ -47,22 +47,15 @@ export const Checkpoints = ({ statistics }: CheckpointsProps) => {
             sigFigs={2}
             tooltipText="The Sharpe Ratio assesses the return of an investment compared to the std. dev. of returns. A higher Sharpe ratio indicates higher returns at greater predictability."
           />
-          <StatCard
-            title="Recent Calmar Ratio (%)"
-            item={scores["short-calmar"]}
-            isPercentage={true}
-            sigFigs={2}
-            tooltipText="Similar to the Calmar ratio, the Recent Calmar Ratio differs in that it only considers daily returns within the past 5 days."
-          />
-        </SimpleGrid>
-        <SimpleGrid mb="lg" cols={4}>
           <StatCard
             title="Statistical Confidence"
             item={scores.statistical_confidence}
             isPercentage={true}
             sigFigs={2}
             tooltipText="Statistical Confidence (t-statistic) is the measure of statistical likelihood of the daily return distribution to come from a random distribution. The higher the value, the lower the probability that the returns are likely random."
           />
+        </SimpleGrid>
+        <SimpleGrid mb="lg" cols={4}>
           <StatCard
             title="Sortino Ratio"
             item={scores.sortino}
@@ -76,14 +69,6 @@ export const Checkpoints = ({ statistics }: CheckpointsProps) => {
             sigFigs={2}
             tooltipText="Estimated Annualized Return, based on daily returns from the past 90 days."
           />
-          <StatCard
-            disabled
-            title="Short-Term Return (%)"
-            item={scores.short_return}
-            isPercentage={true}
-            sigFigs={2}
-            tooltipText="Estimated Annualized Return, based on daily returns from the past 7 days."
-          />
         </SimpleGrid>
       </Box>
 

diff --git a/miner_objects/miner_dashboard/src/components/Contribution/Contribution.tsx b/miner_objects/miner_dashboard/src/components/Contribution/Contribution.tsx
@@ -16,7 +16,7 @@ export function Contribution({ statistics }: StatisticsProps) {
     sharpe,
     sortino,
     statistical_confidence,
-    "short-calmar": shortCalmar,
+    return:returnScore,
   } = scores;
 
   return (
@@ -45,22 +45,22 @@ export function Contribution({ statistics }: StatisticsProps) {
             <Progress.Label>Sharpe</Progress.Label>
           </Progress.Section>
           <Progress.Section
-            value={shortCalmar.overall_contribution * 100}
+            value={sortino.overall_contribution * 100}
             color="#C95E2F"
           >
-            <Progress.Label>Recent Calmar</Progress.Label>
+            <Progress.Label>Sortino</Progress.Label>
           </Progress.Section>
           <Progress.Section
-            value={sortino.overall_contribution * 100}
+            value={statistical_confidence.overall_contribution * 100}
             color="#CF6638"
           >
-            <Progress.Label>Sortino</Progress.Label>
+            <Progress.Label>Statistical Confidence</Progress.Label>
           </Progress.Section>
           <Progress.Section
-            value={statistical_confidence.overall_contribution * 100}
+            value={returnScore.overall_contribution * 100}
             color="#D26F44"
           >
-            <Progress.Label>Statistical Confidence</Progress.Label>
+            <Progress.Label>Return</Progress.Label>
           </Progress.Section>
         </Progress.Root>
       </Box>

diff --git a/miner_objects/miner_dashboard/src/components/Statistics/Statistics.tsx b/miner_objects/miner_dashboard/src/components/Statistics/Statistics.tsx
@@ -58,23 +58,9 @@ export const Statistics = ({ statistics, positions }: StatisticsProps) => {
 
           <Group justify="space-between" align="center" mb="xs">
             <Text size="xs" c="gray">
-              Approximate Drawdown
+              Max Drawdown
             </Text>
-            <Text size="xs">{toRemainingPercent(drawdowns.approximate, 2)}</Text>
-          </Group>
-
-          <Group justify="space-between" align="center" mb="xs">
-            <Text size="xs" c="gray">
-              Recent Drawdown
-            </Text>
-            <Text size="xs">{toRemainingPercent(drawdowns.recent, 2)}</Text>
-          </Group>
-
-          <Group justify="space-between" align="center" mb="xs">
-            <Text size="xs" c="gray" fw="bold">
-              Effective Drawdown
-            </Text>
-            <Text size="xs">{toRemainingPercent(drawdowns.effective, 2)}</Text>
+            <Text size="xs">{toRemainingPercent(drawdowns.max_drawdown, 2)}</Text>
           </Group>
         </Card>
 

diff --git a/miner_objects/miner_dashboard/src/types/index.ts b/miner_objects/miner_dashboard/src/types/index.ts
@@ -1,16 +1,13 @@
 export interface Challenge {
   percentile: number;
-  target_score: number;
   value: number;
-  target_percentile: number;
 }
 
 export interface ChallengeMetric {
   omega: Challenge;
-  overall: Challenge;
-  return_long: Challenge;
-  return_short: Challenge;
-  sharpe_ratio: Challenge;
+  calmar: Challenge;
+  return: Challenge;
+  sharpe: Challenge;
   sortino: Challenge;
   statistical_confidence: Challenge;
 }
@@ -69,16 +66,12 @@ export interface Score {
 }
 
 export interface Scores {
-  risk_adjusted_return: Score;
-  short_risk_adjusted_return_dict: Score;
+  return: Score;
   omega: Score;
   sortino: Score;
   statistical_confidence: Score;
   sharpe: Score;
   calmar: Score;
-  return: Score;
-  short_return: Score;
-  "short-calmar": Score;
 }
 
 // unused
@@ -108,9 +101,7 @@ export interface PenalizedScores {
 }
 
 export interface Drawdowns {
-  approximate: number;
-  effective: number;
-  recent: number;
+  max_drawdown: number;
 }
 
 export interface StatisticsData {
@@ -127,6 +118,11 @@ export interface StatisticsData {
 
 export interface Statistics {
   data: StatisticsData[];
+  constants: Constants;
+}
+
+export interface Constants {
+  CHALLENGE_PERIOD_PERCENTILE_THRESHOLD: number;
 }
 
 export interface Positions {

diff --git a/miner_stats_comparison.png b/miner_stats_comparison.png
diff --git a/neurons/validator.py b/neurons/validator.py
@@ -280,7 +280,7 @@ def rc_priority_fn(synapse: template.protocol.ValidatorCheckpoint) -> float:
             priority_fn=gp_priority_fn,
         )
         self.axon.attach(
-            forward_fn=self.get_data,
+            forward_fn=self.get_dash_data,
             blacklist_fn=gd_blacklist_fn,
             priority_fn=gd_priority_fn,
         )
@@ -618,7 +618,7 @@ def should_fail_early(self, synapse: template.protocol.SendSignal | template.pro
         elif method == SynapseMethod.CHECKPOINT:
             allowed, wait_time = self.checkpoint_rate_limiter.is_allowed(sender_hotkey)
         else:
-            msg = "Received synapse does not match one of expected methods for: receive_signal, get_positions, get_data, or receive_checkpoint"
+            msg = "Received synapse does not match one of expected methods for: receive_signal, get_positions, get_dash_data, or receive_checkpoint"
             bt.logging.trace(msg)
             synapse.successfully_processed = False
             synapse.error_message = msg
@@ -805,11 +805,12 @@ def get_positions(self, synapse: template.protocol.GetPositions,
         bt.logging.info(msg)
         return synapse
 
-    def get_data(self, synapse: template.protocol.GetDashData,
+    def get_dash_data(self, synapse: template.protocol.GetDashData,
                       ) -> template.protocol.GetDashData:
         if self.should_fail_early(synapse, SynapseMethod.DASHBOARD):
             return synapse
 
+        now_ms = TimeUtil.now_in_millis()
         miner_hotkey = synapse.dendrite.hotkey
         error_message = ""
         try:
@@ -835,6 +836,10 @@ def get_data(self, synapse: template.protocol.GetDashData,
             bt.logging.error(error_message)
             synapse.successfully_processed = False
         synapse.error_message = error_message
+        processing_time_s_3_decimals = round((TimeUtil.now_in_millis() - now_ms) / 1000.0, 3)
+        bt.logging.info(
+            f"Sending dash data back to miner [{miner_hotkey}]. Synapse Message: {synapse.error_message}. "
+            f"Process time {processing_time_s_3_decimals} seconds.")
         return synapse
 
     def receive_checkpoint(self, synapse: template.protocol.ValidatorCheckpoint) -> template.protocol.ValidatorCheckpoint: