About symbols standardization

[toyan]

It's time to decide on the standardization rules for exchange's instruments and their types.
The are some requests about this, and personally I'm in.
I have a proposal to define some rules and implement some corresponding functions.
First of all, we have to define 'CURRENCY' type.
It is needed for:

• balances (portfolio) feeds. We have BTC, ETH, USDT, etc units in portfolio regardless of their rates - we just have it;
• options feeds. Option always expires to CURRENCY, not to pair.
  Then we need to define strict rules for user inputs for symbols, something like this:

'''
    This code illustrates a proposal for normalization standards
    for unifying symbols and instruments notations inside Cryptofeed
'''

CURRENCY= 'currency'
SPOT= 'spot' 
PERPETUAL= 'perpetual'
FUTURES= 'futures'
OPTION= 'option'
CALL = 'call'
PUT = 'put'
STRIKE = 'strike'

def parse_symbol(symbol: str, delimiter: str):
    if symbol.count(delimiter) == 0:
        # Defines symbol as CURRENCY type (BTC, ETH, etc.)
    
        type = CURRENCY
        print(f'"{symbol}" is {type}')
    
    elif symbol.count(delimiter) == 1:
        # Defines PAIR for spot markets
        # I propose this is always assume SPOT instrument type for spot markets like BTC-USD, ETH-USDT and so on
        # Left part - base currency, right part - quote currency

        type = SPOT
        base = symbol.split(delimiter)[0]
        quote = symbol.split(delimiter)[1]

        print(f'"{symbol}" is {type} pair, base currency is {base}, quote currency is {quote}')
    
    elif symbol.count(delimiter) == 2:
        # This is always FUTURES instrument type for derivatives markets.
        # Instrument has base and quote currencies and the date of expiration
        base = symbol.split(delimiter)[0]
        quote = symbol.split(delimiter)[1]

        # And here we need to separate two types of futures: perpetual and classic futures:
        if 'PERP' in symbol.split(delimiter)[2]:     #    BTC-USDT-PERP, ETH-USD-PERP
            # Instrument type is PERPETUAL (never expired futures subtype)
            type = PERPETUAL
            
            print(f'"{symbol}" is {type} futures, base currency is {base}, quote currency is {quote}')

        elif symbol[-2:].isdigit():    # Symbol has two delimiters and two last symbols are digits (short year of expiration date)
            
            type = FUTURES
            # Instrument type is FUTURES, and I propose to follow strict standard
            # for futures notation like this: 'BTC-USD-31DEC21', where are:
            expire_year = symbol[-2:]              # assume last two digits always is year in short notation: '21' = 2021, '22' = 2022
            expire_month = symbol[-5:-2]           # month of expiration: JUL, SEP, DEC and so on
                                                   # digits like 07, 09, 12 is not suitable because of regional differencies in standards
                                                   # '101121': 21 is a year, but where is a month and where is a day? It depends. )) 
            expire_day = symbol[-7:-5]             # two digits of the day of expiration: '03', '12', '31' 
            
            print(f'"{symbol}" is {type}, base currency is {base}, quote currency is {quote}, expiration date is {expire_day}-{expire_month}-20{expire_year}')

        else:
            # Throw Exception
            print(f'Wrong symbol input format: {symbol}')

    elif symbol.count(delimiter) == 3:             # Symbol has three delimiters and this is always OPTION type
        type = OPTION                              # 'BTC-50000-31DEC21-C', 'ETH-4000-15MAR22-C'
        base = symbol.split(delimiter)[0]          # Unlike of futures, options have only base cyrrency
                                                   # Expiration date format is the same as futures type has
        
        expire_year = symbol.split(delimiter)[2][-2:]               
        expire_month = symbol.split(delimiter)[2][-5:-2]
        expire_day = symbol.split(delimiter)[2][-7:-5]               

        if symbol.split(delimiter)[3].upper() == "C":
            option_type = CALL                  
        elif symbol.split(delimiter)[3].upper() == "P":
            option_type = PUT
        else:
            # Throw Exception
            print(f'Wrong option type: {symbol.split(delimiter)[3]}')

        if symbol.split(delimiter)[1].isdigit():
            option_strike = symbol.split(delimiter)[1] 
        else:
            # Throw Exception
            print(f'Wrong option strike: {symbol}')

        print(f'"{symbol}" is {option_type} {type} with strike {option_strike}, base currency is {base}, expiration date is {expire_day}-{expire_month}-20{expire_year}')

for symbol in ['BTC', 'BTC-USDT', 'BTC-USDT-PERP', 'BTC-USD-31DEC21', 'BTC-50000-15MAR22-C']:
    parse_symbol(symbol, '-')

[toyan]

If proposal above will be accepted by headquarter, I can implement some corresponding functions like check_input_symbols.
Also I implemented some authenticated channels support for Deribit and Bybit exchanges, and now I'm waiting for decision about symbol normalisation before I send a pull request.

[bmoscon]

@toyan I recently implemented some changes regarding this. Have a look and let me know what you think

[bmoscon]

I didn't implement exactly everything you mention above, but I did get a lot of them, so let me know

[toyan]

Excellent implementation, it fully corresponds to what I was talking about, thank you!
You made a mistake in specifying the instrument type in the Bybit class, I fixed it, see my implementation in pull request.

[toyan]

@bmoscon There are some things regarding symbol standardization I would like to discuss.
While implementing ACC_BALANCES for Deribit, I have had some difficulties.
It is suitable to specify symbols as CURRENCY in 'symbols' list while adding a feed to Feedhandler for ACC_BALANCES feed.
So we have to have that normalized symbols in exchange's symbols list.
Assuming base currency in pairs always are portfolio currency, I used this to collect it:

        currencies = []
        for entry in data:
            for e in entry['result']:
                base = e['base_currency']
                if base not in currencies:
                    currencies.append(base)

And then

        for currency in currencies:
            s = Symbol(currency, currency, type=CURRENCY)
            ret[s.normalized] = currency
            info['instrument_type'][s.normalized] = CURRENCY

CURRENCY type does not have 'quote', but quote is positional parameter for Symbol type.
I think Symbol class args should looks as
s = Symbol(type, base, quote=quote... (and all rest args are optional) )

[bmoscon]

that seems reasonable. Since most exchanges support spot or spot only, i made type an kwarg and defaulted it to spot, but it could be a required arg too.