convert_Schnitzer_codes_in_icd_10_to_icd_10_cm.Rmd

---
title: Translating ICD-9-CM (and the previously translated ICD-9 and ICD-10) codes from Schnitzer et al., 2011, to ICD-10-CM
author: "Jan Savinc"
date: "`r format(Sys.time(), '%d %B, %Y')`"
output: 
  html_document:
    toc: true
    toc_float: true
    code_folding: hide
editor_options: 
  chunk_output_type: console
bibliography: bibliography.bib
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```


# Introduction

The aim of this document is to produce an updated list of codes suggestive of maltreatment [@schnitzer_2011] - these were originally compiled for ICD-9-CM, and as part of the CHASe study [@dougall_2020] I translated them to ICD-9 and ICD-10, but for its use with US data, an ICD-10-CM version is needed.


# Libraries

We'll use `tidyverse` for data processing, and `icd` for its convenient functions for checking if codes exist in the ICD catalog. `knitr` is useful for printing nice tables in this document. `readxl` is for reading excel files!

```{r}
library(tidyverse)
library(icd)
library(knitr)
library(readxl)
```


## `icd` settings

`icd` will download data to a pre-defined location - we'll change that to a subfolder of this project.

```{r}
if (!dir.exists("./icd_data")) {
  dir.create("./icd_data")
}
set_icd_data_dir(path = "./icd_data")  # tell icd to download data here
# download_all_icd_data()  # tell icd to download everything
get_icd10cm2018()  # TODO: work out how this might work!
# get_icd10who2016()
```


## General Equivalence Mappings for the CM versions of ICD

The latest GEM files were downloaded from the [CDC website](ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/Publications/ICD10CM/2018/Dxgem_2018.zip). The 2018 release was used because it was the most recent at the time of writing.

```{r}
gem_icd9cm <- 
  read_fwf("./raw/2018_I9gem.txt", 
           fwf_cols(
             source = c(1, 5), 
             target = c(7, 13), 
             approximate=c(15,15), 
             no_map=c(16,16), 
             combination=c(17,17), 
             scenario=c(18,18), 
             choice_list=c(19,19) 
             )
           )

gem_icd10cm <- 
  read_fwf("./raw/2018_I10gem.txt", 
           fwf_cols(
             source = c(1, 7), 
             target = c(9, 13), 
             approximate=c(15,15), 
             no_map=c(16,16),  
             combination=c(17,17), 
             scenario=c(18,18), 
             choice_list=c(19,19) 
             )
           )

## for our purposes the distinction between forward & backward maps isn't relevant, so we can combine the two to simplify finding matches
gem_combined <- 
  bind_rows(
    gem_icd9cm %>% rename(icd9cm=source, icd10cm=target) %>% mutate(direction = "f"),
    gem_icd10cm %>% rename(icd9cm=target, icd10cm=source) %>% mutate(direction = "b")
  )
```

## Previously translated ICD-10 codes

```{r}
previously_translated_codes_file <- "./processed_ICD_codes/Schnitzer_et_al_2011_ICD_crossmapping_for_appendix_with_codes_and_descriptions.xlsx"

codes_translated <-
  map(
    .x = excel_sheets(previously_translated_codes_file),
    .f = ~read_excel(path = previously_translated_codes_file, sheet = .x)
  ) %>% set_names(x = ., nm = excel_sheets(previously_translated_codes_file))

schnitzer_age_requirements <- read_csv("./processed_ICD_codes/schnitzer_et_al_2011_empirical_inclusions_age_index.csv")
schnitzer_code_categories <- read_csv("./processed_ICD_codes/schnitzer_et_al_2011_empirical_inclusions_categories_index.csv") %>%
  rename(maltreatment_category = maltreatment_type_schnitzer2011)
```


# Translation

I've combined the following sources of information:

* General Equivalence Mappings (GEM) - these were searched for the matching ICD-9-CM code, and then the forwad & backward matches were combined, usually up to 4 characters long (ICD-10-CM includes a lot of detail in characters paste the 4th)
* the use of the `icd` package in R, specifically the function `explain_code()` in combination with e.g. `as.icd9cm()` to specify which ICD version code was being supplied;
* looking up ICD-10-CM chapters online, e.g. see here for [chapter 19 of ICD-10-CM, version 2016)](https://icd.codes/icd10cm/chapter19)


```{r}
codes_translated$inclusions %>%
  select(condition, icd10, comment_icd10) %>%
  mutate(
    icd10who = as.icd10who(icd10)#,
    # icd10cm = explain_code(icd10who)
    )

## helper function to find ICD-10-CM equivalents in the GEM for ICD-9-CM codes
## icd9cm_code is an ICD-9-CM given as a character string or number, and include decimal dot
## character_depth indicates how detailed codes to display, default is 4 e.g. S12.1
## this finds both forward & backward matches to the icd-9-cm code, truncates the matches to the requested number of characters, and shows unique matches & their descriptions
find_icd10cm_equivalent <- function(icd9cm_code, character_depth = 4) {
  icd9cm_code <- str_remove_all(string = icd9cm_code, pattern="\\.")
  gem_combined %>% 
    filter(str_detect(icd9cm, paste0("^",icd9cm_code))) %>% 
    select(icd10cm) %>%
    mutate(
      icd10cm = str_sub(icd10cm, 1, character_depth),
      ) %>% 
    distinct %>% 
    group_by(icd10cm) %>%
    summarise(meaning = icd::explain_code(as.icd10cm(str_remove_all(icd10cm, pattern = "X")))) %>%
    arrange(icd10cm)
}
```

## Checking through codes

The below contains the results of my checking through codes, without much commentary.

```{r}
icd::explain_code("614.9")
find_icd10cm_equivalent("614.9")
icd::explain_code("N739")


icd::explain_code("9224")
icd::explain_code("S302")

icd::explain_code("V715")
icd::explain_code("Z044")

## observation for alleged abuse/neglext
icd::explain_code("V7181")
icd::explain_code("Z047")
icd::explain_code("Z0471")
icd::explain_code("Z0472")
gem_combined %>% filter(str_detect(icd9cm,"V7181"))

## retinal haemorrhage
icd::explain_code("36281")
gem_combined %>% filter(str_detect(icd9cm,"36281"))
icd::explain_code("H356")

## rib fractures
icd::explain_code("8070")
icd::explain_code("8071")
icd::explain_code("S223") # one rib
icd::explain_code("S224") # multiple ribs

icd::explain_code("811")
icd::explain_code("S421")

## traumatic subdural hemorrhage
## with or without open wound is distinguished
icd::explain_code("8522")
icd::explain_code("8523")
icd::explain_code("S065")
find_icd10cm_equivalent("8522")

# other/unspecified intracranial hemmorh
icd::explain_code("853")
icd::explain_code("8530")
find_icd10cm_equivalent("853")
icd::explain_code("S068")
icd::explain_code("S06893")


## stomach injury originally specified as injury with open wound but not in description so i think we can translate to more general code
icd::explain_code("8631")
icd::explain_code("S363")

## assault
icd::explain_code("E965")  # firearms
icd::explain_code("X93")
icd::explain_code("X94")
icd::explain_code("X95")
icd::explain_code("X96")
icd::explain_code("E966")  # cutting/piercing
icd::explain_code("X99")
icd::explain_code("E9682")  # blunt/thrown object
icd::explain_code("Y00")
gem_combined %>% filter(str_detect(icd9cm,"^E965")) %>% mutate(icd10cm=str_sub(icd10cm,1,4)) %>% select(icd10cm) %>% distinct
## assault NOS
icd::explain_code("E9689")
icd::explain_code("Y09")

# TODO: this needs looking into!
## undet intent other means
icd::explain_code("E988")
gem_combined %>% filter(str_detect(icd9cm,"^E988")) %>% mutate(icd10cm=str_sub(icd10cm,1,4)) %>% select(icd10cm) %>% distinct
gem_combined %>% filter(str_detect(icd9cm,"^E988")) %>% mutate(icd10cm=str_sub(icd10cm,1,4)) %>% select(icd10cm) %>% distinct %>% group_by(icd10cm) %>% summarise(icd10cm = str_replace_all(icd10cm, pattern="X", replacement = ""), meaning = explain_code(icd10cm))
# find_icd10cm_equivalent("E988")
icd::explain_code("Y26")

# TODO: this is also complicated - icd-10 splits vertebrae across body systems
# vertebral fracture
icd::explain_code("805")
find_icd10cm_equivalent(805)

icd::explain_code("S12")
# S12.0,S12.1,S12.2,S12.7,S12.9,S22.0,S22.1,S32.0,S32.7

explain_code("852")
find_icd10cm_equivalent(icd9cm_code = "852")

## intrathoracic injury NEC
explain_code("862")
find_icd10cm_equivalent(icd9cm_code = "862")

## small intestine injury
explain_code("8632")
explain_code("8633")
find_icd10cm_equivalent(icd9cm_code = "8632")
find_icd10cm_equivalent(icd9cm_code = "8633")

## spleen injury
explain_code("865")
find_icd10cm_equivalent(icd9cm_code = "865")

## spinal cord injury
explain_code("952")
find_icd10cm_equivalent(icd9cm_code = "952")
tibble(code=c("S14.0", "S14.1", "S24.0", "S24.1", "S34.0", "S34.1", "S34.3", "T06.0", "T06.1", "T09.3")) %>% group_by(code) %>% summarise(explain_code(code))
# NOTE:  S34.3 should probably be included in the ICD-10 codes also!

# malnutrition
explain_code("262")
# find_icd10cm_equivalent("262")
gem_combined %>% filter(str_detect(icd9cm,"^262")) %>% mutate(icd10cm=str_sub(icd10cm,1,4)) %>% select(icd10cm) %>% distinct

# caries
explain_code("5210")
find_icd10cm_equivalent("5210")
explain_code("K02")

# Solar radiation dermatitis ("Contact dermatitis and other eczema due to solar radiation")
explain_code("6927")
find_icd10cm_equivalent("6927")
explain_code("L578")

# pelvic fract
explain_code("808")
find_icd10cm_equivalent("808")
tibble(code=paste0("S",320:329)) %>% group_by(code) %>% summarise(explain_code(code))
explain_code("S32")
explain_code("S321") # sacrum
explain_code("S322") # coccyx


# Traumatic pneumohemothorax
explain_code("860")
find_icd10cm_equivalent("860")

# heart or lung inj
explain_code("861")
find_icd10cm_equivalent("861")
explain_code("S26")
explain_code("S273")
explain_code("S274")
explain_code("S275")
explain_code("S276")

## I just did the rest by fly here, except for a few more...
## 2nd hand smoke - this actually exists but you need to find it by keyword
gem_combined %>% filter(str_detect(icd9cm,"^E8694")) %>% mutate(icd10cm=str_sub(icd10cm,1,4)) %>% select(icd10cm) %>% distinct
## 
gem_combined %>% filter(str_detect(icd9cm,"^E9800")) %>% mutate(icd10cm=str_sub(icd10cm,1,4)) %>% select(icd10cm) %>% distinct

## Neurologic hereditary, degenerative, and other disorders
map_df(
  .x = paste0(330:344),
  .f = ~find_icd10cm_equivalent(.x)
) %>% arrange(icd10cm) %>% print(n=nrow(.))
```

## E988

The 4th digit categories in ICD-9-CM were:

`r tibble(icd9cm=paste0("E",9880:9889)) %>% group_by(icd9cm) %>% mutate(meaning=explain_code(icd9cm))`

* jumping: `Y30, Y31`
* burns or fire: `Y26, Y27`
* scald: `Y27`
* cold exposure: none
* electrocution: none
* crashing motor vehicle: `Y32`
* crashing aircraft: none
* caustic substances except poisoning: none
* other specified means: ?
* unspecified means: `Y33`


For finding equivalents to the ICD-9-CM code E988, `E988 Injury by other and unspecified means, undetermined whether accidentally or purposely inflicted`, I consulted the tabular ICD-10-CM list available on the CDC website for keyword searching; beyond the sections above, I only found the following general code:

* Y33 Other specified events, undetermined intent

In conclusion, the codes are then `Y26, Y27, Y30, Y31, Y32, Y33`.

## Defining the translations to ICD-10-CM

Below I've used the `tibble:tribble()` function to define the tables of translations by hand, in combination with the `datapasta::tribble_paste()` function which pastes a requested table into a format for hand editing.

```{r}
translations_icd10cm <- list()

# the tribble below was made by using datapasta::tribble_paste()
# datapasta::tribble_paste(input_table = codes_translated$inclusions %>% select(condition, icd9cm) %>% mutate(icd10cm = "", comment_icd10cm = ""))
# datapasta::tribble_paste(input_table = inclusions_icd10cm %>% mutate_at(vars(icd10cm,comment_icd10cm), ~if_else(is.na(.),"",.)))
translations_icd10cm$inclusions <- tibble::tribble(
                                      ~condition,              ~icd9cm, ~icd10cm, ~comment_icd10cm,
                                "Genital herpes",              "054.1",       "A60",            "",
                          "Gonococcal infection",                "098",       "A54",            "",
      "Pelvic inflammatory disease, unspecified",              "614.9",       "N73.9",          "not sure the same conditions were grouped as unspecified between ICD-9-CM and ICD-10-CM",
                   "Contusion of genital organs",              "922.4",       "S30.2",          "",
                "Observation after alleged rape",              "V71.5",       "Z04.4",          "",
                 "Observation for abuse/neglect",             "V71.81",       "Z04.7",          "No equivalent in ICD-10; ICD-10-CM specifies 'physical abuse', whereas ICD-9-CM specifies 'suspected abuse & neglect'.",
                            "Retinal hemorrhage",             "362.81",       "H35.6",          "No equivalent in ICD-9;",
                                  "Rib fracture",       "807.0, 807.1",       "S22.3, S22.4",   "ICD-10-CM doesn't specify open or closed fracture unlike ICD-9-CM.",
                              "Scapula fracture",                "811",       "S42.1",           "",
                 "Traumatic subdural hemorrhage",              "852.2",       "S06.5",            "ICD-9-CM specifies 'no mention of open intracranial wound'; ICD-10-CM doesn't make that distinction.",
     "Other/unspecified intracranial hemorrhage",              "853.0",       "S06.8",               NA,
                                "Stomach injury",              "863.1",       "S36.3",               "ICD-9-CM specified open wound into cavity; I've useda the more general ICD-10-CM code for stomach injury.",
                                       "Assault", "E965, E966, E968.2",       "X93-X96, X99, Y00",               "",
                                  "Assault, NOS",             "E968.9",       "Y09",               "",
              "Undetermined intent, other means",               "E988",       "Y26, Y27, Y30, Y31, Y32, Y33",               "Several modalities of injury that were coded as 'other or unspecified' in ICD-9-CM have their own 3-character codes in ICD-10-CM",
                          "Skull vault fracture",                "800",       "S02.0",          NA,
                            "Vertebral fracture",                "805",       "S12, S22.0",               "Note: the description in Schnitzer et al. (2011) says 'Vertebral fracture', even though the provided Code ICD-9-CM 805 is for cervical vertebral fractures specifically. I interpreted this going by description so thoracic vertebrae are included; the later inclusion for Pelvic fracture covers lumbar vertebrae in the same spirit.",
             "Traumatic subarachnoid hemorrhage",              "852.0",       "S06.6",               "",
                     "Intrathoracic injury, NEC",                "862",       "S27.8, S27.9",               NA,
                        "Small intestine injury",       "863.2, 863.3",       "S36.4",               NA,
                                 "Spleen injury",                "865",       "S36.0",               NA,
                            "Spinal cord injury",                "952",       "S14.0, S14.1, S24.0, S24.1, S34.0, S34.1, S34.3",               "In ICD-9-CM, spinal cord injury locations are specified at 4th digit level; in ICD-10-CM they are specified with separate 3-character codes, grouped by body systems.",
                     "Other severe malnutrition",                "262",       "E43",               NA,
                                 "Dental caries",              "521.0",       "K02",               NA,
                    "Solar radiation dermatitis",              "692.7",       "L57.8",               NA,
                               "Pelvic fracture",                "808",       "S32",               "ICD-10-CM includes lumbar vertebrae with pelvic fractures, this is included to go with the broad interpertation of the above code for vertebral fractures.",
                    "Traumatic pneumohemothorax",                "860",       "S27.0, S27.1, S27.2",               "The description given in Schnitzer et al. (2011) says 'Traumatic pneumothorax', but the provided ICD-9-CM code includes hemothorax also, so this was included.",
                          "Heart or lung injury",                "861",       "S26, S27.3, S27.4, S27.5, S27.6",               "ICD-10 groups lung injuries and intrathoracic injuries together, so only lung-related codes in S27 are included.",
                                "GI injury, NEC",              "863.8",       "S36.2, S36.8, S36.9",               "ICD-9 separately specifies 4th digit codes for stomach, small intenstine, colon or rectum, so to match the Other code we include ICD-10 codes for gastrointestinal injuries excluded by those
",
                                  "Liver injury",                "864",       "S36.1",               NA,
                                 "Kidney injury",                "866",       "S37.0",               NA,
                                  "Burn of head",                "941",       "T20",               NA,
                                 "Burn of trunk",                "942",       "T21",               NA,
                                   "Burn of leg",                "945",       "T24, T25",               "ICD-10-CM treats ankle and foot as a separate code so that's included",
                        "Burn of multiple sites",                "946",       NA,               "ICD-10-CM denotes multiple burn sites as a 6th digit, specifying for each individual burn site that there are other burn sites present; These cases are already covered by the other codes denoting burns!",
                 "Poisoning by drugs/medicinals",            "960-979",       "T36-T50a",               "T36-T50 in ICD-10-CM incldues undetermined intent codes, which were specified by Schnitzer et al. 2011 but with different exclusion codes (see below); therefore we need to exclude 6th character code '4' to exclude undetermined intent poisoning!",
                "Drowning, non-fatal submersion",              "994.1",       "T75.1",               NA,
                     "Second-hand tobacco smoke",             "E869.4",       "Z77.22",               "Different code to ICD-10!",
                             "Swimming accident",             "E910.2",       "W68, W69",               NA,
                       "Bathtub (near) drowning",             "E910.4",       "W65",               NA,
                         "Other (near) drowning",             "E910.8",       "W73",               NA,
               "Accidental (near) drowning, NOS",             "E910.9",       "W74",               NA,
                          "Unarmed fight, brawl",             "E960.0",       "Y04",               "Y04 in ICD-10-CM is slightly more widely defined than E960.0 in ICD-9-CM but it conceptually matches the description.",
                "Undetermined intent, poisoning",               "E980",       "T36-T50b",               "Undetermined intent in posioning codes in ICD-10-CM is denoted by 6th character '4' in poisoning codes T36-T50, e.g. T36.0X4",
                  "Undetermined intent, firearm",               "E985",       "Y22, Y23, Y24, Y25",               NA,
                       "Household circumstances",                "V60",       "Z59",               NA
     )

translations_icd10cm$exclusions <-
# datapasta::tribble_paste(input_table = codes_translated$exclusions %>% select(icd9cm) %>% mutate(icd10cm = "", comment_icd10cm = "") %>% distinct)
tibble::tribble(
                     ~icd9cm, ~icd10cm, ~comment_icd10cm,
                       "767",       "P10-P15, P52.4, P52.6, P52.8, P52.9", "ICD-10 codes for Birth trauma plus several codes for non-traumatic brain haemorrhages that were included in ICD-9-CM 767 birth trauma due to hypoxia/anoxia (but excluding intraventricular & subarachnoid haemorrhages which werent included in ICD-9-CM code 767)",
                       "765",       "P07",               "Disorders relating to short gestation and low birthweight",
                     "771.2",       "P352",               "Congenital herpes simplex",
                     "098.4",       "A543",               "Gonococcal infection of eye",
                     "771.6",       "P391",               "Neonatal conjunctivitis and dacryocystitis",
                    "756.51",       "Q780",               "Osteogenesis imperfecta",
                    "E960.1",       "T74.2",               "ICD-9-CM E960.1 Rape mapped to T74.2 Sexual abuse",
                    "E968.4",       "Y07, X58",             "ICD-9-CM 'Assault by criminal neglect' maps to ICD-10-CM code X58 Exposure to other specified factors (which is used for neglect); also Y07 is for denoting the perpetrator of assault, maltreatment and neglect",
                   "286–287",       "D65-D69",               "Coagulation, purpura, & haemorrhagic disorders",
                 "E800–E819",       "V01-V99",               "MVA - motor vehicle accidents; these are defined more broadly in ICD-10-CM ('Transport accidents') than ICD-9-CM but that's acceptable.",
                 "E890–E897",       "X00-X08",               "ICD-9-CM ACCIDENTS CAUSED BY FIRE AND FLAMES (but excluding Other & NOS codes) mapped to ICD-10-CM Exposure to smoke, fire and flames",
                 "E870–E876",       "Y62-Y69",               "Misadventures during medical care",
             "733.10–733.19",       "M84.4, M84.5, M84.6, M80.0",               "This is specified as a range but captures entire range of ICD-9-CM 733.1 for Pathological fracture; this includes M80.0 for Age-related osteoporosis with current pathological fracture",
     "E810–E813, E815–E819b",       "V20-V99",               "ICD-9 MOTOR VEHICLE TRAFFIC ACCIDENTS (E810-E819) excluding cases where child was pedestrian or cyclist (.6 or .7); ICD-10 transport accidents except ones where victim was pedestrian or pedal cyclist (i.e. V01-V19); this includes motorcycle rider injuries V20-V29 - these should be considered as indicative of neglect for children <10 y.o.!"
     )



translations_icd10cm$malnutrition_exclusions <-
# datapasta::tribble_paste(input_table = codes_translated$malnutrition_exclusions %>% select(condition, icd9cm) %>% mutate(icd10cm = "", comment_icd10cm = "") %>% distinct)
tibble::tribble(
                                                                               ~condition,         ~icd9cm, ~icd10cm, ~comment_icd10cm,
                                               "Infections of the gastrointestinal tract",         "009.0",       "A09",     "Infectious gastroenteritis and colitis, unspecified",
                                                                           "Tuberculosis",   "010.0–018.9",       "A15-A19",  "A15-A19 Tubercolosis",
                                                                            "HIV disease",           "042",       "B20", 	    "B20 HIV",
                                                                        "Viral hepatitis",  "070.00–070.9",       "B15-B19",  "B15-B19 Viral hepatitis",
                                                                             "Malignancy",  "140.0–208.91",       "C00-C96, C7A, C7B",        "Covers entire C__ range of malignant neoplasms",
                                                                         "Hypothyroidism",     "243–244.9",       "E00-E03, E89.0",   "E00-E03 congenital & acquired hypothyroidism; E89.0 postprocedural hypothyroidism",
                                                                      "Diabetes mellitus", "250.00–250.93",       "E08-E13",   "E08-E13 Diabetes mellitus",
                                                                  "Parathyroid disorders",   "252.0–252.9",       "E20, E21, E89.2",  "E20-E21 Parathyroid disorders; E89.0 Postprocedural hypoparathyroidism",
                          "Disorders of the pituitary gland and its hypothalamic control",   "253.0–253.9",       "E22, E23, E89.3",               "E22 Hyperfunction of pituitary gland; E23 Hypofunction and other disorders of pituitary gland;	E89.2 Postprocedural hypoparathyroidism",
                                                            "Inborn errors of metabolism",   "270.0–275.9",       "E70-E88, D89, M10",   "E70-E90 Metabolic disorders; However, there's matches in other categories; D89 Other disorders involving the immune mechanism, not elsewhere classified; M10 Gout; E20.1 Pseudohypoparathyroidism already covered above",
                                                                    "Lactose intolerance",         "271.3",       "E73",               NA,
                                                                        "Cystic fibrosis",        "277.00",       "E84",               NA,
                                                                     "Mental retardation",       "317–319",       "F70-F79",         "F70-F79 Intellectual disabilities",
                               "Neurologic hereditary, degenerative, and other disorders",  "330.0–344.42",       "G00-G99, F84.2, R52",               "this range is composed of: HEREDITARY AND DEGENERATIVE DISEASES OF THE CENTRAL NERVOUS SYSTEM (330-337);  PAIN (338); OTHER HEADACHE SYNDROMES (339); OTHER DISORDERS OF THE CENTRAL NERVOUS SYSTEM (340-349) except 348-349, which covers Other conditions of brain and 349 Other and unspecified disorders of the nervous system; not clear why those were excluded - we'll cover the entire G00-G99 range; also matches R52 Pain and F84.2 Rett's syndrome; there are also a range of E75XX codes that are already captured above",
                                                               "Intracerebral hemorrhage",           "431",       "I61",               NA,
                                             "Polyarteritis nodosa and allied conditions",   "446.0–446.7",       "M30, M31",	"Polyarteritis nodosa and allied conditions to ICD-10 M30 Polyarteritis nodosa and related conditions and M31 Other necrotising vasculopathies",
                                                                                 "Asthma", "493.00–493.92",       "J45",               NA,
                                                                "Gastroesophageal reflux",        "530.81",       "K21",               NA,
                                                             "Inflammatory bowel disease",   "555.0–558.9",       "K50-K52",               "Inflammatory bowel disease: NONINFECTIOUS ENTERITIS AND COLITIS (555-558) - Noninfective enteritis and colitis (K50-K52)",
                                                                        "Biliary disease",   "575.0–576.9",       "K82-K83",               "K82 Other diseases of gallbladder; K83 Other diseases of biliary tract",
                                                                              "Cirrhosis",   "571.0–571.9",       "K70-K76",               "K70-K76 Diseases of liver (K77 excluded - diseases of liver classified elsewhere)",
                                                               "Pancreatic insufficiency",         "577.8",       "K86.8",               NA,
                                                               "Intestinal malabsorption",   "579.0–579.9",       "K90, K91",               "K90 intestinal malabsorption; K91 Postprocedural disorders of digestive system, not elsewhere classified",
                                                                 "Renal tubular acidosis",         "588.8",       "N25.8",               NA,
                                                            "Chronic renal insufficiency",         "593.9",       "N18, N19, N28.9",               "Schnitzer specified Chronic renal insufficiency, but listed code 593.9 Disorders of kidney and ureter, unspecified; I'm assuming they mean the former, which maps to N18 & possibly N19, but we'll also include N28.9 Disorder of kidney and ureter, unspecified",
                                                                "Urinary tract infection",         "599.0",       "N39.0",               NA,
                                                  "Diffuse diseases of connective tissue",   "710.0–710.9",       "M32-M36",               "M32-M36  Systemic connective tissue disorders: based on matching ICD-9-CM code headings to ICD-10-CM & regex_prefixes generated via GEMs",
                          "Rheumatoid arthritis and other inflammatory polyarthropathies",   "714.0–714.9",       "M05, M06, M08, M12.0",               "based on GEMs, we also include M12.0 Chronic postrheumatic arthropathy",
                                                            "Cardiac disease, congenital",   "745.0–747.9",       "Q20-Q28, P29.3",               "Q20-Q28 Congenital malformations of the circulatory system; also includes P29.3 Persistent foetal circulation",
                                                                 "Cleft palate/cleft lip", "749.00–749.25",       "Q35, Q36, Q37",               NA,
                                                                       "Pyloric stenosis",         "750.5",       "Q40.0",               NA,
                                                                 "Hirschsprung's disease",         "751.3",       "Q43.1",               NA,
                                                              "Chromosomal abnormalities",   "758.0–758.9",       "Q90-Q99",               "Q90-Q99 	Chromosomal abnormalities, not elsewhere classified",
                                                                 "Fetal alcohol syndrome",        "760.71",       "Q86.0, P04.3",               "Q86.0 Fetal alcohol syndrome (dysmorphic); P04.3 Fetus and newborn affected by maternal use of alcohol",
                                             "Birth trauma: subdural/cerebral hemorrhage",         "767.0",       "P10, P11.0, P11.2, P11.9, P52",               "GEM based: P10 Intracranial laceration and haemorrhage due to birth injury;  P11.0 Cerebral edema due to birth injury, P11.1 Other specified brain damage due to birth injury'; P11.2 Unspecified brain damage due to birth injury,  P11.9 Birth injury to central nervous system, unspecified,  P52 Intracranial nontraumatic hemorrhage of newborn",
                                                             "Bronchopulmonary dysplasia",         "770.7",       "P27.1",               "Although the ICD-9-CM code is specified more widely (Chronic respiratory disease arising in the perinatal period), I've taken the more specific description for P27.1 Bronchopulmonary dysplasia originating in the perinatal period",
                                                                   "Perinatal infections",  "771.0–771.89",       "P35-P39",               NA,
                                                            "Intraventricular hemorrhage", "772.10–772.14",       "P10, P52",               "This is already covered above but I've spelled it out anyway for clarity!",
     "Subarachnoid, subdural, extradural, other/unspecified, hemorrhage following injury", "852.00–853.19",       "S063-S068",               "I've interpreted this to mean any traumatic intracranial hemorrhage, and these are roughly covered in S063-S068",
                                                                         "Lead poisoning",   "984.0–984.9",       "T56.0",               NA
     )

translations_icd10cm$additional_requirements <-
# datapasta::tribble_paste(input_table = codes_translated$additional_requirements %>% select(icd9cm) %>% mutate(icd10cm = "", comment_icd10cm = "") %>% distinct)
tibble::tribble(
            ~icd9cm, ~icd10cm, ~comment_icd10cm,
      "480.0–487.8",       "J10-J18",               "pneumonia & influenza (not avian) - I excluded J09 as that's what avian influenza falls under",
      "490.0–491.9",       "J40, J41, J42",               "chronic bronchitis & bronchitis not specified as acute or chronic",
     "466.0–466.19",       "J20, J21, J68",               "Acute bronchitis and bronchiolitis; added ICD-10 J68 respiratory conditions due to exposure to chemicals",
      "493.0–493.9",       "J45",               "Asthma",
      "381.0–381.4",       "H65, H66",               "Otitis media was specified in Schnitzer et al., 2011, so that's what I searched for in ICD-10-CM"
     )
```

# Compiling translated codes

The above listed codes are specified in a compact format, using code ranges such as C00-C97, for example. In addition to that, some codes are excluded on the basis of having a particular 6th character code in ICD-10. First these will be combined with the ICD-9-CM to ICD-9 and ICD-10 translation to provide a complete, compact listing of translated codes; next, the codes will be resolved so that individual codes are available for algorithmic use.


## Compact listing

```{r}
codes_translated$inclusions <-
  codes_translated$inclusions %>%
  left_join(schnitzer_age_requirements, by="inclusion_index") %>%
  left_join(schnitzer_code_categories, by="inclusion_index")

codes_translated_including_icd10cm <- 
  imap(
    .x = codes_translated,
    .f = ~left_join(x = .x, y = translations_icd10cm[[.y]])
  )
## save as excel file
openxlsx::write.xlsx(x = codes_translated_including_icd10cm, file = "./processed_ICD_codes/Schnitzer_et_al_2011_ICD_crossmapping_including_ICD-10-CM.xlsx")
```


## Resolving codes listed as ranges to individual codes

To resolve a code range like `X20-X30`, we need to interpret the range and produce individual codes, e.g. `X20, X21, X22, ..., X30`.

```{r}
## the easiest way to specify this function is to write a function that separates a single string "X20-X30" to "X20,X21,X22,...";
## then we use purrr::map_chr() to apply this function to all codes
separate_code_ranges_icd10cm <- function(code_range) {
  separate_single_code_range <- function(code_range) {
    first_character <- toupper(str_sub(code_range, 1, 1))  # extract letter at start of code
    numbers <- parse_number(str_split(code_range, pattern="\\-", simplify = TRUE))
    numbers_interpolated <- seq(from=numbers[1], to=numbers[2], by=1)
    numbers_padded <- str_pad(numbers_interpolated, width = 2, pad = "0")  # pad single-digit numbers so they always have a leading zero
    comma_separated_string <- paste0(first_character, numbers_padded, collapse = ",")
    return(comma_separated_string)
  }
  map_chr(
    .x = code_range,
    .f = function(this_code_range) {
      if (is.na(this_code_range)) return(this_code_range)  # there were some codes that didn't exist, so this is here to not break the function if NAs provided
      
      ## run the separating function if dash present in code, and return unchanged code if not
      if (str_detect(string = this_code_range, pattern = "\\-")) {
        return(separate_single_code_range(this_code_range))
      } else {
        return(this_code_range)
      }
    }
  )
}
## test that the function works as expected...
test_codes <- c("A22","F10-F12","X23.4","Y00-Y04")
expected_separated_codes <- c("A22","F10,F11,F12","X23.4","Y00,Y01,Y02,Y03,Y04")
stopifnot(separate_code_ranges_icd10cm(test_codes)==expected_separated_codes)
```

On to resolving the codes!

I'm keeping the original code in the column `icd10cm` below, and the parsed/resolved codes in `parsed_codes`, for manual review.

```{r}
codes_resolved <- list()

codes_resolved$inclusions <-
  codes_translated_including_icd10cm$inclusions %>%
  select(inclusion_index, icd10cm) %>%
  drop_na(icd10cm) %>%
  mutate(
    undetermined_intent_exclude_flag = str_detect(icd10cm, pattern="a$"),  # detect lowercase a/b at end of code
    undetermined_intent_include_flag = str_detect(icd10cm, pattern="b$")
    ) %>%  # add a flag to undetermined intent codes where we'll need to specify a particular character beyond the 3 characters in the code specified for inclusions
  separate_rows(icd10cm, sep=",") %>%  # separate comma-separated codes into separate rows
  mutate(across(icd10cm, ~trimws(.))) %>%  # remove whitespace before & after codes
  mutate(parsed_codes = separate_code_ranges_icd10cm(icd10cm)) %>%
  separate_rows(parsed_codes, sep=",") %>%
  mutate(
    parsed_codes = str_remove_all(parsed_codes, pattern="\\."),  # remove dot
    regex_prefix = paste0("^",parsed_codes),
    regex_prefix = if_else(condition = undetermined_intent_include_flag, true = paste0(regex_prefix,"..4"), false = regex_prefix), # T36.0X4 or T360X4
    regex_prefix = if_else(condition = undetermined_intent_exclude_flag, true = paste0(regex_prefix,"..[^4]"), false = regex_prefix) # exclude T36.0X4 or T360X4
    ) %>%
  distinct

test_tibble_undetermined_intent <- tibble(code = c("T36015", "T36014231", "T3604"))  # the last code is misspecified - not enough characters!
test_tibble_undetermined_intent %>% fuzzyjoin::regex_left_join(y = codes_resolved$inclusions, by=c("code"="regex_prefix"))

codes_resolved$exclusions <-
  codes_translated_including_icd10cm$exclusions %>%
  select(inclusion_index, exclusion_index, icd10cm) %>%
  drop_na(icd10cm) %>%
  separate_rows(icd10cm, sep=",") %>%  # separate comma-separated codes into separate rows
  mutate(across(icd10cm, ~trimws(.))) %>%  # remove whitespace before & after codes
  mutate(parsed_codes = separate_code_ranges_icd10cm(icd10cm)) %>%
  separate_rows(parsed_codes, sep=",") %>%
  distinct

codes_resolved$malnutrition_exclusions <-
  codes_translated_including_icd10cm$malnutrition_exclusions %>%
  select(malnutrition_index, icd10cm) %>%
  drop_na(icd10cm) %>%
  separate_rows(icd10cm, sep=",") %>%  # separate comma-separated codes into separate rows
  mutate(across(icd10cm, ~trimws(.))) %>%  # remove whitespace before & after codes
  mutate(parsed_codes = separate_code_ranges_icd10cm(icd10cm)) %>%
  separate_rows(parsed_codes, sep=",") %>%
  distinct

codes_resolved$all_exclusions <-
  bind_rows(
    codes_resolved$exclusions,
    codes_resolved$malnutrition_exclusions %>%
      select(-malnutrition_index) %>%
      mutate(
        exclusion_index = max(codes_resolved$exclusions$exclusion_index)+1,
        inclusion_index = codes_resolved$inclusions %>% filter(parsed_codes=="E43") %>% .$inclusion_index
      )
  ) %>%
  mutate(regex_prefix = paste0("^",str_remove_all(parsed_codes, pattern="\\.")))

codes_resolved$additional_requirements <-
  codes_translated_including_icd10cm$additional_requirements %>%
  select(inclusion_index, icd10cm) %>%
  drop_na(icd10cm) %>%
  separate_rows(icd10cm, sep=",") %>%  # separate comma-separated codes into separate rows
  mutate(across(icd10cm, ~trimws(.))) %>%  # remove whitespace before & after codes
  mutate(parsed_codes = separate_code_ranges_icd10cm(icd10cm)) %>%
  separate_rows(parsed_codes, sep=",") %>%
  distinct %>%
  mutate(regex_prefix = paste0("^",str_remove_all(parsed_codes, pattern="\\.")))
```

## Cleaning up codes

Finally, we only keep the minimal information needed - the inclusion/exclusion codes, the regular expression prefix used for matching codes, and the age requirements, and the indexes that tell us what inclusion codes go with exclusion codes or additional inclusions.

```{r}
codes_cleaned <- list()

codes_cleaned$inclusions <-
  codes_resolved$inclusions %>%
  select(inclusion_index, inclusion_code = icd10cm, regex_prefix) %>%
  left_join(schnitzer_age_requirements, by = "inclusion_index") %>%
  left_join(schnitzer_code_categories, by = "inclusion_index")

codes_cleaned$all_exclusions <-
  codes_resolved$all_exclusions %>%
  select(inclusion_index, exclusion_index, exclusion_code = icd10cm, regex_prefix)

codes_cleaned$additional_requirements <-
  codes_resolved$additional_requirements %>%
  select(inclusion_index, additional_inclusion_code = icd10cm, regex_prefix)
```

## Save codes

```{r}
iwalk(
  .x = codes_cleaned,
  .f = function(x,name) {
    filepath = paste0("./processed_ICD_codes/codes_schnitzer2011_icd10cm_",name,".csv")
    write_csv(x = x, path = filepath)
  }
)

## also save in this second location:
if (!dir.exists("../Collaborations/translated_to_icd10cm/codes_translated_to_icd10cm")) dir.create("../Collaborations/translated_to_icd10cm/codes_translated_to_icd10cm")

iwalk(
  .x = codes_cleaned,
  .f = function(x,name) {
    filepath = paste0("../Collaborations/translated_to_icd10cm/codes_translated_to_icd10cm/codes_schnitzer2011_icd10cm_",name,".csv")
    write_csv(x = x, path = filepath)
  }
)
```