Blog: Artificial intelligence and cancer – Fred Hutch News Service
Using AI to guide cancer treatment
Goulart’s team, which included the Hutch’s Drs. Steve Schwartz, Scott Ramsey, Christina Baik and researchers from the University of Kentucky, worked with Hutch NLP engineer Emily Silgard to develop and train an algorithm to plumb the Washington state and Kentucky cancer registries for metastatic lung cancer patients diagnosed and tested for the two mutations between 2011 and 2013.
Cancer patients’ electronic health records, including tumor pathology reports, are customarily fed into state cancer registries that go on to form the National Cancer Institute’s Surveillance, Epidemiology and End Results Program, known as SEER. The SEER database, which serves as a proxy for the U.S. cancer patient population, provides valuable information to researchers on cancer incidence and mortality, as well as patient demographics, tumor characteristics and treatment.
NCI and SEER, which funded the Hutch study, hope to improve the registry by tracking and recording genomic biomarkers, such as EGFR and ALK, crucial information patients and doctors use to make treatment decisions (Hutch researchers are also working with NCI to fill in the recurrence data gaps). Manual abstraction of genetic data is time-consuming and labor intensive. Automating the process could make it more efficient and cost-effective and bump up the value of the national database.
“SEER and the NCI are trying to develop ways to report on important molecular markers in cancer efficiently. One of the questions is, ‘Can they use NLP for that?’” Goulart said. “The ultimate goal is to have a way to identify patients treated in real world scenarios and clinics whose lung cancers carry those mutations.”
After training the NLP algorithms, the research team sent it digging through the electronic pathology reports of Washington’s Cancer Surveillance System, or CSS, housed at Fred Hutch, and the Kentucky Cancer Registry, housed at the University of Kentucky.
If a test for the mutations was found, the NLP algorithm reported whether the patient tested positive or negative. It also reported on the type of test that was used to determine the mutation.
Mutation tracking smackdown
Then the researchers pitted man against machine. Two practicing oncologists dove in and read the records for each of the qualifying cancer patients: 1,634 from Washington and 565 from Kentucky.
“They read all the pathology reports from two registries and generated a gold standard ‘internal validation data set’ of what these EGFR and ALK results were when available,” said Goulart. “Then we had the NLP run the same reports and show their results and compared them with the manually extracted data.”
As it turned out, the NLP algorithm did quite well against actual oncologists in the CSS registry housed at Fred Hutch. On the EGFR mutation, the algorithm scored 97 percent on sensitivity and 98 percent on precision. On ALK, it scored 95 percent on sensitivity and 100 percent on precision. The NLP did not perform as well with the Kentucky registry, however, scoring 29 percent on sensitivity and 48 percent on precision for EGFR mutations and 100 percent on sensitivity and 2 percent on precision for ALK mutations.
The algorithm also identified the type of genetic test used with patients: Mutational analysis was the standard for EGFR testing and FISH, or fluorescence in situ hybridization, was the standard for determining an ALK mutation.
“The appropriate test technique is key for EGFR therapies to work,” said Goulart. “The NLP accurately detects EGFR by mutational analysis and ALK by all techniques, particularly FISH, which is by far the most common ALK test used in practice.”