Evaluation metrics LRGASP – Challenge 1

Your submission: iso_detect_ref_spectra_human_simulation on PacBio data

Background

Challenge 1 is evaluated according to four criteria:

1. Broad GENCODE Annotation
2. Subset of manually curated loci selected by GENCODE
3. sim Lexogen Set 4
4. Simulated data.

The LRGASP uses SQANTI categories to define evaluating features and metrics for Challenge 1.

LRGASP Challenge 1 Definitions:

• Full Splice Match (FSM): Transcripts matching a reference transcript at all splice junctions.
• Incomplete Splice Match (ISM): Transcripts matching consecutive, but not all, splice junctions of the reference transcripts.
• Novel in Catalog (NIC): Transcripts containing new combinations of already annotated splice junctions or novel splice junctions formed from already annotated donors and acceptors.
• Novel Not in Catalog (NNC): Transcripts using novel donors and/or acceptors.
• Reference Match (RM): FSM transcript with 5?? and 3??ends within 50 nts of the TSS/TTS annotation.
• 3?? polyA supported: Transcript with polyA motif support at the 3??end.
• 5?? CAGE supported: Transcript with CAGE support at the 5??end.
• 3?? reference supported: Transcript with 3??end within 50 nts from reference transcript or gene TTS.
• 5?? reference supported: Transcript with 5??end within 50 nts from reference transcript or gene TSS.
• Supported Reference Transcript Model (SRTM): FSM/ISM transcript with 5?? end within 50nts of the TSS or has CAGE support AND 3?? end within 50nts of the TTS or has polyA motif support
• Supported Novel Transcript Model (SNTM): NIC/NNC transcript with 5?? end within 50nts of the TSS or CAGE support AND 3?? end within 50nts of the TTS or polyA motif support AND Illumina read support at novel junctions
• Redundancy: # LR transcript models / reference model
• Intron retention (IR): level Number of IR within the NIC category
• Illumina Splice Junction (SJ): Support % SJ in transcript model with Illumina support
• Full Illumina Splice Junction Support: % transcripts in category with all SJ supported
• % Novel Junctions: # of new junctions / total # junctions
• % Non-canonical junctions: # of non-canonical junctions / total # junctions
• % Non-canonical transcripts % transcripts with at least one nc junction
• Intra-priming: Evidence of intra-priming (see SQANTI ref)
• RT-switching: Evidence of RT-switching (see SQANTI ref)

This document shows the performance of your pipeline for criteria 4. Critical data for evaluation according to 2. and 4. will be made available after the closure of the challenge, and therefore pre-evaluation reports cannot be provided. Note you???re your criterion 1 metrics reported here have been calculated using GENCODE human v38 and mouse M27 releases while the final evaluation will use human v39 and mouse M28 to be released after completing of the challenge.

Evaluation of detected transcripts for Challenge 1

Global overview

Value Number of genes detected 34162 Number of known genes detected 22295 Number of transcripts detected 88816 Number of transcripts associated to a known gene 66845 Number of unique SJ detected 329914

Absolute value Relative value (%) Novel SJ 5095 0.02 Non-canonical SJ 140489 0.43

Evaluation of FSM

Absolute value Relative value (%) Number of isoforms 45797 - Reference Match 44606 97.4 5’ reference supported (transcript) 44824 97.88 3’ reference supported (transcript) 45379 99.09 5’ reference supported (gene) 45084 98.44 3’ reference supported (gene) 45725 99.84 Supported Reference Transcript Model (SRTM) 45043 98.35 Reference redundancy Level 1.02 -

Evaluation of ISM

Absolute value Relative value (%) Number of isoforms 9417 - 5’ reference supported (transcript) 255 2.71 3’ reference supported (transcript) 8612 91.45 5’ and 3’ reference supported (gene) 1371 14.56 5’ reference supported (gene) 1557 1557 3’ reference supported (gene) 9179 97.47 Supported Reference Transcript Model (SRTM) 1371 14.56 Reference redundancy Level 1.11 -

Evaluation NIC

Absolute value Relative value (%) Number of isoforms 6522 - 5’ and 3’ reference supported (gene) 1620 24.84 5’ reference supported (gene) 1822 27.94 3’ reference supported (gene) 2175 33.35 Intron retention incidence 1349 20.68

Evaluation NNC

Absolute value Relative value (%) Number of isoforms 5109 - 5’ and 3’ reference supported (gene) 4088 80.02 5’ reference supported (gene) 4255 83.28 3’ reference supported (gene) 4826 94.46 Non-canonical SJ incidence 3321 65 Full Illumina SJ support 5109 100 RT-switching incidence 934 18.28

Evaluation of Simulation

Simulated transcripts were grouped according to different thresholds and attributes, so metrics were calculated regarding to these ground truth setttings. These sets of ground truth transcripts are:

• All simulated transcripts (Using GENCODE + novelty as reference).
• Only GENCODE transcripts simulated.
• Only GENCODE transcripts simulated with at least 1 TPM expression.
• Only GENCODE transcripts simulated with at least 5 TPM expression.
• Only those transcripts that were added from a different organism (novelty).

The following metrics and definitions apply to simulated transcripts.

• Number of isoforms simulated: Total number of simulated isoforms taken into account as ground truth.
• True Positive detections (TP): Isoforms identified as Reference Match of a simulated transcript.
• Partial True Positive detections (PTP): Isoforms identified as ISM or FSM_non_RM
• False Negative (FN): Transcripts that were expected to be detected but they were not.
• False Positive (FP): Transcripts that were detected but they weren’t simulated.
• Sensitivity: TP/Number of isoforms simulated
• Precision: RM/Detected isoforms
• Non_redundant Precision: TP/ detected isoforms
• Positive Detection Rate: unique(TP+PTP)/Number of isoforms simulated
• False Discovery Rate: (FP + PTP)/Detected isoforms
• False Detection Rate: (FP)/Detected isoforms
• Redundancy: (Detected isoforms associated to simulated transcripts)/(Simulated transcripts detected)

Evaluation of all simulated transcripts

Value Number of isoforms simulated 48192 True Positive detections (TP) 44292 Number of transcripts associated to TP (Reference Match) 44580 Partial True Positive detections (PTP) 8858 Number of transcripts associated to PTP 10180 False Negative (FN) 3124 False Positive (FP) 34056 Sensitivity 0.92 Precision 0.5 Non Redundant Precision 0.5 Positive Detection Rate 0.94 False Discovery Rate 0.48 False Detection Rate 0.38 Redundancy 1.22

Evaluation of all GENCODE simulation

Value Number of isoforms simulated 41964 True Positive detections (TP) 39078 Number of transcripts associated to TP (Reference Match) 39112 Partial True Positive detections (PTP) 7276 Number of transcripts associated to PTP 8054 False Negative (FN) 2232 False Positive (FP) 41650 Sensitivity 0.93 Positive Detection Rate 0.95 Redundancy 1.19

Evaluation of only GENCODE transcripts simulated

Value Number of isoforms simulated 28414 True Positive detections (TP) 26607 Number of transcripts associated to TP (Reference Match) 26615 Partial True Positive detections (PTP) 5509 Number of transcripts associated to PTP 6172 False Negative (FN) 1370 False Positive (FP) 56029 Sensitivity 0.94 Positive Detection Rate 0.95 Redundancy 1.21

Evaluation of only GENCODE transcripts simulated with TPM >= 5

Value Number of isoforms simulated 16196 True Positive detections (TP) 15250 Number of transcripts associated to TP (Reference Match) 15252 Partial True Positive detections (PTP) 3303 Number of transcripts associated to PTP 3754 False Negative (FN) 683 False Positive (FP) 69810 Sensitivity 0.94 Positive Detection Rate 0.96 Redundancy 1.23

Evaluation of novelty

Value Number of isoforms simulated 6228 True Positive detections (TP) 5214 Number of transcripts associated to TP (Reference Match) 5468 Partial True Positive detections (PTP) 1582 Number of transcripts associated to PTP 2126 False Negative (FN) 892 False Positive (FP) 81222 Sensitivity 0.84 Positive Detection Rate 0.86 Redundancy 1.42