Evaluation metrics LRGASP – Challenge 1

Your submission: mouse_simulation_pb on PacBio data

Background

Challenge 1 is evaluated according to four criteria:

1. Broad GENCODE Annotation
2. Subset of manually curated loci selected by GENCODE
3. sim Lexogen Set 4
4. Simulated data.

The LRGASP uses SQANTI categories to define evaluating features and metrics for Challenge 1.

LRGASP Challenge 1 Definitions:

• Full Splice Match (FSM): Transcripts matching a reference transcript at all splice junctions.
• Incomplete Splice Match (ISM): Transcripts matching consecutive, but not all, splice junctions of the reference transcripts.
• Novel in Catalog (NIC): Transcripts containing new combinations of already annotated splice junctions or novel splice junctions formed from already annotated donors and acceptors.
• Novel Not in Catalog (NNC): Transcripts using novel donors and/or acceptors.
• Reference Match (RM): FSM transcript with 5?? and 3??ends within 50 nts of the TSS/TTS annotation.
• 3?? polyA supported: Transcript with polyA motif support at the 3??end.
• 5?? CAGE supported: Transcript with CAGE support at the 5??end.
• 3?? reference supported: Transcript with 3??end within 50 nts from reference transcript or gene TTS.
• 5?? reference supported: Transcript with 5??end within 50 nts from reference transcript or gene TSS.
• Supported Reference Transcript Model (SRTM): FSM/ISM transcript with 5?? end within 50nts of the TSS or has CAGE support AND 3?? end within 50nts of the TTS or has polyA motif support
• Supported Novel Transcript Model (SNTM): NIC/NNC transcript with 5?? end within 50nts of the TSS or CAGE support AND 3?? end within 50nts of the TTS or polyA motif support AND Illumina read support at novel junctions
• Redundancy: # LR transcript models / reference model
• Intron retention (IR): level Number of IR within the NIC category
• Illumina Splice Junction (SJ): Support % SJ in transcript model with Illumina support
• Full Illumina Splice Junction Support: % transcripts in category with all SJ supported
• % Novel Junctions: # of new junctions / total # junctions
• % Non-canonical junctions: # of non-canonical junctions / total # junctions
• % Non-canonical transcripts % transcripts with at least one nc junction
• Intra-priming: Evidence of intra-priming (see SQANTI ref)
• RT-switching: Evidence of RT-switching (see SQANTI ref)

This document shows the performance of your pipeline for criteria 4. Critical data for evaluation according to 2. and 4. will be made available after the closure of the challenge, and therefore pre-evaluation reports cannot be provided. Note you???re your criterion 1 metrics reported here have been calculated using GENCODE human v38 and mouse M27 releases while the final evaluation will use human v39 and mouse M28 to be released after completing of the challenge.

Evaluation of detected transcripts for Challenge 1

Global overview

Value Number of genes detected 41429 Number of known genes detected 15102 Number of transcripts detected 158641 Number of transcripts associated to a known gene 46330 Number of unique SJ detected 498755

Absolute value Relative value (%) Novel SJ 320553 0.64 Non-canonical SJ 333508 0.67

Evaluation of FSM

Absolute value Relative value (%) Number of isoforms 23026 - Reference Match 22277 96.75 5’ reference supported (transcript) 22569 98.02 3’ reference supported (transcript) 22614 98.21 5’ reference supported (gene) 22675 98.48 3’ reference supported (gene) 22721 98.68 Supported Reference Transcript Model (SRTM) 22443 97.47 Reference redundancy Level 1.01 -

Evaluation of ISM

Absolute value Relative value (%) Number of isoforms 1424 - 5’ reference supported (transcript) 263 18.47 3’ reference supported (transcript) 1062 74.58 5’ and 3’ reference supported (gene) 158 11.1 5’ reference supported (gene) 424 424 3’ reference supported (gene) 1118 78.51 Supported Reference Transcript Model (SRTM) 158 11.1 Reference redundancy Level 1.04 -

Evaluation NIC

Absolute value Relative value (%) Number of isoforms 232 - 5’ and 3’ reference supported (gene) 124 53.45 5’ reference supported (gene) 139 59.91 3’ reference supported (gene) 140 60.34 Intron retention incidence 42 18.1

Evaluation NNC

Absolute value Relative value (%) Number of isoforms 21648 - 5’ and 3’ reference supported (gene) 1807 8.35 5’ reference supported (gene) 2917 13.47 3’ reference supported (gene) 2096 9.68 Non-canonical SJ incidence 20816 96.16 Full Illumina SJ support 21648 100 RT-switching incidence 1367 6.31

Evaluation of Simulation

Simulated transcripts were grouped according to different thresholds and attributes, so metrics were calculated regarding to these ground truth setttings. These sets of ground truth transcripts are:

• All simulated transcripts (Using GENCODE + novelty as reference).
• Only GENCODE transcripts simulated.
• Only GENCODE transcripts simulated with at least 1 TPM expression.
• Only GENCODE transcripts simulated with at least 5 TPM expression.
• Only those transcripts that were added from a different organism (novelty).

The following metrics and definitions apply to simulated transcripts.

• Number of isoforms simulated: Total number of simulated isoforms taken into account as ground truth.
• True Positive detections (TP): Isoforms identified as Reference Match of a simulated transcript.
• Partial True Positive detections (PTP): Isoforms identified as ISM or FSM_non_RM
• False Negative (FN): Transcripts that were expected to be detected but they were not.
• False Positive (FP): Transcripts that were detected but they weren’t simulated.
• Sensitivity: TP/Number of isoforms simulated
• Precision: RM/Detected isoforms
• Non_redundant Precision: TP/ detected isoforms
• Positive Detection Rate: unique(TP+PTP)/Number of isoforms simulated
• False Discovery Rate: (FP + PTP)/Detected isoforms
• False Detection Rate: (FP)/Detected isoforms
• Redundancy: (Detected isoforms associated to simulated transcripts)/(Simulated transcripts detected)

Evaluation of all simulated transcripts

Value Number of isoforms simulated 27152 True Positive detections (TP) 22241 Number of transcripts associated to TP (Reference Match) 22244 Partial True Positive detections (PTP) 1885 Number of transcripts associated to PTP 1979 False Negative (FN) 4257 False Positive (FP) 134418 Sensitivity 0.82 Precision 0.14 Non Redundant Precision 0.14 Positive Detection Rate 0.84 False Discovery Rate 0.86 False Detection Rate 0.85 Redundancy 1.06

Evaluation of all GENCODE simulation

Value Number of isoforms simulated 20152 True Positive detections (TP) 16023 Number of transcripts associated to TP (Reference Match) 16023 Partial True Positive detections (PTP) 1158 Number of transcripts associated to PTP 1200 False Negative (FN) 3598 False Positive (FP) 141418 Sensitivity 0.8 Positive Detection Rate 0.82 Redundancy 1.04

Evaluation of only GENCODE transcripts simulated

Value Number of isoforms simulated 15608 True Positive detections (TP) 13539 Number of transcripts associated to TP (Reference Match) 13539 Partial True Positive detections (PTP) 845 Number of transcripts associated to PTP 880 False Negative (FN) 1804 False Positive (FP) 144222 Sensitivity 0.87 Positive Detection Rate 0.88 Redundancy 1.04

Evaluation of only GENCODE transcripts simulated with TPM >= 5

Value Number of isoforms simulated 9428 True Positive detections (TP) 8771 Number of transcripts associated to TP (Reference Match) 8771 Partial True Positive detections (PTP) 493 Number of transcripts associated to PTP 515 False Negative (FN) 591 False Positive (FP) 149355 Sensitivity 0.93 Positive Detection Rate 0.94 Redundancy 1.05

Evaluation of novelty

Value Number of isoforms simulated 7000 True Positive detections (TP) 6218 Number of transcripts associated to TP (Reference Match) 6221 Partial True Positive detections (PTP) 727 Number of transcripts associated to PTP 779 False Negative (FN) 659 False Positive (FP) 151641 Sensitivity 0.89 Positive Detection Rate 0.91 Redundancy 1.1