Your submission: ES_long_short on ONT+Illumina data
Background
Challenge 1 is evaluated according to four criteria:
- Broad GENCODE Annotation
- Subset of manually curated loci selected by GENCODE
- SIRV Lexogen Set 4
- Simulated data.
The LRGASP uses SQANTI categories to define evaluating features and metrics for Challenge 1.
LRGASP Challenge 1 Definitions:
- Full Splice Match (FSM): Transcripts matching a reference transcript at all splice junctions.
- Incomplete Splice Match (ISM): Transcripts matching consecutive, but not all, splice junctions of the reference transcripts.
- Novel in Catalog (NIC): Transcripts containing new combinations of already annotated splice junctions or novel splice junctions formed from already annotated donors and acceptors.
- Novel Not in Catalog (NNC): Transcripts using novel donors and/or acceptors.
- Reference Match (RM): FSM transcript with 5´ and 3´ends within 50 nts of the TSS/TTS annotation.
- 3´ polyA supported: Transcript with polyA motif support at the 3´end.
- 5´ CAGE supported: Transcript with CAGE support at the 5´end.
- 3´ reference supported: Transcript with 3´end within 50 nts from reference transcript or gene TTS.
- 5´ reference supported: Transcript with 5´end within 50 nts from reference transcript or gene TSS.
- Supported Reference Transcript Model (SRTM): FSM/ISM transcript with 5´ end within 50nts of the TSS or has CAGE support AND 3´ end within 50nts of the TTS or has polyA motif support
- Supported Novel Transcript Model (SNTM): NIC/NNC transcript with 5´ end within 50nts of the TSS or CAGE support AND 3´ end within 50nts of the TTS or polyA motif support AND Illumina read support at novel junctions
- Redundancy: # LR transcript models / reference model
- Intron retention (IR): level Number of IR within the NIC category
- Illumina Splice Junction (SJ): Support % SJ in transcript model with Illumina support
- Full Illumina Splice Junction Support: % transcripts in category with all SJ supported
- % Novel Junctions: # of new junctions / total # junctions
- % Non-canonical junctions: # of non-canonical junctions / total # junctions
- % Non-canonical transcripts % transcripts with at least one nc junction
- Intra-priming: Evidence of intra-priming (see SQANTI ref)
- RT-switching: Evidence of RT-switching (see SQANTI ref)
This document shows the performance of your pipeline for criteria 1. and 3. Critical data for evaluation according to 2. and 4. will be made available after the closure of the challenge, and therefore pre-evaluation reports cannot be provided. Note you’re your criterion 1 metrics reported here have been calculated using GENCODE human v38 and mouse M27 releases while the final evaluation will use human v39 and mouse M28 to be released after completing of the challenge.
Evaluation of detected transcripts for Challenge 1
Here, SIRVs are not taken into account for none of the metrics.
Global overview
| Number of genes detected |
18384 |
| Number of known genes detected |
13410 |
| Number of transcripts detected |
34605 |
| Number of transcripts associated to a known gene |
28884 |
| Number of unique SJ detected |
114074 |
|
| Novel SJ |
7598 |
0.07 |
| Non-canonical SJ |
2098 |
0.02 |
|
Evaluation of FSM
| Number of isoforms |
19578 |
- |
| Reference Match |
13028 |
66.54 |
| 5’ reference supported (transcript) |
15217 |
77.72 |
| 3’ reference supported (transcript) |
15216 |
77.72 |
| 5’ reference supported (gene) |
16066 |
82.06 |
| 3’ reference supported (gene) |
16049 |
81.97 |
| 5’ CAGE supported |
277 |
1.41 |
| 3’ polyA supported |
13571 |
69.32 |
| Supported Reference Transcript Model (SRTM) |
14715 |
75.16 |
| Intra-priming |
880 |
4.49 |
| Reference redundancy Level |
1.02 |
- |
|
Evaluation of ISM
| Number of isoforms |
1093 |
- |
| 5’ reference supported (transcript) |
248 |
22.69 |
| 3’ reference supported (transcript) |
488 |
44.65 |
| 5’ and 3’ reference supported (gene) |
256 |
23.42 |
| 5’ reference supported (gene) |
453 |
453 |
| 3’ reference supported (gene) |
699 |
63.95 |
| 5’ CAGE supported |
6 |
0.55 |
| 3’ polyA supported |
826 |
75.57 |
| Supported Reference Transcript Model (SRTM) |
355 |
32.48 |
| Intra-priming |
18 |
1.65 |
| Reference redundancy Level |
1.08 |
- |
|
Evaluation NIC
| Number of isoforms |
3458 |
- |
| 5’ and 3’ reference supported (gene) |
2353 |
68.05 |
| 5’ reference supported (gene) |
2777 |
80.31 |
| 3’ reference supported (gene) |
2888 |
83.52 |
| 5’ CAGE supported |
67 |
1.94 |
| 3’ polyA supported |
3131 |
90.54 |
| Supported Novel Transcript Model (SNTM) |
2395 |
69.26 |
| Intra-priming |
37 |
1.07 |
| Intron retention incidence |
504 |
14.57 |
|
Evaluation NNC
| Number of isoforms |
4755 |
- |
| 5’ and 3’ reference supported (gene) |
1374 |
28.9 |
| 5’ reference supported (gene) |
2962 |
62.29 |
| 3’ reference supported (gene) |
2449 |
51.5 |
| 5’ CAGE supported |
49 |
1.03 |
| 3’ polyA supported |
3072 |
64.61 |
| Supported Novel Transcript Model (SNTM) |
1317 |
27.7 |
| Non-canonical SJ incidence |
1726 |
36.3 |
| Full Illumina SJ support |
2778 |
58.42 |
| Intra-priming |
321 |
6.75 |
| RT-switching incidence |
620 |
13.04 |
|
Evaluation of Spike-Ins (SIRVs)
The following metrics and definitions apply to SIRV transcripts:
- SIRV transcript: Transcript mapping to a SIRV sequence
- Reference SIRV (rSIRV): Lexogen SIRV model
- True Positive detections (TP): rSIRVs identified as RM
- Partial True Positive detections (PTP): rSIRVs identified as ISM or FSM_non_RM
- False Negative (FN): rSIRVs without FSM or ISM
- False Positive (FP): NIC + NNC + antisense + fusion SIRV_transcripts
- Sensitivity: TP/rSIRVs
- Precision: RM/ SIRV_transcripts
- Non_redundant Precision: TP/ SIRV_transcripts
- Positive Detection Rate: unique(TP+PTP)/rSIRVs
- False Discovery Rate: (FP + PTP)/SIRV_transcripts
- False Detection Rate: (FP)/SIRV_transcripts
- Redundancy: (FSM + ISM)/unique(TP+PTP)
ATENTION If in this chunk of the evaluation all the results are 0, please, check if the reference genome and/or transcriptome used for building your transcript-model contain information about spike-ins.
| SIRV transcripts |
95 |
| True Positive detections (TP) |
69 |
| SIRV transcripts associated to TP (Reference Match) |
69 |
| Partial True Positive detections (PTP) |
6 |
| SIRV transcripts associated to PTP |
8 |
| False Negative (FN) |
9 |
| False Positive (FP) |
17 |
| Sensitivity |
0.82 |
| Precision |
0.73 |
| Non Redundant Precision |
0.73 |
| Positive Detection Rate |
0.89 |
| False Discovery Rate |
0.24 |
| False Detection Rate |
0.18 |
| Redundancy |
1.04 |
|