Build databses. GTF / GFF details - SnpEff & SnpSift
Author: Pablo Cingolani
Description: SnpEff and SnpSift

Skip to content

# Building databases: GTF / GFF details

In this section we show some specific details on the GTF and GFF file format required by SnpEff to build databases.

Warning

Most people do NOT need to build a database, and can safely use a pre-built one. So unless you are working with a rare, custom, or new genomes you most likely don't need to do it either.

## Summary

As seen in the previous Building databases, there are three main steps when building a database:

1. Step 1: Configure a new genome in SnpEff's config file `snpEff.config`.
2. Step 2: Build using gene annotations and reference sequences
3. Step 3: Checking the database: SnpEff will check the database by comparing predicted protein sequences and CDS sequences with ones provided by the user.

In this section we'll go into the details of the GTF and GFF format requirements for Step 2. As a general rule, GTF format is preferred over GFF, so if your genome provides both GTF anf GFF, use GTF whenever possible.

## GTF format example

This is a snippet example from a GTF file that fulfills SnpEff's requirements. The example (from ENSEMBL's human genome GTF file) shows the definition of one gene, one transcript and it's exons, as well as the trancript's start codon, stop codon, and UTR regions.

    # Note that tabs have been replaced by spaces for readability
    chr1    ensembl  gene          10472288        10630758        .       +       .       gene_id "ENSG00000142655.13"; gene_type "protein_coding"; gene_name "PEX14";
    chr1    ensembl  transcript    10474950        10630758        .       +       .       gene_id "ENSG00000142655.13"; transcript_id "ENST00000356607.9"; transcript_type "protein_coding";
    chr1    ensembl  exon          10474950        10475002        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  CDS           10474967        10475002        .       +       0       transcript_id "ENST00000356607.9";
    chr1    ensembl  start_codon   10474967        10474969        .       +       0       transcript_id "ENST00000356607.9";
    chr1    ensembl  exon          10495274        10495321        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  CDS           10495274        10495321        .       +       0       transcript_id "ENST00000356607.9";
    chr1    ensembl  exon          10536213        10536297        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  CDS           10536213        10536297        .       +       0       transcript_id "ENST00000356607.9";
    chr1    ensembl  exon          10599238        10599366        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  CDS           10599238        10599366        .       +       2       transcript_id "ENST00000356607.9";
    chr1    ensembl  exon          10618332        10618417        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  CDS           10618332        10618417        .       +       2       transcript_id "ENST00000356607.9";
    chr1    ensembl  exon          10623019        10623121        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  CDS           10623019        10623121        .       +       0       transcript_id "ENST00000356607.9";
    chr1    ensembl  exon          10624340        10624437        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  CDS           10624340        10624437        .       +       2       transcript_id "ENST00000356607.9";
    chr1    ensembl  exon          10627272        10627363        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  CDS           10627272        10627363        .       +       0       transcript_id "ENST00000356607.9";
    chr1    ensembl  exon          10629531        10630758        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  CDS           10629531        10629984        .       +       1       transcript_id "ENST00000356607.9";
    chr1    ensembl  stop_codon    10629985        10629987        .       +       0       transcript_id "ENST00000356607.9";
    chr1    ensembl  UTR           10474950        10474966        .       +       .       transcript_id "ENST00000356607.9";
    chr1    ensembl  UTR           10629985        10630758        .       +       .       transcript_id "ENST00000356607.9";

For a more detailed example, check ENSEMBL's GTF files, for instance this one for GRCh38.107 (the human genome)

## GTF format details

The full GTF format specification is beyond the scope of this section, and it is assumed you are familiar with it. It's probably a good idea to take a look at the format specification before reading the rest of this section, here are some links:

- ENSEMBL GTF format specification
- Brent's lab GTF format

### GTF File name

SnpEff expects the GTF file to be located at

    $SNPEFF_HOME/data/GENOME_NAME/genes.gtf

where:

- `$SNPEFF_HOME` is the directory where SnpEff is installed (usually `$HOME/snpEff`)
- `GENOME_NAME` is the genome name of the genome you are trying to build, which MUST match the name you added in the config file `snpEff.config`

Note: The file name can be `genes.gff.gz` if it's compressed using `gzip`.

### GTF lines

In a nutshell, GTF files are text files and each line is parsed separately.

Lines that start with `#` are treated as comments (i.e. ignored).

Each (non-comment) line is parsed as a tab-separate list of fields, for example:

    #!genome-build GRCh38.p13
    1       protein_coding  gene         69091   70008   .       +       .       gene_id "ENSG00000186092"; gene_name "OR4F5"; gene_source "ensembl_havana"; gene_biotype "protein_coding";
    1       protein_coding  transcript   69091   70008   .       +       .       gene_id "ENSG00000186092"; transcript_id "ENST00000335137"; gene_name "OR4F5"; gene_source "ensembl_havana"; gene_biotype "protein_coding"; transcript_name "OR4F5-001"; transcript_source "ensembl_havana"; tag "CCDS"; ccds_id "CCDS30547";
    1       protein_coding  exon         69091   70008   .       +       .       gene_id "ENSG00000186092"; transcript_id "ENST00000335137"; exon_number "1"; gene_name "OR4F5"; gene_source "ensembl_havana"; gene_biotype "protein_coding"; transcript_name "OR4F5-001"; transcript_source "ensembl_havana"; tag "CCDS"; ccds_id "CCDS30547"; exon_id "ENSE00002319515";
    1       protein_coding  CDS          69091   70005   .       +       0       gene_id "ENSG00000186092"; transcript_id "ENST00000335137"; exon_number "1"; gene_name "OR4F5"; gene_source "ensembl_havana"; gene_biotype "protein_coding"; transcript_name "OR4F5-001"; transcript_source "ensembl_havana"; tag "CCDS"; ccds_id "CCDS30547"; protein_id "ENSP00000334393";
    1       protein_coding  start_codon  69091   69093   .       +       0       gene_id "ENSG00000186092"; transcript_id "ENST00000335137"; exon_number "1"; gene_name "OR4F5"; gene_source "ensembl_havana"; gene_biotype "protein_coding"; transcript_name "OR4F5-001"; transcript_source "ensembl_havana"; tag "CCDS"; ccds_id "CCDS30547";
    1       protein_coding  stop_codon   70006   70008   .       +       0       gene_id "ENSG00000186092"; transcript_id "ENST00000335137"; exon_number "1"; gene_name "OR4F5"; gene_source "ensembl_havana"; gene_biotype "protein_coding"; transcript_name "OR4F5-001"; transcript_source "ensembl_havana"; tag "CCDS"; ccds_id "CCDS30547";

It should be noted that lines do NOT have a specific order. Usually information defining a gene tends to be together, but this is not required by the GTF format.

### GTF fields

The nine tab-separated fields in each (non-comment) line are:

1. seqname: name of the chromosome or scaffold; chromosome names can be given with or without the 'chr' prefix. Important note: the seqname must be one used within Ensembl, i.e. a standard chromosome name or an Ensembl identifier such as a scaffold ID, without any additional content such as species or assembly. See the example GFF output below.
2. source: name of the program that generated this feature, or the data source (database or project name)
3. feature: feature type name, e.g. Gene, transcript, exon, etc.
4. start: Start position* of the feature, with sequence numbering starting at 1.
5. end: End position* of the feature, with sequence numbering starting at 1.
6. score: A floating point value.
7. strand: defined as + (forward), or - (reverse).
8. frame: One of '0', '1' or '2'. '0' indicates that the first base of the feature is the first base of a codon, '1' that the second base is the first base of a codon, and so on..
9. attribute: A semicolon-separated list of tag-value pairs, providing additional information about each feature.

Info

Also note that `'.'` denotes an empty field, you cannot just use an empty string to denote an empty field.

### GTF field requirements

SnpEff requires that the fields are:

1. seqname: Must match the name of the chromosome / scaffold in the reference genome sequence FASTA file (with or without a `chr` prepend).
2. source: This field is ignored by SnpEff.
3. feature: These feature types, such as `gene`, `exon`, `cds`, etc. See details in section GTF Feature
4. start: One-based chromosome position of feature start (base included).
5. end: One-based chromosome position of feature end (base included).
6. score: SnpEff ignores this field.
7. strand: Considered negative strand if `'-'`, otherwise interpreted as positive strand.
8. frame: Interpreted as 'phase', can be `{0, 1, 2}`. If empty (`'.'`) or `-1` is interpreted as "missing". See "GTF Frame details" section below.
9. attribute: Attribute list, see GTF Attributes section below

### GTF Feature

These feature types will be translated to SnpEff entities the following way (case ignored):

Feature Type | Feature value
--- | ---
GENE | gene, protein
TRANSCRIPT | pseudogene, transcript, mrna, trna, snorna, rrna, ncrna, mirna, snrna, pseudogenic_transcript
EXON | exon, pseudogenic_exon
CDS | cds
START_CODON | start_codon
STOP_CODON | stop_codon
UTR5 | five_prime_utr, 5'-utr, 5'utr, 5utr
UTR3 | three_prime_utr, 3'-utr, 3'utr, 3utr
INTRON_CONSERVED | intron_CNS, intron_cns
INTERGENIC_CONSERVED | inter_cns

### GTF Frame

The frame field indicates the number of bases that should be removed from the beginning of this feature to reach the first base of the next codon. This is typically used in `EXON` or `CDS` features within in coding genes.

Possible values are:

- `0`: indicates that the feature begins with a whole codon at the 5' most base.
- `1`: means that there is one extra base (the third base of a codon) before the first whole codon and
- `2`: means that there are two extra bases (the second and third bases of the codon) before the first codon
- `.`: Missing value, SnpEff will inferr this value from the feature's coordinates

Info

Sometimes this is called 'phase' instead of frame, to distinguish form the "coding base modulo 3" definition.

Frame correction SnpEff performs a "frame correction". If the frame value calculated using the feature (exon) coordinates differs from the one given in the `start / end` coordinates, the coordinates will be corrected.

This correction is performed in two stages, for each transcript:

i) First exon is corrected by adding a fake 5'UTR ii) Other exons are corrected by changing the start (or end) coordinates. We drop bases from either the `start` coordinate (if the exon is on the positive strand) or `end` coordinate (if the exon is on the negative strand) until the frame matches the one from the GTF.

Check zero frames: If all frames are zero, there is a high chance that the frame values are incorrectly labeled as zero instead of "missing values" (i.e. '.'). SnpEff will check if all frame values are zero. If there are more than `MIN_TOTAL_FRAME_COUNT` frmae values set (by default 10) and all of them are zero, it will show a warning.

### GTF Attributes

The "attributes" field is parsed as a semicolon-separated list of key-value pairs, providing additional information about each feature.

Required attributes are:

Feature type | Required attributes | Optional attributes
--- | --- | ---
GENE | ID / GeneID, GeneBioType, | GeneName
TRANSCRIPT | ID / TranscriptID, TranscriptBioType, ParentID | `transcript_support_level`, `transcript_version`
CDS, EXON, STOP_CODON, START_CODON | ID, ParentID / TranscriptID |
UTR, UTR5, UTR3 | ID, ParentID / TranscriptID |
INTRON_CONSERVED | ID, TranscriptID / TranscriptID |
INTERGENIC_CONSERVED | ID |

### GTF Attribute: ID

The attribute name can be (not case sensitive, in search order):

- `id`
- `gene_id` (if feature type is `GENE`)
- `transcript_id` (if feature type is `TRANSCRIPT`)
- `exon_id` (if feature type is `EXON`),
- `db_xref`
- `name`

If none is available, SnpEff will generate an ID as

    feature + "_" + chromosomeName + "_" + start + "_" + end

where `feature` is the parse "Feature type" .

### GTF Attribute: ParentId

The attribute name can be (not case sensitive, in search order):

- `parent`
- `gene` (if feature type is `TRANSCRIPT` or `INTRON_CONSERVED`)
- same as TranscriptId (if feature type is any of 'EXON', 'CDS', 'START_CODON', 'STOP_CODON', 'UTR3', or 'UTR5')

Warning

The value of `ParentID` must match exactly the `ID` of the parent feature (e.g. the `ParentID` for a transcript, must match the ID of the parent gene). It is a common mistake in some GTF / GFF files to add or remove some characters. If the IDs don't match, the GTF/GFF file is invalid for SnpEff.

### GTF Attribute: GeneId

The attribute name can be (not case sensitive, in search order):

- `gene_id`
- `id` (if feature type is `GENE`)

`GeneId` value must be a unique ID for each gene in the genome. If the value is repeated, SnpEff will add a dot ('.') followed by an integer number to make is unique.

### GTF Attribute: TranscriptId

The attribute name can be (not case sensitive, in search order):

- `transcript_id`
- `id` (if feature type is `TRANSCRIPT`)
- or the same as `ParentID` if feature type is `EXON`

### GTF Attribute: GeneName

The attribute name can be (not case sensitive, in search order):

- `gene_name`
- `name` (if feature type is `GENE`)

### GTF Attribute: BioType

The attribute name can be (not case sensitive): `biotype`

Possible values are:

BioType | Possible attribute values
--- | ---
protein_coding | mrna, protein, cds, trna, start_codon, stop_codon, five_prime_utr, 5'-utr, 5'utr, 5utr, three_prime_utr, 3'-utr, 3'utr, 3utr
transcribed_processed_pseudogene | pseudogenic_transcript, pseudogenic_exon
lincRNA | ncrna
rRNA | rrna
miRNA | mirna
snRNA | snrna
snoRNA | snorna
prime3_overlapping_ncrna | 3prime_overlapping_ncrna

If the BioType field is not found, the GTF source field will be parsed, otherwise the feature type will be parsed.

### GTF Attribute: GeneBioType

This is similar to BioType, used specifically for feature type `GENE` The attribute name can be (not case sensitive, in search order):

- `gene_biotype`
- `gene_type`
- `biotype`

Attribute values are parsed the same maner as `BioType`.

If `GeneBioType` is `protein_coding`, then the gene is assumed to be a protein coding (all transcripts within will be also considered protein coding).

### GTF Attribute: TranscriptBioType

This is similar to BioType, used specifically for feature type `TRANSCRIPT`

The attribute name can be (not case sensitive, in search order):

- `transcript_biotype`
- `transcript_type`
- `biotype`

Attribute values are parsed the same maner as `BioType`.

If `TranscriptBioType` is `protein_coding`, then the transcript is assumed to be a protein coding transcript.

## GFF

SnpEff treats GFF files the same way as GTF files.

The GFF format is more flexible / lax than GTF. Unfortunately, this extra flexibility also means that it is difficult to find GFF files that fulfill the requirements to build a genomic database, as many people add the information in different ways.

Info

Generally GTF files are preferred to build databases

GFF3 is the currently supported version, the old GFF2 format is deprecated

### GFF File name

SnpEff expects the GFF file to be located at

    $SNPEFF_HOME/data/GENOME_NAME/genes.gff

where:

- `$SNPEFF_HOME` is the directory where SnpEff is installed (usually `$HOME/snpEff`)
- `GENOME_NAME` is the genome name of the genome you are trying to build, which MUST match the name you added in the config file `snpEff.config`

Note: The file name can be `genes.gff.gz` if it's compressed using `gzip`.

### GTF lines and fields

GFF lines and fields are very similar to GTF ones. The main difference is that the attributes field is formatted as semi-colon separated `key=value` pairs (in GTF `key` and `value` are separated by a space instead of an `=` sign).

Info

Other than the minor difference in attributes formatting, SnpEff parses and interprets all the fields and attributes exactly the same way as in GTF files.

### GFF genome sequence

GFF files can have the reference genome sequence in the same file. After a special comment `##FASTA` you can concatenate the whole genome FASTA file. For example (see GFF3 Sequence Section):

    ##gff-version 3
    ctg123 . exon            1300  1500  .  +  .  ID=exon00001
    ctg123 . exon            1050  1500  .  +  .  ID=exon00002
    ctg123 . exon            3000  3902  .  +  .  ID=exon00003
    ctg123 . exon            5000  5500  .  +  .  ID=exon00004
    ctg123 . exon            7000  9000  .  +  .  ID=exon00005
    ##FASTA
    >ctg123
    cttctgggcgtacccgattctcggagaacttgccgcaccattccgccttg
    tgttcattgctgcctgcatgttcattgtctacctcggctacgtgtggcta
    tctttcctcggtgccctcgtgcacggagtcgagaaaccaaagaacaaaaa
    aagaaattaaaatatttattttgctgtggtttttgatgtgtgttttttat
    aatgatttttgatgtgaccaattgtacttttcctttaaatgaaatgtaat
    cttaaatgtatttccgacgaattcgaggcctgaaaagtgtgacgccattc
    ...

This makes it easier to distribute the genome reference toghether with the genome annotations in one file.