目录

FASTQ 文件格式


fastq 格式 是一种基于文本的格式,用于存储生物序列(通常是核苷酸序列)及其相应的碱基质量分数。为了简洁起见,序列字母和质量分数都用一个ASCII字符编码。

它最初是由威康信托桑格研究所(Wellcome Trust Sanger Institute)开发的,用于捆绑FASTA格式的序列及其质量数据。最近,已成为测序数据格式的一种标准,用于存储Illumina基因组分析仪等高通量测序仪器的输出 [1]


1. 格式


FASTQ文件中的每个序列通常占四行。

一个包含单个序列的FASTQ文件类似如下:

@SEQ_ID
GATTTGGGGTTCAAAGCAGTATCGATCAAATAGTAAATCCATTTGTTCAACTCACAGTTT
+
!''*((((***+))%%%++)(%%%%).1***-+*''))**55CCF>>>>>>CCCCCCC65

原始的Sanger FASTQ文件还允许对序列和质量字符串进行包装(多行分割),但通常不建议这样做,因为使用“@”和“+”作为标记会使解析变得复杂(这些字符也可能出现在质量字符串中)。


2. 识别符


Illumina公司序列标识符

Illumina软件中的序列使用系统标识符:

@HWUSI-EAS100R:6:73:941:1973#0/1
HWUSI-EAS100R 仪器名称
6 flowcell lane
73 tile number within the flowcell lane
941 'x'-coordinate of the cluster within the tile
1973 'y'-coordinate of the cluster within the tile
#0 index number for a multiplexed sample (0 for no indexing)
/1 the member of a pair, /1 or /2 (paired-end or mate-pair reads only)

Illumina 1.4以后的版本使用#NNNNNN代替#0作为多路标识,其中NNNNNN是多路标记的序列。

在Casava 1.8中,@的格式发生了变化:

@EAS139:136:FC706VJ:2:2104:15343:197393 1:Y:18:ATCACG
EAS139 the unique instrument name
136 the run id
FC706VJ the flowcell id
2 flowcell lane
2104 tile number within the flowcell lane
15343 'x'-coordinate of the cluster within the tile
197393 'y'-coordinate of the cluster within the tile
1 the member of a pair, 1 or 2 (paired-end or mate-pair reads only)
Y Y if the read is filtered, N otherwise
18 0 when none of the control bits are on, otherwise it is an even number
ATCACG index sequence

注意,Illumina软件的最新版本输出的是一个样本号(取自样本表),而不是索引序列。例如,在批处理的第一个示例中可能出现以下头部:

@EAS139:136:FC706VJ:2:2104:15343:197393 1:N:18:1


NCBI Sequence Read Archive(SRA,序列读取存档)


从INSDC1) 序列读取归档文件中的FASTQ文件,通常如下:

@SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=36
GGGTGATGGCCGCTGCCGATGGCGTCAAATCCCACC
+SRR001666.1 071112_SLXA-EAS1_s_7:5:1:817:345 length=36
IIIIIIIIIIIIIIIIIIIIIIIIIIIIII9IG9IC

在本例中,有一个ncbi分配的标识符,描述包含来自Solexa/Illumina(如上所述)的原始标识符和读取长度。测序采用双端模式(~500bp insert size),见SRR001666。


3. 序列质量


质量值 $Q$ 是 $p$(即,对应的碱基测序错误的概率)整数映射。使用了两种方式编码。第一个是用来评估碱基可靠性的桑格标准,也称为Phred质量评分:

$$ Q_{\text{sanger}}=-10\,\log _{10}p $$

另一种是Solexa (即, Illumina Genome Analyzer提供的软件)早些时候使用了不同的映射,编码的是 $p/(1-p)$而不是probability $p$:

$$ Q_{\text{solexa to v.1.3}}=-10\,\log _{10}{\frac {p}{1-p}} $$

这两种映射在较高的质量值上是渐近相同的,在较低的质量水平上略有差异(即,约p > 0.05,或等效Q < 13)。(见下图)

Figure 1. Relationship between Q and p using the Sanger (red) and Solexa (black) equations.The vertical dotted line indicates p = 0.05, or equivalently, Q ≈ 13.

4. 编码


相同字母在不同格式代表不同的PHRED score:


  SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS.....................................................
  ..........................XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX......................
  ...............................IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII......................
  .................................JJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJJ.....................
  LLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLL....................................................
  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~
  |                         |    |        |                              |                     |
 33                        59   64       73                            104                   126
  0........................26...31.......40                                
                           -5....0........9.............................40 
                                 0........9.............................40 
                                    3.....9..............................41 
  0.2......................26...31........41                              

 S - Sanger        Phred+33,  raw reads typically (0, 40)
 X - Solexa        Solexa+64, raw reads typically (-5, 40)
 I - Illumina 1.3+ Phred+64,  raw reads typically (0, 40)
 J - Illumina 1.5+ Phred+64,  raw reads typically (3, 41)
     with 0=unused, 1=unused, 2=Read Segment Quality Control Indicator (bold) 
     (Note: See discussion above).
 L - Illumina 1.8+ Phred+33,  raw reads typically (0, 41)

5. 模拟


已经有不少模拟仿真测序序列的工具[2],[3],他们的对比可见此文献[4]

6. 数据压缩


在FASTQ格式中(压缩前),质量值约占所需磁盘空间的一半,因此对质量值进行压缩可以显著降低存储需求,加快测序数据的分析和传输。无损压缩和有损压缩最近都在文献中被报道。例如,QualComp[5]算法使用用户指定的速率(每个质量值的比特数)执行有损压缩。基于速率失真理论结果,分配比特数,使原始(未压缩)质量值与重构(压缩后)质量值之间的MSE(均方误差)最小。其他压缩质量值的算法包括SCALCE[6]和Fastqz[7],都是无损压缩算法,提供了一种可选的受控有损转换方法。例如,SCALCE根据观察到“邻近”的质量值在一般情况下是相似的,从而减小了字母表的大小,有关的基准测试,参见[8]



参考文献


[1] Cock, P. J. A.; Fields, C. J.; Goto, N.; Heuer, M. L.; Rice, P. M. (2009). “The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants”. Nucleic Acids Research. 38 (6): 1767–1771. doi:10.1093/nar/gkp1137. PMC 2847217. PMID 20015970.
[2] Huang, W; Li, L; Myers, J. R.; Marth, G. T. (2012). “ART: A next-generation sequencing read simulator”. Bioinformatics. 28 (4): 593–4. doi:10.1093/bioinformatics/btr708. PMC 3278762. PMID 22199392.
[3] Pratas, D; Pinho, A. J.; Rodrigues, J. M. (2014). “XS: A FASTQ read simulator”. BMC Research Notes. 7: 40. doi:10.1186/1756-0500-7-40. PMC 3927261. PMID 24433564.
[4] Escalona, Merly; Rocha, Sara; Posada, David (2016). “A comparison of tools for the simulation of genomic next-generation sequencing data”. Nature Reviews Genetics. 17 (8): 459–69. doi:10.1038/nrg.2016.57. PMC 5224698. PMID 27320129.
[5] Ochoa, Idoia; Asnani, Himanshu; Bharadia, Dinesh; Chowdhury, Mainak; Weissman, Tsachy; Yona, Golan (2013). “Qual Comp: A new lossy compressor for quality scores based on rate distortion theory”. BMC Bioinformatics. 14: 187. doi:10.1186/1471-2105-14-187. PMC 3698011. PMID 23758828.
[6] Hach, F; Numanagic, I; Alkan, C; Sahinalp, S. C. (2012). “SCALCE: Boosting sequence compression algorithms using locally consistent encoding”. Bioinformatics. 28 (23): 3051–7. doi:10.1093/bioinformatics/bts593. PMC 3509486. PMID 23047557
[8] M. Hosseini, D. Pratas, and A. Pinho. 2016. A survey on data compression methods for biological sequences. Information 7(4):(2016): 56
1)
International Nucleotide Sequence Database Collaboration,收集和传播包含DNA和RNA序列的数据库。它涉及下列数据库: DNA Data Bank of Japan(DDBJ,日本)、GenBank(美国)和European Nucleotide Archive(英国)。https://en.wikipedia.org/wiki/International_Nucleotide_Sequence_Database_Collaboration
2)
Seqanswer's topic of skruglyak, dated January 2011