#+TITLE:FAST:Fast Analysis of Sequences Toolbox Cookbook
#+AUTHOR: Katherine C.H. Amrine \& David H. Ardell
#+LANGUAGE:  en
#+TEXT:      These examples are executable from the installation directory.
#+OPTIONS:   H:5 num:t toc:t \n:nil @:t ::t |:t ^:t *:t TeX:t d:nil
#+LATEX_HEADER: \usepackage[colorlinks=true,urlcolor=SteelBlue4,linkcolor=Firebrick4]{hyperref}
#+STARTUP: align

* Tutorials

** Prelude 

*** The FAST definition of "FastA format"

FastA format began with the FastA search utilities of William
Pearson. For FAST, "fasta format" means what is conventionally called
"multi-fasta" format of sequence or alignment data, largely as
implementated in BioPerl in the module =Bio::SeqIO::fasta=. 

In the FAST implementation of "fasta format", multiple sequence
records may appear in a single file or input stream. Sequence data may
contain gap characters. The logical elements of a sequence record are
its /identifier/, /description/ and /sequence/. The /identifier/
(indicated with =id= in the example here) and /description/ (=desc=)
together make the /identifier line/ of a sequence record, that must
begin with the sequence record start symbol =>= on a single line. The
/description/ begins after the first block of white-space on this line
(indicated with =<space>=). The /sequence/ of a record
appears immediately after its identifier line and may continue over
multiple lines until the next record. 

In FAST, the description may be broken into multiple /fields/ defined
by a /delimiter/ (indicated with =<delim>=). FAST uses a "one-based"
indexing of fields as indicated here:

#+BEGIN_EXAMPLE
>seq1-id<space>seq1-desc-field1<delim>seq1-desc-field2<delim>...
seq1-sequence
seq1-sequence
...
seq1-sequence
>seq2-id<space>seq2-desc-field1<delim>seq2-desc-field2<delim>...
seq2-sequence
seq2-sequence
...
seq2-sequence
#+END_EXAMPLE

*** Use =man= pages for full documentation

All FAST utilities follow UNIX conventions in having default and
optional behaviors. For more information about how to use and modify
the behavior of any FAST utility such as =faswc=, consult its manual
page with /e.g./:

#+BEGIN_SRC shell
man faswc
#+END_SRC

or alternatively:

#+BEGIN_SRC shell
perldoc faswc
#+END_SRC


** Example 1: Prototyping a pipeline to cut, reverse complement, and translate a gene by coordinate from a genome

*** Calculating sequence length

Chromosome 1 from the /Saccharomyces cerevisiae/ genome is available
in =t/data/chr01.fsa=. By default, =faslen= calculates the lengths of
sequence records on its input, and outputs its input, augmenting
sequence descriptions with its calculations using the tag (or name)
=length= and a (name,value) separator =:=, as in =length:872=. We can
therefore easily obtain the length of this chromosome sequence as
follows:

#+BEGIN_SRC shell
faslen t/data/chr01.fsa | egrep ">"
#+END_SRC

Alternatively, =faswc -c= will output the length of the chromosome
directly to =STDOUT=:

#+BEGIN_SRC shell
faswc -c t/data/chr01.fsa
#+END_SRC

*** Cut out a subsequence by coordinate with =fascut=

=fascut= will cut a subsequence by coordinate. For example, suppose we
know that the location of gene =YAR030C= in yeast chromosome 1 begins
186512 and ends 186853 on the minus strand. Let's cut this from our
chromosome. The following code will extract this subsequence in fasta
format to =STDOUT=:

#+BEGIN_SRC shell
fascut 186512..186853 t/data/chr01.fsa
#+END_SRC

*** Computing reverse complement of a sequence with =fasrc=

Knowing that this is on the minus strand, we need to obtain the
reverse complement of this sequence.  =fasrc= will compute this.  The
following code will take the output of =fascut= as its input and
return the reverse complemeht in fasta file to =STDOUT=:

#+BEGIN_SRC shell
fascut 186512..186853 t/data/chr01.fsa | fasrc
#+END_SRC

*** Translating a sequence with =fasxl=

To translate this sequence, we extend the pipeline with the =fasxl= utility:

#+BEGIN_SRC shell
fascut 186512..186853 t/data/chr01.fsa | fasrc | fasxl
#+END_SRC

Examine the output, we will see that the peptide starts with a
methionine, and ends with a stop codon, indicated by the =*= character
by default. 

*** Computing codon usage with =fascodon=

If we are interested in the codon usage of our gene, we can edit the
last command-line (by typing =up-arrow= on most UNIX shells) and
replace =fasxl= with =fascodon= at the end of our pipeline. =fascodon=
reprints the input sequence with the counts of each codon with 
information on starts and stops appended to the identifier. With the following
code, we can see that the most frequently used codon in this example
is =AAT= (encoding an Arginine)

#+BEGIN_SRC shell
fascut 186512..186853 t/data/chr01.fsa | fasrc | fascodon
#+END_SRC 

Appending the =-t= option to the command will give the codon frequencies
for both the input sequence and the reverse complement in a verbose
table format as follows 

#+BEGIN_SRC shell
fascut 186512..186853 t/data/chr01.fsa | fasrc | fascodon -t
#+END_SRC


*** Computing base composition with =fascomp=

=fascomp= will return the base/protein composition of a sequence
as an addition to the identifier. If
we are interested in the normalized (option =-n=) base composition of the first
chromosome in a clean table format (option =-t=), we can run the following:

#+BEGIN_SRC shell
fascomp -nt t/data/chr01.fsa
#+END_SRC


** Example 2: Reformatting, selecting and transforming alignments in FAST

*** Reformatting alignment data with =fasconvert=

A file with protein sequences that match a search for "P450" is
available in =t/data/P450.fas= under the FAST installation
directory. Another file contains this data aligned using =clustalw=
with the name =P450.clustalw2.aln=. The =fasconvert= tool can convert
from fasta to many formats, or from many formats to fasta, including
clustalw to fasta as showin in the following example

#+BEGIN_SRC shell
fasconvert -i clustalw -f t/data/P450.clustalw2.aln
#+END_SRC

The previous command automatically saves its output to an output file
of the same basename and an extension of =.fas= in the same
directory of the original file. The
=faswc= utility will count the total number of sequences and 
total number of nucleotides in a fasta file
To look at the length of all sequences, use the
following code.

#+BEGIN_SRC shell
faswc t/data/P450.clustalw2.fas
#+END_SRC

which outputs  

=9      5013 t/data/P450.clustalw2.fas=

=9      5013 total=

to  =STDOUT=.

*** Selecting sequences with =fasgrep=

We can subset the output in many ways to get information we are
interested in, for example, if we want to get the original sequence
with the gi number "86475799", we can use =fasgrep=, which will pull
out sequences that match a Perl regular expression. By default,
=fasgrep= attempts to match sequence identifiers:

#+BEGIN_SRC shell
fasgrep "86475799" t/data/P450.fas
#+END_SRC

We can retrieve the aligned version of this sequence as it has the
same identifier

#+BEGIN_SRC shell
fasgrep "86475799" t/data/P450.clustalw2.fas
#+END_SRC

*** Reformatting gap characters with =fastr=

=fastr= may be useful when we must change specific characters based on
the requirements of a bioinformatic program. For example, to reformat
gap characters in a fasta-format alignment from "-" to ".".

#+BEGIN_SRC shell
fastr -s "-" "." t/data/P450.clustalw2.fas
#+END_SRC 

*** Degapping sites with =alncut= 

=alncut= also allows for editing of alignments based on their gap
profile. This utility is useful in many applications, including 
selecting gap-free sites for input into phylogenetic softwares.
To remove sites with at least one gap in all sequences, we
can do the following:

#+BEGIN_SRC shell
alncut -g t/data/P450.clustalw2.fas
#+END_SRC

We can then determine the length of the alignment by executing:

#+BEGIN_SRC shell
alncut -g t/data/P450.clustalw2.fas | faslen | head -1 
#+END_SRC

And if we are interested in retaining only unique sequences, 
/fasuniq/ appended to the output will collapse duplicate sequences to
one, appending all of the identifiers to one large identifier. 

#+BEGIN_SRC shell
alncut -g t/data/P450.clustalw2.fas | faslen | fasuniq
#+END_SRC

** Example 3: partitioning files based on NCBI taxonomy

The =fastax= tool is a powerful tool when one wants to partition
data based on their taxonomic affiliations. We can partition large
datasets for subset analyses, statistical comparisons, and other
applications and preparation of data. =fastax= depends on the user
supplying a file with the tree structure already defined. In this
example, we will use NCBI taxonomy. The files necessary include
a nodes file (in this case /nodes.dmp/) and a names file (in this
case, /names.dmp/). The nodes file consists of a line for each
taxonomic entry in NCBI with information about its class (superfamily,
genus, etc) and its parent node, indexed by its numeric identifier. 
These files were retrieved from NCBI in a zipped package via FTP 
located at pub/taxonomy/taxdump.tar.gz. The names.dmp file will 
link the numeric identifier to any specific name that the entry can 
be named including its scientific name, common name, and alternative 
spellings accepted by NCBI. We will not create our own nodes and 
names files, but note that it can be done if the user disagrees with 
the NCBI taxonomic structure, or requires more specific partitions 
of their data. 

The importance of structured sequence tags comes into play in this 
example. As described above, the line above the sequence in a fasta
file is indexed by a ">" character followed by the identifier, followed
by a space, and then everything else is located in the description. 
=fastax= will need more structure around the taxonomic classification. 
The program, by default, will search by the description field, but the 
descrption field will have it's own structure. Note, if our description
field only contains the species, or the TaxID, then we don't need
to worry about structuring the description. If it is not the only thing
in our description, then we need to modify the description a bit, or
determine if there is a delimiter that already exists between the 
TaxID and the other components of the description. 

*** Reformat the description to allow for fastax sorting

If we look at our P450.fas file, we see that the description 
consists of "P450" and then a space, and then a square bracket "[",
the species name, and then a closing square bracket "]". There is 
currently no identifier that is unique surrounding only the species
name. Open bracket and closed bracket are two different characters,
and using the open bracket as a delimiter will give you the species
name and the closed bracket in the second field, and this will not 
match correctly. We will first change our description field to one
delimiter using the handy =fastr= tool. Arbitrarily, we will chose 
the double quote character for our description delimiter.

#+BEGIN_SRC shell
fastr -d "[]" "\"" t/data/P450.fas
#+END_SRC

Now in standard output, we will see fasta file-formatted text with
the species name in the description in the sequence tag surrounded
by quotes. In this file, the beginning of the description is in 
field one, and the species in field two. We can use this information
to construct a command to pull out the sequences that are in the 
taxonomic "Pooideae" tribe. Assuming that nodes.dmp and names.dmp are
in the same working directory, we can run the following. 

#+BEGIN_SRC shell
fastr -d "[]" "\"" t/data/P450.fas | 
      fastax -S \" -f 2 t/data/nodes.dmp t/data/names.dmp "Pooideae"
#+END_SRC

The output of this pipeline should be five sequences, including P450 sequences
from the /Triticum aestivum/ and /Lolium rigidum/ species (classified as species
belonging to the /Pooideae/ tribe). 

** Example 4: Retrieve a subset of sequences based on a list of identifiers

=fasgrep= is a useful tool for retrieving subsets of sequences from large
fasta files. Often fasta files will contain an identifier line, and then 
one line following with a sequence corresponding to the previous identifier. 
If this is always the case, parsing fasta files is fairly simple. When 
the sequence that follows the identifier exists on multiple lines, the task
of subsetting sequences becomes more challenging. If we have a list of 
sequence identifiers in the file /ids.txt/, we can write a bash wrapper, 
incorporating the unix =cat= command, for the fasgrep command to search 
for this subset of sequences as follows:

#+BEGIN_SRC shell
for i in $(cat ids.txt); do fasgrep $i sequences.fas; done > subset.fas
#+END_SRC

Now we have a subest of sequences located in /subset.fas/ that correspond
to the identifiers listed in /ids.txt/. 

*** Note on searching for identifiers
Fasgrep works with perl regular expression syntax.It is often beneficial 
to code the identifiers in /ids.txt/ with some sort of a line or a word 
anchor. For example, if you are looking for a complete identifier like
 =sequence_1=
but have sequences in your file named =sequence_11= and =sequence_12=, 

#+BEGIN_SRC shell
fasgrep "sequence_1" sequences.fas
#+END_SRC

will return all three sequences. If
you'd like to just look for =sequence_1=, adding a line anchor, such as:

#+BEGIN_SRC shell
fasgrep "sequence_1$" sequences.fas
#+END_SRC

will return =sequence_1= and skip =sequence_11= and =sequence_12=. This
only works if =sequence_1= is at the end of your identifier. If you are
looking for =sequence_1= within a larger identifier, you may need to 
take extra precautions to not retrieve other names that are more
specific to the structure of your identifiers.
