Options for local sequence searches.

Local implementations of NCBI software and the NCBI databases are available at UTHSCSA. These databases can be searched as an alternative to searching at NCBI. Because only UTHSCSA users can access these services, the local search is generally faster than the turnaround from NCBI. Additionally, other programs, including HMMER and SAM access the databases. The local tools can also be used to gain some extra capabilities such as:

Running a batch of blast searches at once.
Running multiple iterations of Psi-Blast without having to intervene at each iteration.
Incorporating your own sequences into the database to be the target of a reverse Psi-Blast search.
Tweaking parameters beyond limits allowed at the remote servers.
Running scripted control of searches to automate repetitive tasks.
Creating and testing new HMM or PSSM models for protein families.
Accessing some additional utilities from the Blast suite not available at NCBI.

Note to GCG users:

The default local databases provided in the GCG system are not regularly updated and hence should be considered obsolete. See notes below for accessing the same local daily updated databases out of GCG as used by the local blast suite.

Programs available

Netblast - web interface to:

Blast family

BlastP - protein key against protein database
BlastX - translated nucleotide key against protein database
BlastN - nucleotide key against nucleotide database
TblastN - protein key against translated nucleotide database
TblastX - translated nucleotide key against translated nucleotide database

Psi-Blast, and Phi-Blast
RPS-Blast (same as CD-search on NCBI Psi-Blast page) [*Currently a problem exists with the Neblast interface to RPS-Blast; use the standalone version or the NCBI web site instead*].
Megablast
Blast 2 sequences

Standalone Blast Suite (run from the unix command prompt)

as above - Blast family, Psi/Phi-Blast, RPS-Blast, Blast 2 sequences.
plus - Tools for regular expression searching, database construction, extraction of sequence from local databases, making RPS models, search of PSSMs versus translated DNA, and clustering of a database by Blast matching criterion. See the Table below.

For either method, you will need a username and password for bcf. If you do not have one, apply here.

NetBlast family.

NOTE: as of 7/12/2006, the local NetBlast implementation was out of order. If you wish to use this program, inquire to Dr. Dememler.

Use these for the following purposes:

To get web based Blast family, or Psi-Blast, searches with turnaround faster than at NCBI. Note: turnaround times at NCBI and locally both vary with usage. Currently, the local search is running about 5 times faster than at NCBI.

Features:

Provides searches against the same as at NCBI.

Usage:

If you have a bcf account, use that username and password. Otherwise obtain a username and password here. Note: Accounts that until recently were for the computer "bioinf" have been transferred to bcf.
Point browser to https://bcf.uthscsa.edu/blast/ and enter username and password when prompted.
Select program from top page and fill in the sequence and options just like searching at NCBI.

Databases available:

The same nucleotide and protein databases at NCBI are available at UTHSCSA. They are updated periodically directly from NCBI. Databases for RPS-Blast are NCBI's remake of Pfam, Smart, COGS, KOGS, and an option to search All (which includes a few more protein families than the union of the other 3).

Comparison to other methods:

For all but RPS-BLAST, average return time currently is about 5 times faster than from NCBI.
The NetBlast version of RPS-BLAST is currently very sluggish (~ 4 minutes through Pfam). Standalone rpsblast (below) takes about 5 sec., as does turnaround from the NCBI CDD site. Turnaround from NCBI can vary widely.
The ability to limit the search to a subset of a database defined by an Entrez descriptor is available for the Blast family, but not yet for Psi-Blast. Employing this option causes a call to NCBI Entrez, so this option will make the local search slow down or fail if connections to NCBI are slowed or severed.
Extra capabilities over NCBI web site:

Will compute Smith Waterman alignment over finds in Psi-Blast. This mainly slightly alters E values.

There are some capabilities not available locally, but available through the NCBI web site:

No Entrez query limitation in Psi-Blast. To do this locally, one could use the command line version of Psi-Blast (blastpgp) and limit the search by an Entrez-generated gi list..
Can not substitute an accession number for the sequence in the input box.
Can not specify a numerical subset of the sequence entered in box to use as search key.
Can not alter alignment view without repeating search; can not alter alignment view between Psi-Blast iterations.
Psi-Blast can not accept pre formed PSSM or return the PSSM. The local command line version of Psi-Blast (blastpgp) can do so.
Only uses single window layout.
Psi-Blast can not toggle gi number vs. accession name in output. The command line version can do so.
There is no CD search box in local Psi-Blast. The RPS search option on the top page of local NetBlast does this search instead.
RPS results are not dynamically linked to alignments as they are at the NCBI site.

Command line Blast versions (Standalone Blast).

Note: These programs are constantly updated from NCBI. Sometimes there are small changes to the command syntax. If the suggested syntax does not work, try typing the command followed by a <space> <dash> to see a synopsis of the syntax for the currently installed version.

Use these to:

Run a batch of blast or Psi-Blast searches at once. Just put the batch of keys in a multi-fasta file and supply that as the key: blastall -p blastp -i keys.fa -d $BLAST_DB/nr -o results.txt
Running multiple iterations of Psi-Blast without having to intervene at each iteration. e.g.. blastpgp -i key.fa -d $BLAST_DB/nr -j8 -o results.txt
Incorporating your own sequences into the database to be the target of a reverse Psi-Blast search. Use -d "$BLAST/nr my_seqs", where myseqs refers to your own library formatted by formatdb.
Tweaking parameters beyond limits allowed at the remote servers. e.g.. blastall -p blastn -i LINE1.fa -v 10000 -b 10000 -d $BLAST_DB/nt -o result.out; I want to see 10000 hits to the repetitive sequence LINE1.
Running scripted control of searches to automate repetitive tasks. e.g. this script was designed to start a reverse Psi-Blast search of 5 iterations from every known phage gene against the global library plus a new virus we sequenced. It saves each search independently and logs which files contain a hit to one of the genes in the new virus. Note: the paths to the libraries in the script are not current.
Creating and testing new HMM or PSSM models for protein families.

Databases available

The same nucleotide and protein databases at NCBI are available at UTHSCSA. They are updated nightly directly from NCBI. To refer to the databases from the command line blast programs, use the path $BLAST_DB. More than one database can be searched at the same time by enclosing the names in quotes: e.g. -d "$BLAST_DB/human_genomic $BLAST_DB/other_genomic".. See this link for the current distribution of sequences among the different database divisions. If there is an extension, it must be included in the -d option. Eg. -d $BLAST_DB/drosoph.aa

Example: blastall -i key.fa -d $BLAST_DB/nr -p blastp -out results
Do not rely on nr being found by default if you leave out the -d option in the command line version.
To get the list of update dates for all databases, type ls -l $BLAST_DB/*.*in
The databases are stored in a format specific for blast searching. Only the nr protein database is also routinely kept as $BLAST_DB/nr in the fasta format for access by other programs. To recreate a fasta-formatted version of one of the other databases for access by another program, use the fastacmd command as illustrated below for the gss division:

fastacmd -DT -d $BLAST_DB/gss -o gss

Access

Log into a bcf account using ssh.
No graphics interface is necessary.
Your root directory should have a file named .ncbirc with the contents:

This file should have been added to your home directory at setup time. You can only see files named with a leading . with the ls -a command.

Documentation for standalone programs

The readme files indicated in the table below can be read with vi.
Each program gives a synopsis of its options if the name is typed followed by a space and a single -. There are many available options in the synopsis obtained in that way which are not otherwise obvious. eg. fastacmd -


Program	Documentation file NOTE: the documentation described below is now somewhat dated, but the files can still be read in /home/hardies/oldblast Newer documentation is at this site	Summary of functions
blastall	README.bls	Run blastp, blastn, blastx, tblastn or tblastx searches. Permits batch of searches at once from a multifasta file for the key search of multiple databases at once search of database restricted to a list of gi numbers Examples: blastall -i key.fa -d $BLAST_DB/nr -p blastp -o result.out blastall -i key.fa -d"$BLAST_DB/nr my_local_library" -p blastp -o result.out blastall -i key.fa -d $BLAST_DB/nr -i gi.lst -p blastp -o result.out
blastall	README.bls	Run psitblastn search. Key is a position specific matrix generated by blastpgp. The key is searched against a nucleotide database translated in all six frames.
fastacmd	README.bls	Retrieve sequence by ID from local database to a fasta file. e.g.. fastacmd -s NP_640322 -pT -d $BLAST_DB/nr -o NP_640322.fa The -pT parameter is necessary for the protein nr database to avoid confusion with the nucleotide nr database. Retrieve multiple sequences specified by a file of IDs to a multifasta file e.g.. fastacmd -i gilist.txt -d $BLAST_DB/est -o <output filename> Restore a formatted database to a multifasta file e.g.. fastacmd -DT -cT -d $BLAST_DB/euk_genome -o <output filename> Count the sequences in a database Extracting a subrange of sequence from large nucleotide entries e.g.. given the following source descriptor in a bacterial protein entry: /coded_by="complement(NC_004431.1:5193864..5194652) Retrieve gene plus surrounding DNA by: fastacmd -s NC_004431 -L 5193500,5195000 -d $BLAST_DB/prok_genomes -o <output filename>
blastpgp	README.bls	Conduct Psi-Blast search in non-interactive mode Permits: batch of searches at once from a multifasta file of keys. search of multiple databases at once inclusion of user-created databases, allowing inclusion of private sequences in reverse Psi-Blast strategies Example blastpgp -i key.fa -d "$BLAST_DB/nr mylib" -j 8 -o results.out restriction of database searched to a list of gi numbers (for increased sensitivity). Get the gi list from NCBI Entrez using a web browser, e.g. retrieve all bacterial gi's with Entrez query bacteria[orgn], select "gi list" in the display box, and download. blastpgp -i key.fa -d $BLAST_DB -l list.gi -j 8 -o results.out creation of PSSM in one database, and then use in another. creation of PSSM for use by blastall in psitblastn mode. creation of PSSM for conversion to form used by rpsblast. Conducts PHI-Blast search A requirement to match a regular expression specifying a sequence motif is added to the first round of a Psi-Blast search. Note: command line Psi-Blast requires you to specify the regular expression in a file of a particular format (see documentation). NetBlast lets you just put the regular expression in a box. Syntax of of regular expressions for blast is described at http://www.ncbi.nlm.nih.gov/blast/html/PHIsyntax.html
seedtop	README.bls	Searches a database for match to a regular expression specifying a sequence motif. Can also search a library of patterns against a sequence. You would have to obtain the library (from the Procite database, for example). NetBlast only lets you do this in conjunction with a Blast search. Command line blast let's you do it separately.
bl2seq	README.bls	Blast two sequences against each other. Can be done in blastp, blastn, blastx, tblastx, or tblastn modes.
formatdb	README.formatdb	Creates a user-defined blast searchable database from a multifasta file. To get rid of redundancies that block formatdb over a list of sequences retrieved by Entrez: Set Entrez to retrieve refseq only. Retrieve gi list fastacmd -i gi.lst -d $BLAST_DB/nr -tT -pT -o library formatdb -i library -oT -pT
megablast	README.mbl	Speeds up a blastn search between two very long nucleotide sequences at the cost of assuming near identity. Mainly used for overlapping clones.
blastclust	README.bcl	Organizes a database into sets of homologous sequences
rpsblast	README.rps	Searches a sequence against a library of protein family models (mainly derived from the Pfam and smart databases). Databases are in /ncbi/rpsblast and allowable names are Pfam, Smart, Cog, Kog, and All. The case must be matched. e.g. rpsblast -i key.fa -d /ncbi/rps-blast/All -o <output filename> NOTE: the rpsblast database has not been maintained. If you need to do local rpsblast, contact Dr. Demeler. Permits: Inclusion of user-specified protein family models created with blastpgp. Search of a nucleotide sequence translated in all 6 frames.
impala	README.imp	Searches a protein family model (derived from blastpgp) against a database of sequences.
copymat makemat	README.rps	Programs used to convert PSSMs generated by blastpgp to protein family models used by rpsblast and impala.
fmerge		Merges two databases (as multifasta files) with removal of redundant gi's. It's a two step process resulting in addition of fasta entries from update_file to the multifasta file oldlib: fmerge -t 1 -n oldlib -i index.oldlib fmerge -t 2 -m update_file -i index.oldlib Look in fmerge.log to see summary of the process.

Access to databases from GCG

The GCG programs blast, and psiblast can be redirected to search the daily updated local databases by adding the parameter -INfile2=$BLAST_DB/<database name> to the command line.

To retrieve specific sequences from the daily updated databases, instead of the GCG fetch function, use fastacmd -s <accession number> -d $BLAST_DB/<database name> -o <output file> to retrieve the sequence. If the database is protein nr, also include -pT in the parameter list to avoid confusion with the nucleotide nr database. The file retrieved will be in fasta format. Use GCG command fromfasta to convert it to GCG format, if desired.

We do not at the moment have a way to make the daily updated databases available to non-blast GCG programs like FastA. We recommend the Blast suite in place of those programs in any case.

Last update 7/12/06; Steve Hardies