SeekDeep

SeekDeep is one command line program that contains several programs within that all combined together make up the SeekDeep targeted sequencing analysis pipeline. Currently installs only MAC, Ubuntu, and other UNIX based systems is supported.

Downloading code

back to top
Source code for the program is hosted at github and can be downloaded by either using git (recommended) or by downloading a compressed folder of the code
The recommended approach is to use git to clone the repository for the code because this is the easiest way to keep the code up to date.

Code

cd ~
git clone https://github.com/bailey-lab/SeekDeep
cd SeekDeep

Or downloading compressed folder, latest version is v3.0.1, this is not recommended as this will make it harder to keep the code up to date

Code

cd ~
wget https://github.com/bailey-lab/SeekDeep/archive/v3.0.1.tar.gz
tar zxvf v3.0.1.tar.gz 
cd SeekDeep-v3.0.1/

Installing Dependencies

Compliers

You need to install either clang or g++-10 or greater to compile SeekDeep, the latest g++ (g++-10) is recommended as it will provide the fastest binary.

Clang

For installing on Ubuntu the repository needs to be added for you version of Ubuntu before it can be installed via apt-get install. clang-3.8 or higher is required. So depending on your version of Ubuntu, which can be check with lsb_release -sc, add the following to /etc/apt/sources.list or to a file in /etc/apt/sources.list.d/ which can be named llvm.list to keep track of what is in there
For Ubuntu version 14.04 trusty

Code

#add to /etc/apt/sources.list 
#or create /etc/apt/sources.list.d/llvm.list and add there
deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.8 main
deb-src http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.8 main
deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty main
deb-src http://llvm.org/apt/trusty/ llvm-toolchain-trusty main

Or For Ubuntu version 16.04 xenial

Code

#add to /etc/apt/sources.list 
#or create /etc/apt/sources.list.d/llvm.list and add there
deb http://llvm.org/apt/xenial/ llvm-toolchain-xenial-3.8 main
deb-src http://llvm.org/apt/xenial/ llvm-toolchain-xenial-3.8 main
deb http://llvm.org/apt/xenial/ llvm-toolchain-xenial main
deb-src http://llvm.org/apt/xenial/ llvm-toolchain-xenial main

Or rather than adding those lines by hand you can use the below commands to create and add to /etc/apt/sources.list.d/llvm.list (version will be determined)

Code

echo "deb http://llvm.org/apt/"$(lsb_release -sc)"/ llvm-toolchain-"$(lsb_release -sc)"-3.8 main" | sudo tee /etc/apt/sources.list.d/llvm.list
echo "deb-src http://llvm.org/apt/"$(lsb_release -sc)"/ llvm-toolchain-"$(lsb_release -sc)"-3.8 main" | sudo tee -a /etc/apt/sources.list.d/llvm.list
echo "deb http://llvm.org/apt/"$(lsb_release -sc)"/ llvm-toolchain-"$(lsb_release -sc)" main" | sudo tee -a /etc/apt/sources.list.d/llvm.list
echo "deb-src http://llvm.org/apt/"$(lsb_release -sc)"/ llvm-toolchain-"$(lsb_release -sc)" main" | sudo tee -a /etc/apt/sources.list.d/llvm.list

Now just add the gpg key and then install with apt-get

Code

#for libstdc++
sudo add-apt-repository ppa:ubuntu-toolchain-r/test

# the gpg key needs to be added for apt-get to understand 
wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key |sudo apt-key add -

sudo apt-get update
sudo apt-get install clang-3.8

Gcc

Code

sudo add-apt-repository ppa:ubuntu-toolchain-r/test

sudo apt-get update
sudo apt-get install g++-10

Libraries

back to top
A few external libraries are needed for the libraries that SeekDeep uses. As of 2.5.2 only cmake (needed by bamtools, restbed,and jsoncpp) is needed.

To get the newest cmake, the following ppa can be added and then install via apt-get install

Code

sudo add-apt-repository ppa:george-edison55/cmake-3.x
sudo apt-get update
sudo apt-get install cmake

Or you can use the setup script from SeekDeep to install SeekDeep. You will then need to add SeekDeep/bin, where it will install cmake, to your PATH variable before continuing.

Code

./setup.py --libs cmake:3.7.2 --symlinkBin

To get git

Code

sudo apt-get update
sudo apt-get install git

Brand New Ubuntu Machine

For a newly installed Ubuntu environment, the following script is used and may be of use rather then installing just what is needed like above.

Code

#!/usr/bin/env bash

##!!!!!!!!!! Use your owntime zone here !!!!!!!!!!!
echo America/New_York | sudo tee  /etc/timezone  && sudo dpkg-reconfigure --frontend noninteractive tzdata
export DEBIAN_FRONTEND=noninteractive
sudo apt-get update && sudo apt-get dist-upgrade -y && sudo apt-get -y autoremove
sudo apt-get install -y build-essential software-properties-common libcurl4-openssl-dev
sudo apt-get install -y git make

sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y
sudo apt-get update && sudo apt-get install -y g++-10

cd ~
git clone https://github.com/bailey-lab/SeekDeep
cd SeekDeep

./setup.py --libs cmake:3.7.2 --symlinkBin
echo "" >> ~/.profile && echo "#Add SeekDeep bin to your path" >> ~/.profile && echo "export PATH=\"$(pwd)/bin:\$PATH\"" >> ~/.profile
. ~/.profile
./setup.py --addBashCompletion
./install.sh 7

#add other tools 
./setup.py --libs muscle:3.8.31 --symlinkBin --overWrite

For non-sudo users

back to top
For non-sudo users on a Ubuntu machine if you cannot get your admin to install the dependencies above I would recommend using linuxbrew which is a useful software for local users anyways as it contains more up to date software than the default Ubuntu repositories.

The install is easy and depends only that your machine has ruby,curl, git, and python-setuptools. If you don’t have those on your machine you may need your admin to run the following

Code

sudo apt-get install build-essential curl file git

You then can run the following to install

Code

sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)"

And then to add brew to your path to use brew itself and the tools it install

Code

PATH="$HOME/.linuxbrew/bin:$PATH"

And to make it so it’s automatically added at login add to your ~/.profile or ~/.bash_profile depending which file exists

Code

echo 'export PATH="$HOME/.linuxbrew/bin:$PATH"' >>~/.profile

Now you can use brew to install dependencies and then compfile SeekDeep,

Code

brew update && brew upgrade
brew install gcc cmake 

#brew can't install clang so usee g++ 
CC=gcc CXX=g++ ./configure.py
./setup.py --compfile compfile.mk --outMakefile makefile-common.mk 
make -j 4

Configuring

back to top
Once code has been downloaded it needs to be compiled but first come dependencies have to be downloaded and a compiler has to be chosen. SeekDeep requires > g++-10 or clang++-3.8 to be compiled.

Code

#configure and choose compilers
#no arguments assumes g++-10 or whatever environmental CC and CXX are set to 
./configure.py

#compiling with gcc on ubuntu
./configure.py -CC gcc-7 -CXX g++-10

Installing Further Dependencies Libraries

back to top
Calling configure will create a file called compfile.mk which will contain info on the libraries needed and what compilers to use, this is give to setup.py to download the libraries SeekDeep relay on. This may take some time especially if being done on a machine that doesn’t have many cores. Some of these dependencies might be worked out in future releases making install easier.

Code

# will install dependent libaries in a folder called external
./setup.py --compfile compfile.mk --outMakefile makefile-common.mk

This command should report on whether anything failed to install, if this does happen, contact Nick Hathaway at Nicholas.Hathaway@umassmed.edu to complain to him his install is annoying and therefore failing, and send along any errors messages that can be found.

Compiling

back to top
If the setup.py was able to install everything, next is to make SeekDeep itself which takes a simple make command and will compile SeekDeep in a directory called bin/ in the SeekDeep directory

Code

make

To use more cores to speed up make use -j, below would use 4 cores

Code

make -j 4

Again if this fails to compile, complain to Nick (Nicholas.Hathaway@umassmed.edu) that he should get his act together to make this an easier install.

Adding SeekDeep to path

back to top
To call SeekDeep from the command line it will need to be called either with it’s full name or adding the bin directory in the SeekDeep directory to your environmental path. This is done by modifying your ~/.profile file (on MAC it’s ~/.bash_profile). Also this won’t take effect until this file is sourced either on your next log in or by called source ~/.profile

Code

#add the following to your ~/.profile or ~/.bash_profile and replace /home/user/ with the path where SeekDeep was downloaded
export PATH=/home/user/SeekDeep/bin/:$PATH

Adding auto flag completion

back to top
A lot of the flags in SeekDeep can be long and annoying to type in so auto flag completion has been added in the folder SeekDeep/etc/bash_completion.d in a file called SeekDeep. The content of this file should be added to ~/.bash_completion though your shell environment must already be set up for bash completion which is true for most Ubuntu. There is also a command in ./setup.py that will add this to .bash_completion for you (it simply does what was said above and puts the contents of the file in SeekDeep/etc/bash_completion.d/SeekDeep to ~/.bash_completion)

Code

./setup.py --addBashCompletion

Adding additional tools.

muscle is a third party tools that SeekDeep call and should be installed as

Code

#add other tools 
./setup.py --libs muscle:3.8.31 --symlinkBin --overWrite

Calling SeekDeep commands

To get a list of commands that SeekDeep can call simply call it without any arguments

Code

SeekDeep

Version 3.0.2-dev
Programs
Use SeekDeep [PROGRAM] --help to see more details about each program
Commands are not case sensitive
SeekDeep
1) extractor
2) extractorByKmerMatching
3) extractorPairedEnd
4) kluster
5) makeSampleDirectories
6) processClusters
7) qluster
SeekDeepServer
1) genProjectConfig
2) popClusteringViewer
SeekDeepUtils
01) benchmarkControlMixturesOnProcessedClustersDir
02) benchmarkMultiTarAmpControlMixtures
03) benchmarkTarAmpControlMixtures
04) combineBasicResultsFiles
05) deRepPopClusDir
06) dryRunQualityFiltering
07) gatherInfoOnTargetedAmpliconSeqFile
08) genTargetInfoFromGenomes
09) getPossibleSampleNamesFromRawInput
10) primersToFasta
11) rBind
12) replaceUnderscores
13) runMultipleCommands
14) SampleBarcodeFileToSeekDeepInput
15) setupTarAmpAnalysis
16) variantCallOnSeqAndProtein

Then to call a sub-command just type it’s name after SeekDeep, and without adding arguments will automatically print it’s help message

Code

SeekDeep qluster

#or with -help/-h 
SeekDeep qluster --help

Also the commands and flags of SeekDeep are not case sensitive so all the below would work

Code

SeekDeep qluster --help 
SeekDeep qluster --HELP
SeekDeep qluster --HeLP
SeekDeep qluster --HeLp
SeekDeep QLUSTER --HeLP
SeekDeep qluster --HeLp
SeekDeep qlUstEr --HeLp

# SeekDeep SeekDeep is one command line program that contains several programs within that all combined together make up the SeekDeep targeted sequencing analysis pipeline. Currently installs only MAC, Ubuntu, and other UNIX based systems is supported. # Downloading code [back to top](#TOC) Source code for the program is hosted at [github](https://github.com/bailey-lab/SeekDeep) and can be downloaded by either using git (recommended) or by downloading a compressed folder of the code The recommended approach is to use git to clone the repository for the code because this is the easiest way to keep the code up to date. ```{r downloading-git, echo=T, eval=FALSE, engine='bash'} cd ~ git clone https://github.com/bailey-lab/SeekDeep cd SeekDeep ``` **Or** downloading compressed folder, latest version is v3.0.1, this is not recommended as this will make it harder to keep the code up to date ```{r downloading-tar, echo=T, eval=FALSE, engine='bash'} cd ~ wget https://github.com/bailey-lab/SeekDeep/archive/v3.0.1.tar.gz tar zxvf v3.0.1.tar.gz cd SeekDeep-v3.0.1/ ``` # Installing Dependencies [back to top](#TOC) ## Compliers You need to install either clang or g++-10 or greater to compile SeekDeep, the latest g++ (g++-10) is recommended as it will provide the fastest binary. ### Clang For installing on Ubuntu the repository needs to be added for you version of Ubuntu before it can be installed via apt-get install. clang-3.8 or higher is required. So depending on your version of Ubuntu, which can be check with `lsb_release -sc`, add the following to /etc/apt/sources.list or to a file in /etc/apt/sources.list.d/ which can be named llvm.list to keep track of what is in there For Ubuntu version 14.04 trusty ```{r , echo=T, eval=FALSE, engine='bash'} #add to /etc/apt/sources.list #or create /etc/apt/sources.list.d/llvm.list and add there deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.8 main deb-src http://llvm.org/apt/trusty/ llvm-toolchain-trusty-3.8 main deb http://llvm.org/apt/trusty/ llvm-toolchain-trusty main deb-src http://llvm.org/apt/trusty/ llvm-toolchain-trusty main ``` **Or** For Ubuntu version 16.04 xenial ```{r , echo=T, eval=FALSE, engine='bash'} #add to /etc/apt/sources.list #or create /etc/apt/sources.list.d/llvm.list and add there deb http://llvm.org/apt/xenial/ llvm-toolchain-xenial-3.8 main deb-src http://llvm.org/apt/xenial/ llvm-toolchain-xenial-3.8 main deb http://llvm.org/apt/xenial/ llvm-toolchain-xenial main deb-src http://llvm.org/apt/xenial/ llvm-toolchain-xenial main ``` **Or** rather than adding those lines by hand you can use the below commands to create and add to /etc/apt/sources.list.d/llvm.list (version will be determined) ```{r , echo=T, eval=FALSE, engine='bash'} echo "deb http://llvm.org/apt/"$(lsb_release -sc)"/ llvm-toolchain-"$(lsb_release -sc)"-3.8 main" | sudo tee /etc/apt/sources.list.d/llvm.list echo "deb-src http://llvm.org/apt/"$(lsb_release -sc)"/ llvm-toolchain-"$(lsb_release -sc)"-3.8 main" | sudo tee -a /etc/apt/sources.list.d/llvm.list echo "deb http://llvm.org/apt/"$(lsb_release -sc)"/ llvm-toolchain-"$(lsb_release -sc)" main" | sudo tee -a /etc/apt/sources.list.d/llvm.list echo "deb-src http://llvm.org/apt/"$(lsb_release -sc)"/ llvm-toolchain-"$(lsb_release -sc)" main" | sudo tee -a /etc/apt/sources.list.d/llvm.list ``` Now just add the gpg key and then install with apt-get ```{r , echo=T, eval=FALSE, engine='bash'} #for libstdc++ sudo add-apt-repository ppa:ubuntu-toolchain-r/test # the gpg key needs to be added for apt-get to understand wget -O - http://llvm.org/apt/llvm-snapshot.gpg.key |sudo apt-key add - sudo apt-get update sudo apt-get install clang-3.8 ``` ### Gcc [back to top](#TOC) For Ubuntu, the repository for the newest gcc needs to be added and g++-10 or higher can be installed ```{r , echo=T, eval=FALSE, engine='bash'} sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install g++-10 ``` ### Libraries [back to top](#TOC) A few external libraries are needed for the libraries that SeekDeep uses. As of 2.5.2 only cmake (needed by bamtools, restbed,and jsoncpp) is needed. To get the newest cmake, the following ppa can be added and then install via apt-get install ```{r , echo=T, eval=FALSE, engine='bash'} sudo add-apt-repository ppa:george-edison55/cmake-3.x sudo apt-get update sudo apt-get install cmake ``` Or you can use the setup script from SeekDeep to install SeekDeep. You will then need to add SeekDeep/bin, where it will install cmake, to your PATH variable before continuing. ```{r , echo=T, eval=FALSE, engine='bash'} ./setup.py --libs cmake:3.7.2 --symlinkBin ``` To get git ```{r , echo=T, eval=FALSE, engine='bash'} sudo apt-get update sudo apt-get install git ``` ### Brand New Ubuntu Machine For a newly installed Ubuntu environment, the following script is used and may be of use rather then installing just what is needed like above. ```{r , echo=T, eval=FALSE, engine='bash'} #!/usr/bin/env bash ##!!!!!!!!!! Use your owntime zone here !!!!!!!!!!! echo America/New_York | sudo tee /etc/timezone && sudo dpkg-reconfigure --frontend noninteractive tzdata export DEBIAN_FRONTEND=noninteractive sudo apt-get update && sudo apt-get dist-upgrade -y && sudo apt-get -y autoremove sudo apt-get install -y build-essential software-properties-common libcurl4-openssl-dev sudo apt-get install -y git make sudo add-apt-repository ppa:ubuntu-toolchain-r/test -y sudo apt-get update && sudo apt-get install -y g++-10 cd ~ git clone https://github.com/bailey-lab/SeekDeep cd SeekDeep ./setup.py --libs cmake:3.7.2 --symlinkBin echo "" >> ~/.profile && echo "#Add SeekDeep bin to your path" >> ~/.profile && echo "export PATH=\"$(pwd)/bin:\$PATH\"" >> ~/.profile . ~/.profile ./setup.py --addBashCompletion ./install.sh 7 #add other tools ./setup.py --libs muscle:3.8.31 --symlinkBin --overWrite ``` ### For non-sudo users [back to top](#TOC) For non-sudo users on a Ubuntu machine if you cannot get your admin to install the dependencies above I would recommend using [linuxbrew](http://brew.sh/) which is a useful software for local users anyways as it contains more up to date software than the default Ubuntu repositories. The install is easy and depends only that your machine has ruby,curl, git, and python-setuptools. If you don't have those on your machine you may need your admin to run the following ```{r , echo=T, eval=FALSE, engine='bash'} sudo apt-get install build-essential curl file git ``` You then can run the following to install ```{r , echo=T, eval=FALSE, engine='bash'} sh -c "$(curl -fsSL https://raw.githubusercontent.com/Linuxbrew/install/master/install.sh)" ``` And then to add brew to your path to use brew itself and the tools it install ```{r , echo=T, eval=FALSE, engine='bash'} PATH="$HOME/.linuxbrew/bin:$PATH" ``` And to make it so it's automatically added at login add to your `~/.profile` or `~/.bash_profile` depending which file exists ```{r , echo=T, eval=FALSE, engine='bash'} echo 'export PATH="$HOME/.linuxbrew/bin:$PATH"' >>~/.profile ``` Now you can use brew to install dependencies and then compfile SeekDeep, ```{r , echo=T, eval=FALSE, engine='bash'} brew update && brew upgrade brew install gcc cmake #brew can't install clang so usee g++ CC=gcc CXX=g++ ./configure.py ./setup.py --compfile compfile.mk --outMakefile makefile-common.mk make -j 4 ``` # Configuring [back to top](#TOC) Once code has been downloaded it needs to be compiled but first come dependencies have to be downloaded and a compiler has to be chosen. SeekDeep requires > g++-10 or clang++-3.8 to be compiled. ```{r configuring, echo=T, eval=FALSE, engine='bash'} #configure and choose compilers #no arguments assumes g++-10 or whatever environmental CC and CXX are set to ./configure.py #compiling with gcc on ubuntu ./configure.py -CC gcc-7 -CXX g++-10 ``` ## Installing Further Dependencies Libraries [back to top](#TOC) Calling configure will create a file called compfile.mk which will contain info on the libraries needed and what compilers to use, this is give to setup.py to download the libraries SeekDeep relay on. This may take some time especially if being done on a machine that doesn't have many cores. Some of these dependencies might be worked out in future releases making install easier. ```{r install_dependencies, echo=T, eval=FALSE, engine='bash'} # will install dependent libaries in a folder called external ./setup.py --compfile compfile.mk --outMakefile makefile-common.mk ``` This command should report on whether anything failed to install, if this does happen, contact Nick Hathaway at Nicholas.Hathaway@umassmed.edu to complain to him his install is annoying and therefore failing, and send along any errors messages that can be found. # Compiling [back to top](#TOC) If the setup.py was able to install everything, next is to make SeekDeep itself which takes a simple `make` command and will compile SeekDeep in a directory called bin/ in the SeekDeep directory ```{r compiling, echo=T, eval=FALSE, engine='bash'} make ``` To use more cores to speed up make use -j, below would use 4 cores ```{r compiling-threaded, echo=T, eval=FALSE, engine='bash'} make -j 4 ``` Again if this fails to compile, complain to Nick (Nicholas.Hathaway@umassmed.edu) that he should get his act together to make this an easier install. # Adding SeekDeep to path [back to top](#TOC) To call SeekDeep from the command line it will need to be called either with it's full name or adding the bin directory in the SeekDeep directory to your environmental path. This is done by modifying your ~/.profile file (on MAC it's ~/.bash_profile). Also this won't take effect until this file is sourced either on your next log in or by called `source ~/.profile` ```{r adding_to_path, echo=T, eval=FALSE, engine='bash'} #add the following to your ~/.profile or ~/.bash_profile and replace /home/user/ with the path where SeekDeep was downloaded export PATH=/home/user/SeekDeep/bin/:$PATH ``` # Adding auto flag completion [back to top](#TOC) A lot of the flags in **SeekDeep** can be long and annoying to type in so auto flag completion has been added in the folder SeekDeep/etc/bash_completion.d in a file called SeekDeep. The content of this file should be added to ~/.bash_completion though your shell environment must already be set up for bash completion which is true for most Ubuntu. There is also a command in `./setup.py` that will add this to .bash_completion for you (it simply does what was said above and puts the contents of the file in SeekDeep/etc/bash_completion.d/SeekDeep to ~/.bash_completion) ```{r auto_complete, echo=T, eval=FALSE, engine='bash'} ./setup.py --addBashCompletion ``` # Adding additional tools. muscle is a third party tools that SeekDeep call and should be installed as ```{r, echo=T, eval=FALSE, engine='bash'} #add other tools ./setup.py --libs muscle:3.8.31 --symlinkBin --overWrite ``` # Calling SeekDeep commands To get a list of commands that SeekDeep can call simply call it without any arguments ```{r callingSeekDeep, echo=T, eval=FALSE, engine='bash'} SeekDeep ``` ```{r, echo=F, eval=T, engine='bash'} SeekDeep ``` Then to call a sub-command just type it's name after SeekDeep, and without adding arguments will automatically print it's help message ```{r , echo=T, eval=FALSE, engine='bash'} SeekDeep qluster #or with -help/-h SeekDeep qluster --help ``` ```{r, echo=F, eval=F, engine='bash'} SeekDeep qluster --help | gsed -r "s/\x1B\[([0-9]{1,2}(;[0-9]{1,2})?)?[m|K]//g" | head -15 echo .... ``` Also the commands and flags of SeekDeep are not case sensitive so all the below would work ```{r, engine='bash',comment="",eval=FALSE} SeekDeep qluster --help SeekDeep qluster --HELP SeekDeep qluster --HeLP SeekDeep qluster --HeLp SeekDeep QLUSTER --HeLP SeekDeep qluster --HeLp SeekDeep qlUstEr --HeLp ```