Tip: Single Cell Analysis in python - Doublet Detection notes

 Had some issues, so ran the following commands:

# remotes::install_github("davismcc/scater")  in R prompt in the conda environment from linux terminal

# BiocManager::install("scDblFinder") - in R prompt in conda environment from linux terminal

Tip: the item to be deployed exceeds the maximum deployment size

Tip: the item to be deployed exceeds the maximum deployment size  

When deploying R Shiny app, I constantly had this pop-up that I exceeding the maximum deployment size which was around 1.3Gb.

But, my file sizes all combined was around ~320 Mb.

I realized that the .RData was around ~1.1Gb. Once I identified and removed this file, the deployment size was brought within the acceptable limits, resolving the issue.

Cutoffs and Thresholds for single nuclei and single cell RNAseq data

 

paper : https://www.biorxiv.org/content/biorxiv/early/2021/06/25/2021.06.25.449944.full.pdf

Data Downloads with SFTP Commands

sftp> lpwd

Local working directory: /home/user/

sftp> lcd /home/user/Data/

sftp> lpwd

Local working directory: /home/user/Data/

for d in abc842  abc9829 abc8424; do var=`echo $d | sed 's/abc/Sample_ABC/g'`; echo "get -R $var"; done

get -R Sample_ABC8425

get -R Sample_ABC9829

get -R Sample_ABC8424

paste the above get-R commands in the sftp command prompt to download the data to server

Resolved: Error: object ‘LayerData<-’ is not exported by 'namespace:SeuratObject'

Signac gives me :

Error: object ‘LayerData<-’ is not exported by 'namespace:SeuratObject'

Execution halted

I updated my SeuratObject (4.1.3     -> 5.0.0    ) [CRAN]

Notes on snapatac2

When I ran:

data = snap.pp.import_data(

    fragment_file,

    chrom_sizes=snap.genome.hg38,

    #file="2_filtered_data/ATAC/atac_test.h5ad",  # Optional

    sorted_by_barcode=False,)

------------------------

fragment_file being fragments.tsv.gz and fragments.tsv generated the error

PanicException: called `Result::unwrap()` on an `Err` value: Custom { kind: Other, error: "Unknown frame descriptor" }

instead i used fragments.srt.bed.gz generated from :

time sort -k4,4 fragments.tsv | gzip - > fragments.srt.bed.gz,

resolved the issue

Issues with running scArches

The scarches command below:

scarches_model = sca.models.SCVI.load_query_data( adata=adata_to_map_augmented, reference_model="./reference_model", freeze_dropout=True)

gave the following error:

RuntimeError: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver.

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver

Copyright (c) 2005-2021 NVIDIA Corporation

Built on Thu_Nov_18_09:45:30_PST_2021

Cuda compilation tools, release 11.5, V11.5.119

Build cuda_11.5.r11.5/compiler.30672275_0

import torch

print(torch.version.cuda)

12.1

So, I ran the following:

pip install light-the-torch
ltt install torch torchvision

Looks like my cuda and torch versions are not same. it is scary to update nvidia drivers without backing up. Coz this may result in issues. So thought I will not delve into cuda for now.

But again I got this error:

RuntimeError: Detected that PyTorch and torchvision were compiled with different CUDA major versions. PyTorch has CUDA Version=12.1 and torchvision has CUDA Version=11.8. Please reinstall the torchvision that matches your PyTorch install.
I did the following:
conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia

This time again another error:
undefined symbol when importing torchaudio with pytorch...

Now, did the following, the above error was resolved
pip install -U torch torchaudio --no-cache-dir

Instead of gpu I relied on CPU. 
scarches_model = sca.models.SCVI.load_query_data(
    adata=adata_to_map_augmented,
    reference_model="./reference_model",
    freeze_dropout=True,
    use_gpu=False
)
Now, I have only a warning as below:
WARNING:jax._src.xla_bridge:An NVIDIA GPU may be present on this machine, but a CUDA-enabled jaxlib is not installed. Falling back to cpu.

Blank Screen Cursor at Top Left Ubuntu 22.04 LTS

I changed to Nouveau driver and I could not login 

cannot login to tty using CTRL+ALT+[F1 to F7] - Blank screen still persists

Reboot to recovery mode 

- Shift did not work at all

- ESC took me to grub mode - but nothing worked! ( i think i pressed ESC too long!)

- ESC until I saw logo (not ubuntu) on the monitor - 

https://support.starlabs.systems/kb/guides/using-recovery-mode

https://linuxconfig.org/how-to-uninstall-the-nvidia-drivers-on-ubuntu-20-04-focal-fossa-linux

Then I pressed enter for option: Press Enter for maintenance

nvidia-smi

ubuntu-drivers autoinstall

apt-get update

ubuntu-drivers devices

apt install nvidia-driver-535

apt-get purge nvidia*

sudo ubuntu-drivers autoinstall 

sudo reboot

This still did not result in correct configuration. I changed the driver from 535 to 470, then it started working!

Notes on adata object

mdata["GEX"].X = mdata["GEX"].layers['counts'].copy()

sc.pp.normalize_total(mdata["GEX"], target_sum=1e4)

sc.pp.log1p(mdata["GEX"])

sc.pp.highly_variable_genes(mdata["GEX"], n_top_genes=2000, batch_key='batch')

sc.pp.neighbors(mdata["GEX"])

WARNING: You’re trying to run this on 13953 dimensions of `.X`, if you really want this, set `use_rep='X'`.

         Falling back to preprocessing with `sc.pp.pca` and default params.

In the above, I did not explicitly calculate the pca. So, scanpy is letting me know that it is calculating the PCA and then use that to calculate the neighbors. This makes sense, just because we have ~13K genes does not means we capture more information. Too much high dimensional data is difficult even computationally deal with. So, doing a PCA makes sense here. 

My conda environments for single cell analysis

Environment 1 - Automatic annotation_celltypist (successful)

 

 conda create --name annotation_celltypist

 conda activate annotation_celltypist

 conda install -c conda-forge mamba

 mamba install -c conda-forge scanpy python-igraph leidenalg
 pip install jupyter

 mamba install -c bioconda -c conda-forge celltypist

 pip install -U scarches

 pip install urllib - was automatically installed via scarches I think

 pip install "pandas<2.0.0"

=========================================

Environment 2 - scanpy (successful)

conda create --name sc_manual_annotation
conda activate sc_manual_annotation
conda install -c conda-forge mamba
mamba install -c conda-forge scanpy
pip install jupyter

=========================================

Environment 3 - counts_to_clustering + nichenet (successful)

conda create --name singlecell
conda activate singlecell
conda install -c conda-forge mamba
mamba install -c conda-forge scanpy python-igraph leidenalg
mamba install -c conda-forge altair matplotlib numpy pandas seaborn
pip install jupyter
pip install anndata2ri
pip install sccoda

In linux:
sudo apt install libcurl4-openssl-dev
sudo apt-get install libmkl-rt
sudo apt install libfontconfig1-dev
sudo apt-get install libcairo2-dev
sudo apt-get install  libharfbuzz-dev libfribidi-dev
sudo apt-get install libfreetype6-dev libpng-dev libtiff5-dev libjpeg-dev
sudo apt install cmake

In R studio:
install.packages("SoupX")
install.packages("systemfonts", dependencies = TRUE)
remotes::install_github("davismcc/scater", dependencies = TRUE)
install.packages("Cairo")
install.packages("textshaping")
install.packages("ragg")
install.packages("ggraster")
BiocManager::install("scDblFinder")
BiocManager::install("BiocParallel")
BiocManager::install("scry")
install.packages("ggpubr")
library("devtools"); install_github("lme4/lme4",dependencies=TRUE)
BiocManager::install("ComplexHeatmap")
devtools::install_github("saeyslab/nichenetr")
install.packages("tidyverse")

## advise to self: dont install R packages in jupyter notebook install in Rstudio

 =========================================

Environment 4 - CITE-Seq data analysis (in Apple M2 Macbook) - successful

brew install --cask mambaforge 

mamba create -n scvi_mamba

mamba activate scvi_mamba

mamba install -y -c conda-forge  python=3.9 scanpy python-igraph leidenalg altair matplotlib numpy pandas seaborn scvi-tools muon

pip install jupyter

pip install --user scikit-misc

Issue 1: Sometimes mamba activate does not work in mac m2 notebooks and it would just throw only the following lines when trying to activate an environment.

% mamba activate scvi_mamba

Run 'mamba init' to be able to run mamba activate/deactivate

and start a new shell session. Or use conda to activate/deactivate.

Then, copy the following contents in .bash_profile to .zhsrc file

# >>> conda initialize >>>

# !! Contents within this block are managed by 'conda init' !!

......

# <<< conda initialize <<<

Now 

source ~/.zshrc

Issue 2: 

NotImplementedError: The operator 'aten::lgamma.out' is not currently implemented for the MPS device. If you want this op to be added in priority during the prototype phase of this feature, please comment on https://github.com/pytorch/pytorch/issues/77764. As a temporary fix, you can set the environment variable `PYTORCH_ENABLE_MPS_FALLBACK=1` to use the CPU as a fallback for this op. WARNING: this will be slower than running natively on MPS.

Solution: In the terminal run this before opening Jupyter notebook

% export PYTORCH_ENABLE_MPS_FALLBACK=1

Else, in the Jupyter notebook

% env PYTORCH_ENABLE_MPS_FALLBACK=1

In my case, I aborted my notebook and ran the command in terminal which then resolve the issue.

Environment 5 - RNA velocity - successful

conda create -n scvelo_env python=3.9
conda activate scvelo_env
conda install numpy scipy cython numba matplotlib scikit-learn h5py click
pip install notebook
pip install scanpy
pip install leidenalg
pip install scvelo
pip install pandas==1.1.5 
pip install numpy==1.21.1

pip install git+https://github.com/theislab/cellrank.git@main

pip install pandas==1.1.5 

pip install numpy==1.21.1

pip install rpy2

cellrank==2.0.0 scanpy==1.9.3 anndata==0.9.2 numpy==1.24.4 numba==0.57.1 scipy==1.10.1 pandas==2.0.3 pygpcca==1.0.4 scikit-learn==1.1.3 statsmodels==0.14.0 scvelo==0.3.0 pygam==0.8.0 matplotlib==3.7.1 seaborn==0.12.2

Environment 5 - Differential Gene Expression - successful

conda create -n deg

conda activate deg

 conda install -c conda-forge scanpy python-igraph leidenalg
 pip install jupyter

pip install diffxpy