Running multiple parallel scripts from a bash script using GNU parallel

 Supposedly, if we want to run any perl/any language program in bash on mutiple input files (say 1000 inputs files) and if the number of threads in my computer is either 48 or 64 only, sometimes there might be overload which can lead to Resource temporarily unavailable error.

So, to circumvent this problem, we want to run a batch of 48 scripts/commands parallely without overloading the resources. This can be done using GNU parallel. The advantage of GNU parallel is, By default, parallel runs as many jobs in parallel as there are CPU cores.

In order to do this, we will create a file with commands and pass it to GNU parallel.

time for d in $(ls */*fq); 

do 

echo "perl /storage/apps/SNP_Validation_Scripts/tools/Q20_Q30_Stats_wo_functions.pl $d";

 done >parallel_script_Q20_Q30_stats.sh

parallel < parallel_script_Q20_Q30_stats.sh

And thats it!