Mar 31 1998
                                                         V.Rud   (MSU)
                                                         
       get_for_batch - tool to download files for fbatch
       __________________________________________________

  'get_for_batch' is modified version of 'get_from_fmss' script
  created to download files onto fbatch temporary disks. That could 
  be useful if you are going to process some data only once, so you 
  don't need to keep them on our local disk.
  
  'get_for_batch' script is in $SELEX_TOOLS_BIN, gets available by
  'setup -d selex_tools' (or 'setup -d off781' on fsgi03). Also one
  need do 'setup fmss'. 
                                           
  All of fbatch *_disk queues have allocated temporary disks:
   
  Queue       Space(GB)
  
  30min_disk  2.0
    4hr_disk  2.0
   12hr_disk  2.0
  1day_queue  2.0
  
  ALL of them are accessable NOT from 'fsgi03', but only from another SGI
  computers (4 SGI R4400 machines).                           ^^^^^^^
  
  But on fsgi03 we have our own local disk and queue:

  Queue       Space(GB)
  
  e781_disk  3.4    for each of 3 simultanious jobs
  
  Let me start with fbatch submitting example:
  
  unsetup off781
  fbatch_sub -q 4hr_disk -R irix run_my_batch 
  
  This command will submitt job to any SGI less loaded host into '4hr_disk'
  queue which has 2.0 GB disk. One needs to use 'unsetup off781' because
  unsetup in script below will fail on NON fsgi03 hosts (local fsgi03 disk
  will be not accessable).
  
  Users script 'run_my_batch' have to call 'get_for_batch' at the beginning:

run_my_batch:
______________________________________  
#!/bin/sh
#

exec_host=`uname -n`
echo Exec_host is $exec_host

# Make setup available internally:

  if [ -r /usr/local/etc/setups.sh ]; then
  	. /usr/local/etc/setups.sh
  	  echo Setup installed
  fi
#
# Setup fmss:
#
  setup fmss

if [ "$exec_host" != "fsgi03" ]; then

# To use AFS version of products:

  PRODUCTS="/afs/fnal.gov/files/home/room2/e781/products/ups_database/declared /afs/fnal/products/ups_database/declared"
  export PRODUCTS

  printenv PRODUCTS

# Setup AFS version of off781:

  setup -z /afs/fnal.gov/files/home/room2/e781/products/ups_database/declared/ -d off781
  
fi
echo OFF781_DIR=$OFF781_DIR

  get_for_batch -t charm -run 8915_002:7

  soap.exe -f my.cmd
______________________________________  
  
  'get_for_batch' script has three groups of arguments (you may invoke
  it  with -h argument to look on usage and examples):  
  
  '-t <type>' have to be the FIRST  argument to specify type of runs 
              to be processed (charm|calib|prim).
  
  Second group of arguments gives run numbers and file numbers for this type
  
  -run <run> in some obvious format: '8915_002:7' means that you request 
             for run 8915 files from 2 to 7 ('8915_002' - only one 2nd file).
  
  You may request any number of runs:
  
  get_for_batch -t charm -run 8915_002:7 -run 8920_001:32 # two runs at once
  
  If you are going to process files with nonstandard names you may use
  another group of arguments for single files:
  
  -f <file> where file name could be with range (like 'my_file_10822_0(01:17)')
  
  This way you may also request any number of runs:

  get_for_batch -f pz001_charm_run010315_001.out1 # single file

  get_for_batch -f pz001_charm_run010315_001.out1 -f 'pz001_charm_run0103(25:45)_001.out1' 
  
  By last request 1+21=22 files will be 'downloaded' - see below.
  
  But remember that each data file is about 200 MB long and batch disk space is 
  limited.
  
  'get_for_batch' checks available disk space before downloading next file.
  If it see that space is not enough it stops.
  
  For requested files 'get_for_batch' looks for files firstly in FMSS and than 
  in needfile like 'get_from_fmss' does.
  
  IMPORTANT: fbatch *_disk computers have NO access to our local disks 
  (/usr/e781/*) on fsgi03, but they HAVE access to AFS.
  
  So, if you would like to process files from our local disk, include them 
  please into 'perm_disk.cmd' file in your running directory and they will 
  be copied to the batch disk:

perm_disk.cmd:
______________ 
!____________________________________________________
disk in /usr/e781/data01/standard/charm_run008915_001
!____________________________________________________

  'get_for_batch' produces in output file 'files_batch.cmd' list of all
  copied and downloaded files:

files_batch.cmd:
________________ 
disk in /tmp/spacall.pid16990/charm_run008915_001
disk in /tmp/spacall.pid16990/charm_run008915_002

  For SOAP include call to 'files_batch.cmd' into your 'my.cmd' file: 

my.cmd:
!_______________________
........................
in files_batch.cmd
ana 0
exit
!_______________________

  Output from 'get_for_batch' is mostly the same as from 'get_from_fmss', 
  only info about disk usage and list of downloaded files added.
 
  Any comments and suggestions are welcome (rud@fnal).
____________________________________________________________________

Updated on Oct 17, 1999