|
|
Fermilab |
|
MEMORANDUM OF UNDERSTANDING
Between the
MINOS Experiment and the Computing Division
February, 2000
TABLE OF CONTENTS
INTRODUCTION 3
I. PERSONNEL AND INSTITUTIONS 3
II. Fermilab Computing Division 3
III. SPECIAL CONSIDERATIONS 5
SIGNATURES 6
APPENDIX I - E875 PREP REQUEST 7
APPENDIX II - E875 DATA ACQUISITION REQUEST 8
APPENDIX III - DAQ SYSTEM DESCRIPTION 9
APPENDIX IV - COMPUTING OFF-LINE ANALYSIS MODEL 12
COMPUTING DIVISION RESOURCE ANALYSIS XX
This is a memorandum of understanding between the Fermi National Accelerator Laboratory Computing Division and the experimenters of MINOS (E-875). The memorandum is intended solely for the purpose of providing a budget estimate and a work allocation for Fermilab, the funding agencies and the participating institutions. It reflects an arrangement that currently is satisfactory to the parties; however, it is recognized and anticipated that changing circumstances of the evolving research program will necessitate revisions. The parties agree to negotiate amendments to this memorandum which will reflect such required adjustments.
I. PERSONNEL AND INSTITUTIONS
Spokesperson: S. Wojcicki, Stanford
Deputy Spokesperson: D. Ayres, ANL
MINOS Project Manager: A. Byon-Wagner, FNAL
MINOS Computing Off-line liaison: J. Urheim, Minnesota
MINOS Computing On-line liaison: ???
Relevant MINOS System Managers:
Electronics J. Thron, Argonne
(Institutions: Harvard, Oxford, Argonne, Fermilab)
Trigger and DAQ System G. Pearce, RAL
(Institutions: RAL)
Database P. Border, Minnesota
(Institutions: Minnesota, IHEP-Protvino)
Detector Control System T. Alexopoulos, Wisconsin
(Institutions: Wisconsin, Minnesota)
PREP Electronics ???
(Institutions: Texas A&M, Fermilab)
II. Fermilab Computing Division
2.1 The Computing Division liaison is A. Para.
2.2 The attached plan contains the experiment's present understanding of its analysis model from code development, through data logging, production, data analysis and Monte Carlo. A more detailed quantitative description is given in Appendix IV. The Computing Division cannot guarantee, at this time, that these resources can be made available. The Computing Division, guided by priorities set by management, will attempt to allocate on a quarterly basis, the available resources. The present request and amendments will be used in attempting to plan the laboratory's computing acquisition strategies.
Type of Funds Equipment Operating
database and other software products designated
as supported by the Computing Division. $ xxx K
2.6 Maintenance and update services for operating systems and packages at versions levels supported by the Computing Division. $ xxx K
2.7 Equipment as specified in Appendix I and II.
PREP equipment $ xxx K
DAQ equipment $ xxx K
2.8 Maintenance and repair of PREP and DAQ equipment. $ xxx K
III. SPECIAL CONSIDERATIONS
3.5 At the completion of the experiment:
The spokesperson is responsible for the return of all PREP, equipment, Computing equipment and non-PREP data acquisition electronics. If the return is not completed after a period of one year after the end of running the Spokesperson will be required to furnish, in writing, an explanation for any non-return.
SIGNATURES
_________________________________
M. Kasemann, Head of Fermilab Computing Division
________________________________
A. Byon-Wagner, MINOS Manager
________________________________
S. Wojcicki, MINOS Spokesperson
APPENDIX I - E875 PREP REQUEST
(
COLLECTED PREP LIST SPREADSHEET WILL BE FILLED HERE
APPENDIX II - E875 DATA ACQUISITION REQUEST
(JOnathan)
COLLECTED ONLINE DAQ RELATED EQUIPMENT LIST FROM cd SHOULD BE FILLED HERE
APPENDIX III - DAQ SYSTEM DESCRIPTION
(JONATHAN)
oNLINE DAQ SYSTEM DESCRIPTION SHOULD BE WRITTEN IN HERE, including list of equipment from CD needed and MINOS milestones
APPENDIX IV - COMPUTING OFF-LINE ANALYSIS MODEL
INTRODUCTION
This appendix deals with the offline computing needs for the MINOS experiment. The offline needs can be broken into the following areas:
This document will address the resources required in each of the preceding areas. These resources include number of tapes, amount of CPU, disk space and number of people.
EXPERIMENTAL DATA
In Tables I and II we list the data volumes per year that we expect to record to tape for MINOS from the far and near detectors respectively assuming a high energy beam. The number of neutrino interactions from the far detector is small, on the order of 22,000 per year. The rate from the far detector is dominated by 1Hz of cosmic ray m interactions. The neutrino interaction rate from the near detector will be somewhat larger, namely 105 Hz for the high energy beam. Of this 105 Hz, a small fraction, < 1 Hz , is in the central 25cm of the target region and will be used for neutrino oscillation studies. A somewhat larger region will be used for studies of conventional neutrino physics (about 4 Hz). There will also be 11 Hz of events where a neutrino interacts upstream in the rock and produces a m in the detector. The near detector DAQ system is capable of recording the full 250 Hz of cosmic rays m seen by the near detector but it is expected that we will record only a fraction of these for full reconstruction and this is reflected in the numbers in Table II. The far detector assumes 3´ 107 seconds in one year (cosmic rays are always there) and the near detector assumes an effective year of 107 seconds. For simplicity, 1Kbyte º 1000 bytes. The data is expected to expand by about a factor of 5 after processing.
|
Sample |
Rate/second (Hz) |
Events/year |
Event Size (Kbytes) |
Data Volume /year (GB) |
Processed Data Volume /year (GB) |
|
n events |
0.0022 |
22000 |
2.4 |
0.05 |
0.25 |
|
Cosmic ray m |
1 |
3´ 107 |
2.4 |
72 |
360 |
|
Total |
1 |
3´ 107 |
72 |
360 |
Table I Event rates for the far detector
|
Sample |
Rate/second (Hz) |
Events/year (107) |
Event Size (Kbytes) |
Data Volume /year (GB) |
Processed Data Volume /year (TB) |
|
n events |
105 |
105 |
0.8 |
840 |
4.2 |
|
m events |
11 |
11 |
0.8 |
88 |
0.44 |
|
Cosmic ray m |
11 |
11 |
0.8 |
88 |
0.44 |
|
Total |
127 |
127 |
1016 |
5.08 |
|
|
Oscillation n |
0.26 |
0.26 |
0.8 |
2 |
0.01 |
|
Conventional n |
4 |
4 |
0.8 |
32 |
0.16 |
Table II Event rates for the near detector
The total processed data volume per year from both near and far detectors is about 5.5 Terabytes. Assuming a tape capacity of 20 Gbytes (current STK technology) we would require about 300 tapes/year to archive the complete data from both detectors. This is about 5% of the current STK library per year. The physics data, both neutrino oscillation and conventional, will be stored permanently on disk (about 170 Gbytes/year). How much of the remaining data is stored permanently on disk is yet to be determined. As an example, given current disk capacities and prices of 50 Gbytes and $28/Gbyte we would require about 110 disks/year for the full 5.5 Terabytes at a cost of about $160,000 per year for JBOD. We expect disk capacities to continue to increase and prices to continue to drop making it feasible to store all the data on disk by the time MINOS begins data taking, should this be desirable.
Data will be recorded on disk locally at the far detector and transferred over the network to Fermilab/FCC (Feynman Computing Center) for archiving to tape. There will also be data archiving facilities at the far detector. The needs here are modest and the technology will be chosen later. The near detector data will be logged to disk locally at the experimental hall and then transferred to FCC to be archived to tape. The rates are 100 Kbyte/sec and 2.4 Kbyte/sec for the near and far detectors respectively.
ONLINE MONITORING AND CALIBRATION
Calibration Database
The current plans are to use the ORACLE database for storing the calibration information. We will require the following resources from Fermilab
DATA COPYING
The data copying needs are not yet known. The samples for oscillation studies are small and can probably be transferred over the network. Collaborators at overseas institutions such as the UK and Russia may want a complete copy of all the data. In the worst case, if we assume that we need to make 5 copies of the complete sample then this will be 27.5 TB per year which is about 460 tapes assuming an export media such as Exabyte Mammoth and 60 Gbyte cartridges. This is a modest amount (about 3-4 weeks) which should be able to be satisfied by a central tape copy facility.
OFFLINE DATA PROCESSING
The offline processing for both the near and far detectors will be done at Fermilab. A summary of the processing needs is given in Table III. These numbers are based on the existing MINOS code which is in Fortran. The code is currently being re-written using Object Oriented techniques and C++. This may cause some slow down of the code. For the purposes of this document the processing time per event will be given in SpecInt seconds per event (SpecInt sec/event) and the CPU requirements will be given in SpecInt. We are using the SpecInt95 measurements that are available at http://www.spec.org. To calculate the number of CPUs we have used an Intel motherboard SE440BX2 550 MHz Pentium III rated at 22 SpecInt95. In the early stages it will be necessary to reprocess the data as the algorithms are being refined at the same time as new data is being processed. Two complete processing passes through the data are assumed.
|
Sample |
SpecInt sec/event |
Events/year (107) |
SpecInt /year |
Number of CPUs/year |
|
Near detector |
7 |
127 |
600 |
27 |
|
Far detector |
18 |
3 |
36 |
2 |
|
Total |
25 |
130 |
636 |
29 |
Table III Processing needs for near and far detectors
OFFLINE ANALYSIS
We plan to perform physics analysis of the data at Fermilab as well as at other MINOS collaborating institutions.
We expect Fermilab to provide a central machine(s) in FCC with access to a central mass storage system for data that does not fit on permanently on disk. This machine(s) will act as a data server with the data being available via AFS or similar distributed means so that it is accessible to the collaborators offsite and to desktops at Fermilab. This machine(s) will also act as a central location for the MINOS software. We expect that people will make extensive use of desktop computing given the modest size of most of the data samples and the continued growth of disk capacity.
MONTE CARLO GENERATION AND STORAGE
There are two types of Monte Carlo required for MINOS, simulation of neutrino interactions in the detector for oscillation measurements/conventional neutrino physics and simulation of the neutrino beam to understand features of the beam such as beam profiles, flux etc. In both cases the requirements are not precisely known so the numbers here are based on present knowledge. We assume here that we will generate the samples at Fermilab but the possibility may also exist to generate them at collaborating institutions and ship them to Fermilab for storage.
Physics Monte Carlo
For studies of oscillations we expect to need about 10 times more Monte Carlo events than data for the far detector, this is a negligible amount (220,000 events/year). For the near detector we require that the statistical accuracy should be negligible compared to the statistical accuracy of the far detector data. Hence a sample equal to the near detector data sample will be sufficient (the relevant rate is 0.26 Hz in the central 25 cm radius of the target region). For studies of conventional neutrino physics the Monte Carlo sample needs to be about twice the data sample size as the measurements will be dominated by systematic uncertainties. The current execution time for the simulation is 103 SpecInt sec/event. If we include the reconstruction then it becomes 110 SpecInt sec/event to simulate and reconstruct a physics event. The needs per year are summarized in Table IV. To calculate the number of CPUs we have used an Intel motherboard SE440BX2 550 MHz Pentium III rated at 22 SpecInt95.
|
Events/year |
8.8 ´ 107 |
|
SpecInt/year |
323 |
|
Number CPUs/ year |
15 |
|
Event size |
57 Kbytes |
|
Data stored/year |
5 TB |
Table IV Physics Monte Carlo needs per year
This data will also need to be stored in the central mass storage system at Fermilab and be accessible on the MINOS central analysis machine. The samples for the oscillations studies will be kept permanently on disk (about 500 Gbytes/year).
Beam Monte Carlo
Currently a single run takes about two CPU weeks on an SGI O200 180 MHz CPU rated at 8 SpecInt95 to obtain sufficient statistics. The data volume produced per run is about 240 Mbytes. It is expected that a factor of 10 times longer runs will be needed for the physics analysis and that a few hundred of these runs will be required. To calculate the number of CPUs we have used an Intel motherboard SE440BX2 550 MHz Pentium III rated at 22 SpecInt95. The needs are summarized in Table V.
|
SpecInt sec/run |
9.7´ 107 |
|
Runs/year |
200 |
|
SpecInt/year |
645 |
|
Number CPUs/year |
30 |
|
Storage/year |
480 Gbytes |
Table V Beam Monte Carlo needs per year
About 17 CPUs per year would be required to simulate the approximately 108 events that these 200 runs would produce using the detector simulation.
The overall data volumes and processing needs for MINOS for 5 years of data taking are summarized in Table VI.
|
Full Data Sample |
Events |
Raw Data |
Processed Data |
Processing |
||
|
Near detector |
63.5´ 108 |
5.5 TB |
27.5 TB |
3000 SpecInt |
||
|
Far detector |
1.5´ 108 |
0.36 TB |
1.8 TB |
180 SpecInt |
||
|
Full Monte Carlo samples |
Events |
Processed Data |
Processing |
|||
|
Physics Monte Carlo |
4.4´ 108 |
25 TB |
1615 SpecInt |
|||
|
Beam Monte Carlo |
1000 runs |
2.4 TB |
3225 SpecInt |
|||
|
n Oscillation samples |
Events |
Processed Data |
Processing |
|||
|
Near detector data |
13´ 106 |
15 GB |
31 SpecInt |
|||
|
Far detector data |
110,000 |
0.25 GB |
negligible |
|||
|
Monte Carlo |
40´ 106 |
2.3 TB |
110 SpecInt |
|||
|
Conventional n samples |
||||||
|
Near detector data |
2´ 108 |
800 GB |
94 SpecInt |
|||
|
Monte Carlo |
4´ 108 |
23 TB |
1465 SpecInt |
|||
Table VI Summary of data volumes and processing needs for 5 years of data taking
SOFTWARE and PERSONNEL RESOURCES
The MINOS experiment has a working Fortran simulation and reconstruction package that is currently being used for detector optimization and physics studies. The decision was made in summer 1999 to replace this system with a new Object Oriented system based on C++. It was felt that, although there is a working Fortran system, the tools it is based on such as ZEBRA and ADAMO would not be supported over the life of the experiment and it would therefore be sensible to make the transition to an OO system.
The software group in MINOS is in the process of specifying what this new system will look like. This means it is hard to make definitive and complete requests for personnel and support. However, we can say some things about this area.