sdssOBJfileReadCat

top summary usage flags examples distribution details changes future bugs credits

SUMMARY

This program converts FITS binary table data into a standard output stream of ASCII column data. This may be used for reading by humans, to produce ASCII data files, or for piping into other programs. The format of the output is controlled by command line arguments.

The program was specifically designed to read Sloan Digital Sky Survey (SDSS) target selection object catalog (tsOBJC) FITS files which is a major component of the output of the SDSS photometric pileline (PHOTO). It contains a few "special" features to facilitate reading of these particular files, e.g. filtering according to the value of certain "named" flag bits.

USAGE

Here we give 3 forms of valid command line usage. The 1st two are rather trivial and the 3rd is what you want to use. Those things in blue are what you have to type, those things in red are variables for which you must substitute appropriate values, and those things in black are indications of format: that which is enclosed in [ ] is optional, the | separate forms which should not be used together, and the ... indicates that more constructs of the same form are allowed.
sdssOBJfileReadCat -usage
sdssOBJfileReadCat -version
sdssOBJfileReadCat [ -K | -kKEY1 ...] [ -C | -cCOL1 ...] [ -tTABLE1 ...]
[ [ +FLAG1 | -FLAG1 ] ... ] [ -verbose ] FILENAME
The order of the command line arguments does not matter. No whitespace is allowed after -k, -c, -t, +, or - but there must be whitespace between the arguments. The meaning of the arguments are
-usage print to stderr a usage message much like this one.
-version print to stderr the version number (1.2.1 currently).
-K print to stdout all keywords, values, and comments. Overrides any -k flags.
-C print to stdout all column labels. Overrides any -c or -t flags. No tabular data is printed.
-kKEY request keyword KEY and value to be printed to stdout. The case of KEY does not matter. Use -K to determine valid values for KEY.
-cCOL request column with label COL be printed to stdout. The case of COL does matter. Use -C to determine valid values for COL.
-tTABLE request the column data from table # TABLE be printed to stdout. TABLE must be a positive integer. If no tables are specified, but some columns are, then data from all tables are printed in order.
+FLAG restrict printing of column data to objects where the bit in objc_flags or status corresponding to flag FLAG is set to 1.
-FLAG restrict printing of column data to objects where the bit in objc_flags or status corresponding to flag FLAG is set to 0.
FLAG must be all upper case. Allowable values are given below.
-verbose print out some informative messages (labeled FYI) to stderr. Useful for debugging. If the program dies this is a way to identify where the problem is.
FILENAME name of the FITS files from which data is read. Only one file is allowed. This can be either a Unix compressed and gzip'd FITS file.
On one computer and with one particular tsOBJC file compress reduces tsOBJC file size by 30% and slows execution by a factor 3, gzip reduces file size by 45% and slows execution by a factor 2. Use gzip!

The keywords, columns, and tables will be printed in the order they are requested. Keywords (if any) printed first then a table, the columns left to right. The rows corresponding to the rows in the 1st table requested, then the 2nd table requested, etc. If no tables are requested then all the tables in the file will be printed in the order in the file. If you don't want to see the tabular data then don't request any columns.

If a requested keyword does not exist in the file then a WARNING will be printed and the program continues.

If a requested column does not exist in the file then an ERROR is printed and the program stops.

Valid FITS keywords may contain "A-Z", "0-9", "-", and "_". If a requested keyword contains "a-z" this is converted to upper case. Otherwise invalid keyword requests cause a WARNING message and are ignored.

Anticipated problems with the file will cause a message labeled ERROR to be printed to stderr and the the program will stop. This includes errors and messages found by the FITSIO library.

There are a variety of other WARNING messages printed to stderr which do not stop the program

top summary usage flags examples distribution details changes future bugs credits

FLAGS

The allowed values of the FLAG argument are given below as well as a description of their meaning (taken from some other document obviously written by an Englishman). These give the meaning of the bits (1 for true, 0 for false) of the 32 bit integer stored in the column objc_flags and status. For objc_flags the bits signify, in order from least to most significant bit:
CANONICAL_CENTER used canonical, not local, centre
BRIGHT detected by Bright Objects
EDGE object is too close to edge of frame
BLENDED object is/was blended
CHILD object is a child
PEAKCENTER given centre position of peak pixel
NODEBLEND no deblending attempted
NOPROFILE too small to estimate a profile
NOPETRO no Petrosian radius
MANYPETRO more than one Petrosian radius
NOPETRO_BIG no Petrosian radius as object is too big
DEBLEND_TOO_MANY_PEAK too many peaks to deblend
CR contains a CR pixel
MANYR50 more than one 50% radius
MANYR90 more than one 90% radius
BAD_RADIAL some low S/N radial points
INCOMPLETE_PROFILE r_P includes off-frame pixels
INTERP object contains interpolated pixels
SATUR object contains saturated pixels
NOTCHECKED object contains NOTCHECKED pixels
SUBTRACTED object had wings subtracted
NOSTOKES object has no measured stokes params
BADSKY sky level so bad that object is -ve
PETROFAINT >= 1 Petrosian radius too faint
TOO_LARGE object is too large
DEBLENDED_AS_PSF deblender treated obj as PSF
DEBLEND_PRUNED deblender pruned peak list
ELLIPFAINT Centre's fainter than desired elliptical isophote
BINNED1 object was found in 1x1 binned image
BINNED2 object was found in 2x2 binned image
BINNED4 object was found in 4x4 binned image
SPARE1 Spare value; largest flag value

For status the bits signify,in order from least to most significant bit:

SET This object's status has been set in reference to its own run. 1
GOOD Good as determined by its object flags. Absence implies bad. 2
DUPLICATE This object has one or more duplicate detections in an adjacent field of the same Frames Pipeline Run. 3
OK_RUN Located within the primary range of rows for this field. This is usable object. 5
RESOLVED This object has been resolved against other runs. 6
PSEGMENT Belongs to a PRIMARY segment. This does not imply that this is a primary object. 7
FIRST_FIELD Belongs to the first field in its segment. Used to distinguish objects in fields shared by two segments. 9
OK_SCANLINE Lies within valid nu range for its scanline. 10
OK_STRIPE Lies within valid eta range for its stripe. 11
SECONDARY This is a secondary survey object. 13
PRIMARY This is a primary survey object. 14
TARGET This is a spectroscopic target. 15

The meaning of the bits of this flag were taken from the file $TARGET_DIR/common/arObjectStatus.h (on 13 Jul 2000). The 3rd column above tells which bit is set for each flag (1-32) and we see that most bits are not used for anything, but one can effect access them using the flag, Status04 for bit 4 or Status32. This can be useful if these bits are used later on and this program is not updated to reflect this change. Note that our bit numbering (1-32) differs from another convention (0-31).

top summary usage flags examples distribution details changes future bugs credits

Examples

Here are some examples of usage of sdssOBJfileReadCat. The sample FITS file used, ts77_1_2.fit (which contains the 1st two fields (or field 115 & 116 depending on how you count) of run 77 column 1, is available (4 MB uncompressed). A normal tsOBJC file is ~800MB uncompressed, but remember you don't have to uncompress FITS files to read them. See if you get the same results with the sample file.

The command line sdssOBJfileReadCat -K ts77_1_2.fit lists all the keywords in the primary HDU with output:

SIMPLE  =                    T
BITPIX  =                    8
NAXIS   =                    0
EXTEND  =                    T
RUN     =                   77
CAMCOL  =                    1
FIELD0  =                  115
NFIELDS =                    2
FILTERS = 'u g r i z'
COBJ_ID = '1998-10-22T16:50:57 04673' 
Note there are only 2 tables (frames, fields, whatever) is this file.

The command line sdssOBJfileReadCat -C -kRUN -kNFIELDS ts77_1_2.fit lists only two keywords but all of the column labels:

77                                                                     RUN     
2                                                                      NFIELDS 
# columns in table         112
       1                                             parent     1xINTEGER*4   
       2                                                 id     1xINTEGER*4   
       3                                             nchild     1xINTEGER*4   
       4                                          objc_type     1xINTEGER*4   
       5                                              catid     1xINTEGER*4   
       6                                         objc_flags     1xINTEGER*4   
       7                                          objc_rowc     1xREAL*4      
       8                                       objc_rowcerr     1xREAL*4      
       9                                          objc_colc     1xREAL*4      
      10                                       objc_colcerr     1xREAL*4      
      11                                               rowc     5xREAL*4      
      12                                            rowcerr     5xREAL*4      
      13                                               colc     5xREAL*4      
      14                                            colcerr     5xREAL*4      
      15                                                sky     5xREAL*4      
      16                                             skyerr     5xREAL*4      
      17                                          psfcounts     5xREAL*4      
      18                                       psfcountserr     5xREAL*4      
      19                                        fibercounts     5xREAL*4      
      20                                     fibercountserr     5xREAL*4      
      21                                        petrocounts     5xREAL*4      
      22                                     petrocountserr     5xREAL*4      
      23                                           petrorad     5xREAL*4      
      24                                        petroraderr     5xREAL*4      
      25                                           petror50     5xREAL*4      
      26                                        petror50err     5xREAL*4      
      27                                           petror90     5xREAL*4      
      28                                        petror90err     5xREAL*4      
      29                                                  q     5xREAL*4      
      30                                               qerr     5xREAL*4      
      31                                                  u     5xREAL*4      
      32                                               uerr     5xREAL*4      
      33                                           iso_rowc     5xREAL*4      
      34                                        iso_rowcerr     5xREAL*4      
      35                                       iso_rowcgrad     5xREAL*4      
      36                                           iso_colc     5xREAL*4      
      37                                        iso_colcerr     5xREAL*4      
      38                                       iso_colcgrad     5xREAL*4      
      39                                              iso_a     5xREAL*4      
      40                                           iso_aerr     5xREAL*4      
      41                                          iso_agrad     5xREAL*4      
      42                                              iso_b     5xREAL*4      
      43                                           iso_berr     5xREAL*4      
      44                                          iso_bgrad     5xREAL*4      
      45                                            iso_phi     5xREAL*4      
      46                                         iso_phierr     5xREAL*4      
      47                                        iso_phigrad     5xREAL*4      
      48                                              r_dev     5xREAL*4      
      49                                           r_deverr     5xREAL*4      
      50                                             ab_dev     5xREAL*4      
      51                                          ab_deverr     5xREAL*4      
      52                                            phi_dev     5xREAL*4      
      53                                         phi_deverr     5xREAL*4      
      54                                         counts_dev     5xREAL*4      
      55                                      counts_deverr     5xREAL*4      
      56                                              r_exp     5xREAL*4      
      57                                           r_experr     5xREAL*4      
      58                                             ab_exp     5xREAL*4      
      59                                          ab_experr     5xREAL*4      
      60                                            phi_exp     5xREAL*4      
      61                                         phi_experr     5xREAL*4      
      62                                         counts_exp     5xREAL*4      
      63                                      counts_experr     5xREAL*4      
      64                                       counts_model     5xREAL*4      
      65                                    counts_modelerr     5xREAL*4      
      66                                            texture     5xREAL*4      
      67                                             star_l     5xREAL*4      
      68                                              exp_l     5xREAL*4      
      69                                              dev_l     5xREAL*4      
      70                                            fracpsf     5xREAL*4      
      71                                              flags     5xINTEGER*4   
      72                                               type     5xINTEGER*4   
      73                                              nprof     5xINTEGER*4   
      74                                           profmean    75xREAL*4      
      75                                            profmed    75xREAL*4      
      76                                            proferr    75xREAL*4      
      77                                             status     1xINTEGER*4   
      78                                                 ra     1xREAL*8      
      79                                                dec     1xREAL*8      
      80                                             lambda     1xREAL*8      
      81                                                eta     1xREAL*8      
      82                                                  l     1xREAL*8      
      83                                                  b     1xREAL*8      
      84                                           offsetra     5xREAL*4      
      85                                          offsetdec     5xREAL*4      
      86                                         primtarget     1xINTEGER*4   
      87                                          sectarget     1xINTEGER*4   
      88                                          reddening     5xREAL*4      
      89                                  propermotionmatch     1xINTEGER*4   
      90                                  propermotiondelta     1xREAL*4      
      91                                       propermotion     1xREAL*4      
      92                                         firstmatch     1xINTEGER*4   
      93                                         firstdelta     1xREAL*4      
      94                                          firstpeak     1xREAL*4      
      95                                           firstint     1xREAL*4      
      96                                           firstrms     1xREAL*4      
      97                                         firstmajor     1xREAL*4      
      98                                         firstminor     1xREAL*4      
      99                                            firstpa     1xREAL*4      
     100                                         rosatmatch     1xINTEGER*4   
     101                                         rosatdelta     1xREAL*4      
     102                                           rosatcps     1xREAL*4      
     103                                           rosathr1     1xREAL*4      
     104                                           rosathr2     1xREAL*4      
     105                                           rosatext     1xREAL*4      
     106                                         rosatflags     1xINTEGER*4   
     107                                           priority     1xINTEGER*4   
     108                                             prirun     1xINTEGER*4   
     109                                             pricol     1xINTEGER*4   
     110                                           prifield     1xINTEGER*4   
     111                                           prirerun     1xINTEGER*4   
     112                                              priid     1xINTEGER*4   

The command line sdssOBJfileReadCat -cid -cra -cdec -t2 ts77_1_2.fit lists the object id #, right ascension, and declination of all objects in table 2 of file

          1  0.21270849308000001D+03 -0.10471545799999999D+01
          2  0.21270849308000001D+03 -0.10471545799999999D+01
          3  0.21270849308000001D+03 -0.10471545799999999D+01
          4  0.21271336486999999D+03 -0.10449860000000000D+01
          5  0.21270701552000000D+03 -0.10513217699999999D+01

......................

        921  0.21269784283999999D+03 -0.11352878200000001D+01
        922  0.21282421335999999D+03 -0.95877422000000001D+00
        923  0.21280411975000001D+03 -0.11176868400000000D+01
        924  0.21280486594999999D+03 -0.92578313999999995D+00
        925  0.21285138001999999D+03 -0.11236230700000001D+01
        926  0.21273926746999999D+03 -0.10530256499999999D+01
Here we have only listed the first and last 5 objects, there are 926 objects in this table.

The command line sdssOBJfileReadCat -verbose +CR +BLENDED -EDGE -cpetrorad ts77_1_2.fit lists the Petrosian radius (pixel units for all 5 filters) of all objects which were flagged as being cosmic rays, blended, but not "too close" to a frame edge from all the tables:
FYI: Processed arguments
FYI: Opening FITS file
FYI: Writing keywords
FYI: Verifying requested tables
FYI: # tables in file:           2
FYI: Moving to table           1
FYI: Verifying columns
FYI: # columns in table         112
FYI: objc_flags in column #           6
FYI: petrorad in column #          23
FYI: petrorad is           5 x REAL*4      
FYI: # of rows in this table         816
FYI: buffering rows in groups of          28
  0.1541127E+01  0.1556261E+01  0.1180500E+01  0.1141653E+01  0.1440764E+01
  0.1426809E+01  0.1546609E+01  0.1130794E+01  0.1101963E+01  0.1508232E+01
  0.2969211E+01  0.2967758E+01  0.9909164E+00  0.1072843E+01  0.3693898E+01
  0.2969211E+01  0.2967758E+01  0.1696858E+01  0.2969003E+01  0.2968338E+01
  0.2969211E+01  0.1740986E+01  0.1170673E+01  0.1116420E+01  0.5784102E+01
  0.1130343E+02  0.2967758E+01  0.1233499E+01  0.1186841E+01  0.3106080E+01
FYI: Moving to table           2
FYI: Verifying columns
FYI: # of rows in this table         926
FYI: buffering rows in groups of          28
  0.1539270E+01  0.2568938E+01  0.1953130E+01  0.2439866E+01  0.2472842E+03
  0.1539040E+01  0.2567672E+01  0.2011595E+01  0.2442673E+01  0.1340042E+01
  0.1641415E+01  0.1630719E+01  0.1514477E+01  0.1585134E+01  0.1761094E+01
  0.1641094E+01  0.1631115E+01  0.1514623E+01  0.1585233E+01  0.1760960E+01
  0.1477162E+01  0.2229292E+01  0.1785995E+01  0.1424159E+01  0.1233918E+01
  0.1477170E+01  0.2233846E+01  0.1807890E+01  0.1397873E+01  0.1233657E+01
  0.1607237E+01  0.2027235E+01  0.1712467E+01  0.2857276E+01  0.1385738E+01
  0.1609272E+01  0.2026796E+01  0.1712423E+01  0.2859180E+01  0.1385715E+01
  0.1529824E+01  0.1480958E+01  0.1100140E+01  0.1081247E+01  0.1286339E+01
  0.2969210E+01  0.4941751E+01  0.4212746E+01  0.4246148E+01  0.4741463E+01
  0.4375863E+02  0.5980228E+01  0.7196271E+01  0.1136478E+01  0.1448311E+01
  0.7357306E+01  0.2967757E+01  0.1710757E+01  0.2396447E+01  0.1130010E+02
  0.2832305E+00  0.2198863E+01  0.2744453E+01  0.1296306E+01  0.1445121E+01
  0.4582955E+01  0.2967757E+01  0.1428394E+01  0.3524470E+01  0.2968337E+01
  0.4582955E+01  0.4580713E+01  0.9494182E-01  0.2969002E+01  0.1186033E+01
FYI: # of rows read:        1742
FYI: File Done

This is executed with -verbose flag and that shown in red is what is sent to stderr, will that in black to stdout. This is output of one column which itself has width 5, containing 5 real numbers.

Improper Usage

If one executes the command line sdssOBJfileReadCat +BRIGHT ts77_1_2.fit one only obtains the warning
WARNING: No output requested
in stderr, since one hasn't requested output with the -K, -k, -C, or -c flags.

With the command sdssOBJfileReadCat -cid ts77_1_2.fit ts77_2_2.fit one finds this in stderr

WARNING: Only one FITS file read at a time
ERROR: Problem with argument: ts77_2_2.fit
Usage: sdssOBJfileReadCat [ -verbose]
            [ -K | -kKEY1 -kKEY2 ...] [ -C | -cCOL1 -cCOL2 ...]
                [ [ +FLAG1 | -FLAG1 ] [ +FLAG2 | -FLAG2 ] ... ]
                              [ -tTABLE1 -tTABLE2 ...] FILENAME

       sdssOBJfileReadCat -version

       sdssOBJfileReadCat -usage

for more details see
      http://www-astro-theory.fnal.gov/Personal/stebbins/Software/SDSS/
Here one is trying to process two files, but only one file may be processed at a time.

With the command line sdssOBJfileReadCat -cID ts77_1_2.fit one finds this in stderr

ERROR: No column labeled ID
since case matters for column labels, one shouldn't have capitalized ID.

With the command line sdssOBJfileReadCat -c id ts77_1_2.fit one finds this in stderr

ERROR: Problem with argument: id
since no space is allowed between "-c" and "id".

top summary usage flags examples distribution details changes future bugs credits

DISTRIBUTION

The intent is to distribute sdssOBJfileReadCat as stand-alone binary executables. Shared libraries have not been used to my knowledge. Executables are presently available for the platforms marked in green below. Operating system version for systems on which they have been used, but the executables will probably work on other versions. as well. Click to get the executable.


IRIX
5.3

IRIX
6.3

IRIX64*
6.5
Redhat
i686 Linux
2.2

OSF1
4.0

SunOS

HPUX

AIX
v1.1.2 Yes v1.2.0 Yes Yes No No No
*IRIX64 refers to 64 bit architecture using R8000 or R10000 processors.

The source files are also available. While the subroutines are not very well documented, one should feel free to use the code for other purposes. The code compiles with GNU's g77 as well as GNU's f2c (use the fort77 as a front end) so it should be very portable to any Unix systems. The fort77 compiled version seems to run faster than the g77 compiled version which runs faster than that compiled with native IRIX f77. It requires the CFITSIO library which is free and can be obtained elsewhere. Here v1.42 was used, but assume the newer v2.0 would work as well.

top summary usage flags examples distribution details changes future bugs credits

SOME DETAILS

Specification of input file format

While this program was written for the target selection object catalog files output by SDSS's photometric pipeline it may work with other files. This program should work with any FITS file of the same form as SDSS object files. These consist of a Primary HDU (Header/Data Unit) containing keywords and values followed by one or more HDUs containing 2-d binary tables with the same column structure. The number of rows in each table may vary. Variable width columns are not supported. The one part of this program which is fine tuned to tsCOBJ files is the [+|-]FLAG command line option which need not be used.

The [+|-]FLAG option requires there to be column labeled objc_flags and/or status containing one 4 byte integer per row. Of course for the FLAG values to make sense the meaning of the bits in this integer must be the same as in a tsOBJC file. The flag selection was included in the Fortran code rather than an external filter because the external filter implementation I came up with slowed the code by large factor.

This program will not access any "data" in the primary HDU nor will it access any keywords in subsequent HDUs except for NAXIS2 which gives the # of rows, TTYPE, which gives the label of the columns, and TFORM which which gives the data format (e.g. 5E for 5 REAL*4 numbers.

This program cannot access any column which does not have a label (i.e. TTYPE). If more than one column has the same label then only the 1st column with that label can be accessed. Whitespace is ignored in column labels. If one requests two or more columns with the same label (up to whitespace) then the output will contain multiple identical columns.

The column structure being the same means that the TFORMs and TTYPEs and their order are the same in every table. This program really only cares that the position, TFORM, and of course TTYPE of the requested columns are the same in each of the requested tables. The TFORM and TTYPE of unrequested columns is irrelevant as is the format of HDUs which correspond to unrequested tables.

CHANGES/IMPROVEMENTS

  • v1.1.2 (1 December 1999)
  • While this program expects the keyword NFIELDS in the primary header to tell it how many HDU's the file contains, it no longer requires it. This allows you to read many more types of FITS files which do not use NFIELDS but have more than one table. If NFIELDS is not present it gives a warning as things may go wrong.
  • If you specify to read all binary tables by omitting the -t flag, then the program will exit with an error after it has read the last table which is present.
  • If you specify, with the -t flag, a table beyond the number of tables in the file then it will read all the tables requested up to the 1st invalid one and at that point exit with an error.
  • v1.2.0 (14 July 2000)
  • The progam now allows easy selection on the bits of the status column, e.g. PRIMARY, TARGET, etc.
  • , in the same way as one could select on bits in the objc_flags column.
  • v1.2.1 (14 December 2000)
  • Fixed various bugs which prevented listing of CHARACTER data columns in binary tables (this had never been tested as was stated in previous versions of the documentation). Schlegel spectroscopic redshift data files, ~/spZ...fits, contain CHARACTER data columns.
  • BUGS and LIMITATIONS

    All known bugs have been fixed, but undoubtedly some remain. Testing has not been as extensive as it might, but real use is probably the best test. Send bug reports to stebbins@fnal.gov.

    Some possible bugs and known limitations are:

  • The program generally assumes that the FITS files contains one or more binary tables all with the same column structure (see details). If the file contains more than one "type" of binary table, as long as one specifies only tables with the same structure (using the -t flag), it should work. Even if the file contains HDUs which are not binary tables, as long as one avoids these HDUs (again using the -t flag) it should work.
  • It has not been tested on "bit" or "byte" binary FITS data types (the tsOBJC files contain no data of this type) and so it may not work for these columns.
  • Will not work with variable width data columns. It will work if binary table contains such columns so long as you do not try to read them. I have a suspicion that a variation of the tsOBJC files containing variable width FITS binary table columns may be used in some distributions of the SDSS data. This program will fail (with an error) if you try to read these columns in these files.
  • This program was written in Fortran 77 and does not dynamically allocate memory. If a single column is outrageously wide the program will fail (with an error). One would have to increase the value of the parameter OutputBufferSize and/or ColumnBufferSize and recompile to read such a file.
  • Individual elements of column (more than one element for a vector column) with CHARACTER datatype cannot have length more than 99 characters. If it does and one tries to read this column the program may give erroneous output without error or warning.
  • Vector columns with CHARACTER datatype cannot have length more than 1000 elements.
  • No columns labeled either "C" or "K" may be read.
  • One cannot request a FILENAME which begins with either "-" or "+". One can always work around this by including more of the path of the file on input, i.e. "./-.fit" rather than "-.fit".
  • FUTURE DEVELOPMENT

  • Write an awk-script interface to make object selection easier.
  • Interface with graphics and analysis programs.
  • Implement output to a smaller FITS file with reduced data content.
  • CREDITS

    Written by Albert Stebbins with help from Bob Nichol and Gordon Richards.

    top summary usage flags examples distribution details changes future bugs credits

    stebbins@pavo.fnal.gov
    Last modified: Fri Dec 15 17:56:42 CST