| top | summary | usage | flags | examples | distribution | details | changes | future | bugs | credits |
This program converts FITS binary table data into a standard output stream of ASCII column data. This may be used for reading by humans, to produce ASCII data files, or for piping into other programs. The format of the output is controlled by command line arguments.
The program was specifically designed to read Sloan Digital Sky Survey (SDSS) target selection object catalog (tsOBJC) FITS files which is a major component of the output of the SDSS photometric pileline (PHOTO). It contains a few "special" features to facilitate reading of these particular files, e.g. filtering according to the value of certain "named" flag bits.
| sdssOBJfileReadCat | -usage |
| sdssOBJfileReadCat | -version |
| sdssOBJfileReadCat |
[ -K |
-kKEY1
...] [ -C |
-cCOL1
...] [ -tTABLE1
...] [ [ +FLAG1 | -FLAG1 ] ... ] [ -verbose ] FILENAME |
| -usage | print to stderr a usage message much like this one. |
| -version | print to stderr the version number (1.2.1 currently). |
| -K | print to stdout all keywords, values, and comments. Overrides any -k flags. |
| -C | print to stdout all column labels. Overrides any -c or -t flags. No tabular data is printed. |
| -kKEY | request keyword KEY and value to be printed to stdout. The case of KEY does not matter. Use -K to determine valid values for KEY. |
| -cCOL | request column with label COL be printed to stdout. The case of COL does matter. Use -C to determine valid values for COL. |
| -tTABLE | request the column data from table # TABLE be printed to stdout. TABLE must be a positive integer. If no tables are specified, but some columns are, then data from all tables are printed in order. |
| +FLAG | restrict printing of column data to objects where the bit in objc_flags or status corresponding to flag FLAG is set to 1. |
| -FLAG | restrict printing of column data to objects where the bit in objc_flags or status corresponding to flag FLAG is set to 0. |
| FLAG must be all upper case. Allowable values are given below. | |
| -verbose | print out some informative messages (labeled FYI) to stderr. Useful for debugging. If the program dies this is a way to identify where the problem is. |
| FILENAME | name of the FITS files from which data is read. Only one file is allowed. This can be either a Unix compressed and gzip'd FITS file. |
| On one computer and with one particular tsOBJC file compress reduces tsOBJC file size by 30% and slows execution by a factor 3, gzip reduces file size by 45% and slows execution by a factor 2. Use gzip! |
The keywords, columns, and tables will be printed in the order they are requested. Keywords (if any) printed first then a table, the columns left to right. The rows corresponding to the rows in the 1st table requested, then the 2nd table requested, etc. If no tables are requested then all the tables in the file will be printed in the order in the file. If you don't want to see the tabular data then don't request any columns.
If a requested keyword does not exist in the file then a WARNING will be printed and the program continues.
If a requested column does not exist in the file then an ERROR is printed and the program stops.
Valid FITS keywords may contain "A-Z", "0-9", "-", and "_". If a requested keyword contains "a-z" this is converted to upper case. Otherwise invalid keyword requests cause a WARNING message and are ignored.
Anticipated problems with the file will cause a message labeled ERROR to be printed to stderr and the the program will stop. This includes errors and messages found by the FITSIO library.
There are a variety of other WARNING messages printed to stderr which do not stop the program
| top | summary | usage | flags | examples | distribution | details | changes | future | bugs | credits |
The allowed values of the FLAG argument are given below as well as a description of their meaning (taken from some other document obviously written by an Englishman). These give the meaning of the bits (1 for true, 0 for false) of the 32 bit integer stored in the column objc_flags and status. For objc_flags the bits signify, in order from least to most significant bit:
| CANONICAL_CENTER | used canonical, not local, centre |
| BRIGHT | detected by Bright Objects |
| EDGE | object is too close to edge of frame |
| BLENDED | object is/was blended |
| CHILD | object is a child |
| PEAKCENTER | given centre position of peak pixel |
| NODEBLEND | no deblending attempted |
| NOPROFILE | too small to estimate a profile |
| NOPETRO | no Petrosian radius |
| MANYPETRO | more than one Petrosian radius |
| NOPETRO_BIG | no Petrosian radius as object is too big |
| DEBLEND_TOO_MANY_PEAK | too many peaks to deblend |
| CR | contains a CR pixel |
| MANYR50 | more than one 50% radius |
| MANYR90 | more than one 90% radius |
| BAD_RADIAL | some low S/N radial points |
| INCOMPLETE_PROFILE | r_P includes off-frame pixels |
| INTERP | object contains interpolated pixels |
| SATUR | object contains saturated pixels |
| NOTCHECKED | object contains NOTCHECKED pixels |
| SUBTRACTED | object had wings subtracted |
| NOSTOKES | object has no measured stokes params |
| BADSKY | sky level so bad that object is -ve |
| PETROFAINT | >= 1 Petrosian radius too faint |
| TOO_LARGE | object is too large |
| DEBLENDED_AS_PSF | deblender treated obj as PSF |
| DEBLEND_PRUNED | deblender pruned peak list |
| ELLIPFAINT | Centre's fainter than desired elliptical isophote |
| BINNED1 | object was found in 1x1 binned image |
| BINNED2 | object was found in 2x2 binned image |
| BINNED4 | object was found in 4x4 binned image |
| SPARE1 | Spare value; largest flag value |
For status the bits signify,in order from least to most significant bit:
| SET | This object's status has been set in reference to its own run. | 1 |
| GOOD | Good as determined by its object flags. Absence implies bad. | 2 |
| DUPLICATE | This object has one or more duplicate detections in an adjacent field of the same Frames Pipeline Run. | 3 |
| OK_RUN | Located within the primary range of rows for this field. This is usable object. | 5 |
| RESOLVED | This object has been resolved against other runs. | 6 |
| PSEGMENT | Belongs to a PRIMARY segment. This does not imply that this is a primary object. | 7 |
| FIRST_FIELD | Belongs to the first field in its segment. Used to distinguish objects in fields shared by two segments. | 9 |
| OK_SCANLINE | Lies within valid nu range for its scanline. | 10 |
| OK_STRIPE | Lies within valid eta range for its stripe. | 11 |
| SECONDARY | This is a secondary survey object. | 13 |
| PRIMARY | This is a primary survey object. | 14 |
| TARGET | This is a spectroscopic target. | 15 |
The meaning of the bits of this flag were taken from the file
$TARGET_DIR/common/arObjectStatus.h (on 13 Jul 2000). The 3rd column
above tells which bit is set for each flag (1-32) and we see that most bits are
not used for anything, but one can effect access them using the flag,
Status04 for bit 4 or Status32. This can be useful if these bits
are used later on and this program is not updated to reflect this change. Note
that our bit numbering (1-32) differs from another convention (0-31).
| top | summary | usage | flags | examples | distribution | details | changes | future | bugs | credits |
Here are some examples of usage of sdssOBJfileReadCat. The sample FITS file used, ts77_1_2.fit (which contains the 1st two fields (or field 115 & 116 depending on how you count) of run 77 column 1, is available (4 MB uncompressed). A normal tsOBJC file is ~800MB uncompressed, but remember you don't have to uncompress FITS files to read them. See if you get the same results with the sample file.
The command line sdssOBJfileReadCat -K ts77_1_2.fit lists all the keywords in the primary HDU with output:
SIMPLE = T BITPIX = 8 NAXIS = 0 EXTEND = T RUN = 77 CAMCOL = 1 FIELD0 = 115 NFIELDS = 2 FILTERS = 'u g r i z' COBJ_ID = '1998-10-22T16:50:57 04673'Note there are only 2 tables (frames, fields, whatever) is this file.
The command line sdssOBJfileReadCat -C -kRUN -kNFIELDS ts77_1_2.fit lists only two keywords but all of the column labels:
77 RUN
2 NFIELDS
# columns in table 112
1 parent 1xINTEGER*4
2 id 1xINTEGER*4
3 nchild 1xINTEGER*4
4 objc_type 1xINTEGER*4
5 catid 1xINTEGER*4
6 objc_flags 1xINTEGER*4
7 objc_rowc 1xREAL*4
8 objc_rowcerr 1xREAL*4
9 objc_colc 1xREAL*4
10 objc_colcerr 1xREAL*4
11 rowc 5xREAL*4
12 rowcerr 5xREAL*4
13 colc 5xREAL*4
14 colcerr 5xREAL*4
15 sky 5xREAL*4
16 skyerr 5xREAL*4
17 psfcounts 5xREAL*4
18 psfcountserr 5xREAL*4
19 fibercounts 5xREAL*4
20 fibercountserr 5xREAL*4
21 petrocounts 5xREAL*4
22 petrocountserr 5xREAL*4
23 petrorad 5xREAL*4
24 petroraderr 5xREAL*4
25 petror50 5xREAL*4
26 petror50err 5xREAL*4
27 petror90 5xREAL*4
28 petror90err 5xREAL*4
29 q 5xREAL*4
30 qerr 5xREAL*4
31 u 5xREAL*4
32 uerr 5xREAL*4
33 iso_rowc 5xREAL*4
34 iso_rowcerr 5xREAL*4
35 iso_rowcgrad 5xREAL*4
36 iso_colc 5xREAL*4
37 iso_colcerr 5xREAL*4
38 iso_colcgrad 5xREAL*4
39 iso_a 5xREAL*4
40 iso_aerr 5xREAL*4
41 iso_agrad 5xREAL*4
42 iso_b 5xREAL*4
43 iso_berr 5xREAL*4
44 iso_bgrad 5xREAL*4
45 iso_phi 5xREAL*4
46 iso_phierr 5xREAL*4
47 iso_phigrad 5xREAL*4
48 r_dev 5xREAL*4
49 r_deverr 5xREAL*4
50 ab_dev 5xREAL*4
51 ab_deverr 5xREAL*4
52 phi_dev 5xREAL*4
53 phi_deverr 5xREAL*4
54 counts_dev 5xREAL*4
55 counts_deverr 5xREAL*4
56 r_exp 5xREAL*4
57 r_experr 5xREAL*4
58 ab_exp 5xREAL*4
59 ab_experr 5xREAL*4
60 phi_exp 5xREAL*4
61 phi_experr 5xREAL*4
62 counts_exp 5xREAL*4
63 counts_experr 5xREAL*4
64 counts_model 5xREAL*4
65 counts_modelerr 5xREAL*4
66 texture 5xREAL*4
67 star_l 5xREAL*4
68 exp_l 5xREAL*4
69 dev_l 5xREAL*4
70 fracpsf 5xREAL*4
71 flags 5xINTEGER*4
72 type 5xINTEGER*4
73 nprof 5xINTEGER*4
74 profmean 75xREAL*4
75 profmed 75xREAL*4
76 proferr 75xREAL*4
77 status 1xINTEGER*4
78 ra 1xREAL*8
79 dec 1xREAL*8
80 lambda 1xREAL*8
81 eta 1xREAL*8
82 l 1xREAL*8
83 b 1xREAL*8
84 offsetra 5xREAL*4
85 offsetdec 5xREAL*4
86 primtarget 1xINTEGER*4
87 sectarget 1xINTEGER*4
88 reddening 5xREAL*4
89 propermotionmatch 1xINTEGER*4
90 propermotiondelta 1xREAL*4
91 propermotion 1xREAL*4
92 firstmatch 1xINTEGER*4
93 firstdelta 1xREAL*4
94 firstpeak 1xREAL*4
95 firstint 1xREAL*4
96 firstrms 1xREAL*4
97 firstmajor 1xREAL*4
98 firstminor 1xREAL*4
99 firstpa 1xREAL*4
100 rosatmatch 1xINTEGER*4
101 rosatdelta 1xREAL*4
102 rosatcps 1xREAL*4
103 rosathr1 1xREAL*4
104 rosathr2 1xREAL*4
105 rosatext 1xREAL*4
106 rosatflags 1xINTEGER*4
107 priority 1xINTEGER*4
108 prirun 1xINTEGER*4
109 pricol 1xINTEGER*4
110 prifield 1xINTEGER*4
111 prirerun 1xINTEGER*4
112 priid 1xINTEGER*4
The command line sdssOBJfileReadCat -cid -cra -cdec -t2 ts77_1_2.fit lists the object id #, right ascension, and declination of all objects in table 2 of file
1 0.21270849308000001D+03 -0.10471545799999999D+01
2 0.21270849308000001D+03 -0.10471545799999999D+01
3 0.21270849308000001D+03 -0.10471545799999999D+01
4 0.21271336486999999D+03 -0.10449860000000000D+01
5 0.21270701552000000D+03 -0.10513217699999999D+01
921 0.21269784283999999D+03 -0.11352878200000001D+01
922 0.21282421335999999D+03 -0.95877422000000001D+00
923 0.21280411975000001D+03 -0.11176868400000000D+01
924 0.21280486594999999D+03 -0.92578313999999995D+00
925 0.21285138001999999D+03 -0.11236230700000001D+01
926 0.21273926746999999D+03 -0.10530256499999999D+01
Here we have only listed the first and last 5 objects, there are 926 objects in
this table.
The command line
sdssOBJfileReadCat -verbose +CR +BLENDED -EDGE
-cpetrorad ts77_1_2.fit lists the Petrosian radius (pixel units for
all 5 filters) of all objects which were flagged as being cosmic rays, blended,
but not "too close" to a frame edge from all the tables:
FYI: Processed arguments FYI: Opening FITS file FYI: Writing keywords FYI: Verifying requested tables FYI: # tables in file: 2 FYI: Moving to table 1 FYI: Verifying columns FYI: # columns in table 112 FYI: objc_flags in column # 6 FYI: petrorad in column # 23 FYI: petrorad is 5 x REAL*4 FYI: # of rows in this table 816 FYI: buffering rows in groups of 28 0.1541127E+01 0.1556261E+01 0.1180500E+01 0.1141653E+01 0.1440764E+01 0.1426809E+01 0.1546609E+01 0.1130794E+01 0.1101963E+01 0.1508232E+01 0.2969211E+01 0.2967758E+01 0.9909164E+00 0.1072843E+01 0.3693898E+01 0.2969211E+01 0.2967758E+01 0.1696858E+01 0.2969003E+01 0.2968338E+01 0.2969211E+01 0.1740986E+01 0.1170673E+01 0.1116420E+01 0.5784102E+01 0.1130343E+02 0.2967758E+01 0.1233499E+01 0.1186841E+01 0.3106080E+01 FYI: Moving to table 2 FYI: Verifying columns FYI: # of rows in this table 926 FYI: buffering rows in groups of 28 0.1539270E+01 0.2568938E+01 0.1953130E+01 0.2439866E+01 0.2472842E+03 0.1539040E+01 0.2567672E+01 0.2011595E+01 0.2442673E+01 0.1340042E+01 0.1641415E+01 0.1630719E+01 0.1514477E+01 0.1585134E+01 0.1761094E+01 0.1641094E+01 0.1631115E+01 0.1514623E+01 0.1585233E+01 0.1760960E+01 0.1477162E+01 0.2229292E+01 0.1785995E+01 0.1424159E+01 0.1233918E+01 0.1477170E+01 0.2233846E+01 0.1807890E+01 0.1397873E+01 0.1233657E+01 0.1607237E+01 0.2027235E+01 0.1712467E+01 0.2857276E+01 0.1385738E+01 0.1609272E+01 0.2026796E+01 0.1712423E+01 0.2859180E+01 0.1385715E+01 0.1529824E+01 0.1480958E+01 0.1100140E+01 0.1081247E+01 0.1286339E+01 0.2969210E+01 0.4941751E+01 0.4212746E+01 0.4246148E+01 0.4741463E+01 0.4375863E+02 0.5980228E+01 0.7196271E+01 0.1136478E+01 0.1448311E+01 0.7357306E+01 0.2967757E+01 0.1710757E+01 0.2396447E+01 0.1130010E+02 0.2832305E+00 0.2198863E+01 0.2744453E+01 0.1296306E+01 0.1445121E+01 0.4582955E+01 0.2967757E+01 0.1428394E+01 0.3524470E+01 0.2968337E+01 0.4582955E+01 0.4580713E+01 0.9494182E-01 0.2969002E+01 0.1186033E+01 FYI: # of rows read: 1742 FYI: File DoneThis is executed with -verbose flag and that shown in red is what is sent to stderr, will that in black to stdout. This is output of one column which itself has width 5, containing 5 real numbers.
WARNING: No output requestedin stderr, since one hasn't requested output with the -K, -k, -C, or -c flags.
With the command sdssOBJfileReadCat -cid ts77_1_2.fit ts77_2_2.fit one finds this in stderr
WARNING: Only one FITS file read at a time
ERROR: Problem with argument: ts77_2_2.fit
Usage: sdssOBJfileReadCat [ -verbose]
[ -K | -kKEY1 -kKEY2 ...] [ -C | -cCOL1 -cCOL2 ...]
[ [ +FLAG1 | -FLAG1 ] [ +FLAG2 | -FLAG2 ] ... ]
[ -tTABLE1 -tTABLE2 ...] FILENAME
sdssOBJfileReadCat -version
sdssOBJfileReadCat -usage
for more details see
http://www-astro-theory.fnal.gov/Personal/stebbins/Software/SDSS/
Here one is trying to process two files, but only one file may be processed at
a time.
With the command line sdssOBJfileReadCat -cID ts77_1_2.fit one finds this in stderr
ERROR: No column labeled IDsince case matters for column labels, one shouldn't have capitalized ID.
With the command line sdssOBJfileReadCat -c id ts77_1_2.fit one finds this in stderr
ERROR: Problem with argument: idsince no space is allowed between "-c" and "id".
| top | summary | usage | flags | examples | distribution | details | changes | future | bugs | credits |
The intent is to distribute sdssOBJfileReadCat as stand-alone binary executables. Shared libraries have not been used to my knowledge. Executables are presently available for the platforms marked in green below. Operating system version for systems on which they have been used, but the executables will probably work on other versions. as well. Click to get the executable.
IRIX 5.3 |
IRIX 6.3 |
IRIX64* 6.5 |
Redhat i686 Linux 2.2 |
OSF1 4.0 |
SunOS |
HPUX |
AIX |
| v1.1.2 | Yes | v1.2.0 | Yes | Yes | No | No | No |
The source files are also available. While the subroutines are not very well documented, one should feel free to use the code for other purposes. The code compiles with GNU's g77 as well as GNU's f2c (use the fort77 as a front end) so it should be very portable to any Unix systems. The fort77 compiled version seems to run faster than the g77 compiled version which runs faster than that compiled with native IRIX f77. It requires the CFITSIO library which is free and can be obtained elsewhere. Here v1.42 was used, but assume the newer v2.0 would work as well.
| top | summary | usage | flags | examples | distribution | details | changes | future | bugs | credits |
While this program was written for the target selection object catalog files output by SDSS's photometric pipeline it may work with other files. This program should work with any FITS file of the same form as SDSS object files. These consist of a Primary HDU (Header/Data Unit) containing keywords and values followed by one or more HDUs containing 2-d binary tables with the same column structure. The number of rows in each table may vary. Variable width columns are not supported. The one part of this program which is fine tuned to tsCOBJ files is the [+|-]FLAG command line option which need not be used.
The [+|-]FLAG option requires there to be column labeled objc_flags and/or status containing one 4 byte integer per row. Of course for the FLAG values to make sense the meaning of the bits in this integer must be the same as in a tsOBJC file. The flag selection was included in the Fortran code rather than an external filter because the external filter implementation I came up with slowed the code by large factor.
This program will not access any "data" in the primary HDU nor will it access any keywords in subsequent HDUs except for NAXIS2 which gives the # of rows, TTYPE, which gives the label of the columns, and TFORM which which gives the data format (e.g. 5E for 5 REAL*4 numbers.
This program cannot access any column which does not have a label (i.e. TTYPE). If more than one column has the same label then only the 1st column with that label can be accessed. Whitespace is ignored in column labels. If one requests two or more columns with the same label (up to whitespace) then the output will contain multiple identical columns.
The column structure being the same means that the TFORMs and TTYPEs and their order are the same in every table. This program really only cares that the position, TFORM, and of course TTYPE of the requested columns are the same in each of the requested tables. The TFORM and TTYPE of unrequested columns is irrelevant as is the format of HDUs which correspond to unrequested tables.
All known bugs have been fixed, but undoubtedly some remain. Testing has not been as extensive as it might, but real use is probably the best test. Send bug reports to stebbins@fnal.gov.
Some possible bugs and known limitations are:
Written by Albert Stebbins with help from Bob Nichol and Gordon Richards.
| top | summary | usage | flags | examples | distribution | details | changes | future | bugs | credits |