Notes on Dervish

Table of Contents

1 Introduction

Dervish is a collection of libraries and corresponding tcl bindings on which much of the SDSS software infrastructure is built. This document demonstrates some of the most basic things that can be done with it using the tcl interface. I wrote it after I had been away from using it for a while, and found myself needing reminders. I expect to need to do this occasionally, and this document is supposed to help with that.

The most useful way to think about Dervish is as a C library with some tcl bindings. If you think about it that way, its workings are fairly intuitive. If you think of it as tcl extended to have a variety of features that astronomers might want, the results may be frustrating.

These notes assume a previous knowledge of tcl. See the official tcl tutorial if you need help there. Note that most of the resources on the web are designed for tcl version 8.0 or later, while Dervish is built on tcl 7.4; not everyithing will work the same.

There is also some out of date tutorial material for Dervish and related SDSS tools on the SDSS Primers page.

In most cases, you can get help on a specific commend using the help command, or by using the "-help" argument on the command itself.

2 Handles, Schema, and Chains

Dervish lets you allocate, use, and deallocate memory for data structures, not just strings as is usual in tcl. A dervish "handle" allows you to reference the memory allocated for a structure, and a "schema" describes what is found there:

sdsslnx33$ dervish
Executing commands in /sdss/ups/prd/dervish/v8_18/Linux-2-4-2-3-2/etc/dervishStartup.tcl:
dervish> typedef struct {char name[80]; int x; int y} SAMP;
dervish> genericNew SAMP
h0
dervish> schemaPrint h0
name {char[80]}
x int
y int
dervish> exprPrint -header -noquote h0
h0                   SAMP
(h0).name
(h0).x               0
(h0).y               0
dervish> handleSet h0.name Joe
dervish> handleSet h0.x 10
dervish> handleSet h0.y 42
dervish> exprPrint -header -noquote h0
h0                   SAMP
(h0).name            Joe
(h0).x               10
(h0).y               42
dervish> exprGet h0.x
10
dervish> exprGet h0
{name Joe} {x 10} {y 42}
dervish> handleSet h0 { Fred 2 3}
dervish> exprPrint h0
(h0).name            "Fred"
(h0).x               2
(h0).y               3
dervish> handleDel h0
dervish> exit
sdsslnx33$

If you want to use a structure defined in a C header file, you can use makeio instead of typedef.

You can view and edit one of these structures interactively using structsEdit. (Ctrl-x saves changes to memory, and Ctrl-c exits.)

A "chain" is a doubly linked list of these structures. You can interactively look through a chain using chainPage. (Ctrl-n and Ctrl-p go to the next and previous elements of the chain, respectively.) Commands that load and save data often do such as chains. You can get specific elements of a chain using chainElementGetByPos.

3 Yanny par files

You can load a Yanny param file into a chain using param2Chain, and write to one using chain2Param.

dervish> param2Chain  opCamera-52639.par h
h0 h1
dervish> keylget h mjd
52641
dervish> puts $h
{mjd {52641}} {scalef {16.596119193140311}}

Note that there are two type of structures in this par file, so two chains were allocated. You can look through them interactively using

dervish> chainPage h0

and

dervish> chainPage h1

4 FITS headers

You can read a FITS header using hdrReadAsFits:

dervish> set h [hdrReadAsFits [hdrNew] image.fits]
h0
dervish> set ra [hdrGetAsDbl $h RA]
189.34690000000001
dervish> hdrDel $h

The "-hdu" option will let you get the header for an extension HDU.
Other header utilities can be found using:
dervish> info command hdr*

5 Regions and FITS images

To load a FITS image into memory first create a handle for the region using regNew, and then load it using regReadAsFits. You can then display the image. When you are done, be sure do deallocate the memory. For example:

dervish> set im [regReadAsFits [regNew] image.fits]
h0
dervish> saoDisplay $im
1
dervish> regDel $im

Other region utilities can be found using:

dervish> info command reg*

6 FITS binary tables

You can load a FITS binary table into a chain using fits2Schema, and write to a fits file using schema2Fits.

sdsslnx33$ dervish
Executing commands in /sdss/ups/prd/dervish/v8_18/Linux-2-4-2-3-2/etc/dervishStartup.tcl:
dervish> set h [hdrNew]
h0
dervish> fits2Schema tsObj-001339-5-40-0075.fit OBJ $h -hdu 1
h3
dervish> chainElementGetByPos h3 1
h2
dervish> exprGet h2.ra
260.994934505629
dervish> exit

You can also look through the chain interactively using

dervish> chainPage h3

7 More on Chains

Chains have operations corresponding to typical list tools. This transcript should provide some clues:

sdsslnx33$ dervish
Executing commands in /sdss/ups/prd/dervish/v8_18/Linux-2-4-2-3-2/etc/dervishStartup.tcl:
dervish> typedef struct {char name[80]; int x; int y} SAMP;
dervish> genericNew SAMP
h0
dervish> genericNew SAMP
h1
dervish> handleSet h0 { Fred 2 3}
dervish> handleSet h0 { Saturn 1001 2001}
dervish> chainNew GENERIC
h2
dervish> handleSet h1 { Saturn 1001 2001}
dervish> chainElementAddByPos h2 h1
dervish> chainElementAddByPos h2 h0
dervish> chainElementGetByPos h2 1
h0
dervish> chainElementGetByPos h2 0
h1
dervish> chainCursorNew h2
h3
dervish> chainWalk h2 h3 THIS
h1
dervish> chainWalk h2 h3 NEXT
h0
dervish> chainWalk h2 h3 THIS
h0
dervish> chainWalk h2 h3 PREVIOUS
h1
dervish> chainWalk h2 h3 PREVIOUS
dervish> chainWalk h2 h3 THIS
h0
dervish> chainCursorDel h2 h3
dervish> chainCopy h2
h3
dervish> chainDel h2
dervish> exprGet h0
{name Saturn} {x 1001} {y 2001}
dervish> chainDestroy h3
0
dervish> exprGet h0
Error: Unknown handle: h0

Error: h is not open
p_shTclHandleAddrGet: unknown handle name h0.
dervish> exit
sdsslnx33$

There are many other chain utilities. These can be listed using:

dervish> info command chain*

8 Plotting

Dervish uses an interface to pgPlot for plotting, using fairly direct mappings to the C and Fortran interfaces, as discribed in the pgPlot subroutine list and the pgPlot subroutine discription pages. Here is an example of its use in Dervish:

sdsslnx33$ dervish
Executing commands in /sdss/ups/prd/dervish/v8_18/Linux-2-4-2-3-2/etc/dervishStartup.tcl:
dervish> set pg [pgstateNew]
h0
dervish> pgstateSet $pg -device "/XWINDOW"
dervish> pgstateOpen $pg
Open /XWINDOW
h0
dervish> set xmin 0; set xmax 10; set ymin 0; set ymax 100
100
dervish> set color 1; set symbol 2
2
dervish> pgSci $color
dervish> pgEnv $xmin $xmax $ymin $ymax 0 0
 Type <RETURN> for next page:
dervish> pgLabel "X values" "Y values"
dervish> set color 2
2
dervish> pgSci $color
dervish> set x 5; set y 8
8
dervish> pgPoint $x $y $symbol
dervish> set x 8; set y 80
80
dervish> pgPoint $x $y $symbol
dervish> pgstateClose $pg
Close /XWINDOW
 Type <RETURN> for next page:
dervish> exit
sdsslnx33$

You can got a list of pgPlot routines from within Dervish:

dervish> info command pg*

and use the standard Dervish help facility to get usage:

dervish> help pgText

Note that pgPlot uses a separate process to show plots in X Windows. If you are using ssh to log into a remote host, and are making plots from that host, it is sometimes helpful to start it locally rather than remotely, so that ssh does not wait for the process to end when you try to log out. You start the server thus:

bash$ pgxwin_server &

9 Vectors

A Dervish "vector" is a one dimensional array of floats. One can operate on all elements of a vector at once:

sdsslnx33$ dervish
Executing commands in /sdss/ups/prd/dervish/v8_18/Linux-2-4-2-3-2/etc/dervishStartup.tcl:
dervish> vectorExprEval {{42 7 4344 2}}
h0
dervish> vectorExprPrint h0
42 7 4344 2
dervish> vectorExprEval 2*h0
h1
dervish> vectorExprPrint h1
84 14 8688 4
dervish> vectorExprPrint h0
42 7 4344 2
dervish> vectorExprSet h0 3*h0
h0
dervish> vectorExprPrint h0
126 21 13032 6
dervish> vectorExprDel h0
dervish> vectorExprDel h1
dervish> exit
sdsslnx33$

Note that the distinguishing feature between vectorExprEval and vectorExprSet is that the former will allocate the new vector for you, but the later requires that it already exist. As usual, you can get a list of vector commands using

dervish> info command v*

One can create vectors from elements in a chain of structures using vFromChain, and set elements in a chain of structures from a vector using vToChain, which makes them particularly useful for operating on columns in FITS tables or Yanny par files:

sdsslnx33$ dervish
Executing commands in /sdss/ups/prd/dervish/v8_18/Linux-2-4-2-3-2/etc/dervishStartup.tcl:
dervish> hdrNew
h0
dervish> fits2Schema tsObj-001339-5-40-0075.fit OBJ h0 -hdu 1
h3
dervish> vFromChain h3 rowv
h2
dervish> vMean h2
-505.0784
dervish> vectorExprEval h2*2
h1
dervish> vMean h1
-1010.157
dervish> hdrDel h0
dervish> vectorExprDel h1
dervish> vectorExprDel h2
dervish> chainDestroy h3
0
dervish> exit
sdsslnx33$

10 Argument parsing and writing help strings

Dervish inherits two different help string handling systems from its dependencies. One supplies a help string when a user invokes the help command, the other when the user runs the procedure and supplies "-help" as one of the arguments. Because not all procedures in all products handle the "-help" argument properly, and some interpret it as some other argument, users are safer trying the former first. When writing a precedure, devopers should make sure both are supplied. Here is an example:

lappend thisFiles_procs doSomething
proc doSomething { args } {
 set opts [list [list [info level 0] "The proc is an example of how to do som
ething, but does nothing."]]
 lappend opts [list {<happyArg>} STRING "" happy "The desired value for happy
"]
 lappend opts [list {<sadArg>} STRING "" sad "The desired value for sad."]
 if {[shTclParseArg $args $opts [info level 0]] == 0} {
  return
 }

 puts $happy
 puts $sad

}

# This goes at the end of the source file
set_ftclHelp samples thisFiles_procs ;# perform ftclHelpDefine's on all listed p
rocs

Note that the opts argument of shTclParseArg is a list of lists. The first element of this list is a two element list, the first element of which is the name of the procedure (determined automagically above using info), and the second is a text description of the command. The remaining elements of this list are lists describing the options. These lists contain the following elements:

  1. The label for the argument given in the help string. This string must be surrounded by angle brackets ("<happy>"), in which case the argument is required; preceeded by a dash ("-happy"), in which case the argument may either be present or absent; or surrounded by square brackets ("[happy]"), in which case the default value (given by element 3 in this list) is used as a default.
  2. The type of the argument, which may be INTEGER, CONSTANT, STRING, or DOUBLE.
  3. The default value for the variable. If there is no default value, a placeholder must be used.
  4. The name of the tcl variable to be set by this option.
  5. A text description of the option to be displayed in the help string.

11 Memory management

Given that we can allocate and deallocate memory manually in Dervish, the possibility of memory leaks is an issue. Dervish provides some tools for debugging them, the most useful of which may be memStatsPrint.

sdsslnx33$ dervish
Executing commands in /sdss/ups/prd/dervish/v8_18/Linux-2-4-2-3-2/etc/dervishStartup.tcl:
dervish> memStatsPrint

Number of memory allocation requests:     0
Number of memory de-allocation requests:  0
Total bytes currently in use:             0
Total bytes in Free Memory Pool:          0
Total bytes allocated by malloc():        0
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        0.00  %
Percentage of memory allocation requests
  satisfied from the Operating System:    0.00  %

dervish> typedef struct {char name[80]; int x; int y} SAMP;
dervish> genericNew SAMP
h0
dervish> memStatsPrint

Number of memory allocation requests:     1
Number of memory de-allocation requests:  0
Total bytes currently in use:             88
Total bytes in Free Memory Pool:          0
Total bytes allocated by malloc():        168
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        0.00  %
Percentage of memory allocation requests
  satisfied from the Operating System:    100.00%

dervish> genericDel h0
dervish> memStatsPrint

Number of memory allocation requests:     1
Number of memory de-allocation requests:  1
Total bytes currently in use:             0
Total bytes in Free Memory Pool:          128
Total bytes allocated by malloc():        168
Percentage of memory allocation requests
  satisfied from Free Memory Pool:        0.00  %
Percentage of memory allocation requests
  satisfied from the Operating System:    100.00%

dervish> exit
sdsslnx33$

You can, of course, find the others using

dervish> info command mem*

12 Using regions in a C program

A region is simply an array, usually an image. For example, this program creates a 512 by 512 array, initializes the whole array to 42, prints the value of pixel 128, 128, multiplies the whole array by 10, and prints new value of the pixel.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include "region.h"
#include "shCRegUtils.h"

int
main(int argc, char **argv)
{
  char *image="im";
  REGION *r;

  r=shRegNew(image,512,512,TYPE_U16);
  shRegSetWithDbl(42.0,r);
  printf("The starting value is %g\n", shRegPixGetAsDbl(r,128,128));
  shRegMultWithDbl(10.0,r,r);
  printf("The ending value is %g\n", shRegPixGetAsDbl(r,128,128));
  return 1;
}

It can be compiled and run like this:

sdsslnx33$ setup dervish
sdsslnx33$ gcc -o reg_arith reg_arith.c -DSDSS_LITTLE_ENDIAN -DSTAND_ALONE -Wall \
 -I$DERVISH_DIR/include -I$LIBFITS_DIR/include -L$DERVISH_DIR/lib \
 -L$LIBFITS_DIR/lib -ldervishnotcl -lfits
sdsslnx33$ ./reg_arith
The starting value is 42
The ending value is 420
sdsslnx33$

13 Using chains in a C program

A chain is a doubly linked list. This sample creates a chain of a given structure, prints each member, and dealloctes the memory used.

#include <stdio.h>
#include <stdlib.h>
#include <shChain.h>
#include <shCGarbage.h>

/* This declaration should be in shChain.h, but isn't */
int shChainDestroy(CHAIN *pChain, void (*pDelFunc)(void *));

/* Definition and create and destroy procs for our structure */
typedef struct {
  int bar;
  int baz;
} FOO;

FOO *fooNew(int bar, int baz)
{
  FOO *f;
  f=(FOO *)shMalloc(sizeof(FOO));
  f->bar=bar;
  f->baz=baz;
  return f;
}

void fooDel(FOO *f)
{
  shFree(f);
}

/* Make a short chain of FOOs, print them, and free memory */

int
main(int argc, char **argv)
{
  FOO *f;
  CHAIN *c;
  CURSOR_T cursor;

  c=shChainNew("FOO");

  f=fooNew(1,2);
  shChainElementAddByPos(c, f, "FOO", TAIL, AFTER);
  f=fooNew(11,12);
  shChainElementAddByPos(c, f, "FOO", TAIL, AFTER);
  f=fooNew(21,22);
  shChainElementAddByPos(c, f, "FOO", TAIL, AFTER);

  cursor=shChainCursorNew(c);
  while((f= (FOO*) shChainWalk(c,cursor,NEXT))!=NULL) {
 printf("bar=%d, baz=%d\n",f->bar,f->baz);
  }
  shChainCursorDel(c, cursor);
  shChainDestroy(c,(void (*)(void *))fooDel);

  return 1;

}

It can be compiled and run like this:

sdsslnx33$ setup dervish
sdsslnx33$ gcc -o chain_sample chain_sample.c -DSDSS_LITTLE_ENDIAN -DSTAND_ALONE
 -Wall -I$DERVISH_DIR/include -I$LIBFITS_DIR/include -L$DERVISH_DIR/lib -L$LIBFITS_DIR/lib -ldervishnotcl -lfits -lm
sdsslnx33$ ./chain_sample
bar=1, baz=2
bar=11, baz=12
bar=21, baz=22
sdsslnx33$

Author: Eric H. Neilsen, Jr.

Created: 2014-06-16 Mon 15:31

Emacs 24.3.1 (Org mode 8.2.5c)

Validate