Dealing with Large Files

Large astronomical FITS data files present a challenge for browser-based image display, in two ways:

The nightmare scenario is to run out of browser memory after taking a long time to download the file!

One possibility for dealing with large files is to download a small representation file for display and quick-look analysis, while supporting detailed analysis and image extraction on the original parent file. In JS9, you can:

These schemes are described below.

Automatic Generation of a FITS Representation File

Consider a 12000 x 12000 Hubble Deep Field image. Even using gzip compression, this 300Mb file could take several minutes to download and would require more than 500Mb of memory to display. But you can bin this file by a factor of 24 into a gzip'ed 512 x 512 FITS image with file size of approximately 500Kb. This FITS representation file would download quickly and give a good overview of the entire data set. You can perform quick-look analysis on this file using JS9 plugins. But if you also could re-extract a section from the original file at a higher resolution, you would, in essence, be panning around the original image without the transmission and memory overhead associated with processing the whole file.

JS9 supports FITS representation file generation and display, as well as extraction of binned image sections from the original parent file, using the fits2fits scheme. To utilize it at your site, first configure a Node.js-based Server-side Helper using the standard GNU configure/make setup: 

  # where to find cfitsio and install binaries, what sort of helper to build
  ./configure  --with-cfitsio=/usr/local --prefix=$HOME --with-helper=nodejs
  make
  make install

Once the js9Helper is running, configure fits2fits by setting the following global parameters in your js9prefs.js file:

The server-side helper must support temporary user work directories in which to store the FITS representation files. This is accomplished by setting the globalOpts.workDir property in js9prefs.js to a top-level directory which will contain the individual work sub-directories for each JS9 connection. The workDir path should either be an absolute path, or one that is relative to the JS9 helper directory.

Optionally, you probably will want to set the globalOpts.workDirQuota property (current default value of 50Mb) to limit the amount of space that can be utilized by each user. Note that the workDirQuota is not a hard quota: the js9helper only checks whether the quota has been exceeded before extracting the next section. This means that users can always extract one section that will bring the total size above the quota. Thus, if the quota is 10Mb and the work directory is empty, imsection will extract a 50Mb file. But it will not extract any more files until already-loaded files have been closed in order to bring the total under the quota. The globalOpts.fits2fits property controls whether a FITS representation file is automatically generated. It accepts the following values:

Of course, this property is only checked if JS9 is connected to a server-side helper which has a working directory configured, as described above.

The current default for fits2fits is "size>100", which means that representation files will be generated if the original FITS file has a size greater than 100 Mb. The globalOpts.image and globalOpts.table objects specify the image dimensions and bin factor used to generate the FITS representation file:

Binning is applied after the image section is extracted or the table is binned into an image. Thus, dimensions of 4096 x 4096 and bin 4 will produce a 1024 x 1024 image.

When fit2fits is set up in this way, JS9 will ask the helper to check the file size when loading a FITS URL. If the check returns false, the original FITS file will be loaded internally into JS9. If true, the JS9 helper will extract a representation image (i.e, an image section) from the parent file, store it in the temporary work directory, and then tell JS9 to display the representation file.

The smaller in-memory representation file will be used when performing browser-based analysis such as WCS reprojection, 3D plots, and imexam. But the representation file knows its parent, and will send the parent filename to run server-side analysis tasks. Thus JS9 performs quick-look analysis on the smaller file and more detailed server-side analysis on the original parent file.

In this context, perhaps the most important server-side task is the Bin/Filter/Section plugin available in the View menu. This plugin allows you to extract sections from FITS binary tables and images, with options to choose the image center, image dimensions, and bin factor. For tables, the plugin also allows you to filter rows of the table by means of arithmetic and boolean operations on the table columns.

The Bin/Filter/Section plugin works with both FITS representation files and with in-memory FITS files. If the displayed FITS file is a representation file (i.e., it has a parent file, a JS9 helper is connected, etc.), the plugin will ask the JS9 helper to extract the new image section from the parent. Otherwise, it extracts the section from the in-memory FITS file. This behavior is meant to be transparent to the user. See the Bin/Filter/Section plugin for more information.

NB: Unfortunately, the fits2fits scheme is not available with FITS files that are drag/dropped onto JS9 or files that are opened through the file menu dialog box. Both of these methods call for the browser to load the whole file directly and pass it to JS9 for display. There is some discussion within the browser development community of supporting on-the-fly processing, but nothing definite is planned.

Manual Specification of a FITS Parent File

Assuming the JS9 helper is set up, but fits2fits is set to "never", you can load a small FITS file using JS9.Load() and manually specify a parentFile property in the options object: 

    JS9.Load("js9/fits/f814w_bin24.fits.gz", {colormap:"cool", parentFile: "fits/f814w.fits.gz"})
One important note: the pathname of the URL to load is (as usual) relative to the web page, while the pathname of the parentFile must be relative to the JS9 install directory (i.e., where the JS9 helper was started). In the above example, the web page is one level up from the JS9 install directory. Once the parentFile has been specified, the in-memory file should act as a representation file.

Of course, the two files need to be tied to one another spatially. For server-side data analysis, having agreement between WCS parameters is sufficient. In order to use the Bin/Filter/Section plugin, however, the representation file also must be connected to the parent file using IRAF LTM/LTV keywords, so that the physical coordinates of the representation file point to the file coordinates of the parent. The js9helper program will generate a FITS image section with the correct keywords. The syntax is: 

    js9helper -i parentFile imsection repFile section filter
where section can be: and filter specifies a column filter for binary tables. For example: 
    js9helper -i f814w.fits.gz imsection f814_bin24.fits "12000@0,12000@0,24"
was used to generate the binned representation image in the JS9 Large demo.
Last updated: May 8, 2021