Contents || Index
05/06/2008

Large Data Cubes

With the development of fast SDD detectors, spectral images are becomming too large to completely hold in memory, even on 1 Gb+ machines.  Up to this point, Lispix has stored the entire data cube in memory, and provided some tools to clip, crop, or truncate cubes that are too large, so that they will fit.  This has become increasingly impractical.  Lispix now handles large data cubes using file-based methods, where the disk file is treated somewhat like very large computer memory.  Open these files in the normal way, using File / Open in the Lispix Menu Bar, and use the normal data cube sliders and tools with them.  Note that the old bottom row of buttons in the data cube tool that provided manual file-based means of clipping and cropping, has been eliminated.

Old
New

(You can now choose your button color scheme.)

 

The in-memory data cubes are called image-cubes and vector-cubes, denoting how the data is internally represented in Lispix.  Lispix now has new type of cube called a file-cube.  This will display data cubes of any forseeable size.  The small title starts with "f-cube".    If they have not already been preprocessed, this will then be done before the cube is displayed.  You can preprocess a number of files in batch mode using **Data Cube** / Read / Preprocess Large Data Cubes.

Preprocessing

Preprocessing primarily involves making a copy of the original data cube, but stored in different sequence.  If the original cube is recorded vector-by-vector, the copy is stored image-by-image and has "-xim" appended to the name.  Making this "transposed" copy of the data is both memory and time intensive.  I recommend preprocessing files in batch mode, so that you don't have to wait, and then restarting Lispix to return most of the memory back to the operating system.  The preprocessing can be interrupeted:  the next time the batch processing is run, it will start over on the file.  While the copy of the cube is being generated, the file has a .tmp extension.  It is renamed to .raw when preprocessing is complete.  An "orphin" tmp file should be deleted.

To "batch" pre-process your data cubes, use **Data Cube** / Open / Preprocess Large Data Cubes, which offers two options:

  1. Manually browse and select the data cubes to be preprocessed.  They do not need to be in the same folder.  Lispix shows two progress bars - the top one assumes that the cubes are all about the same size and gives a rough time estimate for the whole group of data cubes.
  2. Set the Lispix Read Directory to whatever folder contains all of the data cubes that you wish to preprocess.  Lispix will search the folder and all sub-directories, preprocessing the data cubes as it finds them.  The progress bar at any time is for the data cube being preprocessed at that time.

You can interrupt the preprocessing at any time by closing the progress bar.  Lispix will continue to "crunch" the cube, as can be seen by the indicator for your hard drive, until the next update of the progress bar was to occur.  If you wish to halt preprocessing immediately, you can try right-clicking on the small blue Franz Lisp icon near the clock at the lower-right of your screen and selecting Interrupt Lisp, or by using Task Manager / Applications / Lx--- / End Task.  Interrupting the preprocessing will leave a .tmp file, which Lispix will delete the next time it preprocesses that file, or which you may trash.  Lispix starts preprocessing fresh on each data cube, so it is a good idea to let it finish if it is nearly done.

File Overhead

In order respone quickly, often faster than with the memory-based cubes, Lispix stores copies and summaries of the data in additional files.  You can open the data by clicking either on the original data file, or on the copy.
Here is an example.  "Raney small.spd" is the original data.  All the other files are auxilliary files made by the preprocessing step.
  1. Original Data:  Raney small.spd
    Contents of the "... -LXD" folder

    The -xim.raw file is the duplicate of the .spd file stored in image order.  It lacks the 2 kb header.  If this duplicate were in vector or spectrum order, then the name would have been "Paney small-xsp".

    The -LXD folder (Lispix Data) contains raw files of various spectra.