Large Data Cubes
Most data cubes now are large, and read into Lispix as f-cubes - file based data rather than memory based data. Even if the data cubes are small enough to be kept in memory, I recommend loading them as f-cubes (Lispix gives you this option) because it is fast and keeps Lispix memory use low. Lispix will likely soon open data cubes only as file-based cubes. Pre-processing is required, but only once, and can be done in 'batch' mode. File cubes require twice the storage space on your hard drive. PCA does not yet work with file-based cubes.
With the development of fast SDD detectors, spectral images are becomming too large to completely hold in memory, even on 1 Gb+ machines. Up to this point, Lispix has stored the entire data cube in memory, and provided some tools to clip, crop, or truncate cubes that are too large, so that they will fit. This has become increasingly impractical. Lispix now handles large data cubes using file-based methods, where the disk file is treated somewhat like very large computer memory. Open these files in the normal way, using File / Open in the Lispix Menu Bar, and use the normal data cube sliders and tools with them. Note that the old bottom row of buttons in the data cube tool that provided manual file-based means of clipping and cropping, has been eliminated.
Old |
New |
 |
(You can now choose your button color scheme.) |
The in-memory data cubes are called image-cubes and vector-cubes, denoting how the data is internally represented in Lispix. Lispix now has new type of cube called a file-cube. This will display data cubes of any forseeable size. The small title starts with "f-cube". If they have not already been preprocessed, this will then be done before the cube is displayed. You can preprocess a number of files in batch mode using **Data Cube** / Read / Preprocess Large Data Cubes.
Preprocessing File Cubes
When you open a data spectral image for the first time, Lispix presents this dialog:
Note:
Lispix may have trouble preprocesing a cube on a networked drive. Here is an error message that I get, when preprocessing a new file cube, which occurs after a few minutes, when Lispix attempts to write the first part of the copy of the cube. If this happens, just copy the cube (and rpl file) to your hard drive, preprocess it, and move all the files back to the networked drive. Lispix should be able to use the now preprocessed file on the networked drive.
Preprocessing Continued
Preprocessing primarily involves making a copy of the original data cube, but stored in different sequence. If the original cube is recorded vector-by-vector, the copy is stored image-by-image and has "-xim" appended to the name. Making this "transposed" copy of the data is both memory and time intensive. I recommend preprocessing files in batch mode, so that you don't have to wait, and then restarting Lispix to return most of the memory back to the operating system. The preprocessing can be interrupeted: the next time the batch processing is run, it will start over on the file. While the copy of the cube is being generated, the file has a .tmp extension. It is renamed to .raw when preprocessing is complete. An "orphin" tmp file should be deleted.
To "batch" pre-process your data cubes, use **Data Cube** / Open / Preprocess Large Data Cubes, which offers two options:
