/ How to compress an existing database


Virtual Planet Builder

How to compress an existing database

Versions of OSG since around November 2008 support the compression of .ive files using the -O "compressed" writer option. I've tested the procedure on a 22GB database and the resulting database was reduced to 13GB.


The idea behind this example is to take an existing database that was not built using compression and use osgconv to compress each .ive file into a new database. This procedure is not needed if one creates the database with compression from the start by passing -O "compressed" to vpbmaster.


* Let old_db = the root of the old database, i.e. the directory where the top-level .ive file is located.

* Assume that the parent dir of old_db is writable.

* Let new_db = the root of the new database. It would be a sibling of the old_db directory.

* Using bash on Linux.


Turn off optimization


Create a script to convert all .ive files in a given directory (reason later)

cd old_db
emacs conv_all_in_dir.sh

Put the following in the script, edit as needed, comments inline


# used with e.g.:
#find . -type d -exec ./conv_all_in_dir.sh {} \;

# the output root, edit as needed
#echo $OUT_ROOT

# 1st parameter is the directory we should process
echo Changing to $IN_DIR
cd $IN_DIR

# find: find all ive files in this directory only, not subdirs
# sed: remove ./ in front of names
# xargs: run for every input file, start multiple processes (-P)
# osgconv -O "compressed": make compressed ive files

find . -maxdepth 1 -name "*.ive" | sed -e s/"\.\/"// | xargs -P 2 -I {} osgconv {} $OUT_ROOT/$IN_DIR/{} -O "compressed"

The script is passed a subdirectory of old_db, it goes through that directory and converts all the .ive files and places them in the correct new_db directory.

If you have a multi-core machine, adjust the -P parameter of xargs to suit.

Make the script executable

chmod u+x ./conv_all_in_dir.sh

Recreate the directory structure of the input database. osgconv does not like to create directories.

find . -type d -exec mkdir -p ../new_db/{} \;

Now, go to each directory of old_db and convert all the .ive files into the proper directory of new_db.

find . -type d -exec ./conv_all_in_dir.sh {} \;


Reason for cd before osgconv

For some reason osgconv must be run with only the input file name, i.e. from the directory where the file is located. If it is run by passing not only the filename, but a whole path to the file, the links to files in other directories get corrupted somehow. When the links are broken, one can only open the database properly from a single directory, e.g.

cd new_db
osgviewer terrain.ive

would work, but

cd ..
osgviewer new_db/terrain.ive

or execution from any other path would not.