Degraded indexing and seach performance on Xapian 1.4/glass compared to /chert
I recently rebuilt the databases on my document server and with that migrated from the chert to the glass format. The document server is based on a 6-core, 16GB container running on a 12-core DL380G7/128GB, the collection is stored on a 7-drive array hosted on a DS4243 JBOD enclosure, in other words relatively up-to-date and by no means sluggish hardware. The original database was created using Recoll 1.19 and had been updated without being reset ever since.
Indexing the collection (420.000 files (mixture of PDF and EPUB) spanning 650GB) took almost 70 hours, far longer than expected. During indexing the container slowed down to a crawl but that was expected given the number of reports about
recollindex hogging system resources.
A far bigger - and for me critical - problem is that search performance on the new database is extremely poor compared to the 'chert' format. The database size for all collections together (divided over 3 sets) is about 90GB using the glass format, ~75GB using chert. Searching for a single word in the largest of the sets (48GB database) using the glass format takes about 60 seconds, the same search using the chert format takes less than 2 seconds, in both cases starting with a cold cache. I tested using the python API through the (python3 version of the) web interface and through the
recollq command line program (the document server is headless). The index was generated using mostly default options:
idxrundir = tmp textfilemaxmbs = 200 nocjk = 1 idxflushmb = 200
I tested both with a freshly indexed database and after compressing it using
xapian-compact, the results were similar.
I re-indexed the database after adding
indexStoreDocText = 0 to
recoll.conf and moving
xapiandb out of the way to get back to a chert database, this resolved the performance issues for now. Since chert support will be dropped in Xapian 1.5 this is not a permanent solution so it would be good to find a way to get back the lost performance using the glass format.
Tests performed on a DL380G7 with 2xX5675@3.07GHz, 128GB, system on the internal 8-drive array, data on external 7-drive (15K spinning rust) array in a DS4243 JBOD.