The YAML catalog is very slow
I just tried something on a Koombook.
It has the following remotes:
$ ideascube catalog remotes list
[Kiwix] Kiwix ZIM content : http://catalog.ideascube.org/kiwix.yml
[OSMybon] Experimental OSM Maps catalog : http://catalog.yohanboniface.me/catalog.yml
[Omeka] Omeka packages : http://catalog.ideascube.org/omeka.yml
[StaticSites] Static sites : http://catalog.ideascube.org/static-sites.yml
[bibliotecamovil] Spanish packages : http://catalog.ideascube.org/bibliotecamovil.yml
That gives a grand total of 214 available packages (which really isn't much) :
$ ideascube catalog list --available | grep -E '^ ' | wc -l
214
Loading the cache of available packages:
>>> from datetime import datetime
>>> import json
>>> import yaml
>>>
>>> with open('/var/cache/ideascube/catalog/catalog.yml', 'r') as f:
... begin = datetime.now()
... data = yaml.safe_load(f)
... end = datetime.now()
...
>>> (end - begin).total_seconds()
4.460645
Dumping it back to the disk:
>>> with open('/var/cache/ideascube/catalog/catalog.yaml2', 'w') as f:
... begin = datetime.now()
... yaml.safe_dump(data, f)
... end = datetime.now()
...
>>> (end - begin).total_seconds()
3.118451
So let's try dumping that same data as JSON instead, then:
>>> with open('/var/cache/ideascube/catalog/catalog.json', 'w') as f:
... begin = datetime.now()
... json.dump(data, f)
... end = datetime.now()
...
>>> (end - begin).total_seconds()
0.101003
And loading the JSON back:
>>> with open('/var/cache/ideascube/catalog/catalog.json') as f:
... begin = datetime.now()
... data = json.load(f)
... end = datetime.now()
...
>>> (end - begin).total_seconds()
0.019905
That's a pretty big difference!