Make clearer output for ideascube catalog
Hello team,
It's hard for us to get what is going on :
TASK [package_management : Install, upgrade or remove a package present in device_list.fact] *** failed: [localhost] (item={u'status': u'present', u'name': u'wikipedia.fr'}) => {"changed": true, "cmd": "grep -m 1 -w \"wikipedia.fr\" /var/ideascube/main/catalog/installed.yml || ideascube catalog install wikipedia.fr ", "delta": "1:46:42.640559", "end": "2017-10-23 15:51:26.228841", "failed": true, "item": {"name": "wikipedia.fr", "status": "present"}, "rc": 1, "start": "2017-10-23 14:04:43.588282", "stderr": "grep: /var/ideascube/main/catalog/installed.yml: No such file or directory\nTraceback (most recent call last):\n File \"/opt/venvs/ideascube/bin/django-admin\", line 11, in <module>\n load_entry_point('Django==1.11.5', 'console_scripts', 'django-admin')()\n File \"/opt/venvs/ideascube/lib/python3.4/site-packages/django/core/management/__init__.py\", line 364, in execute_from_command_line\n utility.execute()\n File \"/opt/venvs/ideascube/lib/python3.4/site-packages/django/core/management/__init__.py\", line 356, in execute\n self.fetch_command(subcommand).run_from_argv(self.argv)\n File \"/opt/venvs/ideascube/lib/python3.4/site-packages/django/core/management/base.py\", line 283, in run_from_argv\n self.execute(*args, **cmd_options)\n File \"/opt/venvs/ideascube/lib/python3.4/site-packages/django/core/management/base.py\", line 330, in execute\n output = self.handle(*args, **options)\n File \"/opt/venvs/ideascube/lib/python3.4/site-packages/ideascube/serveradmin/management/commands/catalog.py\", line 124, in handle\n return super().handle(*args, **options)\n File \"/opt/venvs/ideascube/lib/python3.4/site-packages/ideascube/management/base.py\", line 18, in handle\n options['func'](options)\n File \"/opt/venvs/ideascube/lib/python3.4/site-packages/ideascube/serveradmin/management/commands/catalog.py\", line 184, in install_packages\n options['ids'], keep_downloads=options['keep_downloads'])\n File \"/opt/venvs/ideascube/lib/python3.4/site-packages/ideascube/serveradmin/catalog.py\", line 689, in install_packages\n handler.commit()\n File \"/opt/venvs/ideascube/lib/python3.4/site-packages/ideascube/serveradmin/catalog.py\", line 200, in commit\n et = etree.parse(f)\n File \"src/lxml/etree.pyx\", line 3441, in lxml.etree.parse\n File \"src/lxml/parser.pxi\", line 1855, in lxml.etree._parseDocument\n File \"src/lxml/parser.pxi\", line 1875, in lxml.etree._parseFilelikeDocument\n File \"src/lxml/parser.pxi\", line 1770, in lxml.etree._parseDocFromFilelike\n File \"src/lxml/parser.pxi\", line 1185, in lxml.etree._BaseParser._parseDocFromFilelike\n File \"src/lxml/parser.pxi\", line 598, in lxml.etree._ParserContext._handleParseResultDoc\n File \"src/lxml/parser.pxi\", line 705, in lxml.etree._handleParseResult\n File \"src/lxml/etree.pyx\", line 323, in lxml.etree._ExceptionContext._raise_if_stored\n File \"src/lxml/parser.pxi\", line 376, in lxml.etree._FileReaderContext.copyToBuffer\n File \"/opt/venvs/ideascube/lib/python3.4/encodings/ascii.py\", line 26, in decode\n return codecs.ascii_decode(input, self.errors)[0]\nUnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 279: ordinal not in range(128)", "stdout": "\u001b[36mIDEASCUBE_ID=kb\u001b[0m\n\u001b[36mImporting settings from ideascube.conf.kb\n\u001b[0m\n\rDownloading wikipedia.fr: 0.00% || 16.0 KiB/25.9 GiB (0.0 KiB/s) | ETA: 14:04:50\rDownloading wikipedia.fr: 0.05% || 12.1 MiB/25.9 GiB (12.1 MiB/s) | ETA: 14:41:20\rDownloading wikipedia.fr: 0.09% || 24.9 MiB/25.9 GiB (12.4 MiB/s) | ETA: 14:40:24\rDownloading wikipedia.fr: 0.14% || 38.3 MiB/25.9 GiB (12.8 MiB/s) | ETA: 14:39:28\rDownloading wikipedia.fr: 0.20% || 52.7 MiB/25.9 GiB (13.2 MiB/s) | ETA: 14:38:27\rDownloading wikipedia.fr: 0.25% || 66.8 MiB/25.9 GiB (13.4 MiB/s) | ETA: 14:37:59\rDownloading "Downloading wikipedia.fr: 81.28% || 21.1 GiB/25.9 GiB (10.9 MiB/s) | ETA: 14:45:37", "Downloading wikipedia.fr: 81.31% || 21.1 GiB/25.9 GiB (10.9 MiB/s) | ETA: 14:45:37", "Downloading wikipedia.fr: 81.33% || 21.1 GiB/25.9 GiB (10.8 MiB/s) | ETA: 14:45:38", "Downloading wikipedia.fr: 81.35% || 21.1 GiB/25.9 GiB (10.8 MiB/s) | ETA: 14:45:38", "Downloading wikipedia.fr: 81.38% || 21.1 GiB/25.9 GiB (10.8 MiB/s) | ETA: 14:45:39", "Downloading wikipedia.fr: 81.41% || 21.1 GiB/25.9 GiB (10.8 MiB/s) | ETA: 14:45:39", "Downloading wikipedia.fr: 81.45% || 21.1 GiB/25.9 GiB (10.8 MiB/s) | ETA: 14:45:39", "Downloading
I see two possible improvement here :
- Better error handling from the code, Python traces are not easy to read if not a Python dev.
- Drastically reduce download messages (I did truncate 99% here) which are all written to logs.