Better handle connection errors
Looking at the logs, I've got a few connection errors to misc websites: timeouts on given websites, some 50X errors during the night. Probably they could be handled in a way that doesn't prevent executing the search on the other websites; even better, there could be a fixed number of retries after a connection failure.
Here's an example of an error:
84164:Traceback (most recent call last):
84165- File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
84166- "__main__", fname, loader, pkg_name)
84167- File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
84168- exec code in run_globals
84169- File "/home/user/app/flatisfy/__main__.py", line 197, in <module>
84170- main()
84171- File "/home/user/app/flatisfy/__main__.py", line 186, in main
84172- cmds.import_and_filter(config, load_from_db=False)
84173- File "flatisfy/cmds.py", line 189, in import_and_filter
84174- fetch_details=(not load_from_db))
84175- File "flatisfy/cmds.py", line 53, in filter_flats
84176- details = fetch.fetch_details(config, flat["id"])
84177- File "flatisfy/fetch.py", line 262, in fetch_details
84178- weboob_output = weboob_proxy.info(flat_id)
84179- File "flatisfy/fetch.py", line 210, in info
84180- housing = backend.get_housing(flat_id)
84181- File "/home/user/weboob/modules/entreparticuliers/module.py", line 54, in get_housing
84182- return self.browser.get_housing(_id)
84183- File "/home/user/weboob/modules/entreparticuliers/browser.py", line 125, in get_housing
84184- obj = self.housing.go(data=data).get_housing(obj=obj)
84185- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/url.py", line 101, in go
84186- r = self.browser.location(self.build(**kwargs), params=params, data=data, method=method, headers=headers or {})
84187- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/browsers.py", line 698, in location
84188- response = self.open(*args, **kwargs)
84189- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/browsers.py", line 685, in open
84190- return super(PagesBrowser, self).open(callback=internal_callback, *args, **kwargs)
84191- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/browsers.py", line 574, in open
84192- return super(DomainBrowser, self).open(req, *args, **kwargs)
84193- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/browsers.py", line 340, in open
84194- is_async=is_async)
84195- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/sessions.py", line 159, in send
84196- return func(*args, **kwargs)
84197- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/sessions.py", line 151, in func
84198- resp = sup(*args, **kwargs)
84199- File "/usr/local/lib/python2.7/site-packages/requests-2.17.3-py2.7.egg/requests/sessions.py", line 623, in send
84200- r = adapter.send(request, **kwargs)
84201- File "/usr/local/lib/python2.7/site-packages/requests-2.17.3-py2.7.egg/requests/adapters.py", line 516, in send
84202- raise ReadTimeout(e, request=request)
84203-requests.exceptions.ReadTimeout: HTTPConnectionPool(host='www.entreparticuliers.com', port=80): Read timed out. (read timeout=10.0)