Flatisfy issueshttps://framagit.org/phyks/Flatisfy/-/issues2018-01-18T14:46:07Zhttps://framagit.org/phyks/Flatisfy/-/issues/111Cache images hash2018-01-18T14:46:07ZPhyksphyks@phyks.meCache images hashIn the deep duplicate phase, the computation of images hash is quite CPU demanding and we are repeatingly computing the hashes.
We should put the hashes in the cache as well to compute the hashes only once per image. It should drastical...In the deep duplicate phase, the computation of images hash is quite CPU demanding and we are repeatingly computing the hashes.
We should put the hashes in the cache as well to compute the hashes only once per image. It should drastically reduce the length of the third step, which is by far the longer.Phyksphyks@phyks.mePhyksphyks@phyks.mehttps://framagit.org/phyks/Flatisfy/-/issues/81Package for YunoHost2021-05-16T13:04:00ZPhyksphyks@phyks.mePackage for YunoHosthttps://framagit.org/phyks/Flatisfy/-/issues/114Improve duplicate detection2018-01-21T13:13:41ZPhyksphyks@phyks.meImprove duplicate detectionDuplicates detection is quite basic for the moment. There might be ways to improve it. In particular, photo comparison is very basic.
A few interesting links might be:
* https://www.kaggle.com/c/avito-duplicate-ads-detection
* https://...Duplicates detection is quite basic for the moment. There might be ways to improve it. In particular, photo comparison is very basic.
A few interesting links might be:
* https://www.kaggle.com/c/avito-duplicate-ads-detection
* https://dzone.com/articles/searching-images-by-visual-similarityhttps://framagit.org/phyks/Flatisfy/-/issues/113Free speedup of flats processing2018-01-21T13:18:59ZPhyksphyks@phyks.meFree speedup of flats processingFor now, each passes functions call a bunch of steps with the whole flats list. This means we iterate over *all* the flats at each substep.
See https://framagit.org/phyks/Flatisfy/blob/master/flatisfy/filters/__init__.py#L163-182.
Flat...For now, each passes functions call a bunch of steps with the whole flats list. This means we iterate over *all* the flats at each substep.
See https://framagit.org/phyks/Flatisfy/blob/master/flatisfy/filters/__init__.py#L163-182.
Flats are interdependent so this could be done with a single `for flat in flats_list` iteration at the pass level (e.g. https://framagit.org/phyks/Flatisfy/blob/master/flatisfy/filters/__init__.py#L162).
This single loop could even be passed down to `joblib` for parallel processing: https://pythonhosted.org/joblib/parallel.html.https://framagit.org/phyks/Flatisfy/-/issues/104Do not build from public transport data for unwanted quarters.2018-01-21T09:48:08ZPhyksphyks@phyks.meDo not build from public transport data for unwanted quarters.I set the following postal codes as constraints: 35000, 35200 and 35700.
All those postal codes are in the region "FR-BRE" : https://git.phyks.me/Phyks/flatisfy/blob/master/flatisfy/data_files/__init__.py#L47.
This region is in the quar...I set the following postal codes as constraints: 35000, 35200 and 35700.
All those postal codes are in the region "FR-BRE" : https://git.phyks.me/Phyks/flatisfy/blob/master/flatisfy/data_files/__init__.py#L47.
This region is in the quarter "FR-NW" : https://git.phyks.me/Phyks/flatisfy/blob/master/flatisfy/data_files/__init__.py#L63.
Therefore, I expect flatisfy to only build the quarter "stops_fr-nw.txt".https://framagit.org/phyks/Flatisfy/-/issues/105"+" icon is confusing2018-01-18T10:59:41ZPhyksphyks@phyks.me"+" icon is confusingThe "+" icon is a bit confusing: I thought it meant "Add to following list". What about an eye icon instead ?The "+" icon is a bit confusing: I thought it meant "Add to following list". What about an eye icon instead ?https://framagit.org/phyks/Flatisfy/-/issues/102Use nominatim/Government API when possible to detect exact GPS coordinates2018-01-24T14:18:48ZPhyksphyks@phyks.meUse nominatim/Government API when possible to detect exact GPS coordinatesFor some posts (ex: https://www.selogerneuf.com/annonces/investissement/appartement/rennes-35/112639917/) the actual address is available.
When that's the case, a request to [nominatim](https://nominatim.openstreetmap.org/) or [Adresse ...For some posts (ex: https://www.selogerneuf.com/annonces/investissement/appartement/rennes-35/112639917/) the actual address is available.
When that's the case, a request to [nominatim](https://nominatim.openstreetmap.org/) or [Adresse API](https://adresse.data.gouv.fr/api/) should be made to obtain the longitude/latitude and place precisely the housing on the map.https://framagit.org/phyks/Flatisfy/-/issues/77Add a config option to filter on terms in the description2017-10-29T19:16:41ZPhyksphyks@phyks.meAdd a config option to filter on terms in the descriptionTypically, one may want to only include posts containing "parking".Typically, one may want to only include posts containing "parking".https://framagit.org/phyks/Flatisfy/-/issues/60Better handle connection errors2017-06-20T11:40:10ZPhyksphyks@phyks.meBetter handle connection errorsLooking at the logs, I've got a few connection errors to misc websites: timeouts on given websites, some 50X errors during the night. Probably they could be handled in a way that doesn't prevent executing the search on the other websites...Looking at the logs, I've got a few connection errors to misc websites: timeouts on given websites, some 50X errors during the night. Probably they could be handled in a way that doesn't prevent executing the search on the other websites; even better, there could be a fixed number of retries after a connection failure.
Here's an example of an error:
```
84164:Traceback (most recent call last):
84165- File "/usr/local/lib/python2.7/runpy.py", line 174, in _run_module_as_main
84166- "__main__", fname, loader, pkg_name)
84167- File "/usr/local/lib/python2.7/runpy.py", line 72, in _run_code
84168- exec code in run_globals
84169- File "/home/user/app/flatisfy/__main__.py", line 197, in <module>
84170- main()
84171- File "/home/user/app/flatisfy/__main__.py", line 186, in main
84172- cmds.import_and_filter(config, load_from_db=False)
84173- File "flatisfy/cmds.py", line 189, in import_and_filter
84174- fetch_details=(not load_from_db))
84175- File "flatisfy/cmds.py", line 53, in filter_flats
84176- details = fetch.fetch_details(config, flat["id"])
84177- File "flatisfy/fetch.py", line 262, in fetch_details
84178- weboob_output = weboob_proxy.info(flat_id)
84179- File "flatisfy/fetch.py", line 210, in info
84180- housing = backend.get_housing(flat_id)
84181- File "/home/user/weboob/modules/entreparticuliers/module.py", line 54, in get_housing
84182- return self.browser.get_housing(_id)
84183- File "/home/user/weboob/modules/entreparticuliers/browser.py", line 125, in get_housing
84184- obj = self.housing.go(data=data).get_housing(obj=obj)
84185- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/url.py", line 101, in go
84186- r = self.browser.location(self.build(**kwargs), params=params, data=data, method=method, headers=headers or {})
84187- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/browsers.py", line 698, in location
84188- response = self.open(*args, **kwargs)
84189- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/browsers.py", line 685, in open
84190- return super(PagesBrowser, self).open(callback=internal_callback, *args, **kwargs)
84191- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/browsers.py", line 574, in open
84192- return super(DomainBrowser, self).open(req, *args, **kwargs)
84193- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/browsers.py", line 340, in open
84194- is_async=is_async)
84195- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/sessions.py", line 159, in send
84196- return func(*args, **kwargs)
84197- File "/usr/local/lib/python2.7/site-packages/weboob-1.3-py2.7.egg/weboob/browser/sessions.py", line 151, in func
84198- resp = sup(*args, **kwargs)
84199- File "/usr/local/lib/python2.7/site-packages/requests-2.17.3-py2.7.egg/requests/sessions.py", line 623, in send
84200- r = adapter.send(request, **kwargs)
84201- File "/usr/local/lib/python2.7/site-packages/requests-2.17.3-py2.7.egg/requests/adapters.py", line 516, in send
84202- raise ReadTimeout(e, request=request)
84203-requests.exceptions.ReadTimeout: HTTPConnectionPool(host='www.entreparticuliers.com', port=80): Read timed out. (read timeout=10.0)
```https://framagit.org/phyks/Flatisfy/-/issues/101Allow to define the threshold between a housing and a station2017-12-30T13:25:24ZPhyksphyks@phyks.meAllow to define the threshold between a housing and a station> Station RU is too far from flat 104054993@seloger (160743m > 1500m), discarding it.
`1500m` should be editable through the configuration.> Station RU is too far from flat 104054993@seloger (160743m > 1500m), discarding it.
`1500m` should be editable through the configuration.https://framagit.org/phyks/Flatisfy/-/issues/93Improve detection of public transport stations2017-12-29T14:23:39ZPhyksphyks@phyks.meImprove detection of public transport stationshttps://framagit.org/phyks/Flatisfy/-/issues/76Filter out flats without photos2017-10-29T02:08:04ZPhyksphyks@phyks.meFilter out flats without photosThere should be a config option to discard flats without photos.There should be a config option to discard flats without photos.https://framagit.org/phyks/Flatisfy/-/issues/90Only fetch phone numbers when needed2017-11-29T14:56:06ZPhyksphyks@phyks.meOnly fetch phone numbers when neededPhone numbers fetching require extra API calls, and these are rate-limited. We do not want to fetch and store all the possible phone numbers for all posts.
Two options:
* Either query phone number on demand, from the web UI (similar to ...Phone numbers fetching require extra API calls, and these are rate-limited. We do not want to fetch and store all the possible phone numbers for all posts.
Two options:
* Either query phone number on demand, from the web UI (similar to what is done on housing websites).
* Or simply make all queries in batch at the end of filtering pass, on remaining flats only.
First option has the additional advantage to offer a natural workaround rate limitation.https://framagit.org/phyks/Flatisfy/-/issues/99Handle better postal codes data in rural areas2017-12-30T13:27:48ZPhyksphyks@phyks.meHandle better postal codes data in rural areasSee for instance
```
data
0
name "Bourg En Bresse"
area "FR-SE"
postal_code "01000"
lat 46.2069478835
lng 5.22442559307
id 4450
1
name "Bourg En Bresse"
area "FR-SE"
postal_code "01000"
lat 46.2069478835
lng 5.22442559307
id 4453
2
...See for instance
```
data
0
name "Bourg En Bresse"
area "FR-SE"
postal_code "01000"
lat 46.2069478835
lng 5.22442559307
id 4450
1
name "Bourg En Bresse"
area "FR-SE"
postal_code "01000"
lat 46.2069478835
lng 5.22442559307
id 4453
2
name "St Denis Les Bourg"
area "FR-SE"
postal_code "01000"
lat 46.2069478835
lng 5.22442559307
id 35748
3
name "Francheleins"
area "FR-SE"
postal_code "01090"
lat 46.0869886552
lng 4.79611073763
id 123
4
name "Genouilleux"
area "FR-SE"
postal_code "01090"
lat 46.0869886552
lng 4.79611073763
id 126
5
name "Lurcy"
area "FR-SE"
postal_code "01090"
lat 46.0869886552
lng 4.79611073763
id 168
```https://framagit.org/phyks/Flatisfy/-/issues/65Find some common data source to ease adding support for new cities2017-12-04T15:18:48ZPhyksphyks@phyks.meFind some common data source to ease adding support for new citiesAt the moment, it is quite cumbersome to add support for new cities, as one has to find some dataset of public transport stops for this city and process it.
It would be nice if we could find a common source for these data, across all av...At the moment, it is quite cumbersome to add support for new cities, as one has to find some dataset of public transport stops for this city and process it.
It would be nice if we could find a common source for these data, across all available cities. This could be OSM (but it seems difficult to extract *all* information of a given type from OSM) or Navitia (but their geographic info is difficult to match with zip codes).https://framagit.org/phyks/Flatisfy/-/issues/59Better deduplication of flats2017-06-29T10:04:48ZPhyksphyks@phyks.meBetter deduplication of flatsWhen doing a search for buying a flat, I get a lot of duplicates: the pictures seem to be the same, but the prices are different! This happens frequently, for flats which price has lowered in the last few weeks.
If you tell me where dup...When doing a search for buying a flat, I get a lot of duplicates: the pictures seem to be the same, but the prices are different! This happens frequently, for flats which price has lowered in the last few weeks.
If you tell me where duplicates are filtered and removed, I'm happy to have a look and propose a PR.
Or is there no duplication removal by default, and I need to set it in the config.json file?
Your answer on Github:
```
Duplicates are detected based on common info in the first passes (https://github.com/Phyks/Flatisfy/blob/master/flatisfy/filters/duplicates.py#L92-L175).
In the last pass (third pass, enabled by default), there should be a "deep detection", checking for these kind of flats with different prices and same photos. My use case for now is renting a flat in Paris, so the price was not evolving in time and then the price on the posts was only differing by a few euros.
Basically, this deep detection checks for any possible common info, and gives a score based on this to the pair of flats. If the score is too high, they are considered as being duplicates. All the code is here.
For now, the score threshold is 15, but it is totally empirical and may not be really optimized. :/
```v0.1Phyksphyks@phyks.mePhyksphyks@phyks.mehttps://framagit.org/phyks/Flatisfy/-/issues/66Document and extend API2017-12-05T14:17:54ZPhyksphyks@phyks.meDocument and extend APIhttps://framagit.org/phyks/Flatisfy/-/issues/41Make import faster2017-06-29T10:04:47ZPhyksphyks@phyks.meMake import fasterv0.1https://framagit.org/phyks/Flatisfy/-/issues/49Ctrl + click on table line should open it in new tab2017-06-26T14:27:03ZPhyksphyks@phyks.meCtrl + click on table line should open it in new tabv0.1https://framagit.org/phyks/Flatisfy/-/issues/95Dockerfile should use a correct user id2018-01-05T12:04:55ZPhyksphyks@phyks.meDockerfile should use a correct user idSee https://framagit.org/bnjbvr/kresus/blob/master/docker/entrypoint.sh#L9-11.See https://framagit.org/bnjbvr/kresus/blob/master/docker/entrypoint.sh#L9-11.