Better deduplication of flats
When doing a search for buying a flat, I get a lot of duplicates: the pictures seem to be the same, but the prices are different! This happens frequently, for flats which price has lowered in the last few weeks.
If you tell me where duplicates are filtered and removed, I'm happy to have a look and propose a PR.
Or is there no duplication removal by default, and I need to set it in the config.json file?
Your answer on Github:
Duplicates are detected based on common info in the first passes (https://github.com/Phyks/Flatisfy/blob/master/flatisfy/filters/duplicates.py#L92-L175).
In the last pass (third pass, enabled by default), there should be a "deep detection", checking for these kind of flats with different prices and same photos. My use case for now is renting a flat in Paris, so the price was not evolving in time and then the price on the posts was only differing by a few euros.
Basically, this deep detection checks for any possible common info, and gives a score based on this to the pair of flats. If the score is too high, they are considered as being duplicates. All the code is here.
For now, the score threshold is 15, but it is totally empirical and may not be really optimized. :/