Skip to content

Optimize photo comparisons

Phyks requested to merge better-dedup into master

This commit actually does 3 things and might be a bit controversial:

  • it introduces a way to test performance of a given function by adding a decorator factory for measuring performance.
  • it introduces a photos cache, to prevent useless network requests to fetch pictures (this locally improved the 3rd pass duration from 570 seconds to 70 seconds for my search).
  • it decides that two real estate ads are the same if they either have less than 3 pics and they're all the same, or at least 3 common pics. I don't have data backup this up, but it seemed to work great in my case.

Happy to split this in three smaller commits and/or drop changes that are too controversial to be upstreamed.

Merge request reports