Skip to content

spark-quality with deequ

Adrien Lavoillotte requested to merge instanceofme/spark-etl:deequ-try into master

Initial version with simple constraint definition.

Example YAML config:

fields:
- name: str
  ratio: 0.99
  required: true
  unique: true
  allowed:
  - value1
  - value2
  pattern: "value[12]"
- name: num
  fatal: false
  minimum: 0
  maximum: 100
  noNan: true

Known issues (inherent to deequ):

  • NaN values will fail maximum constraint
  • pattern will fail if it successfully matches an empty string (implemented with regex_extract that returns an empty string when it doesn't match)
Edited by Adrien Lavoillotte

Merge request reports

Loading