spark-quality with deequ
Initial version with simple constraint definition.
Example YAML config:
fields:
- name: str
ratio: 0.99
required: true
unique: true
allowed:
- value1
- value2
pattern: "value[12]"
- name: num
fatal: false
minimum: 0
maximum: 100
noNan: true
Known issues (inherent to deequ):
-
NaN
values will failmaximum
constraint -
pattern
will fail if it successfully matches an empty string (implemented with regex_extract that returns an empty string when it doesn't match)
Edited by Adrien Lavoillotte