README.md 4.83 KB
Newer Older
Miguel Tremblay's avatar
Miguel Tremblay committed
1
# METRo test_suite README file
2

Miguel Tremblay's avatar
Miguel Tremblay committed
3
4
5
6
The `test_suite.py` of METRo runs pre-selected case(s) automatically and output a
forecast file named `roadcast_test_suite_run.xml` for every pre-selected case.
It then compares the forecast file `roadcast_test_suite_run.xml` with the given
benchmark forecast file `roadcast_reference.xml` and displays the comparison
Sasa Zhang's avatar
Sasa Zhang committed
7
result.
8

Miguel Tremblay's avatar
Miguel Tremblay committed
9
## Location
10

Miguel Tremblay's avatar
Miguel Tremblay committed
11
12
`test_suite.py` is located in the directory `$METRoDIR/usr/share/metro/data/test_suite`
where "$METRoDIR" represents the home directory of the software METRo.
13

Sasa Zhang's avatar
Sasa Zhang committed
14
15
16
Each test case contains a subdirectory named "caseDDD", where "DDD" is a three
digits case number (ex: "case001" represents a case having case number 1, and
"case020" represents a case having case number 20).
17

Miguel Tremblay's avatar
Miguel Tremblay committed
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
## How to run test_suite.py?

1. Go to the directory "$METRoDIR/usr/share/metro/data/test_suite" where
   "$METRoDIR" represents the home directory of METRo.

2. Enter "python3 test_suite.py" to run all the cases inside the directory
   "test_suite" and displays the running results in a simplified version.

   OPTIONS
   
   "-v":
           runs all cases inside the directory "test_suite" and displays the
           detailed information of running processes and outputs the running
           results in a verbose version.

   "-c XX XX ..." where XX represents the case number:
           runs the case(s) whose number(s) is/are entered after "-c". You can
           enter as many case numbers as you would like to run, but you have to
           enter at least one case number in order to active "-c".

           Note: "-c" and "-s" cannot be used simultaneously.

   "-s XX XX ..." where XX represents the case number:
           runs by skipping the case(s) whose number(s) is/are entered after
           "-s". You can enter as many case numbers as you would like to skip,
           but you have to enter at least one case number in order to active
           "-s".

           Note: "-c" and "-s" cannot be used simultaneously.

   "-e YY" where YY represents the error tolerance in floating points:
           runs all cases inside the directory "test_suite" and compares
           "roadcast_test_suite_run.xml" with "roadcast_reference.xml" and
           outputs comparison result(s) using YY instead of the default value
           of 0.01 as error of tolerance.

   "--clean":
           deletes all the files named "roadcast_test_suite_run.xml" within the
           directory "test_suite".

           Note: "--clean" can only be used in combination with "-v" which
                 displays all the directory/directories where the
                 "roadcast_test_suite_run.xml" is/are deleted.

   "--help":
           test_suite.py displays all the detailed information regarding above
           mentioned parameters.
65

Miguel Tremblay's avatar
Miguel Tremblay committed
66
67
68
69

## Files in the test directory

### config.json:
Sasa Zhang's avatar
Sasa Zhang committed
70
           Configuration file for the test. It includes three attributes:
Sasa Zhang's avatar
Sasa Zhang committed
71
72
           "description", "addition_to_command_line" and
           "expected_running_result".
Sasa Zhang's avatar
Sasa Zhang committed
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100

           Expected value for "description" can be any strings that you
           would like to comment on the case.

           Expected value for "addition_to_command_line":
           --config filename
           --enable-sunshadow
           --fix-deep-soil-temperature temperature
           --generate-config filename
           --lang fr|en
           --log-file filename
           --output-subsurface-levels
           --roadcast-start-date date
           --silent
           --sunshadow-method method--use-anthropogenic-flux
           --use-infrared-forecast
           --use-sst-sensor-depth
           --use-solarflux-forecast
           --verbose-level level
           --version
           --help
           Please refer to:
           https://framagit.org/metroprojects/metro/wikis/Man_page_(METRo)
           for more details.

           Expected value for "expected_running_result" can only be
           either "SUCCESS" or "FAILURE".

Sasa Zhang's avatar
Sasa Zhang committed
101
102
103
104
105
106
107
           One extra attribute "error_tolerance" can be added and the value of
           this attribute has to be in decimal places (For example, having the
           value of "0.05" as the "error_tolerance", the comparison result
           between "roadcast_test_suite_run.xml" and "roadcast_reference.xml"
           lies outside the range of 0.05; in other words, any comparison
           differences within 0.05 are ignored).

Sasa Zhang's avatar
Sasa Zhang committed
108
109
110
           Note: If you would like to use the default error tolerance of
                 "0.01", there is no need to add this extra attribute in
                 "config.json".
111

Miguel Tremblay's avatar
Miguel Tremblay committed
112
113
114
### forecast.xml, observation.xml and station.xml:

          METRo standard input files and each case should have these three
Sasa Zhang's avatar
Sasa Zhang committed
115
           files in order for METRo to generate possible forecast.
116

Miguel Tremblay's avatar
Miguel Tremblay committed
117
### roadcast_reference.xml:
Sasa Zhang's avatar
Sasa Zhang committed
118
119
           Benchmark forecast file. Only exists for those cases whose attribute
           "expected_running_result" has value of "SUCCESS".
120
121