readme 5.45 KB
Newer Older
Sasa Zhang's avatar
Sasa Zhang committed
1
2
3
-------------------------------------------------------------------------------
                          METRo test_suite README file
-------------------------------------------------------------------------------
4

Sasa Zhang's avatar
Sasa Zhang committed
5
Test suite FAQ of METRo 4.0.0 (December 2019)
6
7


Sasa Zhang's avatar
Sasa Zhang committed
8
======================== What is test_suite.py of METRo? ======================
Sasa Zhang's avatar
Sasa Zhang committed
9
10
11
12
13
The test_suite.py of METRo run pre-selected case(s) automatically and output a
forecast file named "roadcast_test_suite_run.xml" for every pre-selected case.
It then compares the forecast file "roadcast_test_suite_run.xml" with the given
benchmark forecast file "roadcast_reference.xml" and displays the comparison
result.
14
15


Sasa Zhang's avatar
Sasa Zhang committed
16
======================= Where is test_suite.py located? =======================
Sasa Zhang's avatar
Sasa Zhang committed
17
18
19
test_suite.py is located in the directory "test_suite", which is in
"$METRoDIR/usr/share/metro/data/test_suite" where "$METRoDIR" represents the
home directory of the software METRo.
20
21


Sasa Zhang's avatar
Sasa Zhang committed
22
======================= Where is each test case located? ======================
Sasa Zhang's avatar
Sasa Zhang committed
23
24
25
26
The directory "test_suite" contains all the subdirectories for test cases.
Each test case contains a subdirectory named "caseDDD", where "DDD" is a three
digits case number (ex: "case001" represents a case having case number 1, and
"case020" represents a case having case number 20).
27
28


Sasa Zhang's avatar
Sasa Zhang committed
29
====== What are all the possible files included inside a case directory? ======
30
config.json:
Sasa Zhang's avatar
Sasa Zhang committed
31
           Configuration file for the test. It includes three attributes:
Sasa Zhang's avatar
Sasa Zhang committed
32
33
           "description", "addition_to_command_line" and
           "expected_running_result".
Sasa Zhang's avatar
Sasa Zhang committed
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

           Expected value for "description" can be any strings that you
           would like to comment on the case.

           Expected value for "addition_to_command_line":
           --config filename
           --enable-sunshadow
           --fix-deep-soil-temperature temperature
           --generate-config filename
           --lang fr|en
           --log-file filename
           --output-subsurface-levels
           --roadcast-start-date date
           --silent
           --sunshadow-method method--use-anthropogenic-flux
           --use-infrared-forecast
           --use-sst-sensor-depth
           --use-solarflux-forecast
           --verbose-level level
           --version
           --help
           Please refer to:
           https://framagit.org/metroprojects/metro/wikis/Man_page_(METRo)
           for more details.

           Expected value for "expected_running_result" can only be
           either "SUCCESS" or "FAILURE".

Sasa Zhang's avatar
Sasa Zhang committed
62
63
64
65
66
67
68
           One extra attribute "error_tolerance" can be added and the value of
           this attribute has to be in decimal places (For example, having the
           value of "0.05" as the "error_tolerance", the comparison result
           between "roadcast_test_suite_run.xml" and "roadcast_reference.xml"
           lies outside the range of 0.05; in other words, any comparison
           differences within 0.05 are ignored).

Sasa Zhang's avatar
Sasa Zhang committed
69
70
71
           Note: If you would like to use the default error tolerance of
                 "0.01", there is no need to add this extra attribute in
                 "config.json".
72

73
forecast.xml, observation.xml and station.xml:
Sasa Zhang's avatar
Sasa Zhang committed
74
           METRo standard input files and each case should have these three
Sasa Zhang's avatar
Sasa Zhang committed
75
           files in order for METRo to generate possible forecast.
76

77
roadcast_reference.xml:
Sasa Zhang's avatar
Sasa Zhang committed
78
79
           Benchmark forecast file. Only exists for those cases whose attribute
           "expected_running_result" has value of "SUCCESS".
80
81


Sasa Zhang's avatar
Sasa Zhang committed
82
======================== How to run test_suite.py? ============================
Sasa Zhang's avatar
Sasa Zhang committed
83
84
1. Go to the directory "$METRoDIR/usr/share/metro/data/test_suite" where
   "$METRoDIR" represents the home directory of METRo.
85

Sasa Zhang's avatar
Sasa Zhang committed
86
87
2. Enter "python3 test_suite.py" to run all the cases inside the directory
   "test_suite" and displays the running results in a simplified version.
88

Sasa Zhang's avatar
Sasa Zhang committed
89
   OPTIONS
Sasa Zhang's avatar
Sasa Zhang committed
90
   "-v":
Sasa Zhang's avatar
Sasa Zhang committed
91
92
93
           runs all cases inside the directory "test_suite" and displays the
           detailed information of running processes and outputs the running
           results in a verbose version.
94

Sasa Zhang's avatar
Sasa Zhang committed
95
   "-c XX XX ..." where XX represents the case number:
Sasa Zhang's avatar
Sasa Zhang committed
96
97
98
           runs the case(s) whose number(s) is/are entered after "-c". You can
           enter as many case numbers as you would like to run, but you have to
           enter at least one case number in order to active "-c".
99

Sasa Zhang's avatar
Sasa Zhang committed
100
           Note: "-c" and "-s" cannot be used simultaneously.
101

Sasa Zhang's avatar
Sasa Zhang committed
102
   "-s XX XX ..." where XX represents the case number:
Sasa Zhang's avatar
Sasa Zhang committed
103
104
105
106
           runs by skipping the case(s) whose number(s) is/are entered after
           "-s". You can enter as many case numbers as you would like to skip,
           but you have to enter at least one case number in order to active
           "-s".
107

Sasa Zhang's avatar
Sasa Zhang committed
108
           Note: "-c" and "-s" cannot be used simultaneously.
109

110
   "-e YY" where YY represents the error tolerance in floating points:
Sasa Zhang's avatar
Sasa Zhang committed
111
112
113
114
           runs all cases inside the directory "test_suite" and compares
           "roadcast_test_suite_run.xml" with "roadcast_reference.xml" and
           outputs comparison result(s) using YY instead of the default value
           of 0.01 as error of tolerance.
115

116
   "--clean":
Sasa Zhang's avatar
Sasa Zhang committed
117
118
           deletes all the files named "roadcast_test_suite_run.xml" within the
           directory "test_suite".
119

Sasa Zhang's avatar
Sasa Zhang committed
120
121
122
           Note: "--clean" can only be used in combination with "-v" which
                 displays all the directory/directories where the
                 "roadcast_test_suite_run.xml" is/are deleted.
123

Sasa Zhang's avatar
Sasa Zhang committed
124
125
126
   "--help":
           test_suite.py displays all the detailed information regarding above
           mentioned parameters.