global step_container needs explanation
The Step
class has this strange pattern which I can not make sense of:
some classmethods like
@classmethod
def get_exec_graph(cls):
"""return the execution graph of the last step append
"""
return dask.delayed(cls.task_launcher)(step_container[-1].step_tasks)
use the global variable step_container
which is declared as a singleton list without explanation:
step_container = singleton_list()
The problem is that in some way that I did not find, a list
gets to be passed to the task_launcher
method of Step
class. For example, if I print some info in the task_launcher
like this
def task_launcher(self,
*dependencies,
task_name: Optional[str] = None,
log_err: Optional[str] = None,
log_out: Optional[str] = None,
scheduler_type: str = "debug",
pbs_worker_name: str = "i2-worker",
# [...]
**kwargs) -> str:
"""this function encapsulate tasks to be delayed
[...]
"""
print(">>>>>")
print("task_name", end=": ")
print(task_name)
print("self", end=": ")
print(self)
print("type(self)", end=": ")
print(type(self))
if task_name and self.older_i2_state:
if task_name in self.older_i2_state and self.older_i2_state[
task_name] == "done":
return "done"
dependencies_complete = all([dep == "done" for dep in dependencies])
if task_name and dependencies and not dependencies_complete:
task_status = "unlaunchable"
self.save_task_status(task_name, task_status)
return task_status
# [...]
I get
>>>>>
task_name: check_inputs
self: CheckInputsClassifWorkflow
type(self): <class 'iota2.steps.check_inputs_step.CheckInputsClassifWorkflow'>
>>>>>
task_name: preprocessing_T31TCJ
self: SensorsPreprocess
type(self): <class 'iota2.steps.sensors_preprocess.SensorsPreprocess'>
>>>>>
task_name: None
self: ['done']
type(self): <class 'list'>
>>>>>
task_name: common_mask_T31TCJ
self: CommonMasks
type(self): <class 'iota2.steps.common_masks.CommonMasks'>
(note how
task_laucher
get called on the['done']
list)
I think this is related to a bug I'm getting on my new regression builder, but all of this is rather obscure to me and I'm failing to debug myself in this darkness.
Could someone enlighten me?