为了支持工作流的组合与复用,WDL允许将整个工作流作为一个步骤在更大的工作流中执行。当一个工作流调用另一个工作流时,被调用的工作流称为子工作流。需要注意的是,子工作流本身也可以包含子工作流,依此类推,对于工作流可以嵌套的深度没有明确的限制。Cromwell支持执行此类工作流。
然而,单个WDL文件只能包含一个工作流定义。为了引用子工作流,可以使用import指令引入该子工作流,并通过其别名和名称进行引用。有关如何通过导入声明和引用任务及工作流的更多详情,请参阅导入文档。
执行
子工作流的执行方式与任务完全相同。 这意味着如果另一个调用依赖于子工作流的输出,则该调用将在整个子工作流(成功)完成后运行。 例如,在以下情况下:
main.wdl
import "sub_wdl.wdl" as sub
workflow main_workflow {
call sub.hello_and_goodbye { input: hello_and_goodbye_input = "sub world" }
# call myTask { input: hello_and_goodbye.hello_output }
output {
String main_output = hello_and_goodbye.hello_output
}
}
sub_wdl.wdl
task hello {
String addressee
command {
echo "Hello ${addressee}!"
}
output {
String salutation = read_string(stdout())
}
}
task goodbye {
String addressee
command {
echo "Goodbye ${addressee}!"
}
output {
String salutation = read_string(stdout())
}
}
workflow hello_and_goodbye {
String hello_and_goodbye_input
call hello {input: addressee = hello_and_goodbye_input }
call goodbye {input: addressee = hello_and_goodbye_input }
output {
String hello_output = hello.salutation
String goodbye_output = goodbye.salutation
}
}
myTask 只有在 hello_and_goodbye 完成后才会启动(这意味着它的所有调用都已完成),尽管 myTask 只需要 hello_and_goodbye 子工作流中 hello 的输出。
如果 hello_and_goodbye 失败,那么 myTask 将不会被执行。
只有工作流输出在工作流外部可见,这意味着对子工作流产生的输出的引用只有在这些输出在工作流输出部分暴露时才有效。
子工作流在主工作流的上下文中执行,这意味着通常每个工作流只执行一次的操作(设置、清理、输出复制、日志复制等)不会为每个子工作流重新执行。例如,如果在工作流初始化期间创建了资源,子工作流需要共享这同一资源。工作流输出将为主根工作流复制,但不会为中间子工作流复制。
重启、中止和调用缓存的工作方式与任务完全相同。 子工作流运行的所有任务都与其他任务一样适用相同的调用缓存规则。 但工作流本身不会被缓存。这意味着在开启调用缓存的情况下,两次运行完全相同的工作流会触发每个任务单独缓存, 但不会缓存工作流本身。
子工作流执行文件(脚本、输出文件、日志)的根路径将位于父工作流调用目录下。 例如,上述主工作流的执行目录将如下所示:
cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/ <- main workflow id
└── call-hello_and_goodbye <- call directory for call hello_and_goodbye in the main workflow
└── hello_and_goodbye <- name of the sub-workflow
└── a6365f91-c807-465a-9186-a5d3da98fe11 <- sub-workflow id
├── call-goodbye
│ └── execution
│ ├── rc
│ ├── script
│ ├── script.background
│ ├── script.submit
│ ├── stderr
│ ├── stderr.background
│ ├── stdout
│ └── stdout.background
└── call-hello
└── execution
├── rc
├── script
├── script.background
├── script.submit
├── stderr
├── stderr.background
├── stdout
└── stdout.background
元数据
每个子工作流都会有自己独立的工作流ID。该ID会出现在父工作流的元数据中,位于对应子工作流的调用部分下的"subWorkflowId"属性里。
例如,查询上述main_workflow元数据(不包括myTask调用部分),可能会得到如下结果:
GET /api/workflows/v2/1d919bd4-d046-43b0-9918-9964509689dd/metadata
{
"workflowName": "main_workflow",
"submittedFiles": {
"inputs": "{}",
"workflow": "import \"sub_wdl.wdl\" as sub\n\nworkflow main_workflow {\n\n call sub.hello_and_goodbye { input: hello_and_goodbye_input = \"sub world\" }\n \n # call myTask { input: hello_and_goodbye.hello_output }\n \n output {\n String main_output = hello_and_goodbye.hello_output\n }\n}",
"options": "{\n\n}"
},
"calls": {
"main_workflow.hello_and_goodbye": [
{
"executionStatus": "Done",
"shardIndex": -1,
"outputs": {
"goodbye_output": "Goodbye sub world!",
"hello_output": "Hello sub world!"
},
"inputs": {
"hello_and_goodbye_input": "sub world"
},
"end": "2016-11-17T14:13:41.117-05:00",
"attempt": 1,
"start": "2016-11-17T14:13:39.236-05:00",
"subWorkflowId": "a6365f91-c807-465a-9186-a5d3da98fe11"
}
]
},
"outputs": {
"main_output": "Hello sub world!"
},
"workflowRoot": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd",
"id": "1d919bd4-d046-43b0-9918-9964509689dd",
"inputs": {},
"submission": "2016-11-17T14:13:39.104-05:00",
"status": "Succeeded",
"end": "2016-11-17T14:13:41.120-05:00",
"start": "2016-11-17T14:13:39.204-05:00"
}
子工作流ID可以单独查询:
GET /api/workflows/v2/a6365f91-c807-465a-9186-a5d3da98fe11/metadata
{
"workflowName": "hello_and_goodbye",
"calls": {
"sub.hello_and_goodbye.hello": [
{
"executionStatus": "Done",
"stdout": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-hello/execution/stdout",
"shardIndex": -1,
"outputs": {
"salutation": "Hello sub world!"
},
"runtimeAttributes": {
"failOnStderr": false,
"continueOnReturnCode": "0"
},
"cache": {
"allowResultReuse": true
},
"Effective call caching mode": "CallCachingOff",
"inputs": {
"addressee": "sub world"
},
"returnCode": 0,
"jobId": "49830",
"backend": "Local",
"end": "2016-11-17T14:13:40.712-05:00",
"stderr": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-hello/execution/stderr",
"callRoot": "/cromwell/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-hello",
"attempt": 1,
"executionEvents": [
{
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "Pending",
"endTime": "2016-11-17T14:13:39.240-05:00"
},
{
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "RequestingExecutionToken",
"endTime": "2016-11-17T14:13:39.240-05:00"
},
{
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "PreparingJob",
"endTime": "2016-11-17T14:13:39.243-05:00"
},
{
"startTime": "2016-11-17T14:13:39.243-05:00",
"description": "RunningJob",
"endTime": "2016-11-17T14:13:40.704-05:00"
},
{
"startTime": "2016-11-17T14:13:40.704-05:00",
"description": "UpdatingJobStore",
"endTime": "2016-11-17T14:13:40.712-05:00"
}
],
"start": "2016-11-17T14:13:39.239-05:00"
}
],
"sub.hello_and_goodbye.goodbye": [
{
"executionStatus": "Done",
"stdout": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-goodbye/execution/stdout",
"shardIndex": -1,
"outputs": {
"salutation": "Goodbye sub world!"
},
"runtimeAttributes": {
"failOnStderr": false,
"continueOnReturnCode": "0"
},
"cache": {
"allowResultReuse": true
},
"Effective call caching mode": "CallCachingOff",
"inputs": {
"addressee": "sub world"
},
"returnCode": 0,
"jobId": "49831",
"backend": "Local",
"end": "2016-11-17T14:13:41.115-05:00",
"stderr": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-goodbye/execution/stderr",
"callRoot": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-goodbye",
"attempt": 1,
"executionEvents": [
{
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "Pending",
"endTime": "2016-11-17T14:13:39.240-05:00"
},
{
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "RequestingExecutionToken",
"endTime": "2016-11-17T14:13:39.240-05:00"
},
{
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "PreparingJob",
"endTime": "2016-11-17T14:13:39.243-05:00"
},
{
"startTime": "2016-11-17T14:13:39.243-05:00",
"description": "RunningJob",
"endTime": "2016-11-17T14:13:41.112-05:00"
},
{
"startTime": "2016-11-17T14:13:41.112-05:00",
"description": "UpdatingJobStore",
"endTime": "2016-11-17T14:13:41.115-05:00"
}
],
"start": "2016-11-17T14:13:39.239-05:00"
}
]
},
"outputs": {
"goodbye_output": "Goodbye sub world!",
"hello_output": "Hello sub world!"
},
"workflowRoot": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11",
"id": "a6365f91-c807-465a-9186-a5d3da98fe11",
"inputs": {
"hello_and_goodbye_input": "sub world"
},
"status": "Succeeded",
"parentWorkflowId": "1d919bd4-d046-43b0-9918-9964509689dd",
"end": "2016-11-17T14:13:41.116-05:00",
"start": "2016-11-17T14:13:39.236-05:00"
}
也可以通过将URL查询参数expandSubWorkflows设置为true来自动包含子工作流元数据(默认为false)。
GET api/workflows/v2/1d919bd4-d046-43b0-9918-9964509689dd/metadata?expandSubWorkflows=true
{
"workflowName": "main_workflow",
"submittedFiles": {
"inputs": "{}",
"workflow": "import \"sub_wdl.wdl\" as sub\n\nworkflow main_workflow {\n\n call sub.hello_and_goodbye { input: hello_and_goodbye_input = \"sub world\" }\n \n # call myTask { input: hello_and_goodbye.hello_output }\n \n output {\n String main_output = hello_and_goodbye.hello_output\n }\n}",
"options": "{\n\n}"
},
"calls": {
"main_workflow.hello_and_goodbye": [{
"executionStatus": "Done",
"subWorkflowMetadata": {
"workflowName": "hello_and_goodbye",
"calls": {
"sub.hello_and_goodbye.hello": [{
"executionStatus": "Done",
"stdout": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-hello/execution/stdout",
"shardIndex": -1,
"outputs": {
"salutation": "Hello sub world!"
},
"runtimeAttributes": {
"failOnStderr": false,
"continueOnReturnCode": "0"
},
"cache": {
"allowResultReuse": true
},
"Effective call caching mode": "CallCachingOff",
"inputs": {
"addressee": "sub world"
},
"returnCode": 0,
"jobId": "49830",
"backend": "Local",
"end": "2016-11-17T14:13:40.712-05:00",
"stderr": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-hello/execution/stderr",
"callRoot": "cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-hello",
"attempt": 1,
"executionEvents": [{
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "Pending",
"endTime": "2016-11-17T14:13:39.240-05:00"
}, {
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "RequestingExecutionToken",
"endTime": "2016-11-17T14:13:39.240-05:00"
}, {
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "PreparingJob",
"endTime": "2016-11-17T14:13:39.243-05:00"
}, {
"startTime": "2016-11-17T14:13:39.243-05:00",
"description": "RunningJob",
"endTime": "2016-11-17T14:13:40.704-05:00"
}, {
"startTime": "2016-11-17T14:13:40.704-05:00",
"description": "UpdatingJobStore",
"endTime": "2016-11-17T14:13:40.712-05:00"
}],
"start": "2016-11-17T14:13:39.239-05:00"
}],
"sub.hello_and_goodbye.goodbye": [{
"executionStatus": "Done",
"stdout": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-goodbye/execution/stdout",
"shardIndex": -1,
"outputs": {
"salutation": "Goodbye sub world!"
},
"runtimeAttributes": {
"failOnStderr": false,
"continueOnReturnCode": "0"
},
"cache": {
"allowResultReuse": true
},
"Effective call caching mode": "CallCachingOff",
"inputs": {
"addressee": "sub world"
},
"returnCode": 0,
"jobId": "49831",
"backend": "Local",
"end": "2016-11-17T14:13:41.115-05:00",
"stderr": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-goodbye/execution/stderr",
"callRoot": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11/call-goodbye",
"attempt": 1,
"executionEvents": [{
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "Pending",
"endTime": "2016-11-17T14:13:39.240-05:00"
}, {
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "RequestingExecutionToken",
"endTime": "2016-11-17T14:13:39.240-05:00"
}, {
"startTime": "2016-11-17T14:13:39.240-05:00",
"description": "PreparingJob",
"endTime": "2016-11-17T14:13:39.243-05:00"
}, {
"startTime": "2016-11-17T14:13:39.243-05:00",
"description": "RunningJob",
"endTime": "2016-11-17T14:13:41.112-05:00"
}, {
"startTime": "2016-11-17T14:13:41.112-05:00",
"description": "UpdatingJobStore",
"endTime": "2016-11-17T14:13:41.115-05:00"
}],
"start": "2016-11-17T14:13:39.239-05:00"
}]
},
"outputs": {
"goodbye_output": "Goodbye sub world!",
"hello_output": "Hello sub world!"
},
"workflowRoot": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd/call-hello_and_goodbye/hello_and_goodbye/a6365f91-c807-465a-9186-a5d3da98fe11",
"id": "a6365f91-c807-465a-9186-a5d3da98fe11",
"inputs": {
"hello_and_goodbye_input": "sub world"
},
"status": "Succeeded",
"parentWorkflowId": "1d919bd4-d046-43b0-9918-9964509689dd",
"end": "2016-11-17T14:13:41.116-05:00",
"start": "2016-11-17T14:13:39.236-05:00"
},
"shardIndex": -1,
"outputs": {
"goodbye_output": "Goodbye sub world!",
"hello_output": "Hello sub world!"
},
"inputs": {
"hello_and_goodbye_input": "sub world"
},
"end": "2016-11-17T14:13:41.117-05:00",
"attempt": 1,
"start": "2016-11-17T14:13:39.236-05:00"
}]
},
"outputs": {
"main_output": "Hello sub world!"
},
"workflowRoot": "/cromwell-executions/main_workflow/1d919bd4-d046-43b0-9918-9964509689dd",
"id": "1d919bd4-d046-43b0-9918-9964509689dd",
"inputs": {
},
"submission": "2016-11-17T14:13:39.104-05:00",
"status": "Succeeded",
"end": "2016-11-17T14:13:41.120-05:00",
"start": "2016-11-17T14:13:39.204-05:00"
}