docs/TextProcessingPipeline.html · ly-code/taskflow

【升级通知】Gitee 将于 7 月 11 日晚 23 点进行全站升级维护

代码拉取完成,页面将自动刷新

richgiteeai

Watch

不关注关注所有动态仅关注版本发行动态关注但不提醒动态

1 Star 0 Fork 0

ly-code/taskflow

代码 Issues 0 Pull Requests 0 Wiki 统计流水线

服务

加入 Gitee

与超过 1400万开发者一起发现、参与优秀开源项目,私有仓库也完全免费 :)

免费加入

已有帐号? 立即登录

文件

master

分支 (3)

标签 (16)

管理

master

dev

dev-defer

v3.6.0

v3.5.0

v3.4.0

v3.3.0

v3.2.0

v3.1.0

v3.0.0

v2.7.0

v2.6.0

v2.5.0

v2.4.0

v2.3.1

v2.3.0

v2.2.0

v2.1.0

v2.0.0

克隆/下载

HTTPS SSH SVN SVN+SSH 下载ZIP

提示

下载代码请复制以下命令到终端执行

为确保你提交的代码身份被 Gitee 正确识别,请执行以下命令完成配置

git config --global user.name userName 
git config --global user.email userEmail

初次使用 SSH 协议进行代码克隆、推送等操作时,需按下述提示完成 SSH 配置

1 生成 RSA 密钥

2 获取 RSA 公钥内容,并配置到 SSH公钥中

在 Gitee 上使用 SVN,请访问使用指南

使用 HTTPS 协议时,命令行会出现如下账号密码验证步骤。基于安全考虑,Gitee 建议配置并使用私人令牌替代登录密码进行克隆、推送等操作

Username for 'https://gitee.com': userName

Password for 'https://userName@gitee.com': # 私人令牌

分支 3

标签 16

taskflow

docs

TextProcessingPipeline.html

TextProcessingPipeline.html 48.99 KB

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8" />
 <title>Learning from Examples &raquo; Text Processing Pipeline | Taskflow QuickStart</title>
 <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,400i,600,600i%7CSource+Code+Pro:400,400i,600" />
 <link rel="stylesheet" href="m-dark+documentation.compiled.css" />
 <link rel="icon" href="favicon.ico" type="image/vnd.microsoft.icon" />
 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
 <meta name="theme-color" content="#22272e" />
</head>
<body>
<header><nav id="navigation">
 <div class="m-container">
 <div class="m-row">
 
 <a href="https://taskflow.github.io"><img src="taskflow_logo.png" alt="" />Taskflow</a> | <a href="index.html" class="m-thin">QuickStart</a>
 
 <div class="m-col-t-4 m-hide-m m-text-right m-nopadr">
 <a href="#search" class="m-doc-search-icon" title="Search" onclick="return showSearch()"><svg style="height: 0.9rem;" viewBox="0 0 16 16">
 <path id="m-doc-search-icon-path" d="m6 0c-3.31 0-6 2.69-6 6 0 3.31 2.69 6 6 6 1.49 0 2.85-0.541 3.89-1.44-0.0164 0.338 0.147 0.759 0.5 1.15l3.22 3.79c0.552 0.614 1.45 0.665 2 0.115 0.55-0.55 0.499-1.45-0.115-2l-3.79-3.22c-0.392-0.353-0.812-0.515-1.15-0.5 0.895-1.05 1.44-2.41 1.44-3.89 0-3.31-2.69-6-6-6zm0 1.56a4.44 4.44 0 0 1 4.44 4.44 4.44 4.44 0 0 1-4.44 4.44 4.44 4.44 0 0 1-4.44-4.44 4.44 4.44 0 0 1 4.44-4.44z"/>
 </svg></a>
 <a id="m-navbar-show" href="#navigation" title="Show navigation"></a>
 <a id="m-navbar-hide" href="#" title="Hide navigation"></a>
 </div>
 <div id="m-navbar-collapse" class="m-col-t-12 m-show-m m-col-m-none m-right-m">
 <div class="m-row">
 <ol class="m-col-t-6 m-col-m-none">
 <li><a href="pages.html">Handbook</a></li>
 <li><a href="namespaces.html">Namespaces</a></li>
 </ol>
 <ol class="m-col-t-6 m-col-m-none" start="3">
 <li><a href="annotated.html">Classes</a></li>
 <li><a href="files.html">Files</a></li>
 <li class="m-show-m"><a href="#search" class="m-doc-search-icon" title="Search" onclick="return showSearch()"><svg style="height: 0.9rem;" viewBox="0 0 16 16">
 <use href="#m-doc-search-icon-path" />
 </svg></a></li>
 </ol>
 </div>
 </div>
 </div>
 </div>
</nav></header>
<main><article>
 <div class="m-container m-container-inflatable">
 <div class="m-row">
 <div class="m-col-l-10 m-push-l-1">
 <h1>
 <a href="Examples.html">Learning from Examples</a> &raquo;
 Text Processing Pipeline
 </h1>
 <nav class="m-block m-default">
 <h3>Contents</h3>
 <ul>
 <li><a href="#FormulateTheTextProcessingPipelineProblem">Formulate the Text Processing Pipeline Problem</a></li>
 <li>
 <a href="#CreateAParallelTextPipeline">Create a Text Processing Pipeline</a>
 <ul>
 <li><a href="#TextPipelineDefineTheDataBuffer">Define the Data Buffer</a></li>
 <li><a href="#TextPipelineDefineThePipes">Define the Pipes</a></li>
 <li><a href="#TextPipelineDefineTheTaskGraph">Define the Task Graph</a></li>
 <li><a href="#TextPipelineSubmitTheTaskGraph">Submit the Task Graph</a></li>
 </ul>
 </li>
 </ul>
 </nav>
We study a text processing pipeline that finds the most frequent character of each string from an input source. Parallelism exhibits in the form of a three-stage pipeline that transforms the input string to a final pair type.<section id="FormulateTheTextProcessingPipelineProblem"><h2><a href="#FormulateTheTextProcessingPipelineProblem">Formulate the Text Processing Pipeline Problem</a></h2>Given an input vector of strings, we want to compute the most frequent character for each string using a series of transform operations. For example:<pre class="m-console"># input strings
abade
ddddf
eefge
xyzzd
ijjjj
jiiii
kkijk

# output
a:2
d:4
e:3
z:2
j:4
i:4
k:3</pre>We decompose the algorithm into three stages:<ol><li>read a <code>std::string</code> from the input vector</li><li>generate a <code>std::unorder_map&lt;char, size_t&gt;</code> frequency map from the string</li><li>reduce the most frequent character to a <code>std::pair&lt;char, size_t&gt;</code> from the map</li></ol>The first and the third stages process inputs and generate results in serial, and the second stage can run in parallel. The algorithm is a perfect fit to pipeline parallelism, as different stages can overlap with each other in time across parallel lines.</section><section id="CreateAParallelTextPipeline"><h2><a href="#CreateAParallelTextPipeline">Create a Text Processing Pipeline</a></h2>We create a pipeline of three pipes (stages) and two parallel lines to solve the problem. The number of parallel lines is a tunable parameter. In most cases, we can just use <code><a href="http://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency.html" class="m-doc-external">std::thread::hardware_concurrency</a></code> as the line count. The first pipe reads an input string from the vector in order, the second pipe transforms the input string from the first pipe to a frequency map in parallel, and the third pipe reduces the frequency map to find the most frequent character. The overall implementation is shown below:<pre class="m-code">#include &lt;taskflow/taskflow.hpp&gt;
#include &lt;taskflow/algorithm/pipeline.hpp&gt;

// Function: format the map
std::string format_map(const std::unordered_map&lt;char, size_t&gt;&amp; map) {
 std::ostringstream oss;
 for(const auto&amp; [i, j] : map) {
 oss &lt;&lt; i &lt;&lt; &#39;:&#39; &lt;&lt; j &lt;&lt; &#39; &#39;;
 }
 return oss.str();
}

int main() {

tf::Taskflow taskflow(&quot;text-filter pipeline&quot;);
 tf::Executor executor;

const size_t num_lines = 2;

// input data 
 std::vector&lt;std::string&gt; input = {
 &quot;abade&quot;, 
 &quot;ddddf&quot;,
 &quot;eefge&quot;,
 &quot;xyzzd&quot;,
 &quot;ijjjj&quot;,
 &quot;jiiii&quot;,
 &quot;kkijk&quot;
 };

// custom data storage
 using data_type = std::variant&lt;
 std::string, std::unordered_map&lt;char, size_t&gt;, std::pair&lt;char, size_t&gt;
 &gt;;
 std::array&lt;data_type, num_lines&gt; mybuffer;

// the pipeline consists of three pipes (serial-parallel-serial)
 // and up to two concurrent scheduling tokens
 tf::Pipeline pl(num_lines,

// first pipe processes the input data
 tf::Pipe{tf::PipeType::SERIAL, [&amp;](tf::Pipeflow&amp; pf) {
 if(pf.token() == input.size()) {
 pf.stop();
 }
 else {
 printf(&quot;stage 1: input token = %s\n&quot;, input[pf.token()].c_str());
 mybuffer[pf.line()] = input[pf.token()];
 }
 }},
 
 // second pipe counts the frequency of each character
 tf::Pipe{tf::PipeType::PARALLEL, [&amp;](tf::Pipeflow&amp; pf) {
 std::unordered_map&lt;char, size_t&gt; map;
 for(auto c : std::get&lt;std::string&gt;(mybuffer[pf.line()])) {
 map[c]++;
 }
 printf(&quot;stage 2: map = %s\n&quot;, format_map(map).c_str());
 mybuffer[pf.line()] = map;
 }},
 
 // third pipe reduces the most frequent character
 tf::Pipe{tf::PipeType::SERIAL, [&amp;mybuffer](tf::Pipeflow&amp; pf) {
 auto&amp; map = std::get&lt;std::unordered_map&lt;char, size_t&gt;&gt;(mybuffer[pf.line()]);
 auto sol = std::max_element(map.begin(), map.end(), [](auto&amp; a, auto&amp; b){
 return a.second &lt; b.second;
 });
 printf(&quot;stage 3: %c:%zu\n&quot;, sol-&gt;first, sol-&gt;second);
 // not necessary to store the last-stage data, just for demo purpose
 mybuffer[pf.line()] = *sol; 
 }}
 );
 
 // build the pipeline graph using composition
 tf::Task init = taskflow.emplace([](){ std::cout &lt;&lt; &quot;ready\n&quot;; })
 .name(&quot;starting pipeline&quot;);
 tf::Task task = taskflow.composed_of(pl)
 .name(&quot;pipeline&quot;);
 tf::Task stop = taskflow.emplace([](){ std::cout &lt;&lt; &quot;stopped\n&quot;; })
 .name(&quot;pipeline stopped&quot;);

// create task dependency
 init.precede(task);
 task.precede(stop);
 
 // dump the pipeline graph structure (with composition)
 taskflow.dump(std::cout);

// run the pipeline
 executor.run(taskflow).wait();
 
 return 0;
}</pre><section id="TextPipelineDefineTheDataBuffer"><h3><a href="#TextPipelineDefineTheDataBuffer">Define the Data Buffer</a></h3>Taskflow does not provide any data abstraction to perform pipeline scheduling, but give users full control over data management in their applications. In this example, we create an one-dimensional buffer of a <a href="https://en.cppreference.com/w/cpp/utility/variant">std::variant</a> data type to store the output of each pipe in a uniform storage:<pre class="m-code">using data_type = std::variant&lt;
 std::string, std::unordered_map&lt;char, size_t&gt;, std::pair&lt;char, size_t&gt;
&gt;;
std::array&lt;std::array&lt;data_type, num_pipes&gt;, num_lines&gt; mybuffer;</pre><aside class="m-note m-info"><h4>Note</h4>One-dimensional buffer is sufficient because Taskflow enables only one scheduling token per line at a time.</aside></section><section id="TextPipelineDefineThePipes"><h3><a href="#TextPipelineDefineThePipes">Define the Pipes</a></h3>The first pipe reads one string and puts it in the corresponding entry at the buffer, <code>mybuffer[pf.line()]</code>. Since we read in each string in order, we declare the pipe as a serial type:<pre class="m-code">tf::Pipe{tf::PipeType::SERIAL, [&amp;](tf::Pipeflow&amp; pf) {
 if(pf.token() == input.size()) {
 pf.stop();
 }
 else {
 mybuffer[pf.line()] = input[pf.token()];
 printf(&quot;stage 1: input token = %s\n&quot;, input[pf.token()].c_str());
 }
}},</pre>The second pipe needs to get the input string from the previous pipe and then transforms that input string into a frequency map that records the occurrence of each character in the string. As multiple transforms can operate simultaneously, we declare the pipe as a parallel type:<pre class="m-code">tf::Pipe{tf::PipeType::PARALLEL, [&amp;](tf::Pipeflow&amp; pf) {
 std::unordered_map&lt;char, size_t&gt; map;
 for(auto c : std::get&lt;std::string&gt;(mybuffer[pf.line()])) {
 map[c]++;
 }
 mybuffer[pf.line()] = map;
 printf(&quot;stage 2: map = %s\n&quot;, format_map(map).c_str());
}}</pre>Similarly, the third pipe needs to get the input frequency map from the previous pipe and then reduces the result to find the most frequent character. We may not need to store the result in the buffer but other places defined by the application (e.g., an output file). As we want to output the result in the same order as the input, we declare the pipe as a serial type:<pre class="m-code">tf::Pipe{tf::PipeType::SERIAL, [&amp;mybuffer](tf::Pipeflow&amp; pf) {
 auto&amp; map = std::get&lt;std::unordered_map&lt;char, size_t&gt;&gt;(mybuffer[pf.line()]);
 auto sol = std::max_element(map.begin(), map.end(), [](auto&amp; a, auto&amp; b){
 return a.second &lt; b.second;
 });
 printf(&quot;stage 3: %c:%zu\n&quot;, sol-&gt;first, sol-&gt;second);
}}</pre></section><section id="TextPipelineDefineTheTaskGraph"><h3><a href="#TextPipelineDefineTheTaskGraph">Define the Task Graph</a></h3>To build up the taskflow graph for the pipeline, we create a module task out of the pipeline structure and connect it with two tasks that outputs messages before and after the pipeline:<pre class="m-code">tf::Task init = taskflow.emplace([](){ std::cout &lt;&lt; &quot;ready\n&quot;; })
 .name(&quot;starting pipeline&quot;);
tf::Task task = taskflow.composed_of(pl)
 .name(&quot;pipeline&quot;);
tf::Task stop = taskflow.emplace([](){ std::cout &lt;&lt; &quot;stopped\n&quot;; })
 .name(&quot;pipeline stopped&quot;);
init.precede(task);
task.precede(stop);</pre></section><section id="TextPipelineSubmitTheTaskGraph"><h3><a href="#TextPipelineSubmitTheTaskGraph">Submit the Task Graph</a></h3>Finally, we submit the taskflow to the execution and run it once:<pre class="m-code">executor.run(taskflow).wait();</pre>As the second stage is a parallel pipe, the output may interleave. One possible result is shown below:<pre class="m-console">ready
stage 1: input token = abade
stage 1: input token = ddddf
stage 2: map = f:1 d:4 
stage 2: map = e:1 d:1 a:2 b:1 
stage 3: a:2
stage 1: input token = eefge
stage 2: map = g:1 e:3 f:1 
stage 3: d:4
stage 1: input token = xyzzd
stage 3: e:3
stage 1: input token = ijjjj
stage 2: map = z:2 x:1 d:1 y:1 
stage 3: z:2
stage 1: input token = jiiii
stage 2: map = j:4 i:1 
stage 3: j:4
stage 2: map = i:4 j:1 
stage 1: input token = kkijk
stage 3: i:4
stage 2: map = j:1 k:3 i:1 
stage 3: k:3
stopped</pre>We can see seven outputs at the third stage that show the most frequent character for each of the seven strings in order (<code>a:2</code>, <code>d:4</code>, <code>e:3</code>, <code>z:2</code>, <code>j:4</code>, <code>i:4</code>, <code>k:3</code>). The taskflow graph of this pipeline workload is shown below:<div class="m-graph"><svg style="width: 30.800rem; height: 32.700rem;" viewBox="0.00 0.00 308.00 327.00">
<g transform="scale(1 1) rotate(0) translate(4 323)">
<title>Taskflow</title>
<g class="m-cluster">
<title>cluster_p0x7ffd7418c200</title>
<polygon points="8,-8 8,-311 142,-311 142,-8 8,-8"/>
<text text-anchor="middle" x="75" y="-299" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Text Processing Pipeline</text>
</g>
<g class="m-cluster">
<title>cluster_p0x7ffd7418c110</title>
<polygon points="150,-119 150,-311 292,-311 292,-119 150,-119"/>
<text text-anchor="middle" x="221" y="-299" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">m1</text>
</g>
<g class="m-node m-flat">
<title>p0x7bc4000142e8</title>
<ellipse cx="75" cy="-266" rx="56.5233" ry="18"/>
<text text-anchor="middle" x="75" y="-263.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">starting pipeline</text>
</g>
<g class="m-node m-flat">
<title>p0x7bc4000143d0</title>
<polygon points="116,-163 38,-163 34,-159 34,-127 112,-127 116,-131 116,-163"/>
<polyline points="112,-159 34,-159 "/>
<polyline points="112,-159 112,-127 "/>
<polyline points="112,-159 116,-163 "/>
<text text-anchor="middle" x="75" y="-142.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">pipeline [m1]</text>
</g>
<g class="m-edge">
<title>p0x7bc4000142e8&#45;&gt;p0x7bc4000143d0</title>
<path d="M75,-247.8851C75,-228.1928 75,-196.4608 75,-173.439"/>
<polygon points="78.5001,-173.2387 75,-163.2388 71.5001,-173.2388 78.5001,-173.2387"/>
</g>
<g class="m-node m-flat">
<title>p0x7bc4000144b8</title>
<ellipse cx="75" cy="-34" rx="57.8786" ry="18"/>
<text text-anchor="middle" x="75" y="-31.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">pipeline stopped</text>
</g>
<g class="m-edge">
<title>p0x7bc4000143d0&#45;&gt;p0x7bc4000144b8</title>
<path d="M75,-126.6706C75,-109.2373 75,-82.7482 75,-62.5489"/>
<polygon points="78.5001,-62.3566 75,-52.3566 71.5001,-62.3567 78.5001,-62.3566"/>
</g>
<g class="m-node">
<title>p0x7bc400014030</title>
<polygon points="218,-284 184.8992,-266 218,-248 251.1008,-266 218,-284"/>
<text text-anchor="middle" x="218" y="-263.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">cond</text>
</g>
<g class="m-node m-flat">
<title>p0x7bc400014118</title>
<polygon points="212,-163 158,-163 158,-159 154,-159 154,-155 158,-155 158,-135 154,-135 154,-131 158,-131 158,-127 212,-127 212,-163"/>
<polyline points="158,-159 162,-159 162,-155 158,-155 "/>
<polyline points="158,-135 162,-135 162,-131 158,-131 "/>
<text text-anchor="middle" x="185" y="-142.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">rt&#45;0</text>
</g>
<g class="m-edge">
<title>p0x7bc400014030&#45;&gt;p0x7bc400014118</title>
<path stroke-dasharray="5,2" d="M213.6858,-250.1814C208.3593,-230.6507 199.2047,-197.0838 192.6698,-173.1225"/>
<polygon points="195.9696,-171.9194 189.9616,-163.1927 189.2162,-173.7612 195.9696,-171.9194"/>
<text text-anchor="middle" x="205.5" y="-203" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">0</text>
</g>
<g class="m-node m-flat">
<title>p0x7bc400014200</title>
<polygon points="284,-163 230,-163 230,-159 226,-159 226,-155 230,-155 230,-135 226,-135 226,-131 230,-131 230,-127 284,-127 284,-163"/>
<polyline points="230,-159 234,-159 234,-155 230,-155 "/>
<polyline points="230,-135 234,-135 234,-131 230,-131 "/>
<text text-anchor="middle" x="257" y="-142.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">rt&#45;1</text>
</g>
<g class="m-edge">
<title>p0x7bc400014030&#45;&gt;p0x7bc400014200</title>
<path stroke-dasharray="5,2" d="M222.9394,-250.6753C229.2165,-231.2 240.1705,-197.2146 247.9606,-173.0452"/>
<polygon points="251.3461,-173.9504 251.0827,-163.3588 244.6837,-171.8029 251.3461,-173.9504"/>
<text text-anchor="middle" x="241.5" y="-203" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">1</text>
</g>
</g>
</svg>
</div></section></section>
 </div>
 </div>
 </div>
</article></main>
<div class="m-doc-search" id="search">
 <a href="#!" onclick="return hideSearch()"></a>
 <div class="m-container">
 <div class="m-row">
 <div class="m-col-m-8 m-push-m-2">
 <div class="m-doc-search-header m-text m-small">
 <div>Tab / T to search, Esc to close</div>
 <div id="search-symbolcount">&hellip;</div>
 </div>
 <div class="m-doc-search-content">
 <form>
 <input type="search" name="q" id="search-input" placeholder="Loading &hellip;" disabled="disabled" autofocus="autofocus" autocomplete="off" spellcheck="false" />
 </form>
 <noscript class="m-text m-danger m-text-center">Unlike everything else in the docs, the search functionality requires JavaScript.</noscript>
 <div id="search-help" class="m-text m-dim m-text-center">
 Search for symbols, directories, files, pages or
 modules. You can omit any prefix from the symbol or file path; adding a
 <code>:</code> or <code>/</code> suffix lists all members of given symbol or
 directory.
 Use &darr;
 / &uarr; to navigate through the list,
 Enter to go.
 Tab autocompletes common prefix, you can
 copy a link to the result using ⌘
 L while ⌘
 M produces a Markdown link.
 </div>
 <div id="search-notfound" class="m-text m-warning m-text-center">Sorry, nothing was found.</div>
 <ul id="search-results"></ul>
 </div>
 </div>
 </div>
 </div>
</div>
<script src="search-v2.js"></script>
<script src="searchdata-v2.js" async="async"></script>
<footer><nav>
 <div class="m-container">
 <div class="m-row">
 <div class="m-col-l-10 m-push-l-1">
 Taskflow handbook is part of the <a href="https://taskflow.github.io">Taskflow project</a>, copyright © <a href="https://tsung-wei-huang.github.io/">Dr. Tsung-Wei Huang</a>, 2018&ndash;2023. Generated by <a href="https://doxygen.org/">Doxygen</a> 1.8.14 and <a href="https://mcss.mosra.cz/">m.css</a>.
 </div>
 </div>
 </div>
</nav></footer>
</body>
</html>

一键复制编辑原始数据按行查看历史

twhuang 提交于 2023年05月10日 03:18 +08:00 . fixed old async behavior in documentation

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353

<!DOCTYPE html>
<html lang="en">
<head>
 <meta charset="UTF-8" />
 <title>Learning from Examples &raquo; Text Processing Pipeline | Taskflow QuickStart</title>
 <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:400,400i,600,600i%7CSource+Code+Pro:400,400i,600" />
 <link rel="stylesheet" href="m-dark+documentation.compiled.css" />
 <link rel="icon" href="favicon.ico" type="image/vnd.microsoft.icon" />
 <meta name="viewport" content="width=device-width, initial-scale=1.0" />
 <meta name="theme-color" content="#22272e" />
</head>
<body>
<header><nav id="navigation">
 <div class="m-container">
 <div class="m-row">
 <span id="m-navbar-brand" class="m-col-t-8 m-col-m-none m-left-m">
 <a href="https://taskflow.github.io"><img src="taskflow_logo.png" alt="" />Taskflow</a> <span class="m-breadcrumb">|</span> <a href="index.html" class="m-thin">QuickStart</a>
 </span>
 <div class="m-col-t-4 m-hide-m m-text-right m-nopadr">
 <a href="#search" class="m-doc-search-icon" title="Search" onclick="return showSearch()"><svg style="height: 0.9rem;" viewBox="0 0 16 16">
 <path id="m-doc-search-icon-path" d="m6 0c-3.31 0-6 2.69-6 6 0 3.31 2.69 6 6 6 1.49 0 2.85-0.541 3.89-1.44-0.0164 0.338 0.147 0.759 0.5 1.15l3.22 3.79c0.552 0.614 1.45 0.665 2 0.115 0.55-0.55 0.499-1.45-0.115-2l-3.79-3.22c-0.392-0.353-0.812-0.515-1.15-0.5 0.895-1.05 1.44-2.41 1.44-3.89 0-3.31-2.69-6-6-6zm0 1.56a4.44 4.44 0 0 1 4.44 4.44 4.44 4.44 0 0 1-4.44 4.44 4.44 4.44 0 0 1-4.44-4.44 4.44 4.44 0 0 1 4.44-4.44z"/>
 </svg></a>
 <a id="m-navbar-show" href="#navigation" title="Show navigation"></a>
 <a id="m-navbar-hide" href="#" title="Hide navigation"></a>
 </div>
 <div id="m-navbar-collapse" class="m-col-t-12 m-show-m m-col-m-none m-right-m">
 <div class="m-row">
 <ol class="m-col-t-6 m-col-m-none">
 <li><a href="pages.html">Handbook</a></li>
 <li><a href="namespaces.html">Namespaces</a></li>
 </ol>
 <ol class="m-col-t-6 m-col-m-none" start="3">
 <li><a href="annotated.html">Classes</a></li>
 <li><a href="files.html">Files</a></li>
 <li class="m-show-m"><a href="#search" class="m-doc-search-icon" title="Search" onclick="return showSearch()"><svg style="height: 0.9rem;" viewBox="0 0 16 16">
 <use href="#m-doc-search-icon-path" />
 </svg></a></li>
 </ol>
 </div>
 </div>
 </div>
 </div>
</nav></header>
<main><article>
 <div class="m-container m-container-inflatable">
 <div class="m-row">
 <div class="m-col-l-10 m-push-l-1">
 <h1>
 <span class="m-breadcrumb"><a href="Examples.html">Learning from Examples</a> &raquo;</span>
 Text Processing Pipeline
 </h1>
 <nav class="m-block m-default">
 <h3>Contents</h3>
 <ul>
 <li><a href="#FormulateTheTextProcessingPipelineProblem">Formulate the Text Processing Pipeline Problem</a></li>
 <li>
 <a href="#CreateAParallelTextPipeline">Create a Text Processing Pipeline</a>
 <ul>
 <li><a href="#TextPipelineDefineTheDataBuffer">Define the Data Buffer</a></li>
 <li><a href="#TextPipelineDefineThePipes">Define the Pipes</a></li>
 <li><a href="#TextPipelineDefineTheTaskGraph">Define the Task Graph</a></li>
 <li><a href="#TextPipelineSubmitTheTaskGraph">Submit the Task Graph</a></li>
 </ul>
 </li>
 </ul>
 </nav>
<p>We study a text processing pipeline that finds the most frequent character of each string from an input source. Parallelism exhibits in the form of a three-stage pipeline that transforms the input string to a final pair type.</p><section id="FormulateTheTextProcessingPipelineProblem"><h2><a href="#FormulateTheTextProcessingPipelineProblem">Formulate the Text Processing Pipeline Problem</a></h2><p>Given an input vector of strings, we want to compute the most frequent character for each string using a series of transform operations. For example:</p><pre class="m-console"><span class="gp"># </span>input strings
<span class="go">abade</span>
<span class="go">ddddf</span>
<span class="go">eefge</span>
<span class="go">xyzzd</span>
<span class="go">ijjjj</span>
<span class="go">jiiii</span>
<span class="go">kkijk</span>

<span class="gp"># </span>output
<span class="go">a:2</span>
<span class="go">d:4</span>
<span class="go">e:3</span>
<span class="go">z:2</span>
<span class="go">j:4</span>
<span class="go">i:4</span>
<span class="go">k:3</span></pre><p>We decompose the algorithm into three stages:</p><ol><li>read a <code>std::string</code> from the input vector</li><li>generate a <code>std::unorder_map&lt;char, size_t&gt;</code> frequency map from the string</li><li>reduce the most frequent character to a <code>std::pair&lt;char, size_t&gt;</code> from the map</li></ol><p>The first and the third stages process inputs and generate results in serial, and the second stage can run in parallel. The algorithm is a perfect fit to pipeline parallelism, as different stages can overlap with each other in time across parallel lines.</p></section><section id="CreateAParallelTextPipeline"><h2><a href="#CreateAParallelTextPipeline">Create a Text Processing Pipeline</a></h2><p>We create a pipeline of three pipes (stages) and two parallel lines to solve the problem. The number of parallel lines is a tunable parameter. In most cases, we can just use <code><a href="http://en.cppreference.com/w/cpp/thread/thread/hardware_concurrency.html" class="m-doc-external">std::<wbr />thread::<wbr />hardware_concurrency</a></code> as the line count. The first pipe reads an input string from the vector in order, the second pipe transforms the input string from the first pipe to a frequency map in parallel, and the third pipe reduces the frequency map to find the most frequent character. The overall implementation is shown below:</p><pre class="m-code"><span class="cp">#include</span><span class="w"> </span><span class="cpf">&lt;taskflow/taskflow.hpp&gt;</span><span class="cp"></span>
<span class="cp">#include</span><span class="w"> </span><span class="cpf">&lt;taskflow/algorithm/pipeline.hpp&gt;</span><span class="cp"></span>

<span class="c1">// Function: format the map</span>
<span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="w"> </span><span class="nf">format_map</span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">unordered_map</span><span class="o">&lt;</span><span class="kt">char</span><span class="p">,</span><span class="w"> </span><span class="kt">size_t</span><span class="o">&gt;&amp;</span><span class="w"> </span><span class="n">map</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">ostringstream</span><span class="w"> </span><span class="n">oss</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="k">const</span><span class="w"> </span><span class="k">auto</span><span class="o">&amp;</span><span class="w"> </span><span class="p">[</span><span class="n">i</span><span class="p">,</span><span class="w"> </span><span class="n">j</span><span class="p">]</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">map</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">oss</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="n">i</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="sc">&#39;:&#39;</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="n">j</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="sc">&#39; &#39;</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">oss</span><span class="p">.</span><span class="n">str</span><span class="p">();</span><span class="w"></span>
<span class="p">}</span><span class="w"></span>

<span class="kt">int</span><span class="w"> </span><span class="nf">main</span><span class="p">()</span><span class="w"> </span><span class="p">{</span><span class="w"></span>

<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Taskflow</span><span class="w"> </span><span class="n">taskflow</span><span class="p">(</span><span class="s">&quot;text-filter pipeline&quot;</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Executor</span><span class="w"> </span><span class="n">executor</span><span class="p">;</span><span class="w"></span>

<span class="w"> </span><span class="k">const</span><span class="w"> </span><span class="kt">size_t</span><span class="w"> </span><span class="n">num_lines</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="mi">2</span><span class="p">;</span><span class="w"></span>

<span class="w"> </span><span class="c1">// input data </span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">vector</span><span class="o">&lt;</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&gt;</span><span class="w"> </span><span class="n">input</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="s">&quot;abade&quot;</span><span class="p">,</span><span class="w"> </span>
<span class="w"> </span><span class="s">&quot;ddddf&quot;</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">&quot;eefge&quot;</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">&quot;xyzzd&quot;</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">&quot;ijjjj&quot;</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">&quot;jiiii&quot;</span><span class="p">,</span><span class="w"></span>
<span class="w"> </span><span class="s">&quot;kkijk&quot;</span><span class="w"></span>
<span class="w"> </span><span class="p">};</span><span class="w"></span>

<span class="w"> </span><span class="c1">// custom data storage</span>
<span class="w"> </span><span class="k">using</span><span class="w"> </span><span class="n">data_type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">variant</span><span class="o">&lt;</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">unordered_map</span><span class="o">&lt;</span><span class="kt">char</span><span class="p">,</span><span class="w"> </span><span class="kt">size_t</span><span class="o">&gt;</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">pair</span><span class="o">&lt;</span><span class="kt">char</span><span class="p">,</span><span class="w"> </span><span class="kt">size_t</span><span class="o">&gt;</span><span class="w"></span>
<span class="w"> </span><span class="o">&gt;</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">array</span><span class="o">&lt;</span><span class="n">data_type</span><span class="p">,</span><span class="w"> </span><span class="n">num_lines</span><span class="o">&gt;</span><span class="w"> </span><span class="n">mybuffer</span><span class="p">;</span><span class="w"></span>

<span class="w"> </span><span class="c1">// the pipeline consists of three pipes (serial-parallel-serial)</span>
<span class="w"> </span><span class="c1">// and up to two concurrent scheduling tokens</span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeline</span><span class="w"> </span><span class="n">pl</span><span class="p">(</span><span class="n">num_lines</span><span class="p">,</span><span class="w"></span>

<span class="w"> </span><span class="c1">// first pipe processes the input data</span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Pipe</span><span class="p">{</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="o">&amp;</span><span class="p">](</span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeflow</span><span class="o">&amp;</span><span class="w"> </span><span class="n">pf</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">()</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">input</span><span class="p">.</span><span class="n">size</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">pf</span><span class="p">.</span><span class="n">stop</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">&quot;stage 1: input token = %s</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">input</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">()].</span><span class="n">c_str</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">mybuffer</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">()]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">input</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">()];</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="p">}},</span><span class="w"></span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// second pipe counts the frequency of each character</span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Pipe</span><span class="p">{</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">PARALLEL</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="o">&amp;</span><span class="p">](</span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeflow</span><span class="o">&amp;</span><span class="w"> </span><span class="n">pf</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">unordered_map</span><span class="o">&lt;</span><span class="kt">char</span><span class="p">,</span><span class="w"> </span><span class="kt">size_t</span><span class="o">&gt;</span><span class="w"> </span><span class="n">map</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="k">auto</span><span class="w"> </span><span class="n">c</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">get</span><span class="o">&lt;</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&gt;</span><span class="p">(</span><span class="n">mybuffer</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">()]))</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">map</span><span class="p">[</span><span class="n">c</span><span class="p">]</span><span class="o">++</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">&quot;stage 2: map = %s</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">format_map</span><span class="p">(</span><span class="n">map</span><span class="p">).</span><span class="n">c_str</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="n">mybuffer</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">()]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}},</span><span class="w"></span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// third pipe reduces the most frequent character</span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Pipe</span><span class="p">{</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="o">&amp;</span><span class="n">mybuffer</span><span class="p">](</span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeflow</span><span class="o">&amp;</span><span class="w"> </span><span class="n">pf</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">auto</span><span class="o">&amp;</span><span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">get</span><span class="o">&lt;</span><span class="n">std</span><span class="o">::</span><span class="n">unordered_map</span><span class="o">&lt;</span><span class="kt">char</span><span class="p">,</span><span class="w"> </span><span class="kt">size_t</span><span class="o">&gt;&gt;</span><span class="p">(</span><span class="n">mybuffer</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">()]);</span><span class="w"></span>
<span class="w"> </span><span class="k">auto</span><span class="w"> </span><span class="n">sol</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">max_element</span><span class="p">(</span><span class="n">map</span><span class="p">.</span><span class="n">begin</span><span class="p">(),</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="n">end</span><span class="p">(),</span><span class="w"> </span><span class="p">[](</span><span class="k">auto</span><span class="o">&amp;</span><span class="w"> </span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="k">auto</span><span class="o">&amp;</span><span class="w"> </span><span class="n">b</span><span class="p">){</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">second</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">b</span><span class="p">.</span><span class="n">second</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">&quot;stage 3: %c:%zu</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">sol</span><span class="o">-&gt;</span><span class="n">first</span><span class="p">,</span><span class="w"> </span><span class="n">sol</span><span class="o">-&gt;</span><span class="n">second</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="c1">// not necessary to store the last-stage data, just for demo purpose</span>
<span class="w"> </span><span class="n">mybuffer</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">()]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="o">*</span><span class="n">sol</span><span class="p">;</span><span class="w"> </span>
<span class="w"> </span><span class="p">}}</span><span class="w"></span>
<span class="w"> </span><span class="p">);</span><span class="w"></span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// build the pipeline graph using composition</span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Task</span><span class="w"> </span><span class="n">init</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">taskflow</span><span class="p">.</span><span class="n">emplace</span><span class="p">([](){</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="s">&quot;ready</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">;</span><span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">name</span><span class="p">(</span><span class="s">&quot;starting pipeline&quot;</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Task</span><span class="w"> </span><span class="n">task</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">taskflow</span><span class="p">.</span><span class="n">composed_of</span><span class="p">(</span><span class="n">pl</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">name</span><span class="p">(</span><span class="s">&quot;pipeline&quot;</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">tf</span><span class="o">::</span><span class="n">Task</span><span class="w"> </span><span class="n">stop</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">taskflow</span><span class="p">.</span><span class="n">emplace</span><span class="p">([](){</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="s">&quot;stopped</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">;</span><span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">name</span><span class="p">(</span><span class="s">&quot;pipeline stopped&quot;</span><span class="p">);</span><span class="w"></span>

<span class="w"> </span><span class="c1">// create task dependency</span>
<span class="w"> </span><span class="n">init</span><span class="p">.</span><span class="n">precede</span><span class="p">(</span><span class="n">task</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span><span class="n">task</span><span class="p">.</span><span class="n">precede</span><span class="p">(</span><span class="n">stop</span><span class="p">);</span><span class="w"></span>
<span class="w"> </span>
<span class="w"> </span><span class="c1">// dump the pipeline graph structure (with composition)</span>
<span class="w"> </span><span class="n">taskflow</span><span class="p">.</span><span class="n">dump</span><span class="p">(</span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="p">);</span><span class="w"></span>

<span class="w"> </span><span class="c1">// run the pipeline</span>
<span class="w"> </span><span class="n">executor</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">taskflow</span><span class="p">).</span><span class="n">wait</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="mi">0</span><span class="p">;</span><span class="w"></span>
<span class="p">}</span><span class="w"></span></pre><section id="TextPipelineDefineTheDataBuffer"><h3><a href="#TextPipelineDefineTheDataBuffer">Define the Data Buffer</a></h3><p>Taskflow does not provide any data abstraction to perform pipeline scheduling, but give users full control over data management in their applications. In this example, we create an one-dimensional buffer of a <a href="https://en.cppreference.com/w/cpp/utility/variant">std::<wbr />variant</a> data type to store the output of each pipe in a uniform storage:</p><pre class="m-code"><span class="k">using</span><span class="w"> </span><span class="n">data_type</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">variant</span><span class="o">&lt;</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">unordered_map</span><span class="o">&lt;</span><span class="kt">char</span><span class="p">,</span><span class="w"> </span><span class="kt">size_t</span><span class="o">&gt;</span><span class="p">,</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">pair</span><span class="o">&lt;</span><span class="kt">char</span><span class="p">,</span><span class="w"> </span><span class="kt">size_t</span><span class="o">&gt;</span><span class="w"></span>
<span class="o">&gt;</span><span class="p">;</span><span class="w"></span>
<span class="n">std</span><span class="o">::</span><span class="n">array</span><span class="o">&lt;</span><span class="n">std</span><span class="o">::</span><span class="n">array</span><span class="o">&lt;</span><span class="n">data_type</span><span class="p">,</span><span class="w"> </span><span class="n">num_pipes</span><span class="o">&gt;</span><span class="p">,</span><span class="w"> </span><span class="n">num_lines</span><span class="o">&gt;</span><span class="w"> </span><span class="n">mybuffer</span><span class="p">;</span><span class="w"></span></pre><aside class="m-note m-info"><h4>Note</h4><p>One-dimensional buffer is sufficient because Taskflow enables only one scheduling token per line at a time.</p></aside></section><section id="TextPipelineDefineThePipes"><h3><a href="#TextPipelineDefineThePipes">Define the Pipes</a></h3><p>The first pipe reads one string and puts it in the corresponding entry at the buffer, <code>mybuffer[pf.line()]</code>. Since we read in each string in order, we declare the pipe as a serial type:</p><pre class="m-code"><span class="n">tf</span><span class="o">::</span><span class="n">Pipe</span><span class="p">{</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="o">&amp;</span><span class="p">](</span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeflow</span><span class="o">&amp;</span><span class="w"> </span><span class="n">pf</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">if</span><span class="p">(</span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">()</span><span class="w"> </span><span class="o">==</span><span class="w"> </span><span class="n">input</span><span class="p">.</span><span class="n">size</span><span class="p">())</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">pf</span><span class="p">.</span><span class="n">stop</span><span class="p">();</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="k">else</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">mybuffer</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">()]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">input</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">()];</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">&quot;stage 1: input token = %s</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">input</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">token</span><span class="p">()].</span><span class="n">c_str</span><span class="p">());</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="p">}},</span><span class="w"></span></pre><p>The second pipe needs to get the input string from the previous pipe and then transforms that input string into a frequency map that records the occurrence of each character in the string. As multiple transforms can operate simultaneously, we declare the pipe as a parallel type:</p><pre class="m-code"><span class="n">tf</span><span class="o">::</span><span class="n">Pipe</span><span class="p">{</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">PARALLEL</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="o">&amp;</span><span class="p">](</span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeflow</span><span class="o">&amp;</span><span class="w"> </span><span class="n">pf</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">unordered_map</span><span class="o">&lt;</span><span class="kt">char</span><span class="p">,</span><span class="w"> </span><span class="kt">size_t</span><span class="o">&gt;</span><span class="w"> </span><span class="n">map</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="k">for</span><span class="p">(</span><span class="k">auto</span><span class="w"> </span><span class="n">c</span><span class="w"> </span><span class="o">:</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">get</span><span class="o">&lt;</span><span class="n">std</span><span class="o">::</span><span class="n">string</span><span class="o">&gt;</span><span class="p">(</span><span class="n">mybuffer</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">()]))</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="n">map</span><span class="p">[</span><span class="n">c</span><span class="p">]</span><span class="o">++</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">}</span><span class="w"></span>
<span class="w"> </span><span class="n">mybuffer</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">()]</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">map</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">&quot;stage 2: map = %s</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">format_map</span><span class="p">(</span><span class="n">map</span><span class="p">).</span><span class="n">c_str</span><span class="p">());</span><span class="w"></span>
<span class="p">}}</span><span class="w"></span></pre><p>Similarly, the third pipe needs to get the input frequency map from the previous pipe and then reduces the result to find the most frequent character. We may not need to store the result in the buffer but other places defined by the application (e.g., an output file). As we want to output the result in the same order as the input, we declare the pipe as a serial type:</p><pre class="m-code"><span class="n">tf</span><span class="o">::</span><span class="n">Pipe</span><span class="p">{</span><span class="n">tf</span><span class="o">::</span><span class="n">PipeType</span><span class="o">::</span><span class="n">SERIAL</span><span class="p">,</span><span class="w"> </span><span class="p">[</span><span class="o">&amp;</span><span class="n">mybuffer</span><span class="p">](</span><span class="n">tf</span><span class="o">::</span><span class="n">Pipeflow</span><span class="o">&amp;</span><span class="w"> </span><span class="n">pf</span><span class="p">)</span><span class="w"> </span><span class="p">{</span><span class="w"></span>
<span class="w"> </span><span class="k">auto</span><span class="o">&amp;</span><span class="w"> </span><span class="n">map</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">get</span><span class="o">&lt;</span><span class="n">std</span><span class="o">::</span><span class="n">unordered_map</span><span class="o">&lt;</span><span class="kt">char</span><span class="p">,</span><span class="w"> </span><span class="kt">size_t</span><span class="o">&gt;&gt;</span><span class="p">(</span><span class="n">mybuffer</span><span class="p">[</span><span class="n">pf</span><span class="p">.</span><span class="n">line</span><span class="p">()]);</span><span class="w"></span>
<span class="w"> </span><span class="k">auto</span><span class="w"> </span><span class="n">sol</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">max_element</span><span class="p">(</span><span class="n">map</span><span class="p">.</span><span class="n">begin</span><span class="p">(),</span><span class="w"> </span><span class="n">map</span><span class="p">.</span><span class="n">end</span><span class="p">(),</span><span class="w"> </span><span class="p">[](</span><span class="k">auto</span><span class="o">&amp;</span><span class="w"> </span><span class="n">a</span><span class="p">,</span><span class="w"> </span><span class="k">auto</span><span class="o">&amp;</span><span class="w"> </span><span class="n">b</span><span class="p">){</span><span class="w"></span>
<span class="w"> </span><span class="k">return</span><span class="w"> </span><span class="n">a</span><span class="p">.</span><span class="n">second</span><span class="w"> </span><span class="o">&lt;</span><span class="w"> </span><span class="n">b</span><span class="p">.</span><span class="n">second</span><span class="p">;</span><span class="w"></span>
<span class="w"> </span><span class="p">});</span><span class="w"></span>
<span class="w"> </span><span class="n">printf</span><span class="p">(</span><span class="s">&quot;stage 3: %c:%zu</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">,</span><span class="w"> </span><span class="n">sol</span><span class="o">-&gt;</span><span class="n">first</span><span class="p">,</span><span class="w"> </span><span class="n">sol</span><span class="o">-&gt;</span><span class="n">second</span><span class="p">);</span><span class="w"></span>
<span class="p">}}</span><span class="w"></span></pre></section><section id="TextPipelineDefineTheTaskGraph"><h3><a href="#TextPipelineDefineTheTaskGraph">Define the Task Graph</a></h3><p>To build up the taskflow graph for the pipeline, we create a module task out of the pipeline structure and connect it with two tasks that outputs messages before and after the pipeline:</p><pre class="m-code"><span class="n">tf</span><span class="o">::</span><span class="n">Task</span><span class="w"> </span><span class="n">init</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">taskflow</span><span class="p">.</span><span class="n">emplace</span><span class="p">([](){</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="s">&quot;ready</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">;</span><span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">name</span><span class="p">(</span><span class="s">&quot;starting pipeline&quot;</span><span class="p">);</span><span class="w"></span>
<span class="n">tf</span><span class="o">::</span><span class="n">Task</span><span class="w"> </span><span class="n">task</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">taskflow</span><span class="p">.</span><span class="n">composed_of</span><span class="p">(</span><span class="n">pl</span><span class="p">)</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">name</span><span class="p">(</span><span class="s">&quot;pipeline&quot;</span><span class="p">);</span><span class="w"></span>
<span class="n">tf</span><span class="o">::</span><span class="n">Task</span><span class="w"> </span><span class="n">stop</span><span class="w"> </span><span class="o">=</span><span class="w"> </span><span class="n">taskflow</span><span class="p">.</span><span class="n">emplace</span><span class="p">([](){</span><span class="w"> </span><span class="n">std</span><span class="o">::</span><span class="n">cout</span><span class="w"> </span><span class="o">&lt;&lt;</span><span class="w"> </span><span class="s">&quot;stopped</span><span class="se">\n</span><span class="s">&quot;</span><span class="p">;</span><span class="w"> </span><span class="p">})</span><span class="w"></span>
<span class="w"> </span><span class="p">.</span><span class="n">name</span><span class="p">(</span><span class="s">&quot;pipeline stopped&quot;</span><span class="p">);</span><span class="w"></span>
<span class="n">init</span><span class="p">.</span><span class="n">precede</span><span class="p">(</span><span class="n">task</span><span class="p">);</span><span class="w"></span>
<span class="n">task</span><span class="p">.</span><span class="n">precede</span><span class="p">(</span><span class="n">stop</span><span class="p">);</span><span class="w"></span></pre></section><section id="TextPipelineSubmitTheTaskGraph"><h3><a href="#TextPipelineSubmitTheTaskGraph">Submit the Task Graph</a></h3><p>Finally, we submit the taskflow to the execution and run it once:</p><pre class="m-code"><span class="n">executor</span><span class="p">.</span><span class="n">run</span><span class="p">(</span><span class="n">taskflow</span><span class="p">).</span><span class="n">wait</span><span class="p">();</span><span class="w"></span></pre><p>As the second stage is a parallel pipe, the output may interleave. One possible result is shown below:</p><pre class="m-console"><span class="go">ready</span>
<span class="go">stage 1: input token = abade</span>
<span class="go">stage 1: input token = ddddf</span>
<span class="go">stage 2: map = f:1 d:4 </span>
<span class="go">stage 2: map = e:1 d:1 a:2 b:1 </span>
<span class="go">stage 3: a:2</span>
<span class="go">stage 1: input token = eefge</span>
<span class="go">stage 2: map = g:1 e:3 f:1 </span>
<span class="go">stage 3: d:4</span>
<span class="go">stage 1: input token = xyzzd</span>
<span class="go">stage 3: e:3</span>
<span class="go">stage 1: input token = ijjjj</span>
<span class="go">stage 2: map = z:2 x:1 d:1 y:1 </span>
<span class="go">stage 3: z:2</span>
<span class="go">stage 1: input token = jiiii</span>
<span class="go">stage 2: map = j:4 i:1 </span>
<span class="go">stage 3: j:4</span>
<span class="go">stage 2: map = i:4 j:1 </span>
<span class="go">stage 1: input token = kkijk</span>
<span class="go">stage 3: i:4</span>
<span class="go">stage 2: map = j:1 k:3 i:1 </span>
<span class="go">stage 3: k:3</span>
<span class="go">stopped</span></pre><p>We can see seven outputs at the third stage that show the most frequent character for each of the seven strings in order (<code>a:2</code>, <code>d:4</code>, <code>e:3</code>, <code>z:2</code>, <code>j:4</code>, <code>i:4</code>, <code>k:3</code>). The taskflow graph of this pipeline workload is shown below:</p><div class="m-graph"><svg style="width: 30.800rem; height: 32.700rem;" viewBox="0.00 0.00 308.00 327.00">
<g transform="scale(1 1) rotate(0) translate(4 323)">
<title>Taskflow</title>
<g class="m-cluster">
<title>cluster_p0x7ffd7418c200</title>
<polygon points="8,-8 8,-311 142,-311 142,-8 8,-8"/>
<text text-anchor="middle" x="75" y="-299" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">Text Processing Pipeline</text>
</g>
<g class="m-cluster">
<title>cluster_p0x7ffd7418c110</title>
<polygon points="150,-119 150,-311 292,-311 292,-119 150,-119"/>
<text text-anchor="middle" x="221" y="-299" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">m1</text>
</g>
<g class="m-node m-flat">
<title>p0x7bc4000142e8</title>
<ellipse cx="75" cy="-266" rx="56.5233" ry="18"/>
<text text-anchor="middle" x="75" y="-263.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">starting pipeline</text>
</g>
<g class="m-node m-flat">
<title>p0x7bc4000143d0</title>
<polygon points="116,-163 38,-163 34,-159 34,-127 112,-127 116,-131 116,-163"/>
<polyline points="112,-159 34,-159 "/>
<polyline points="112,-159 112,-127 "/>
<polyline points="112,-159 116,-163 "/>
<text text-anchor="middle" x="75" y="-142.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">pipeline [m1]</text>
</g>
<g class="m-edge">
<title>p0x7bc4000142e8&#45;&gt;p0x7bc4000143d0</title>
<path d="M75,-247.8851C75,-228.1928 75,-196.4608 75,-173.439"/>
<polygon points="78.5001,-173.2387 75,-163.2388 71.5001,-173.2388 78.5001,-173.2387"/>
</g>
<g class="m-node m-flat">
<title>p0x7bc4000144b8</title>
<ellipse cx="75" cy="-34" rx="57.8786" ry="18"/>
<text text-anchor="middle" x="75" y="-31.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">pipeline stopped</text>
</g>
<g class="m-edge">
<title>p0x7bc4000143d0&#45;&gt;p0x7bc4000144b8</title>
<path d="M75,-126.6706C75,-109.2373 75,-82.7482 75,-62.5489"/>
<polygon points="78.5001,-62.3566 75,-52.3566 71.5001,-62.3567 78.5001,-62.3566"/>
</g>
<g class="m-node">
<title>p0x7bc400014030</title>
<polygon points="218,-284 184.8992,-266 218,-248 251.1008,-266 218,-284"/>
<text text-anchor="middle" x="218" y="-263.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">cond</text>
</g>
<g class="m-node m-flat">
<title>p0x7bc400014118</title>
<polygon points="212,-163 158,-163 158,-159 154,-159 154,-155 158,-155 158,-135 154,-135 154,-131 158,-131 158,-127 212,-127 212,-163"/>
<polyline points="158,-159 162,-159 162,-155 158,-155 "/>
<polyline points="158,-135 162,-135 162,-131 158,-131 "/>
<text text-anchor="middle" x="185" y="-142.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">rt&#45;0</text>
</g>
<g class="m-edge">
<title>p0x7bc400014030&#45;&gt;p0x7bc400014118</title>
<path stroke-dasharray="5,2" d="M213.6858,-250.1814C208.3593,-230.6507 199.2047,-197.0838 192.6698,-173.1225"/>
<polygon points="195.9696,-171.9194 189.9616,-163.1927 189.2162,-173.7612 195.9696,-171.9194"/>
<text text-anchor="middle" x="205.5" y="-203" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">0</text>
</g>
<g class="m-node m-flat">
<title>p0x7bc400014200</title>
<polygon points="284,-163 230,-163 230,-159 226,-159 226,-155 230,-155 230,-135 226,-135 226,-131 230,-131 230,-127 284,-127 284,-163"/>
<polyline points="230,-159 234,-159 234,-155 230,-155 "/>
<polyline points="230,-135 234,-135 234,-131 230,-131 "/>
<text text-anchor="middle" x="257" y="-142.5" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">rt&#45;1</text>
</g>
<g class="m-edge">
<title>p0x7bc400014030&#45;&gt;p0x7bc400014200</title>
<path stroke-dasharray="5,2" d="M222.9394,-250.6753C229.2165,-231.2 240.1705,-197.2146 247.9606,-173.0452"/>
<polygon points="251.3461,-173.9504 251.0827,-163.3588 244.6837,-171.8029 251.3461,-173.9504"/>
<text text-anchor="middle" x="241.5" y="-203" font-family="Helvetica,sans-Serif" font-size="10.00" fill="#000000">1</text>
</g>
</g>
</svg>
</div></section></section>
 </div>
 </div>
 </div>
</article></main>
<div class="m-doc-search" id="search">
 <a href="#!" onclick="return hideSearch()"></a>
 <div class="m-container">
 <div class="m-row">
 <div class="m-col-m-8 m-push-m-2">
 <div class="m-doc-search-header m-text m-small">
 <div><span class="m-label m-default">Tab</span> / <span class="m-label m-default">T</span> to search, <span class="m-label m-default">Esc</span> to close</div>
 <div id="search-symbolcount">&hellip;</div>
 </div>
 <div class="m-doc-search-content">
 <form>
 <input type="search" name="q" id="search-input" placeholder="Loading &hellip;" disabled="disabled" autofocus="autofocus" autocomplete="off" spellcheck="false" />
 </form>
 <noscript class="m-text m-danger m-text-center">Unlike everything else in the docs, the search functionality <em>requires</em> JavaScript.</noscript>
 <div id="search-help" class="m-text m-dim m-text-center">
 <p class="m-noindent">Search for symbols, directories, files, pages or
 modules. You can omit any prefix from the symbol or file path; adding a
 <code>:</code> or <code>/</code> suffix lists all members of given symbol or
 directory.</p>
 <p class="m-noindent">Use <span class="m-label m-dim">&darr;</span>
 / <span class="m-label m-dim">&uarr;</span> to navigate through the list,
 <span class="m-label m-dim">Enter</span> to go.
 <span class="m-label m-dim">Tab</span> autocompletes common prefix, you can
 copy a link to the result using <span class="m-label m-dim">⌘</span>
 <span class="m-label m-dim">L</span> while <span class="m-label m-dim">⌘</span>
 <span class="m-label m-dim">M</span> produces a Markdown link.</p>
 </div>
 <div id="search-notfound" class="m-text m-warning m-text-center">Sorry, nothing was found.</div>
 <ul id="search-results"></ul>
 </div>
 </div>
 </div>
 </div>
</div>
<script src="search-v2.js"></script>
<script src="searchdata-v2.js" async="async"></script>
<footer><nav>
 <div class="m-container">
 <div class="m-row">
 <div class="m-col-l-10 m-push-l-1">
 <p>Taskflow handbook is part of the <a href="https://taskflow.github.io">Taskflow project</a>, copyright © <a href="https://tsung-wei-huang.github.io/">Dr. Tsung-Wei Huang</a>, 2018&ndash;2023.<br />Generated by <a href="https://doxygen.org/">Doxygen</a> 1.8.14 and <a href="https://mcss.mosra.cz/">m.css</a>.</p>
 </div>
 </div>
 </div>
</nav></footer>
</body>
</html>