std::string::length vs strlen using Google Benchmark

Question

I wrote my first benchmark using Google Benchmark to check what is faster between the use of std::string::length and strlen to compute the length of a std::string.

I would like to know what I can improve in my code to use it as a base for future benchmarks. My main concerns are about:

• the creation of the std::string at the beginning of my functions: is it the right place to do it? Should I use fixtures? Or create a global std::vector<std::string> that contains every strings I'll use for my benchmark?
• the use of benchmark::DoNotOptimize and benchmark::ClobberMemory: I'm not sure I use these functions as I should. The idea is the following: I want to be sure the std::string s is created and stored into memory before entering in my loop and I want to be sure the functions std::string::length and strlen are really called and not optimized. Is it the good way to do it?
• the use of DenseRange to create a benchmark with multiple length of string. At first, I wanted to pass an std::string directly to my function but, apparently, only int can be returned by state.range. Is there another way to pass a string to my functions?

Here is my code:

#include <benchmark/benchmark.h>
#include <string>
#include <cstring>
static void BM_InneficientBasicStringLength(benchmark::State& state)
{
 std::string s;
 for (int i = 0; i < state.range(0); i++)
 s += 'a';
 benchmark::DoNotOptimize(s.data());
 benchmark::ClobberMemory();
 size_t size;
 for (auto _ : state)
 {
 benchmark::DoNotOptimize(size = strlen(s.c_str()));
 benchmark::ClobberMemory();
 }
}
BENCHMARK(BM_InneficientBasicStringLength)->DenseRange(0, 100, 10);
static void BM_InneficientBasicStringLength_Fix(benchmark::State& state)
{
 std::string s;
 for (int i = 0; i < state.range(0); i++)
 s += 'a';
 benchmark::DoNotOptimize(s.data());
 benchmark::ClobberMemory();
 size_t size;
 for (auto _ : state)
 {
 benchmark::DoNotOptimize(size = s.length());
 benchmark::ClobberMemory();
 }
}
BENCHMARK(BM_InneficientBasicStringLength_Fix)->DenseRange(0, 100, 10);
BENCHMARK_MAIN();

It compiles with: g++ -o example example.cpp -lbenchmark -lpthread.

---

EDIT: For your information, the results are better with std::string::length which have a constant complexity while strlen have a linear complexity:

> ./example 
2020年08月14日T14:48:17+02:00
Running ./example
Run on (12 X 3900 MHz CPU s)
CPU Caches:
 L1 Data 32 KiB (x6)
 L1 Instruction 32 KiB (x6)
 L2 Unified 1024 KiB (x6)
 L3 Unified 8448 KiB (x1)
Load Average: 5.18, 6.05, 6.32
----------------------------------------------------------------------------------
Benchmark Time CPU Iterations
----------------------------------------------------------------------------------
BM_InneficientBasicStringLength/0 4.79 ns 4.79 ns 150060775
BM_InneficientBasicStringLength/10 4.66 ns 4.66 ns 151148682
BM_InneficientBasicStringLength/20 5.39 ns 5.39 ns 136288403
BM_InneficientBasicStringLength/30 5.36 ns 5.35 ns 135353692
BM_InneficientBasicStringLength/40 5.19 ns 5.18 ns 137059957
BM_InneficientBasicStringLength/50 5.22 ns 5.22 ns 134737400
BM_InneficientBasicStringLength/60 5.32 ns 5.32 ns 134098029
BM_InneficientBasicStringLength/70 6.65 ns 6.65 ns 107061965
BM_InneficientBasicStringLength/80 6.85 ns 6.85 ns 109856173
BM_InneficientBasicStringLength/90 6.78 ns 6.78 ns 106762788
BM_InneficientBasicStringLength/100 6.66 ns 6.66 ns 98081332
BM_InneficientBasicStringLength_Fix/0 3.01 ns 3.01 ns 231875548
BM_InneficientBasicStringLength_Fix/10 3.06 ns 3.06 ns 221447369
BM_InneficientBasicStringLength_Fix/20 2.96 ns 2.96 ns 237989703
BM_InneficientBasicStringLength_Fix/30 2.98 ns 2.98 ns 234755616
BM_InneficientBasicStringLength_Fix/40 2.99 ns 2.98 ns 231015140
BM_InneficientBasicStringLength_Fix/50 2.94 ns 2.94 ns 223906062
BM_InneficientBasicStringLength_Fix/60 3.04 ns 3.04 ns 225199556
BM_InneficientBasicStringLength_Fix/70 3.07 ns 3.07 ns 230208201
BM_InneficientBasicStringLength_Fix/80 3.02 ns 3.02 ns 219373040
BM_InneficientBasicStringLength_Fix/90 3.10 ns 3.10 ns 219921248
BM_InneficientBasicStringLength_Fix/100 3.10 ns 3.10 ns 238076969

• pacmaninbw: What did you find, could you post your results? I would expect string.length() to be much faster since it may only need to access an internal count that is made while the string is created.
• Loki Astari: Are you trying to imitate perl? for (auto _ : state). Don't be obtuse in your code it is supposed to be human readable.
• Pierre: @MartinYork Almost all of the examples given on the GitHub of the project follow this pattern: github.com/google/benchmark#usage Whatever, what alternatives could I use?
• G. Sliepen: Benchmarking calculating the string length is not representative of real world code. Compilers will be able to optimize this if they know anything about the string. See this example. I also wouldn't trust any results that only take a few nanoseconds, it's likely that despite Google Benchmark's best efforts to avoid that, these low numbers will include overhead from benchmarking itself.
• Pierre: @G.Sliepen One of my concern is to avoid optimizations of the compiler. That's why I use benchmark::DoNotOptimize and benchmark::ClobberMemory. However, this is one of my questions: am I using this functions correctly in order to have a correct measure of strlen and std::string::length. About the duration of my benchmark, that's all the problem with microbenchmarking. The idea is to take a very little piece of code outside of its context to measure its efficiency. To prevent any issue about the reproducibility, we can see each measures are performed at least 98081332 times.

Answer 1

This is a long-winded way to create a string:

 std::string s;
 
 for (int i = 0; i < state.range(0); i++)
 s += 'a';

It would be much simpler to write

 auto const s = std::string(state.range(0), 'a');

(perhaps with a cast if state.range() doesn't return a std::size_t).

---

std::size_t and std::strlen are both misspelt. Although a standard library implementation is allowed to also define these identifiers in the global namespace, that's not required and should not be relied on in a portable program.

---

The two functions are very similar. Consider using a template so that the common parts are only written once.

---

The word "inefficient" is misspelt in identifiers. Although this might seem a petty observation, good spelling can determine whether it's possible to find code in large programs.