title,body,link,tags,view_count,is_answered,score,answer_count
How to use golang Benchmark RunParallel?,"<p>this is my code</p>
<pre class=""lang-golang prettyprint-override""><code>func BenchmarkParallel(b *testing.B) {
    var mu sync.Mutex
    var cnt int
    b.ResetTimer()
    b.SetParallelism(100)

    fmt.Println(&quot;cnt&quot;, cnt)
    b.RunParallel(func(pb *testing.PB) {
        mu.Lock()
        defer mu.Unlock()
        cnt++
        for pb.Next() {
            time.Sleep(time.Nanosecond)
            // time.Sleep(time.Second)
        }
    })
    fmt.Println(&quot;cnt&quot;, cnt)
}
</code></pre>
<p><code>$ go test -bench BenchmarkParallel -benchmem -test.count=1</code></p>
<p>Received the following output below</p>
<pre class=""lang-bash prettyprint-override""><code>cnt 0
cnt 800
BenchmarkParallel-8     cnt 0
cnt 800
cnt 0
cnt 800
cnt 0
cnt 800
cnt 0
cnt 800
 3987668               320.3 ns/op             0 B/op          0 allocs/op
PASS
</code></pre>
<p>Why does fmt.Println execute so many times,<br />
but if sleep is changed to 1 second, cnt is only output once?</p>
<p>I expected to see fmt.Println only once</p>
<pre class=""lang-bash prettyprint-override""><code>BenchmarkParallel
cnt 0
cnt 800
BenchmarkParallel-8            1    1001106668 ns/op
PASS
</code></pre>
",https://stackoverflow.com/questions/75545296/how-to-use-golang-benchmark-runparallel,"['go', 'testing']",161,TRUE,0,1
golang benchmark set testing flags in code,"<p>I have a preexisting custom golang benchmark program; It runs benchmraks using <a href=""https://pkg.go.dev/testing#Benchmark"" rel=""nofollow noreferrer"">testing.Benchmark</a> on a collection of functions.</p>
<p>This program uses <a href=""https://github.com/spf13/cobra"" rel=""nofollow noreferrer"">cobra</a> for input parameters.</p>
<p>How do I use <a href=""https://pkg.go.dev/cmd/go#hdr-Testing_flags"" rel=""nofollow noreferrer"">testing flags</a> in the benchmarks?</p>
<p>I cannot see a way to set them in the code explicitly, and I can't get command line options to work.</p>
",https://stackoverflow.com/questions/74981736/golang-benchmark-set-testing-flags-in-code,"['go', 'testing', 'benchmarking', 'flags']",79,FALSE,1,1
Golang benchmark setup database,"<p>I have to write some benchmarks that require a specific database setup. Something like this:</p>
<pre class=""lang-golang prettyprint-override""><code>func BenchmarkXxx(b *testing.B) {
  fmt.Println(&quot;Setup&quot;)
  dropRecords()
  createDatabaseRecords() // this require a lot of time

  fmt.Println(&quot;Start Test&quot;)
  b.ResetTimer()
  for i := 0; i &lt; b.N; i++ {
    fmt.Println(&quot;Loop&quot;)
    TestMyStuffs()
  }
}
</code></pre>
<p>Running this benchmark I can see in the console that the &quot;Setup&quot; and &quot;Start Test&quot; are printing many times, so the <code>BenchmarkXxx</code> function seems to be called many times. Is there a way to run a setup code (<code>createDatabaseRecords</code> in this example) only one time and only for a specific benchmark?</p>
<p>Is there any sort of &quot;best practice&quot; to do this?</p>
",https://stackoverflow.com/questions/73782483/golang-benchmark-setup-database,"['database', 'go', 'testing', 'benchmarking']",275,TRUE,0,1
How to quickly debug Flutter benchmark test (performance profiling) code?,"<p>I am writing some performance profiling (benchmark) test code as suggested here: <a href=""https://docs.flutter.dev/cookbook/testing/integration/profiling"" rel=""nofollow noreferrer"">https://docs.flutter.dev/cookbook/testing/integration/profiling</a>. However, the documentation suggests I should run it like:</p>
<pre><code>flutter drive --driver=test_driver/perf_driver.dart --target=integration_test/scrolling_test.dart --profile
</code></pre>
<p>As we know, that will need even minutes to run once even if only changed one line of code, because it goes through the whole Android/iOS compilation. That takes a lot of time when debugging the correctness of my test code.</p>
<p>On the other hand, we know Flutter has powerful hot reload/restart which only takes seconds to reload code changes. Can we utilize that to easily debug benchmark test code?</p>
",https://stackoverflow.com/questions/72822705/how-to-quickly-debug-flutter-benchmark-test-performance-profiling-code,"['flutter', 'dart', 'testing']",133,FALSE,0,1
"Flutter benchmark/profiling: Difference between benchmarkWidgets and traceAction, and which one to use?","<p>I am benchmarking Flutter apps but is confused with the two APIs.</p>
<ul>
<li><code>traceAction</code>: Mentioned in <a href=""https://docs.flutter.dev/cookbook/testing/integration/profiling"" rel=""nofollow noreferrer"">official tutorial</a>.</li>
<li><code>benchmarkWidgets</code>: Official doc <a href=""https://api.flutter.dev/flutter/flutter_test/benchmarkWidgets.html"" rel=""nofollow noreferrer"">here</a>, also mentioned in <a href=""https://medium.com/flutter/performance-testing-of-flutter-apps-df7669bb7df7"" rel=""nofollow noreferrer"">some other tutorials</a>.</li>
</ul>
<p>Therefore, I wonder which one should I use? Thanks for any suggestions!</p>
",https://stackoverflow.com/questions/72590472/flutter-benchmark-profiling-difference-between-benchmarkwidgets-and-traceaction,"['flutter', 'dart', 'testing', 'benchmarking']",71,FALSE,0,0
JMH Benchmark first approach,"<p>I am approaching for the first time the benchmark tests for my thesis work and I'm studying JMH.
I'm using the <a href=""https://github.com/spring-petclinic/spring-petclinic-rest"" rel=""nofollow noreferrer"">https://github.com/spring-petclinic/spring-petclinic-rest</a> code to do some tests.
After initializing JMH I obtained the class:</p>
<pre><code>import org.openjdk.jmh.annotations.Benchmark;

public class MyBenchmark {

    @Benchmark
    public void testMethod() {
        // This is a demo/sample template for building your JMH benchmarks. Edit as needed.
        // Put your benchmark code here.

    }

}
</code></pre>
<p>If I run <code>mvn clean package</code> it generates the <code>target/spring-petclinic-rest-2.6.2-perf-tests.jar</code> file and if I do <code>java -jar target/spring-petclinic-rest-2.6.2-perf-tests.jar</code> I obtain some results.</p>
<p>My question is, with this simple <em>MyBenchmark</em> class with the empty <em>testMethod</em> method, what is this benchmark test applied to? What code did he test?</p>
<p>Thank you in advance.</p>
",https://stackoverflow.com/questions/72275708/jmh-benchmark-first-approach,"['java', 'testing', 'benchmarking', 'jmh']",224,TRUE,1,1
Benchmark one specific `Test.Tasty.Bench` benchmark in a module using `cabal bench`,"<p>I have a bunch of benchmarks and I want to run a specific benchmark among them.</p>
<p>I had asked a previous <a href=""https://stackoverflow.com/a/72237547/7588896"">similar question for tests</a>. But the same technique does not work for benchmarks. Here are somethings I tried, where &quot;benchmarkGroupName/benchmName&quot; is the .</p>
<pre class=""lang-sh prettyprint-override""><code>cabal test Module.Name.Here --benchmark-options=&quot;--match=\&quot;&lt;match-string&gt;\&quot;&quot;
cabal test Module.Name.Here --match=&lt;match-string&gt;
cabal test Module.Name.Here --benchmark-options=&quot;--pattern=\&quot;&lt;match-string&gt;\&quot;&quot;
</code></pre>
<p>It says something like</p>
<pre class=""lang-sh prettyprint-override""><code>Invalid option `--match'

Usage: Module.Name.Here [-p|--pattern PATTERN] [-t|--timeout DURATION] 
                           [--stream-size ARG] [-l|--list-tests] 
                           [-j|--num-threads NUMBER] [-q|--quiet] 
                           [--hide-successes] [--color never|always|auto] 
                           [--ansi-tricks ARG] [--baseline ARG] [--csv ARG] 
                           [--svg ARG] [--stdev ARG] [--fail-if-slower ARG] 
                           [--fail-if-faster ARG]
</code></pre>
<p>Nothing works I've also looked at the <a href=""https://cabal.readthedocs.io/en/3.4/setup-commands.html?highlight=benchmark-options#cmdoption-runhaskell-Setup.hs-test-benchmark-options"" rel=""nofollow noreferrer"">documentation</a> but it's not helpful.</p>
",https://stackoverflow.com/questions/72318883/benchmark-one-specific-test-tasty-bench-benchmark-in-a-module-using-cabal-ben,"['haskell', 'testing', 'benchmarking', 'cabal']",183,FALSE,0,0
How to run all Golang benchmark tests including subfolders,"<p>I have a Go application with a number of unit and benchmark tests both in the root and in a subfolder called &quot;message&quot;.</p>
<p>I execute the following command to run all unit tests from the root including the ones in the messages and any other subfolder:</p>
<pre><code>go test ./...
</code></pre>
<p>I want to achieve the same for the benchmark tests, i.e. run them all. The following works for the ones in the root directory:</p>
<pre><code>go test -bench .
</code></pre>
<p>The benchmark tests in the /messages folder are ignored which is expected. So I run the following from the root:</p>
<pre><code>go test -bench ./...
</code></pre>
<p>That's not recognised at all, Go seems to execute the <strong>unit tests</strong> that are located in the root dir. I even tried to specify the message folder in the command as follows:</p>
<pre><code>go test -bench ./message
</code></pre>
<p>...but it also failed. Currently if I want to run the benchmark tests in the message folder I have to cd into that folder and execute</p>
<pre><code>go test -bench .
</code></pre>
<p>like above.</p>
<p>So what's the correct way then? How can I tell Go to find the benchmark tests both in the root and the subfolders? How does the regexp arg work in the case of the -bench flag? Apparently it's different from the regexp for the unit test runner.</p>
",https://stackoverflow.com/questions/70878439/how-to-run-all-golang-benchmark-tests-including-subfolders,"['unit-testing', 'go', 'testing', 'benchmarking']",1427,TRUE,3,2
How to benchmark Redis all operations at once,"<p>When i try to benchmark redis using default benchmark all the operations are executing one after the other is there any way to execute all the operation parallel?</p>
",https://stackoverflow.com/questions/39010434/how-to-benchmark-redis-all-operations-at-once,"['testing', 'redis', 'jmeter', 'performance-testing', 'benchmarking']",65,FALSE,-1,1
Why is go Benchmark function running more than once even though -benchtime=1x?,"<p>I have written a simply Go Benchmark case like this.
It gets a value from the DB, checks that it is zero, increments it and then writes it back to the DB.</p>
<pre><code>func BenchmarkMyStuff(b *testing.B) {
   require.Equal(b, 1, b.N)
   x := GetValueFromDB()
   require.Equal(b, 0, x)
   x++
   WriteValueToDB(x)
}
</code></pre>
<p>This benchmark will only pass when it is run once. That is because it updates the DB value and the check of <code>x == 0</code> will fail. To make sure this doesn't happen, I inserted a <code>require</code> statement at the first line of this function that ensures it will only run once.</p>
<p>And this is how I run this Benchmark:</p>
<pre><code>% go test -cpuprofile cpu.prof -memprofile mem.prof -run=XXX -benchtime=1x -bench BenchmarkMyStuff path/to/this/file
</code></pre>
<p>Notice I do <code>-benchtime=1x</code> to make sure it runs only once. Right? That's what the <a href=""https://pkg.go.dev/cmd/go#hdr-Description_of_testing_flags"" rel=""nofollow noreferrer"">manual says</a></p>
<p>Yet when I run this benchmark, it runs twice. Why?
The first time it passes <code>BenchmarkMyStuff-6</code> and then it fails <code>BenchmarkMyStuff-6</code>.</p>
",https://stackoverflow.com/questions/69257841/why-is-go-benchmark-function-running-more-than-once-even-though-benchtime-1x,"['go', 'testing', 'benchmarking']",782,TRUE,0,1
Android Benchmark module - How to test application,"<p>I have created a new benchmark module in my application to test out performance of my cide.</p>

<p>My Project Structure is like this</p>

<pre><code>Project_name
-app
-benchmark
</code></pre>

<p>app is Android application (com.android.application) while benchmark module is library (com.android.library).</p>

<p>I want to test my Java/Kotlin code as well as performance of my xml's inside my app module.
Now in benchmark samples <a href=""https://github.com/android/performance-samples"" rel=""nofollow noreferrer"">https://github.com/android/performance-samples</a> the module being benchmarked is a library module.</p>

<p>How can I refer my Java/Kotlin files and my xml's inside my benchmark module to test the performance.</p>

<p>Note: I tried putting benchmark tests in the app module but it does not calculate the speed of the methods/layout inflation if used outside benchmark module.</p>
",https://stackoverflow.com/questions/61508629/android-benchmark-module-how-to-test-application,"['android', 'performance', 'testing', 'kotlin', 'benchmarking']",741,TRUE,2,1
Go benchmark by function name,"<p>I have this Benchmark function:</p>

<pre><code>BenchmarkMyTest(b *testing.B) {
}
</code></pre>

<p>And I would like to run only this function not running all other tests, but the command never worked for me.</p>

<pre><code>go test -bench='BenchmarkMyTest'
or
go test -run='BenchmarkMyTest'
</code></pre>

<p>What's the correct way of running one single benchmark function in Go?
It says to use regex but I can't find any documentation.</p>

<p>Thanks,</p>
",https://stackoverflow.com/questions/33252269/go-benchmark-by-function-name,"['testing', 'go', 'benchmarking']",2260,TRUE,1,2
How can I use &quot;testing.Benchmark&quot; in a Benchmark function,"<p>my testing file</p>
<pre class=""lang-golang prettyprint-override""><code>func BenchmarkA(b *testing.B) {
    res := testing.Benchmark(BenchB)
    if res.N &lt;= 25{
        b.Fatalf(&quot;fail!&quot;)
    }
</code></pre>
<pre><code>func BenchB(b *testing.B) {
for i := 0; i &lt;= b.N; i++{
        url := &quot;****&quot;
        contentType := &quot;application/x-www-form-urlencoded&quot;
        data := strings.NewReader(&quot;****&quot;)
        _, err := http.Post(url, contentType, data)
        if err != nil {
            b.Fatal(&quot;fail &quot;)
        }
    }
}
</code></pre>
<p>when i run</p>
<pre><code>go test -bench=./router_test.go
</code></pre>
<p>I get a</p>
<pre><code>fatal error: all goroutines are asleep - deadlock!
</code></pre>
<p>if i use &quot;package main&quot; and &quot;func main{}&quot;, the program can run successfully.</p>
<p>what should i do next? (forgive my poor english)</p>
",https://stackoverflow.com/questions/63257550/how-can-i-use-testing-benchmark-in-a-benchmark-function,"['go', 'testing', 'benchmarking']",64,TRUE,-1,1
apache benchmark server performance test unreasonable result,"<p>I am testing the general performance of my home server using apache benchmark. However, when I test the server with 1 user and 10000 requests the reported requests per second is around 250 (on average) all the time but when I do the test with 10 users and 10000 requests then it reports that the requests per second is around 1134. I have tested these two scenarios multiple times and get this unreasonable result. What I am thinking is, why does it jump from approximately 200 req/sec to 1134 req/sec. Shouldn't the first test also give the same or close result?  Both tests send more than 1000 requests. </p>

<pre><code>ab -c 1 -n 1000 http://168.192.54.8/
This is ApacheBench, Version 2.3 &lt;$Revision: 1706008 $&gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 168.192.54.8 (be patient)
Completed 100 requests
Completed 200 requests
Completed 300 requests
Completed 400 requests
Completed 500 requests
Completed 600 requests
Completed 700 requests
Completed 800 requests
Completed 900 requests
Completed 1000 requests
Finished 1000 requests


Server Software:        Apache/2.4.29
Server Hostname:        168.192.54.8
Server Port:            80

Document Path:          /
Document Length:        9381 bytes

Concurrency Level:      1
Time taken for tests:   3.954 seconds
Complete requests:      1000
Failed requests:        0
Total transferred:      9654000 bytes
HTML transferred:       9381000 bytes
Requests per second:    252.92 [#/sec] (mean)
Time per request:       3.954 [ms] (mean)
Time per request:       3.954 [ms] (mean, across all concurrent requests)
Transfer rate:          2384.49 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.3      1       2
Processing:     2    3   0.6      3       7
Waiting:        1    2   0.5      2       5
Total:          2    4   0.7      4       9

Percentage of the requests served within a certain time (ms)
  50%      4
  66%      4
  75%      4
  80%      4
  90%      5
  95%      5
  98%      6
  99%      6
 100%      9 (longest request)
</code></pre>

<pre><code>ab -c 25 -n 10000 http://168.192.54.8/
This is ApacheBench, Version 2.3 &lt;$Revision: 1706008 $&gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 168.192.54.8 (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        Apache/2.4.29
Server Hostname:        168.192.54.8
Server Port:            80

Document Path:          /
Document Length:        9381 bytes

Concurrency Level:      25
Time taken for tests:   8.788 seconds
Complete requests:      10000
Failed requests:        0
Total transferred:      96540000 bytes
HTML transferred:       93810000 bytes
Requests per second:    1137.91 [#/sec] (mean)
Time per request:       21.970 [ms] (mean)
Time per request:       0.879 [ms] (mean, across all concurrent requests)
Transfer rate:          10727.90 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        1    4  14.7      3    1037
Processing:     3   18   4.9     19      44
Waiting:        2    6   2.9      5      30
Total:          5   22  15.4     23    1050

Percentage of the requests served within a certain time (ms)
  50%     23
  66%     24
  75%     24
  80%     25
  90%     27
  95%     29
  98%     32
  99%     35
 100%   1050 (longest request)

</code></pre>
",https://stackoverflow.com/questions/61108352/apache-benchmark-server-performance-test-unreasonable-result,"['testing', 'server', 'load', 'benchmarking', 'apachebench']",965,FALSE,0,0
Benchmark a function over different inputs (of different sizes) in python efficiently,"<p>I'm trying to benchmark a function over different inputs which have different sizes. </p>

<p>For example I have a function <code>something(array)</code> that takes a numpy array with variable length as an input and a list of possible arguments <code>args = [np.random.rand(3, 2, i) for i in range(1,1000)]</code>. 
What I want to do now is to test the time <code>something</code> needs dependent on the input. </p>

<p>What I currently did is the following:</p>

<pre><code>def iterator(ys):

    times = []
    for i in range(len(ys)):
        start = timer()
        test = something(ys[i])
        end = timer()
        times.append(end - start)

    return times
</code></pre>

<p>With this I get the results I want. The problem is that with this method I do not get any key figures like the variance or mean for each input size. 
I could also do a second loop where I run the <code>iterator</code> several times and use the results to compute the key figures but I'd rather use an established method if there is one.</p>

<p>I tried to apply <code>pytest-benchmark</code> (<a href=""https://pytest-benchmark.readthedocs.io/en/stable/"" rel=""nofollow noreferrer"">https://pytest-benchmark.readthedocs.io/en/stable/</a>) but was not able to find a way where I could pass a list of arguments to a function. </p>

<p>Maybe some of you have a nice solution to this!</p>
",https://stackoverflow.com/questions/59306904/benchmark-a-function-over-different-inputs-of-different-sizes-in-python-effici,"['python', 'testing', 'pytest', 'benchmarking']",930,TRUE,2,1
Is there an standard benchmark for webAssembly?,"<p>Is there any standard benchmark test for webAssembly or program langauage?</p>

<p>I want to migrate my c++ program to webAssembly ，so i must do some benchmark to test the preformence of webAssembly and compared it with c++ preformence.</p>
",https://stackoverflow.com/questions/57016422/is-there-an-standard-benchmark-for-webassembly,"['c++', 'testing', 'benchmarking', 'webassembly']",194,TRUE,1,2
"Run a benchmark in parallel, i.e. simulate simultaneous requests","<p>When testing a database procedure invoked from an API, when it runs sequentially, it seems to run consistently within ~3s. However we've noticed that when <strong>several requests come in at the same time</strong>, this can take much longer, causing time outs. I am trying to reproduce the ""several requests at one time"" case as a <code>go test</code>.</p>

<p>I tried the <code>-parallel 10</code> go test flag, but <a href=""https://media.dev.unee-t.com/2019-04-22/parallel.txt"" rel=""noreferrer"">the timings</a> were the same at ~28s.</p>

<p>Is there something wrong with my <a href=""https://github.com/unee-t/unit/commit/20e4f6a0001ab9a49027f4de0e969c3af3377929"" rel=""noreferrer"">benchmark function</a>?</p>

<pre><code>func Benchmark_RealCreate(b *testing.B) {
    b.ResetTimer()
    for n := 0; n &lt; b.N; n++ {
        name := randomdata.SillyName()
        r := gofight.New()
        u := []unit{unit{MefeUnitID: name, MefeCreatorUserID: ""user"", BzfeCreatorUserID: 55, ClassificationID: 2, UnitName: name, UnitDescriptionDetails: ""Up on the hills and testing""}}
        uJSON, _ := json.Marshal(u)
        r.POST(""/create"").
            SetBody(string(uJSON)).
            Run(h.BasicEngine(), func(r gofight.HTTPResponse, rq gofight.HTTPRequest) {
                assert.Contains(b, r.Body.String(), name)
                assert.Equal(b, http.StatusOK, r.Code)
            })
    }
}
</code></pre>

<p>Else how I can achieve what I am after?</p>
",https://stackoverflow.com/questions/55789373/run-a-benchmark-in-parallel-i-e-simulate-simultaneous-requests,"['go', 'testing', 'benchmarking']",6846,TRUE,6,5
how to pass random inputs from a input file to apache benchmark (abs) for load testing a URL,"<p>I want to pass random inputs for load testing my search URL .
Search texts have to be read from an Input file and appended in the URL and send for the AB load testing .</p>

<p>Eg. a) Input file :
        text1
        text2
        text3
    b) URL :- <code>{{BaseURL}}/apps/{{ApplId}}/courses/search?query={{random text from input file }}</code></p>

<p>I would like to hit this search URL with different/random texts present in the input file .
Currently i am trying the below command which just takes one static URL :-</p>

<pre><code>abs -n 100 -c 100 https://ib3rf43def /courses/search?query=course  &gt; ../../abTestResult.txt
</code></pre>

<p>Thanks for the help in advance .!</p>
",https://stackoverflow.com/questions/49689898/how-to-pass-random-inputs-from-a-input-file-to-apache-benchmark-abs-for-load-t,"['apache', 'testing', 'load-testing', 'apachebench']",731,TRUE,1,1
C# Performance benchmark,"<p>To keep track of performance in our software we measure the duration of calls we are interested in.</p>

<p>for example:</p>

<pre><code>using(var performanceTrack = new PerformanceTracker(""pt-1""))
{
    // do some stuff

    CallAnotherMethod();

    using(var anotherPerformanceTrack = new PerformanceTracker(""pt-1a""))
    {
       // do stuff

       // .. do something
    }

    using(var anotherPerformanceTrackb = new PerformanceTracker(""pt-1b""))
    {
       // do stuff

       // .. do something
    }
    // do more stuff
}
</code></pre>

<p>This will result in something like:</p>

<pre><code>pt-1  [----------------------------] 28ms

      [--]                            2ms from another method

pt-1a   [-----------]                11ms

pt-1b                [-------------] 13ms
</code></pre>

<p>In the constructor of PerformanceTracker I start a stopwatch. (As far as I know it's the most reliable way to measure a duration.) In the dispose method I stop the stopwatch and save the results to application insights.</p>

<p>I have noticed a lot of fluctation between the results. To solve this I've already done the following:</p>

<ul>
<li>Run in release built, outside of visual studio.</li>
<li>Warm up call first, not included in to the statistics.</li>
<li>Before every call (total 75 calls) I call the garbage collector.</li>
</ul>

<p>After this the fluctation is less, but still not very accurate. For example I have run my test set twice. Both times </p>

<p>See here the results in milliseconds.</p>

<blockquote>
  <p>Avg: 782.946666666667        981.68<br>
                                                                                                                                                        Min: 489 vs 513<br>
  Max: 2600 vs 4875<br>
  stdev: 305.854933523003 vs 652.343471128764<br>
  sampleSize: 75 vs 75</p>
</blockquote>

<p>Why is the performance measurement with the stopwatch still giving a lot of variation in the results? I found on SO (<a href=""https://stackoverflow.com/a/16157458/1408786"">https://stackoverflow.com/a/16157458/1408786</a>) that I should maybe add the following to my code:</p>

<pre><code>//prevent the JIT Compiler from optimizing Fkt calls away
long seed = Environment.TickCount;

//use the second Core/Processor for the test
Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(2);

//prevent ""Normal"" Processes from interrupting Threads
Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.High;

//prevent ""Normal"" Threads from interrupting this thread
Thread.CurrentThread.Priority = ThreadPriority.Highest;
</code></pre>

<p>But the problem is, we have a lot of async code. How can I get a reliable performance track in the code? My aim is to discover performance degradation when for example after a check in a method is 10ms slower than before...</p>
",https://stackoverflow.com/questions/53959532/c-performance-benchmark,"['c#', 'performance', 'testing', 'async-await', 'benchmarking']",990,FALSE,1,0
How to send json from apache benchmark without external file?,"<p>I have JSON from postman collection and i am iterating the requests list and calling <code>ab</code> from my script. get is working fine.
I have json of request body, how i can send json with <code>ab</code> without using external file. </p>
",https://stackoverflow.com/questions/53922534/how-to-send-json-from-apache-benchmark-without-external-file,"['testing', 'benchmarking', 'apachebench']",268,FALSE,2,0
How to properly StartTime and StopTime in golang benchmark?,"<p>I wrote a benchmark to test the speed of two Fibonacci number generators, and the source code is <a href=""https://github.com/nosarthur/dp/blob/master/fibonacci/fib_test.go"" rel=""nofollow noreferrer"">here on github</a>.</p>

<pre><code>func BenchmarkFib(b *testing.B) {
    fibFuncs := []struct {
        name string
        f    func(int) int
    }{
        {""recursive"", fibRecu},
        {""iterative"", fibIter},
    }
    for _, fibFunc := range fibFuncs {
        // calculate k'th Fibonacci number
        for k := 10; k &lt; 1001; k *= 10 {
            b.Run(fmt.Sprintf(""%s Fib %v"", fibFunc.name, k), func(b *testing.B) {
                for n := 0; n &lt; b.N; n++ {
                    //                  b.StopTimer()
                    // reset the memo
                    memo = map[int]int{0: 0, 1: 1, 2: 1}
                    //                  b.StartTimer()
                    fibFunc.f(k)
                }
            })
        }
    }
}
</code></pre>

<p>As it is, the benchmark works and the output is </p>

<pre><code>nos (master) fibonacci $ go test -bench .
goos: linux
goarch: amd64
pkg: github.com/nosarthur/dynamicP/fibonacci
BenchmarkFib/recursive_Fib_10-4                  1000000              1256 ns/op
BenchmarkFib/recursive_Fib_100-4                  100000             18256 ns/op
BenchmarkFib/recursive_Fib_1000-4                  10000            206109 ns/op
BenchmarkFib/iterative_Fib_10-4                 10000000               218 ns/op
BenchmarkFib/iterative_Fib_100-4                 5000000               292 ns/op
BenchmarkFib/iterative_Fib_1000-4                2000000               881 ns/op
PASS
ok      github.com/nosarthur/dynamicP/fibonacci 12.208s
</code></pre>

<p>However, I added <code>b.StopTime()</code> and <code>b.StartTime()</code> to exclude the time for resetting the memo. With these two lines un-commented, the benchmark hangs, and the partial output is</p>

<pre><code>nos (master *) fibonacci $ go test -bench .
goos: linux
goarch: amd64
pkg: github.com/nosarthur/dynamicP/fibonacci
BenchmarkFib/recursive_Fib_10-4                  1000000              2139 ns/op
BenchmarkFib/recursive_Fib_100-4                  100000             24775 ns/op
BenchmarkFib/recursive_Fib_1000-4                   5000            239197 ns/op
BenchmarkFib/iterative_Fib_10-4                 ^Csignal: interrupt
FAIL    github.com/nosarthur/dynamicP/fibonacci 269.067s
</code></pre>

<p>What is the proper way to exclude the memo resetting? My go version is 1.10.1</p>
",https://stackoverflow.com/questions/52013803/how-to-properly-starttime-and-stoptime-in-golang-benchmark,"['testing', 'go', 'benchmarking']",2254,TRUE,4,1
List of parameters in JMH benchmark test,"<p>I am writing performance tests for an HTTP server using my custom Java library that is sending REST requests to the server. At the beginning, I am performing data preparation stage in order to have a list of objects that I want to send in requests to my server.</p>

<p>Now, the problem is, that I can test a list of parameters that can be injected into the benchmark function using <code>@Param</code> annotation:</p>

<pre><code>@Param({""param1"", ""param2"", ...})
</code></pre>

<p>The problem is, I would like to achieve the same having Java List of parameters and to avoid having to iterate over them. Could you please give me any advice how I can do it?</p>
",https://stackoverflow.com/questions/51897651/list-of-parameters-in-jmh-benchmark-test,"['java', 'performance', 'testing', 'benchmarking', 'jmh']",2036,TRUE,0,1
How can I benchmark signals in the Qt test framework?,"<p>I'm implementing the Connection Scan Algorithm in Qt C++ and I added some tests to the project using the Qt test framework which is fine. I looked in the docs of the Qt test framework but I can't find a way to benchmark a called asynchronous method and wait for it's signal.</p>

<p>You have QSignalSpy available to test signals but it waits a certain time before it checks if the signal was fired. QBENCHMARK can measure the time to execute a piece of code. I would like to measure the time between the call of the asynchronous method and the signal fired by the thread which the method calls.</p>

<pre><code>QString str1 = ""foo"";
QString str2 = ""foo"";

QBENCHMARK {
    str1.localeAwareCompare(str2);
}
</code></pre>

<p>The code above is fine for synchronous operations but not for asynchronous operations with signals.</p>
",https://stackoverflow.com/questions/51745261/how-can-i-benchmark-signals-in-the-qt-test-framework,"['c++', 'qt', 'testing', 'signals', 'benchmarking']",229,TRUE,0,1
"Golang Benchmark table test, can I extract func from b.Run()?","<p>Following benchmark example tests in golang, I could for examlpe have this test:</p>

<pre><code>// An example benchmark to benchmark a query based on different inputs
func Benchmark_GetProcessingCountForRegions(b *testing.B) {

    benchmarks := []struct {
        region string
    }{
        {""EU""},
        {""US""},
    }

    for _, bm := range benchmarks {
        b.Run(bm.region, func(bbb *testing.B) {
            for i := 0; i &lt; bbb.N; i++ {
                taskDb.GetProcessingCountForRegion(bm.region)
            }
        })

    }
}
</code></pre>

<p>This is following default examples from the web and works for me; testing the taskDb package for performance on the GetProcessingCountForRegion query.</p>

<p>However for readability of this test I'm trying to extract the inside func, the one that sits inside b.Run(), like so, first extract the func and put it aside:</p>

<pre><code>func GetProcessingCountForRegion(testingB *testing.B, region string) {
    for i := 0; i &lt; testingB.N; i++ {
        taskDb.GetProcessingCountForRegion(region)
    }
}
</code></pre>

<p>and then use this func in the actual testfunc like so:</p>

<pre><code>for _, bm := range benchmarks {
        b.Run(bm.region, GetProcessingCountForRegion(*b, bm.region))
}
</code></pre>

<p>However this gives a compile error: 
cannot use *b (type testing.B) as type *testing.B in argument to GetProcessingCountForRegion</p>

<p>removing the * from *b, gives the following compile error:
GetProcessingCountForRegion(b, bm.region) used as value</p>

<p>So what I want is remove the func parameter from the B.Run(..) statement.</p>

<p>What am I missing?</p>
",https://stackoverflow.com/questions/47475905/golang-benchmark-table-test-can-i-extract-func-from-b-run,"['testing', 'go']",481,TRUE,1,2
"Go-lang Testing, What is the meaning of Parallel Setting on Benchmark?","<p>In <a href=""https://golang.org/pkg/testing/"" rel=""nofollow"">https://golang.org/pkg/testing/</a> described that we can use <code>testing.B.RunParallel()</code> function to run benchmark in a <em>parallel setting</em>. I tried to write simple testing code:</p>

<pre><code>func BenchmarkFunctionSome(b *testing.B) {

    for i := 0; i &lt; b.N; i++ {
        SomeFunction()
    }
}
</code></pre>

<p>and then i changed it to use <code>RunParallel()</code></p>

<pre><code>func BenchmarkFunctionSome(b *testing.B) {

    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            SomeFunction()
        }
    })
}
</code></pre>

<p>And the one which used <code>RunParallel()</code> is slower than the first benchmark. </p>

<p>Actually what is the meaning of <em>parallel setting</em> in benchmarking? Why it became slower if i used <code>RunParallel()</code> ?</p>
",https://stackoverflow.com/questions/39104879/go-lang-testing-what-is-the-meaning-of-parallel-setting-on-benchmark,"['testing', 'go']",4292,TRUE,0,2
Hadoop benchmark tests,"<p>I am currently performing some tests in a small cluster (4 nodes) and I would like to know if there are any best practices regarding the node that actually launches the test scripts in the cluster. I am using test tools like hive-testbench and HiBench.</p>

<p>There should be no impact in the performance if I run a spark script in the same machine that is hosting the thrift server, or should I add another machine to the cluster just to launch the test/benchmark scripts?</p>

<p>Thanks in advance.</p>
",https://stackoverflow.com/questions/44595441/hadoop-benchmark-tests,"['hadoop', 'testing', 'benchmarking']",130,FALSE,0,0
go benchmark and gc: B/op alloc/op,"<p>benchmark code: </p>

<pre><code>func BenchmarkSth(b *testing.B) {
    var x []int
    b.ResetTimer()
    for i := 0; i &lt; b.N; i++ {
        x = append(x, i)
    }
}
</code></pre>

<p>result: </p>

<pre><code>BenchmarkSth-4    50000000    20.7 ns/op    40 B/op    0 allocs/op 
</code></pre>

<p>question/s:</p>

<ul>
<li>where did 40 B/op come from? (any way of tracing + instructions is greatly appreciated)</li>
<li>how is it possible to have 40 B/op while having 0 allocs?</li>
<li>which one affects GC and how? (B/op or allocs/op)</li>
<li>is it really possible to have 0 B/op using append?</li>
</ul>
",https://stackoverflow.com/questions/42172360/go-benchmark-and-gc-b-op-alloc-op,"['testing', 'memory', 'go', 'garbage-collection', 'benchmarking']",1455,TRUE,3,1
How to benchmark test java applications?,"<p>I'm currently working on a project that talks on source to source compiler optimisation, what tools are there available for me to benchmark test java and JavaScript applications?</p>

<p>Thank you in advance. </p>
",https://stackoverflow.com/questions/42051470/how-to-benchmark-test-java-applications,"['javascript', 'java', 'testing', 'benchmarking']",88,FALSE,1,0
Could I increase the number of client threads more than the maxThreads of Tomcat when using ab to benchmark?,"<p>In the book <em>Tomcat The Definitive Guide</em> written by Jason Brittain with Ian F.Darwin, when using the <code>ab</code> tool to benchmark, the writters says,  </p>

<blockquote>
  <p>you should benchmark by running a minimum of 100,000 HTTP requests.
  Also , you may configure the test client to spawn as many client threads as you would like,
  but you will not get helpful results if you set it higher than the maxThreads you set for you Connector in your Tomcat's conf/server.xml file.
  By default, it is set to 150.</p>
</blockquote>

<p>Then the writter recommends 149.</p>

<p>In my case, with <strong>149</strong> client threads, the running result is:</p>

<pre><code>[user@apachetomcat ~]$ ab -k -n 100000 -c 149 http://10.138.0.2:8080/test.html
This is ApacheBench, Version 2.3 &lt;$Revision: 1430300 $&gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.138.0.2 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests

Server Software:        
Server Hostname:        10.138.0.2
Server Port:            8080
Document Path:          /test.html
Document Length:        13 bytes
Concurrency Level:      149
Time taken for tests:   45.527 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    99106
Total transferred:      23195530 bytes
HTML transferred:       1300000 bytes
Requests per second:    2196.48 [#/sec] (mean)
Time per request:       67.836 [ms] (mean)
Time per request:       0.455 [ms] (mean, across all concurrent requests)
Transfer rate:          497.54 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   6.8      0      70
Processing:    66   67   5.6     67     870
Waiting:       66   67   5.6     67     870
Total:         66   68   8.8     67     870
Percentage of the requests served within a certain time (ms)
  50%     67
  66%     67
  75%     67
  80%     67
  90%     67
  95%     68
  98%     69
  99%    133
 100%    870 (longest request)
</code></pre>

<p>After increasing to <strong>1000</strong> client threads, the result is:</p>

<pre><code>[user@apachetomcat ~]$ ab -k -n 100000 -c 1000 http://10.138.0.2:8080/test.html
This is ApacheBench, Version 2.3 &lt;$Revision: 1430300 $&gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/
Benchmarking 10.138.0.2 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests
Server Software:        
Server Hostname:        10.138.0.2
Server Port:            8080
Document Path:          /test.html
Document Length:        13 bytes
Concurrency Level:      1000
Time taken for tests:   7.205 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Keep-Alive requests:    99468
Total transferred:      23197340 bytes
HTML transferred:       1300000 bytes
Requests per second:    13879.80 [#/sec] (mean)
Time per request:       72.047 [ms] (mean)
Time per request:       0.072 [ms] (mean, across all concurrent requests)
Transfer rate:          3144.28 [Kbytes/sec] received
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   8.1      0      68
Processing:    66   69  22.3     67    1141
Waiting:       66   69  22.3     67    1141
Total:         66   70  27.5     67    1141
Percentage of the requests served within a certain time (ms)
  50%     67
  66%     67
  75%     68
  80%     68
  90%     69
  95%     71
  98%     87
  99%    139
 100%   1141 (longest request)
</code></pre>

<p>The Requests per second increases from 2196.48/sec to 13879.80/sec, so I think this change is meaningful.<br>
Why does the writer think it's not helpful when we set it higher than the maxThreads?<br>
What does <strong>the incrementation of requests per second</strong> mean in my case?<br>
I'm confused with the requests per second. It's very important to understand the writer's benchmarks in the following chapters of the book.</p>
",https://stackoverflow.com/questions/41741355/could-i-increase-the-number-of-client-threads-more-than-the-maxthreads-of-tomcat,"['apache', 'testing', 'tomcat8']",384,FALSE,0,0
Openssl speed benchmark test,"<p>Recently I need to test openssl suite performance on my machine. I ran speed to get some performance numbers. However I wanted specific performance such as encrytion time and decrytion time but the data I got is in the form of </p>

<pre><code>Doing aes-128 cbc for 3s on 16 size blocks: 24115540 aes-128 cbc's in 3.00s
</code></pre>

<p>I checked speed.c file and it was </p>

<pre><code>if (doit[D_CBC_128_AES])
        {
        for (j=0; j&lt;SIZE_NUM; j++)
            {
                        print_message(names[D_CBC_128_AES],c[D_CBC_128_AES][j],lengths[j]);
            Time_F(START);
            for (count=0,run=1; COND(c[D_CBC_128_AES][j]); count++)
                AES_cbc_encrypt(buf,buf,
                    (unsigned long)lengths[j],&amp;aes_ks1,
                    iv,AES_ENCRYPT);// :O only encryption
            d=Time_F(STOP);
                        print_result(D_CBC_128_AES,j,count,d);//xxxxx

            }
        }
</code></pre>

<p>So it is only doing encryption, not decryption. </p>

<p>Why is it so?
How can I get encryption time, and decryption time of algorithms and their modes? 
Do I have to write code myself for all algorithms?
How can I get cpu cycles used in a particular algorithm?</p>
",https://stackoverflow.com/questions/40043219/openssl-speed-benchmark-test,"['performance', 'testing', 'openssl']",1788,FALSE,0,1
Vector functions of STREAM benchmark,"<p>I am currently doing a small research project for school, where I am to test the memory performance bandwidth of a Hypervisor, compared to the virtualised machines it creates and manages.</p>

<p>Due to the timeframe of the project, only one of the vector functions tested by STREAM will be analysed. My thoughtprocess is to look at the results from the ""Copy"" function, since this is the most basic function, which performs no arithmetic, as stated at the bottom of <a href=""https://www.cs.virginia.edu/stream/ref.html"" rel=""nofollow"">https://www.cs.virginia.edu/stream/ref.html</a>
After all, this is a memory bandwidth performance test.</p>

<p>I have yet though to find any google post that proves, or disproves my theory. Is there anyone here who can shine some light on this topic? </p>
",https://stackoverflow.com/questions/34074315/vector-functions-of-stream-benchmark,"['testing', 'memory', 'stream', 'benchmarking', 'memory-bandwidth']",231,FALSE,1,1
what is the theoretical benchmark upon which to make select the artifical network to predicting data?,"<p>I have trained neural networks for 50 times(training) now I am not sure which network (from 50nets) I should choose to predict my data?Among MSE(mean square error) of network or R-squred or validation performance or test performance?....Thanks for any suggestion</p>
",https://stackoverflow.com/questions/35634195/what-is-the-theoretical-benchmark-upon-which-to-make-select-the-artifical-networ,"['performance', 'validation', 'testing', 'neural-network', 'mse']",123,FALSE,0,1
How to benchmark if I need a reset in each iteration?,"<p>I've written a small Sudoku solver using backtracking. Now I want to benchmark the speed of this function. Here is my current code:</p>

<pre><code>type Board struct {
    Cells [9][9]int
}

func BenchmarkBacktrack(b *testing.B) {
    for i := 0; i &lt; b.N; i++ {
        b.StopTimer()
        // prevent the modification of the orignal board
        copy := &amp;Board{
            Cells: exampleBoard.Cells,
        }
        b.StartTimer()
        copy.Backtrack()
     }
}
</code></pre>

<p>Since <code>&amp;Board</code> is pointer I would solve the Sudoku in the first iteration and in the next one I would backtrack a solved board. Therefore, I reset the board at the beginning of each iteration. <code>exampleBoard</code> is filled with sample values.</p>

<p>Is their a better way to benchmark the function without stopping and restarting the timer over and over?</p>

<p>And wouldn't cost the function calls a small amount of time that impacts the benchmark?</p>
",https://stackoverflow.com/questions/33726201/how-to-benchmark-if-i-need-a-reset-in-each-iteration,"['testing', 'go', 'benchmarking']",1500,TRUE,1,2
Is it possible to dynamically run benchmark tests?,"<p>I have a few different implementations of an interface, and a variety of factors I want to test them under. End goal being to make a grid of results for different implementations in different situations.</p>

<p>I could write a test for each possible combination, but that gets exhausting:</p>

<pre><code>func Benchmark_ImplA_N100_X300(b *testing.B){
   impl := newImplA(100,300)
   runBenchmark(b,impl)
}
</code></pre>

<p>The more combinations I add, the more I have to copy/paste. This gets cumbersome fast.</p>

<p>I would love to do something like:</p>

<pre><code>tests := []testing.InternalBenchmark{}
for _, n := range []int{50,100,500,10000}{
   for _,x := range []int{300,200}{
      for name, gen := range implementations{
         impl = gen(n,x)
         bm := testing.InternalBenchmark{Name: fmt.Sprint(name,x,n)}
         bm.F = func(b *testing.B){
            runBench(b,impl)
         }
         tests = append(tests,bm)
      }
   }
}
testing.RunBenchmarks(anything, tests)
</code></pre>

<p>so that I can update the list(s) of combinations, and benchmarks will magically run in all combinations. I have tries something like this in main and in TestMain, and nothing ever outputs. I am not sure if I am using it wrong, or if the testing package is just doing something funny.</p>

<p>I really don't care if the <code>go test</code> tool can handle it or if there is some other way.</p>
",https://stackoverflow.com/questions/33537552/is-it-possible-to-dynamically-run-benchmark-tests,"['testing', 'go', 'benchmarking']",1142,TRUE,7,2
Can someone suggest me how can i test or benchmark apache storm?,"<p>I need to test the performance of apache storm installed in multi node cluster.
I need some advice how can i do that? What aspects we need to consider?
is there any tools available in documentation?</p>

<p>I am quite new to this platform.
Appreciate if anybody could help</p>
",https://stackoverflow.com/questions/29276287/can-someone-suggest-me-how-can-i-test-or-benchmark-apache-storm,"['performance', 'testing', 'benchmarking', 'apache-storm']",511,FALSE,1,1
how to generate inputs for benchmark programs when doing random testing,"<p>The benchmark programs used in papers about random testing usually takes 
numbers as input. It's easy to random generate numbers but the inputs of real-world programs are usually strings or files. How to generate complex inputs when doing random testing?</p>

<p>Another question is that I'm currently implementing a framework for random testing. Maybe the first step is to test programs which take numbers as input. How can I find such benchmark programs?</p>

<p>I want to test c/c++ programs and need to generate the arguments for the main function. </p>
",https://stackoverflow.com/questions/29223217/how-to-generate-inputs-for-benchmark-programs-when-doing-random-testing,"['testing', 'random']",87,FALSE,0,0
