title,body,link,tags,view_count,is_answered,score,answer_count
Possibilities of stress testing in Cypress,"<p>Might I know what are possibilities of testing in Cypress? If one want to test visiting or creating order for few hundred/thousands of times? I would expect there would be some limitations to address misusing of DDoS attacks, but Cypress as tool might offer some safe and valid solution in this kind of manner.</p>
<p>Thank you for answers!</p>
",https://stackoverflow.com/questions/73746770/possibilities-of-stress-testing-in-cypress,['cypress'],83,True,0,1
CPU stress testing by percentage with native linux command,"<p>I am currently running the following commands to stress test my CPU:</p>
<pre><code>md5sum /dev/zero
</code></pre>
<p>Then I kill the process after 30 seconds.</p>
<pre><code>pkill md5sum
</code></pre>
<p>I am looking to run a similar command but where it lets me stress the CPU by a certain percentage. I cannot use libraries like stress-ng, it has to be be able to run with basic linux distro like ubuntu.  I also can't run a program as such, I have to be able to run it from the command line itself. Any suggestions or help is greatly appreciated.</p>
",https://stackoverflow.com/questions/71953623/cpu-stress-testing-by-percentage-with-native-linux-command,"['linux', 'bash', 'shell', 'command-line-interface', 'cpu']",1029,False,2,1
Concurrent requests not working in Siege - Http stress testing,"<p>I'm testing an app located in a AWS EC2 instance via a POST request.</p>
<p>I'm posting an image with the url in the form of a json string by converting it to base64. All the settings are in place. The problem is that even though I'm sending 10 concurrent requests for the same image, by using the -c parameter, only the first request gets processed and the Siege stress test stops.</p>
<p><a href=""https://i.stack.imgur.com/JcoMy.jpg"" rel=""nofollow noreferrer""><img src=""https://i.stack.imgur.com/JcoMy.jpg"" alt=""enter image description here"" /></a></p>
<p>This is the code I'm using</p>
<pre><code>'siege -c10 -r11 --content-type &quot;application/json&quot; -g 'http://www.hello.com POST {&quot;img&quot;: &quot;test64&quot;, fname: &quot;img.jpg&quot;}'
</code></pre>
",https://stackoverflow.com/questions/71033897/concurrent-requests-not-working-in-siege-http-stress-testing,"['amazon-web-services', 'performance-testing', 'benchmarking', 'stress-testing', 'siege']",140,True,0,1
How to do stress testing for AWS CPUUtiliazation and MemoryUtilization?,"<p>I am using AWS cloud ECS services and created once ECS cluster. In which there is one FargateService and behind that 2 containers are running.
Two alarms have been created for fargate service on threashold 95.</p>
<p>Everything looks fine now Testing part comes into picture. I want test alarms functionality.
is there any easy way in AWS using some AWS service or manual script so that I can increase CPU and Memory use to test alarm functionality .</p>
",https://stackoverflow.com/questions/71885554/how-to-do-stress-testing-for-aws-cpuutiliazation-and-memoryutilization,"['amazon-ecs', 'amazon-cloudwatch']",269,False,1,0
Stress testing Postgres queries using K6 [No space on device error],"<p>I'm trying to use <a href=""https://github.com/grafana/xk6-sql"" rel=""nofollow noreferrer"">k6</a> to stress test postgres queries.</p>
<p>I am running postgres through docker and allocated following resources through settings :
CPUs: 4, Memory: 10 GB, Swap: 2GB and Disk image size: 280 GB.</p>
<p>When I did load testing for a query, everything works for up to 45 virtual users but once I increase that to 50 I'm getting the following error :</p>
<pre><code>GoError: pg: could not resize shared memory segment &quot;/PostgreSQL.440965520&quot; to 182976 bytes: No space left on device
</code></pre>
<br>
<p>There are 3 tables (actor, actor_movies and movies). To keep track of which actor has played a role in which movie. I wanted to do a full-text search on 4 fields and so created a materialized view over the join of these tables, roughly around <strong>3 million rows.</strong></p>
<p>I think that the error says that there's not enough resource, but is there anything that I can do to mitigate this.</p>
<p>Please let me know if the question is not clear and if it's missing something.<br>
Thanks :)</p>
<p><strong>EDIT 1</strong>
As @jjanes mentioned in comments.
Following things worked for me :</p>
<ol>
<li>setting max_parallel_workers_per_gather = 0.</li>
<li>reducing work_mem from 4MB to 2 MB.</li>
<li>Increasing the size of shared_buffers from 128MB to 256MB (it should've 15% of actual ram i.e 1.5 GB).</li>
</ol>
<p>Then I started getting this error, which I believe is another config change. <em>GoError: pq: sorry, too many clients already</em> which can be fixed with another config change.</p>
",https://stackoverflow.com/questions/72367078/stress-testing-postgres-queries-using-k6-no-space-on-device-error,"['postgresql', 'docker', 'stress-testing', 'k6']",489,False,1,0
How can i seperate output in two different file in c++ (Stress Testing),"<p>A few days ago, I learned about stress testing and wanted to ask how to make two different .txt files.</p>
<p>Like one for Output and one for my failed test case.</p>
<p>It would work like this:</p>
<p>output: it would have all the output, correct solution, and wrong solution</p>
<p>failedtestcase.txt: it would have only those test cases detail that have failed and which inputs were given for the particular failed test case.</p>
<p>`</p>
<pre><code>int main() {
    freopen(&quot;output.txt&quot;, &quot;w&quot;, stdout);
    //create a freopen for Failed test case

    srand(time(NULL));
    int t = rand() % 10;
    cout&lt;&lt; &quot;======================================&quot; &lt;&lt; endl;
    cout &lt;&lt; &quot;            Test Case: &quot;&lt;&lt; t &lt;&lt; &quot;           &quot;&lt;&lt;  endl;
    cout&lt;&lt; &quot;======================================&quot; &lt;&lt; endl;
  
    while(t--){
        int n = rand() % 11;
        cout &lt;&lt; &quot;Number of Element: &quot; &lt;&lt;  n &lt;&lt; endl;
        int d = rand() % n;
        cout &lt;&lt; &quot;Kth: &quot; &lt;&lt; d &lt;&lt; endl;
        cout &lt;&lt; &quot;Araray: &quot; &lt;&lt; endl;
        int arr[n];
        for(int i=0; i&lt; n;i++){
            arr[i] = rand() % 11;
            cout&lt;&lt; arr[i] &lt;&lt; &quot; &quot;;
        }
        sort(arr, arr+n);
        cout &lt;&lt; endl;

        KthSAndL(arr, n, d);

        freopen(&quot;FailedTestcase.txt&quot;, &quot;w&quot;, stdout);

        pair&lt;int, int&gt; MySolution = KthSAndL(arr, n, d);
        pair&lt;int, int&gt; ActualSolution = kthSmallestLargest(arr, n, d);

        if(MySolution.first != ActualSolution.first &amp;&amp; MySolution.second != ActualSolution.second){
            cout &lt;&lt; &quot;Wrong Answer&quot; &lt;&lt; endl;
            cout &lt;&lt; &quot;Your Answer: &quot; &lt;&lt; MySolution.first &lt;&lt; &quot; &quot; &lt;&lt; MySolution.second &lt;&lt; endl;
            cout &lt;&lt; &quot;Actual Answer: &quot; &lt;&lt; ActualSolution.first &lt;&lt; &quot; &quot; &lt;&lt; ActualSolution.second &lt;&lt; endl;
        }

        cout &lt;&lt; endl;
        cout&lt;&lt; &quot;======================================&quot; &lt;&lt; endl;
    }

    return 0;
}
</code></pre>
<p>`</p>
<p>This is the code and i'm getting this Kind of output 😑</p>
<p><a href=""https://i.stack.imgur.com/RxTMj.png"" rel=""nofollow noreferrer""><img src=""https://i.stack.imgur.com/RxTMj.png"" alt=""enter image description here"" /></a></p>
<p>I was trying to have my all output in output.txt and only failed test case inputs will be shows in failedtestcase.txt</p>
<p>What I tried:</p>
<ol>
<li>Moving the Lines up down before cout'ing (Might work I thought)</li>
</ol>
<p>and well actully I have no idea what I can do 🤷</p>
",https://stackoverflow.com/questions/74299460/how-can-i-seperate-output-in-two-different-file-in-c-stress-testing,"['c++', 'c++11', 'c++17', 'clion', 'stress-testing']",56,False,0,0
Stress testing mobile applications outline (iOS and Android) as a beginner using Jmeter?,"<p>I'm looking into how to use jmeter (I've done manual testing until now) to perform stress testing on mobile applications.</p>
<p>In particular, I have a student management kind of app. Both (iOS - Swift) and (Android - Java) app are available.
I want to mimic lots of student trying to register for same course, leaving same course, paying etc. I'm not sure where to start, or how to even configure jmeter.
What should I request from the devs? Is android apk + ios testflight version enough or should I request full source code?
I feel lost as this is my first proper tech related job while in college, but the lack of resources is making me afraid.</p>
",https://stackoverflow.com/questions/73870294/stress-testing-mobile-applications-outline-ios-and-android-as-a-beginner-using,"['android', 'ios', 'testing', 'jmeter']",32,True,0,1
"In stress testing ,while executing script on Jmeter tool getting token session expired","<p>while executing script on jmeter getting token session expired error showing in below image, searched on google and tried to solve,it is not solved</p>
<p><a href=""https://i.stack.imgur.com/zlJZv.png"" rel=""nofollow noreferrer""><img src=""https://i.stack.imgur.com/zlJZv.png"" alt=""enter image description here"" /></a></p>
",https://stackoverflow.com/questions/72219103/in-stress-testing-while-executing-script-on-jmeter-tool-getting-token-session-e,['jmeter'],35,False,0,1
"How do I make this PowerShell stress testing script faster? It needs to compile and run C++ files, then compare their output","<p>I'm trying to write a PowerShell script to quickly generate a lot of tests for a C++ program and also verify its output.
main.cpp is the program I'm verifying, brute.cpp is a brute solution I know for sure works and gen.cpp generates tests.
Here's the code:</p>
<pre><code>g++ -o gen.exe gen.cpp
g++ -o brute.exe brute.cpp
g++ -o main.exe main.cpp
for ($i = 0; ; $i++) {
    Write-Output &quot;$i&quot;
    Start-Process .\gen.exe -RedirectStandardOutput &quot;.\cmake-build-debug\in.in&quot; -NoNewWindow -Wait
    Start-Process .\brute.exe -RedirectStandardInput &quot;.\cmake-build-debug\in.in&quot; -RedirectStandardOutput &quot;.\cmake-build-debug\out1&quot; -NoNewWindow -Wait
    Start-Process .\main.exe -RedirectStandardInput &quot;.\cmake-build-debug\in.in&quot; -RedirectStandardOutput &quot;.\cmake-build-debug\out2&quot; -NoNewWindow -Wait
    if ((Get-FileHash  &quot;.\cmake-build-debug\out1&quot; ).Hash -ne (Get-FileHash &quot;.\cmake-build-debug\out2&quot;).Hash) {
        Compare-Object (Get-Content &quot;.\cmake-build-debug\out1&quot;) (Get-Content &quot;.\cmake-build-debug\out2&quot;)
        break
    }
}
</code></pre>
<p>This works but it every test takes around 6 seconds, I would want something around 50 times faster. (the programs themselves take up a matter of miliseconds.)</p>
",https://stackoverflow.com/questions/70623326/how-do-i-make-this-powershell-stress-testing-script-faster-it-needs-to-compile,"['c++', 'powershell', 'stress-testing']",205,False,0,0
Why do my TCP clients fail to connect to my server with a socket exception when stress-testing it with many connections,"<p>When stress-testing a TCP server of mine with a large number of connections, I figured that connection requests will throw a SocketException after some time. The exception displays randomly either</p>
<blockquote>
<p>Only one usage of each socket address (protocol/network address/port) is normally permitted.</p>
</blockquote>
<p>or</p>
<blockquote>
<p>No connection could be made because the target machine actively refused it.</p>
</blockquote>
<p>as its message.</p>
<p>This happens usually but randomly after some seconds and some tens of thousands of connects and disconnects. To connect, I use the local end point <code>IPEndPoint clientEndPoint = new(IPAddress.Any, 0);</code>, which I believe would give me the next free ephemeral port.</p>
<p>To isolate the issue, I wrote this simple program that runs both a TCP server and many parallel clients for a simple counter:</p>
<pre><code>using System.Diagnostics;
using System.Net;
using System.Net.Sockets;

CancellationTokenSource cancellationTokenSource = new();
CancellationToken cancellationToken = cancellationTokenSource.Token;

const int serverPort = 65000;
const int counterRequestMessage = -1;
const int randomCounterResponseMinDelay = 10; //ms
const int randomCounterResponseMaxDelay = 1000; //ms
const int maxParallelCounterRequests = 10000;

#region server

int counterValue = 0;

async void RunCounterServer()
{
    TcpListener listener = new(IPAddress.Any, serverPort);
    listener.Start(maxParallelCounterRequests);
    while (!cancellationToken.IsCancellationRequested)
    {
        HandleCounterRequester(await listener.AcceptTcpClientAsync(cancellationToken));
    }

    listener.Stop();
}

async void HandleCounterRequester(TcpClient client)
{
    await using NetworkStream stream = client.GetStream();
    Memory&lt;byte&gt; memory = new byte[sizeof(int)];

    //read requestMessage
    await stream.ReadAsync(memory, cancellationToken);
    int requestMessage = BitConverter.ToInt32(memory.Span);
    Debug.Assert(requestMessage == counterRequestMessage);

    //increment counter
    int updatedCounterValue = Interlocked.Add(ref counterValue, 1);
    Debug.Assert(BitConverter.TryWriteBytes(memory.Span, updatedCounterValue));

    //wait random timeout
    await Task.Delay(GetRandomCounterResponseDelay());

    //write back response
    await stream.WriteAsync(memory, cancellationToken);

    client.Close();
    client.Dispose();
}

int GetRandomCounterResponseDelay()
{
    return Random.Shared.Next(randomCounterResponseMinDelay, randomCounterResponseMaxDelay);
}

RunCounterServer();

#endregion

IPEndPoint clientEndPoint = new(IPAddress.Any, 0);
IPEndPoint serverEndPoint = new(IPAddress.Parse(&quot;127.0.0.1&quot;), serverPort);
ReaderWriterLockSlim isExceptionEncounteredLock = new(LockRecursionPolicy.NoRecursion);
bool isExceptionEncountered = false;

async Task RunCounterClient()
{
    try
    {
        int counterResponse;
        using (TcpClient client = new(clientEndPoint))
        {
            await client.ConnectAsync(serverEndPoint, cancellationToken);

            await using (NetworkStream stream = client.GetStream())
            {
                Memory&lt;byte&gt; memory = new byte[sizeof(int)];

                //send counter request
                Debug.Assert(BitConverter.TryWriteBytes(memory.Span, counterRequestMessage));
                await stream.WriteAsync(memory, cancellationToken);

                //read counter response
                await stream.ReadAsync(memory, cancellationToken);
                counterResponse = BitConverter.ToInt32(memory.Span);
            }

            client.Close();
        }

        isExceptionEncounteredLock.EnterReadLock();
        //log response if there was no exception encountered so far
        if (!isExceptionEncountered)
        {
            Console.WriteLine(counterResponse);
        }

        isExceptionEncounteredLock.ExitReadLock();
    }
    catch (SocketException exception)
    {
        bool isFirstEncounteredException = false;

        isExceptionEncounteredLock.EnterWriteLock();

        //log exception and note that one was encountered if it is the first one
        if (!isExceptionEncountered)
        {
            Console.WriteLine(exception.Message);
            isExceptionEncountered = true;
            isFirstEncounteredException = true;
        }

        isExceptionEncounteredLock.ExitWriteLock();

        //if this is the first exception encountered, rethrow it
        if (isFirstEncounteredException)
        {
            throw;
        }
    }
}

async void RunParallelCounterClients()
{
    SemaphoreSlim clientSlotCount = new(maxParallelCounterRequests, maxParallelCounterRequests);

    async void RunCounterClientAndReleaseSlot()
    {
        await RunCounterClient();
        clientSlotCount.Release();
    }

    while (!cancellationToken.IsCancellationRequested)
    {
        await clientSlotCount.WaitAsync(cancellationToken);
        RunCounterClientAndReleaseSlot();
    }
}

RunParallelCounterClients();

while (true)
{
    ConsoleKeyInfo keyInfo = Console.ReadKey(true);
    if (keyInfo.Key == ConsoleKey.Escape)
    {
        cancellationTokenSource.Cancel();
        break;
    }
}
</code></pre>
<p>My initial guess is, that I run out of ephemeral ports because I somehow do not free them correctly. I just <code>Close()</code> and <code>Dispose()</code> my <code>TcpClient</code>s in both my client and server code when a request is finished. I thought that would automatically release the port, but when I use <code>netstat -ab</code> in a console, it gives me countless entries like this, even after closing the application:</p>
<pre><code>TCP    127.0.0.1:65000        kubernetes:59996       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:59997       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:59998       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:59999       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60000       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60001       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60002       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60003       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60004       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60005       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60006       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60007       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60009       TIME_WAIT
</code></pre>
<p>Also, my PC sometimes gets a lot of stutter some time after I exit the application. I assume this is due to Windows cleaning up my leaked port usage?</p>
<p>So I wonder, what am I doing wrong here?</p>
",https://stackoverflow.com/questions/71070443/why-do-my-tcp-clients-fail-to-connect-to-my-server-with-a-socket-exception-when,"['c#', 'sockets', '.net-core', 'port']",618,True,0,1
A function to generate up to N virtual unique records,"<p>In my database app stress-testing, I need to be able to execute a function that produces a huge number of unique records.</p>
<p>Is it possible to implement a function in PostgreSql that would take a parameter - number of records to be generated, and then produce that many records of some virtual structure that I could later tweak?</p>
<pre class=""lang-sql prettyprint-override""><code>CREATE FUNCTION generate_records(N)

/*
    generate N unique records of some random structure;
*/
</code></pre>
<p>I want to avoid creating actual tables with many millions of records for automated stress-testing, because I am not testing performance of the database, I am testing data-processing performance of a client that consumes a lot of data.</p>
",https://stackoverflow.com/questions/74560100/a-function-to-generate-up-to-n-virtual-unique-records,['postgresql'],20,True,0,1
SOS: Java Mongo Concurrency issue,"<pre><code>JDK : 11
Spring boot : 2.0.5
Mongo : 5 
</code></pre>
<p>I am currently writing the order-book management logic for a trading application. Where only top 10 buy and top 10 sell orders are stored in a mongo collection. The logic testing done manually results perfectly fine. But on connecting a bot to do stress testing, the logic seems to be breaking. It is storing more than 10 records in the collection.</p>
<p>Apart from that, if buy order price is same as the record in collection, only the quantity has to be added. For example: Buy 1 BTC @ 21000 is already there in the collection, and second user places the same order, the records is updated with 2 Qty @ 21000. But this too seems to not work when stress testing. Below is the code block that adds new order to the order book.</p>
<pre><code>IF required more code blocks can be shared.
</code></pre>
<p>CODE :</p>
<pre><code>public synchronized void saveBuyOrder(PendingOrder buyOrder) {

    OrderBook orderBook;
    Optional&lt;OrderBook&gt; optionalOrderBook = orderBookRepository.findByMarketAndOrderTypeAndPrice(buyOrder.getMarket(), buyOrder.getOrderType(), buyOrder.getPrice());
    List&lt;OrderBook&gt; orderBookList = orderBookRepository.findAllByMarketAndOrderTypeOrderByPriceDesc(buyOrder.getMarket(), buyOrder.getOrderType());

    if (pendingOrderRepository.existsById(buyOrder.getId())) {
        if (optionalOrderBook.isPresent()) {
            orderBook = optionalOrderBook.get();
            orderBook.setVolume(orderBook.getVolume().add(buyOrder.getVolume()));
            orderBookRepository.save(orderBook);

        } else {
            if (!orderBookList.isEmpty()) {
                if (orderBookList.size() &lt; 10) {
                    orderBook = new OrderBook();
                    orderBook.setOrderType(buyOrder.getOrderType());
                    orderBook.setMarket(buyOrder.getMarket());
                    orderBook.setVolume(buyOrder.getVolume());
                    orderBook.setPrice(buyOrder.getPrice());
                    orderBookRepository.save(orderBook);

                } else if (orderBookList.size()==10){

                    int compare = buyOrder.getPrice().compareTo(orderBookList.get(0).getPrice());
                    if (compare &gt; 0) {
                        orderBookRepository.deleteById(orderBookList.get(0).getId());
                        orderBook = new OrderBook();
                        orderBook.setMarket(buyOrder.getMarket());
                        orderBook.setOrderType(buyOrder.getOrderType());
                        orderBook.setVolume(buyOrder.getVolume());
                        orderBook.setPrice(buyOrder.getPrice());
                        orderBookRepository.save(orderBook);
                    }
                }
            } else {
                orderBook = new OrderBook();
                orderBook.setMarket(buyOrder.getMarket());
                orderBook.setPrice(buyOrder.getPrice());
                orderBook.setOrderType(buyOrder.getOrderType());
                orderBook.setVolume(buyOrder.getVolume());
                orderBookRepository.save(orderBook);
            }
        }
    }
    sendOrderBookToSocket(buyOrder.getMarket());
}
</code></pre>
",https://stackoverflow.com/questions/72710329/sos-java-mongo-concurrency-issue,"['java', 'mongodb', 'concurrency']",39,False,0,0
Is there any way in debian/ubuntu to put CPU under load until it reaches a certain temperature?,"<p>I need this for some stress testing with my CPU. It uses Linux-Debian/Ubuntu OS and I was wondering if there's any way that I could put it under load until it reaches a certain temperature.</p>
<p>Are there any commands, packages or bash scripts for this?</p>
<p>Any help is appreciated!</p>
",https://stackoverflow.com/questions/73150890/is-there-any-way-in-debian-ubuntu-to-put-cpu-under-load-until-it-reaches-a-certa,"['linux', 'ubuntu', 'debian', 'cpu', 'temperature']",280,True,-1,2
Kafka Generates Events Based on Schema,"<p>Are there any tools service which will allow users to generate Kafka events based on specific schema ? Wanted to do stress testing on my Kafka Topic.</p>
<p>There is one tool i found : <a href=""https://github.com/confluentinc/kafka-connect-datagen"" rel=""nofollow noreferrer"">kafka-connect-datagen</a>, what here we can't configure the bootstrap server as my topic are spread across multiple cluster ? Looking for some recommendations</p>
",https://stackoverflow.com/questions/74313266/kafka-generates-events-based-on-schema,"['apache-kafka', 'stress-testing']",11,False,0,1
mysqlslap doubles the sleep time,"<p>I'm stress testing our database with a dummy query: <code>Select SLEEP(2);</code>.
Therefore I would expect the queries to return in around 2 seconds. However mysqlslap says it would take 4 seconds to run the query.
Despite saying that, it returns the results after 2 seconds. Is that a bug, or am I doing something wrong?</p>
<p>The time is always doubled, so <code>Select SLEEP(3);</code> results in 6 seconds.</p>
<p>To reproduce:</p>
<pre><code>mysqlslap --user=root --password=xxxxxxx --create-schema=svm_test --query=&quot;SELECT SLEEP(2);&quot; --iterations=5  --host=&quot;127.0.0.1&quot; --port=3307 --concurrency=1

Benchmark
Average number of seconds to run all queries: 4.009 seconds
Minimum number of seconds to run all queries: 4.000 seconds
Maximum number of seconds to run all queries: 4.016 seconds
Number of clients running queries: 1
Average number of queries per client: 1
</code></pre>
<p>I'm running MySQL 5.7.33 on Windows.</p>
",https://stackoverflow.com/questions/71939493/mysqlslap-doubles-the-sleep-time,['mysql'],50,False,0,0
Change Stream MongoDB Measuring Time,"<p>I have a tiny little question here. When listening on a Mongo DB change stream, is there any possibility of getting an update/create time for the changed object? The problem is that I can write an object with the current time, but that will give me the time for writing/updating an object and I need the time specifically from MongoDB. This is to perform stress testing, but writing time can make all the results a bit skewed.</p>
<p>Thanks in advance!</p>
",https://stackoverflow.com/questions/74013031/change-stream-mongodb-measuring-time,"['mongodb', 'changestream']",14,False,0,0
Slow Binary Search Tree Insertion,"<p>I have been creating some data structures to keep my skills sharp. I created a BST and have been stress testing its speed against an array. I noticed that the insertion speed for the BST is far slower than the array.push. I have been using Math.random and adding millions of numbers to both data structures. Oddly the BST is much faster at finding a value then the array.includes/indexOf. Is there a better way to code my insert function or is inserting just a slow part with BST</p>
<p>Here is my code</p>
<pre><code>insert(data) {
    if(this.root === null) {
        this.root = new Node(data)
        return this
    } 
    let current = this.root
    while(current) {
        if(current.data === data) {
            console.log(data + ' Already exists in tree')
            return
        }
        if(data &lt; current.data) {
            if(current.left === null) {
                current.left = new Node(data)
                this.size++
                return this
            } 
            current = current.left
        }
        if(data &gt; current.data) {
            if(current.right === null) {
                current.right = new Node(data)
                this.size++
                return this
            }
            current = current.right
        }
    }
}
</code></pre>
",https://stackoverflow.com/questions/70838839/slow-binary-search-tree-insertion,"['javascript', 'binary-tree']",299,True,1,1
How to Perform Stress/Load test on a chrome extension using Jmeter?,"<p>The title explains it.</p>
<p>I have a chrome extension that I have been working on, which shows relevant dcouments and data from a DB related to the webpage open on the main tab. And now I have to stress test it by checking how many users can it handle at once after logging in and clicking on the search &quot;All&quot; documents.</p>
<p>I have been trying to find some good tutorials, but all i get is testing using different jmeter extensions for chrome.</p>
<p>If jmeter can't be used for stress testing an extension, can you share a better alternative for the task?</p>
",https://stackoverflow.com/questions/72990635/how-to-perform-stress-load-test-on-a-chrome-extension-using-jmeter,"['automated-tests', 'load-testing']",478,False,1,1
What is the proper tools and technics to analyze a core dump file in Linux,"<p>I'm not asking how to find the cause of the crash. Actually there is no crash at all. I can't exclude the possibility of memory leaking but the executable passed Valgrind analysis in the stress testing. However, when it's running in the cloud, with much load, it gradually consumed much memory. Devop had to use kill -6 pid to kill the process and generated a core dump file, then restarted it. With that core dump, what are the good tools and technics to help you locate which part of the code contributed to the very high memory consumption? Thanks!</p>
",https://stackoverflow.com/questions/70628526/what-is-the-proper-tools-and-technics-to-analyze-a-core-dump-file-in-linux,"['linux', 'memory-leaks', 'coredump']",345,False,0,0
Raku NativeCall to Rust FFI CArray Size Limitations - SegFault,"<p>Below is a Minimum(?) Reproducible Example of my code. It is a first pass at stress testing / benchmarking the WIP Raku Module Dan::Polars.</p>
<p>In Rust, I make a libmre.so with this code</p>
<pre><code>  1 use libc::c_char;
  2 use libc::size_t;
  3 use std::slice;
  4 use std::ffi::*; //{CStr, CString,}
  5 
  6 //  Container
  7 
  8 pub struct VecC {
  9     ve: Vec::&lt;String&gt;,
 10 }   
 11 
 12 impl VecC {
 13     fn new(data: Vec::&lt;String&gt;) -&gt; VecC               
 14     {   
 15         VecC {
 16             ve: data,
 17         }   
 18     }   
 19     
 20     fn show(&amp;self) {
 21         println!{&quot;{:?}&quot;, self.ve};
 22     }   
 23 }   
 24     
 25 #[no_mangle]
 26 pub extern &quot;C&quot; fn ve_new_str(ptr: *const *const c_char, len: size_t)                        
 27     -&gt; *mut VecC {    
 28     
 29     let mut ve_data = Vec::&lt;String&gt;::new();
 30     unsafe {
 31         assert!(!ptr.is_null());
 32         
 33         for item in slice::from_raw_parts(ptr, len as usize) {
 34             ve_data.push(CStr::from_ptr(*item).to_string_lossy().into_owned());
 35         };  
 36     };  
 37     
 38     Box::into_raw(Box::new(VecC::new(ve_data)))
 39 }   
 40 
 41 #[no_mangle]
 42 pub extern &quot;C&quot; fn ve_show(ptr: *mut VecC) {
 43     let ve_c = unsafe {
 44         assert!(!ptr.is_null());
 45         &amp;mut *ptr
 46     };  
 47     
 48     ve_c.show();
 49 }  
</code></pre>
<p>and this Cargo.toml</p>
<pre><code>  1 [package]
  2 name = &quot;mre&quot;
  3 version = &quot;0.1.0&quot;
  4 edition = &quot;2021&quot;
  5 
  6 # See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
  7 
  8 [dependencies]
  9 libc = &quot;0.2.126&quot;
 10 
 11 [lib]
 12 name = &quot;mre&quot;
 13 path = &quot;src/lib.rs&quot;
 14 crate-type = [&quot;cdylib&quot;]
</code></pre>
<p>In Raku, I consume libmre.so like this</p>
<pre><code>  1 #!/usr/bin/env raku
  2 use lib '../lib';
  3 
  4 use NativeCall;
  5 
  6 #my $output;    #mre tried to move decl to here to avoid going out of scope
  7 sub carray( $dtype, @items ) {
  8     my $output := CArray[$dtype].new();
  9     loop ( my $i = 0; $i &lt; @items; $i++ ) {
 10         $output[$i] = @items[$i]
 11     }
 12     say $output;
 13     $output
 14 }   
 15     
 16 ### Container Classes that interface to Rust lib.rs ###
 17 
 18 constant $n-path    = '../mre/target/debug/mre';
 19 
 20 class VecC is repr('CPointer') is export {
 21     sub ve_new_str(CArray[Str],size_t) returns VecC is native($n-path) { * }
 22     sub ve_show(VecC) is native($n-path) { * }
 23     
 24     method new(@data) { 
 25         ve_new_str(carray(Str, @data), @data.elems );
 26     }   
 27     
 28     method show {
 29         ve_show(self)
 30     }   
 31 }   
 32 
 33 my \N = 100;   #should be 2e9  #fails between 30 and 100
 34 my \K = 100;   
 35 
 36 sub randChar(\f, \numGrp, \N) {
 37     my @things = [sprintf(f, $_) for 1..numGrp];
 38     @things[[1..numGrp].roll(N)];
 39 }   
 40 
 41 my @data = [randChar(&quot;id%03d&quot;, K, N)];
 42 
 43 my $vec = VecC.new( @data );
 44 $vec.show;
</code></pre>
<p>When \N is &lt;30 this runs fine with output like this:</p>
<pre><code>NativeCall::Types::CArray[Str].new
[&quot;id098&quot;, &quot;id035&quot;, &quot;id024&quot;, &quot;id067&quot;, &quot;id051&quot;, &quot;id025&quot;, &quot;id024&quot;, &quot;id092&quot;, &quot;id044&quot;, &quot;id042&quot;, &quot;id033&quot;, &quot;id004&quot;, &quot;id100&quot;, &quot;id091&quot;, &quot;id087&quot;, &quot;id059&quot;, &quot;id031&quot;, &quot;id063&quot;, &quot;id019&quot;, &quot;id035&quot;]
</code></pre>
<p>When \N is &gt; 50 however, I get:</p>
<pre><code>NativeCall::Types::CArray[Str].new
Segmentation fault (core dumped)
</code></pre>
<p>This is:</p>
<pre><code>Welcome to Rakudo™ v2022.04.
Implementing the Raku® Programming Language v6.d.
Built on MoarVM version 2022.04.
on ubuntu
</code></pre>
<p>Since the benchmark calls for \N to be 2e9, I could use some help to try and resolve this.</p>
<p>You are welcome to use p6steve/raku-dan:polars-2022.02-arm64 (or -amd64) on Docker Hub if you would like to try this at home. Don't forget to go <code>cargo build</code> first time. This includes <code>RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | bash -s -- -y </code></p>
",https://stackoverflow.com/questions/72881881/raku-nativecall-to-rust-ffi-carray-size-limitations-segfault,"['rust', 'raku', 'ffi']",120,True,6,1
Analyse JMeter stress test results,"<p>I am new to jmeter tool and have created a script for stress testing ElasticSearch server.</p>
<p>I know that this question has been asked here earlier but I am still not able to figure out what is the significance of the different parameters in the Jmeter summary report ?</p>
<p>Here are my JMeter setup:</p>
<p>No. of threads: 4000</p>
<p>Ramp-up period : 5</p>
<p>Loop Count: 1</p>
<p>Results :</p>
<p>Average:108782</p>
<p>Min: 5757</p>
<p>Max: 258917</p>
<p>Throughput: 15.4/sec</p>
<p>Can someone please explain what does each of the terms in the response column of the image below signify. Also, which metric can I use to gauge my server capacity.</p>
<p><a href=""https://i.stack.imgur.com/xJBom.png"" rel=""nofollow noreferrer""><img src=""https://i.stack.imgur.com/xJBom.png"" alt=""Result Summary"" /></a></p>
",https://stackoverflow.com/questions/74140192/analyse-jmeter-stress-test-results,"['java', 'elasticsearch', 'jmeter', 'stress-testing']",159,False,0,1
Stress testing mobile applications outline (iOS and Android) as a beginner using Jmeter?,"<p>I'm looking into how to use jmeter (I've done manual testing until now) to perform stress testing on mobile applications.</p>
<p>In particular, I have a student management kind of app. Both (iOS - Swift) and (Android - Java) app are available.
I want to mimic lots of student trying to register for same course, leaving same course, paying etc. I'm not sure where to start, or how to even configure jmeter.
What should I request from the devs? Is android apk + ios testflight version enough or should I request full source code?
I feel lost as this is my first proper tech related job while in college, but the lack of resources is making me afraid.</p>
",https://stackoverflow.com/questions/73870294/stress-testing-mobile-applications-outline-ios-and-android-as-a-beginner-using,"['android', 'ios', 'testing', 'jmeter']",32,True,0,1
Optimize retrieval of records when quering between multiple collections in marklogic,"<p>I am trying to query between 2 collections having a join condition between them,
Say the 2 documents are:</p>
<p>Collection 1: Company (1M documents)</p>
<p>Company Documents:</p>
<pre><code>&lt;Company&gt;
 &lt;CompanyName&gt;XYZ&lt;/CompanyName&gt;  
 &lt;Description&gt;ABC&lt;/Description&gt;
  .
  .
&lt;/Company&gt;
</code></pre>
<p>Collection 2: Employee (1M documents)</p>
<p>XML:</p>
<pre><code>&lt;Employee&gt;
 &lt;EmpName&gt;john&lt;/EmpName&gt;  
 &lt;CompanyName&gt;XYZ&lt;/CompanyName&gt;  
  .
  .
&lt;/Employee&gt;
</code></pre>
<p>The Query I am using is,</p>
<pre><code>for $company in collection('Company'), $employee in collection('Employee')
where ( $company/Company/CompanyName eq $employee/Employee/CompanyName)
return document-uri($company)
</code></pre>
<p>when there are less documents in both collections, able to fetch the records,
but when i do stress testing say, 1M records in both collections, in this case takes a lot of time and results in time limit exceeded exception. Is there a way to optimize this query?</p>
<p>I have tried the following methods,
assuming there exists a range index for CompanyName,</p>
<pre><code>for $company in collection('Company'), $employee in cts:values(cts:path-reference('/Employee/CompanyName'))
where ( $company/Company/CompanyName eq $employee)
return document-uri($company)
</code></pre>
<p>if companyName is unique then the above Query will not optimize much. I need a way which can work for both distinct and duplicate value of elements</p>
",https://stackoverflow.com/questions/74623487/optimize-retrieval-of-records-when-quering-between-multiple-collections-in-markl,"['xml', 'xquery', 'marklogic', 'marklogic-8', 'marklogic-9']",72,True,3,1
How can i seperate output in two different file in c++ (Stress Testing),"<p>A few days ago, I learned about stress testing and wanted to ask how to make two different .txt files.</p>
<p>Like one for Output and one for my failed test case.</p>
<p>It would work like this:</p>
<p>output: it would have all the output, correct solution, and wrong solution</p>
<p>failedtestcase.txt: it would have only those test cases detail that have failed and which inputs were given for the particular failed test case.</p>
<p>`</p>
<pre><code>int main() {
    freopen(&quot;output.txt&quot;, &quot;w&quot;, stdout);
    //create a freopen for Failed test case

    srand(time(NULL));
    int t = rand() % 10;
    cout&lt;&lt; &quot;======================================&quot; &lt;&lt; endl;
    cout &lt;&lt; &quot;            Test Case: &quot;&lt;&lt; t &lt;&lt; &quot;           &quot;&lt;&lt;  endl;
    cout&lt;&lt; &quot;======================================&quot; &lt;&lt; endl;
  
    while(t--){
        int n = rand() % 11;
        cout &lt;&lt; &quot;Number of Element: &quot; &lt;&lt;  n &lt;&lt; endl;
        int d = rand() % n;
        cout &lt;&lt; &quot;Kth: &quot; &lt;&lt; d &lt;&lt; endl;
        cout &lt;&lt; &quot;Araray: &quot; &lt;&lt; endl;
        int arr[n];
        for(int i=0; i&lt; n;i++){
            arr[i] = rand() % 11;
            cout&lt;&lt; arr[i] &lt;&lt; &quot; &quot;;
        }
        sort(arr, arr+n);
        cout &lt;&lt; endl;

        KthSAndL(arr, n, d);

        freopen(&quot;FailedTestcase.txt&quot;, &quot;w&quot;, stdout);

        pair&lt;int, int&gt; MySolution = KthSAndL(arr, n, d);
        pair&lt;int, int&gt; ActualSolution = kthSmallestLargest(arr, n, d);

        if(MySolution.first != ActualSolution.first &amp;&amp; MySolution.second != ActualSolution.second){
            cout &lt;&lt; &quot;Wrong Answer&quot; &lt;&lt; endl;
            cout &lt;&lt; &quot;Your Answer: &quot; &lt;&lt; MySolution.first &lt;&lt; &quot; &quot; &lt;&lt; MySolution.second &lt;&lt; endl;
            cout &lt;&lt; &quot;Actual Answer: &quot; &lt;&lt; ActualSolution.first &lt;&lt; &quot; &quot; &lt;&lt; ActualSolution.second &lt;&lt; endl;
        }

        cout &lt;&lt; endl;
        cout&lt;&lt; &quot;======================================&quot; &lt;&lt; endl;
    }

    return 0;
}
</code></pre>
<p>`</p>
<p>This is the code and i'm getting this Kind of output 😑</p>
<p><a href=""https://i.stack.imgur.com/RxTMj.png"" rel=""nofollow noreferrer""><img src=""https://i.stack.imgur.com/RxTMj.png"" alt=""enter image description here"" /></a></p>
<p>I was trying to have my all output in output.txt and only failed test case inputs will be shows in failedtestcase.txt</p>
<p>What I tried:</p>
<ol>
<li>Moving the Lines up down before cout'ing (Might work I thought)</li>
</ol>
<p>and well actully I have no idea what I can do 🤷</p>
",https://stackoverflow.com/questions/74299460/how-can-i-seperate-output-in-two-different-file-in-c-stress-testing,"['c++', 'c++11', 'c++17', 'clion', 'stress-testing']",56,False,0,0
Web Socket: IIS WebSocket Protocol feature need to be enabled for Socket.io?,"<p>For one of my application, I have implemented Web socket using socket.io and hosted in IIS. Currently socket connection helping to provide two way connection between client (React) and server (node.js).</p>
<p>As I mentioned, I have hosted my application in IIS. I have few doubts regarding Turn Windows features on or off -&gt; Internet Information services -&gt; World wide Web Services -&gt; Application Development Features -&gt; WebSocket Protocol. I have tested my application without enabling this feature, its working fine, but I would like to confirm below stuffs.
<a href=""https://i.stack.imgur.com/MzU55.png"" rel=""nofollow noreferrer"">WebSocket Protocol</a></p>
<ol>
<li><p>Do I really need to enable WebSocket protocol feature to make websocket work in my application? If so, how now it is working fine without enabling (I haven't do performance and stress testing, I may face issue on this).</p>
</li>
<li><p>What if I not enabling this feature? in short what is the actual use of this feature?</p>
</li>
</ol>
<p>It would be helpful if anyone answer the above questions. Thanks in advance.</p>
",https://stackoverflow.com/questions/71285309/web-socket-iis-websocket-protocol-feature-need-to-be-enabled-for-socket-io,"['sockets', 'iis', 'websocket', 'socket.io']",1226,True,0,1
Objects reuse when working with Microsoft Information Protection SDK,"<h2>TL;DR</h2>
<p>Is it safe to share MIP File Engine object between multiple threads, each will use it to read a label from a different file?</p>
<h2>The Longer Version</h2>
<p>I'm working with Microsoft Information Protection SDK (from now on, I'll call it MIP) in order to develop a C# service that checks whether files are correctly labeled.</p>
<p>In order to check this, I need to create 3 objects:</p>
<ol>
<li>MIP Context</li>
<li>Profile Engine</li>
<li>File Engine</li>
</ol>
<p>Microsoft documentation says <a href=""https://learn.microsoft.com/en-us/information-protection/develop/concept-mipcontext"" rel=""nofollow noreferrer"">regarding the MIP context</a> that every process should construct only one context and it should reuse it, but, regarding the other two, I wasn't able to find any mention regarding concurrency / thread safeness / objects reuse.</p>
<p>It is expensive to create them for each request I'm serving. It can take up to 2-3 seconds just to construct the file engine.</p>
<p><strong>What I tested?</strong>
I assumed it is thread safe and used those objects concurrently while stress testing the system and it looks like everything works just fine, which is promising but I need more than that.</p>
<p>Thanks for the help!</p>
",https://stackoverflow.com/questions/73809971/objects-reuse-when-working-with-microsoft-information-protection-sdk,['microsoft-information-protection'],62,False,0,0
self hosted ASP.NET Core Web Api capable of starting and stopping,"<p>I'm attempting to convert an old WCF service to an ASP.NET Core Web API, making use of the CoreWCF package. A key feature of this existing service is that it's being self hosted by an other application and is able to gracefully start &amp; stop, without creating memory leaks.</p>
<p>I have been able to figure out how to start and stop a prototype service. However, after performing some stress testing, it does seem like I've left a memory leak somewhere and I'm sadly out of ideas or available documentation at this point. I'm also considering that an ASP.NET Core Web API just isn't supposed to be used like this and I misunderstood this, if so, be sure to let me know. Also my apologies for the truckload of code, but I'm not sure what's relevant or not to the question.</p>
<p>The code for my prototype service looks like this:</p>
<p>Configuring the webhost:</p>
<pre><code>private void CreateWebHostBuilder(){
    host = WebHost.CreateDefaultBuilder()
        .UseKestrel(options =&gt;
        {
            options.AllowSynchronousIO = true;
            options.ListenLocalhost(Startup.PORT_NR);
            options.ConfigureHttpsDefaults(
             options =&gt; options.ClientCertificateMode = ClientCertificateMode.RequireCertificate
            );
        })
        .ConfigureLogging(logging =&gt; { logging.SetMinimumLevel(LogLevel.Warning); })
        .UseSetting(WebHostDefaults.DetailedErrorsKey, &quot;true&quot;)
        .UseShutdownTimeout(TimeSpan.FromSeconds(1))
        .UseStartup&lt;Startup&gt;()
        .Build();
}
</code></pre>
<p>Inside the <code>Startup</code> class:</p>
<p>Configuring the <code>IApplicationBuilder</code>:</p>
<pre><code>public void Configure(IApplicationBuilder app){
    app.UseServiceModel(builder =&gt;
    {
        // Add the Echo Service
        builder.AddService&lt;EchoService&gt;()                
        // Add service web endpoint                
        .AddServiceWebEndpoint&lt;EchoService, IEchoService&gt;(
            WEB_API_PATH,behavior =&gt; { behavior.HelpEnabled = true;}
        );
     });            
     app.UseMiddleware&lt;SwaggerMiddleware&gt;();            
     app.UseSwaggerUI();
     app.UseAuthentication();
}
</code></pre>
<p>Configuring the services:</p>
<pre><code>public void ConfigureServices(IServiceCollection services){
    services.AddServiceModelWebServices()                
            .AddHostedService&lt;EchoService&gt;()                       
            .AddSingleton(new SwaggerOptions())                  
            .AddAuthentication(CertificateAuthenticationDefaults.AuthenticationScheme)
            .AddCertificate();            
}
</code></pre>
<p>The service interface:</p>
<pre><code>[ServiceContract]
[OpenApiBasePath($&quot;/{Startup.WEB_API_PATH}&quot;)]
public interface IEchoService : IHostedService {
    [OperationContract]
    [WebGet(UriTemplate = &quot;/hello&quot;)]
    [OpenApiOperation(Description = &quot;Method used to receive a friendly \&quot;Hello world\&quot;&quot;,
        Summary = &quot;Hello world&quot;)]
    [OpenApiResponse(Description = &quot;OK Response&quot;, StatusCode = HttpStatusCode.OK)]
    string HelloWorld();        
}
</code></pre>
<p>The implemented service:</p>
<pre><code>public class EchoService : IEchoService {        
    public EchoService() { }

    public string HelloWorld() {
        return &quot;Hello world!&quot;;
    }        
    public Task StartAsync(CancellationToken cancellationToken) {
        return Task.CompletedTask;
    }

    public Task StopAsync(CancellationToken cancellationToken)
    {            
        return Task.CompletedTask;
    }        
}
</code></pre>
<p>Creating and starting the host + services:</p>
<pre><code>public void StartWebService(object obj){
    CreateWebHostBuilder();
    host.StartAsync();
}
</code></pre>
<p>Stopping and disposing the services and host:</p>
<pre><code>public void StopWebService(object obj) {
    host.StopAsync().Wait();
    host.Dispose();                            
}
</code></pre>
<p>So if anyone has any suggestions or tutorial reference, be sure to let me know, any help is welcome.</p>
",https://stackoverflow.com/questions/74570117/self-hosted-asp-net-core-web-api-capable-of-starting-and-stopping,"['asp.net-web-api', 'memory-leaks', 'self-host-webapi', 'corewcf']",138,False,1,0
Where is the incerased RSS memory in java process?,"<p>We have a spring cloud system running in K8S enviroment, and I do stress testing on one of service for finding whethere memory leak exist in the service.</p>
<p>There are 3 tools we use to monitor memory change in the server's process:<br />
1 Prometheus and Grafana<br />
2 &quot;top&quot; (top -p )<br />
3 java NMT tool (jcmd  VM.native_memory summary)</p>
<p>During the testing,  found that there is a confused report when RSS memory changed apparently:<br />
From report of Grafana, the RSS increased 1% about 10MB, that coninstance with the &quot;top&quot; output basicly.<br />
BUT from the NMT output, there seems total committed memory decrease 1MB instead of increasing with RSS.</p>
<p>For example:</p>
<pre><code>&quot;top&quot; output is:

2022/10/19 16:00
RSS     VSZ
**855276**  7108367

2022/10/19 16:05
RSS     VSZ
**862876**  7108367
</code></pre>
<pre><code>NMT output is：
2022/10/19 16:00
Total: reserved=2363158KB, committed=**869232KB**

2022/10/19 16:05
Total: reserved=2362056KB, committed=**868365KB**
</code></pre>
<p>I have already known that committed memory in java might not equals to RSS in Linux. But the confused point is when committed memory decreasing in JVM, the RSS is incresing conversely.
Where is the increasing memory got from &quot;top&quot;, because NMT do not give the answer, seems the incereasing memory not produced by JVM?</p>
<p>The other messages of the system:<br />
spring boot 2.1.3.RELEASE<br />
spring cloud 2.1.3.RELEASE<br />
JDK 1.8.0_202<br />
Linux 4.19<br />
GCC 7.3.0<br />
GLibc 2.17</p>
",https://stackoverflow.com/questions/74119656/where-is-the-incerased-rss-memory-in-java-process,"['linux', 'memory', 'jvm']",88,False,0,0
Fastest way to call a function millions of times in Python,"<p>I have a function <code>readFiles</code> that I need to call 8.5 million times (essentially stress-testing a logger to ensure the log rotates correctly). I don't care about the output/result of the function, only that I run it N times as quickly as possible.</p>
<p>My current solution is this:</p>
<pre><code>from threading import Thread
import subprocess

def readFile(filename):
    args = [&quot;/usr/bin/ls&quot;, filename]
    subprocess.run(args)

def main():
    filename = &quot;test.log&quot;
    threads = set()
    for i in range(8500000):
        thread = Thread(target=readFile, args=(filename,)
        thread.start()
        threads.add(thread)
    
    # Wait for all the reads to finish
    while len(threads):
        # Avoid changing size of set while iterating
        for thread in threads.copy():
            if not thread.is_alive():
                threads.remove(thread)
</code></pre>
<p><code>readFile</code> has been simplified, but the concept is the same. I need to run readFile 8.5 million times, and I need to wait for all the reads to finish. Based on my mental math, this spawns ~60 threads per second, which means it will take ~40 hours to finish. Ideally, this would finish within 1-8 hours.</p>
<p>Is this possible? Is the number of iterations simply too high for this to be done in a reasonable span of time?</p>
<p>Oddly enough, when I wrote a test script, I was able to generate a thread about every ~0.0005 seconds, which should equate to ~2000 threads per second, but this is not the case here.</p>
<p>I considered iteration 8500000 / 10 times, and spawning a thread which then runs the readFile function 10 times, which should decrease the amount of time by ~90%, but it caused some issues with blocking resources, and I think passing a lock around would be a bit complicated insofar as keeping the function usable by methods that don't incorporate threading.</p>
<p>Any tips?</p>
",https://stackoverflow.com/questions/71624190/fastest-way-to-call-a-function-millions-of-times-in-python,"['python', 'multithreading', 'performance', 'multiprocessing']",372,False,0,2
Node.js vs ASP.Net Core: Response times when doing I/O heavy operations under stress test?,"<p>Let's assume we are stress testing 2 servers that perform database read/write operations without caching, and make network calls. One is a Node.js server, and the other is an ASP.Net Core one that utilizes Tasks. Assume that both perform the same operations, receive the same requests/second, and run on machines with equal processing power.</p>
<p>I've read that Node.js is very performant on I/O operations, but I can't get my head around it. Sure, the awaited db and network calls run in a non-blocking way, however each of these calls are handled by an individual thread from a limited thread pool. This doesn't sound performant to me at all. I would be forced to implement a caching mechanism to mitigate this, which is something I don't really like. So either Node.js is not the greatest choice for these kinds of operations, or I have incorrect knowledge.</p>
<p>As for ASP.NET Core, I don't know the internals of it, but I'm pretty sure it doesn't have the thread limitation issues Node.js has, so it should have shorter response times by logic. Yet I still can't know for sure if this is the case due to resource consumption and context switching cost concerns.</p>
<p>So which of these 2 would theoretically have shorter response times, and why exactly?</p>
",https://stackoverflow.com/questions/72664821/node-js-vs-asp-net-core-response-times-when-doing-i-o-heavy-operations-under-st,"['node.js', 'performance', 'asp.net-core', 'asynchronous', 'io']",311,False,0,0
Why do my TCP clients fail to connect to my server with a socket exception when stress-testing it with many connections,"<p>When stress-testing a TCP server of mine with a large number of connections, I figured that connection requests will throw a SocketException after some time. The exception displays randomly either</p>
<blockquote>
<p>Only one usage of each socket address (protocol/network address/port) is normally permitted.</p>
</blockquote>
<p>or</p>
<blockquote>
<p>No connection could be made because the target machine actively refused it.</p>
</blockquote>
<p>as its message.</p>
<p>This happens usually but randomly after some seconds and some tens of thousands of connects and disconnects. To connect, I use the local end point <code>IPEndPoint clientEndPoint = new(IPAddress.Any, 0);</code>, which I believe would give me the next free ephemeral port.</p>
<p>To isolate the issue, I wrote this simple program that runs both a TCP server and many parallel clients for a simple counter:</p>
<pre><code>using System.Diagnostics;
using System.Net;
using System.Net.Sockets;

CancellationTokenSource cancellationTokenSource = new();
CancellationToken cancellationToken = cancellationTokenSource.Token;

const int serverPort = 65000;
const int counterRequestMessage = -1;
const int randomCounterResponseMinDelay = 10; //ms
const int randomCounterResponseMaxDelay = 1000; //ms
const int maxParallelCounterRequests = 10000;

#region server

int counterValue = 0;

async void RunCounterServer()
{
    TcpListener listener = new(IPAddress.Any, serverPort);
    listener.Start(maxParallelCounterRequests);
    while (!cancellationToken.IsCancellationRequested)
    {
        HandleCounterRequester(await listener.AcceptTcpClientAsync(cancellationToken));
    }

    listener.Stop();
}

async void HandleCounterRequester(TcpClient client)
{
    await using NetworkStream stream = client.GetStream();
    Memory&lt;byte&gt; memory = new byte[sizeof(int)];

    //read requestMessage
    await stream.ReadAsync(memory, cancellationToken);
    int requestMessage = BitConverter.ToInt32(memory.Span);
    Debug.Assert(requestMessage == counterRequestMessage);

    //increment counter
    int updatedCounterValue = Interlocked.Add(ref counterValue, 1);
    Debug.Assert(BitConverter.TryWriteBytes(memory.Span, updatedCounterValue));

    //wait random timeout
    await Task.Delay(GetRandomCounterResponseDelay());

    //write back response
    await stream.WriteAsync(memory, cancellationToken);

    client.Close();
    client.Dispose();
}

int GetRandomCounterResponseDelay()
{
    return Random.Shared.Next(randomCounterResponseMinDelay, randomCounterResponseMaxDelay);
}

RunCounterServer();

#endregion

IPEndPoint clientEndPoint = new(IPAddress.Any, 0);
IPEndPoint serverEndPoint = new(IPAddress.Parse(&quot;127.0.0.1&quot;), serverPort);
ReaderWriterLockSlim isExceptionEncounteredLock = new(LockRecursionPolicy.NoRecursion);
bool isExceptionEncountered = false;

async Task RunCounterClient()
{
    try
    {
        int counterResponse;
        using (TcpClient client = new(clientEndPoint))
        {
            await client.ConnectAsync(serverEndPoint, cancellationToken);

            await using (NetworkStream stream = client.GetStream())
            {
                Memory&lt;byte&gt; memory = new byte[sizeof(int)];

                //send counter request
                Debug.Assert(BitConverter.TryWriteBytes(memory.Span, counterRequestMessage));
                await stream.WriteAsync(memory, cancellationToken);

                //read counter response
                await stream.ReadAsync(memory, cancellationToken);
                counterResponse = BitConverter.ToInt32(memory.Span);
            }

            client.Close();
        }

        isExceptionEncounteredLock.EnterReadLock();
        //log response if there was no exception encountered so far
        if (!isExceptionEncountered)
        {
            Console.WriteLine(counterResponse);
        }

        isExceptionEncounteredLock.ExitReadLock();
    }
    catch (SocketException exception)
    {
        bool isFirstEncounteredException = false;

        isExceptionEncounteredLock.EnterWriteLock();

        //log exception and note that one was encountered if it is the first one
        if (!isExceptionEncountered)
        {
            Console.WriteLine(exception.Message);
            isExceptionEncountered = true;
            isFirstEncounteredException = true;
        }

        isExceptionEncounteredLock.ExitWriteLock();

        //if this is the first exception encountered, rethrow it
        if (isFirstEncounteredException)
        {
            throw;
        }
    }
}

async void RunParallelCounterClients()
{
    SemaphoreSlim clientSlotCount = new(maxParallelCounterRequests, maxParallelCounterRequests);

    async void RunCounterClientAndReleaseSlot()
    {
        await RunCounterClient();
        clientSlotCount.Release();
    }

    while (!cancellationToken.IsCancellationRequested)
    {
        await clientSlotCount.WaitAsync(cancellationToken);
        RunCounterClientAndReleaseSlot();
    }
}

RunParallelCounterClients();

while (true)
{
    ConsoleKeyInfo keyInfo = Console.ReadKey(true);
    if (keyInfo.Key == ConsoleKey.Escape)
    {
        cancellationTokenSource.Cancel();
        break;
    }
}
</code></pre>
<p>My initial guess is, that I run out of ephemeral ports because I somehow do not free them correctly. I just <code>Close()</code> and <code>Dispose()</code> my <code>TcpClient</code>s in both my client and server code when a request is finished. I thought that would automatically release the port, but when I use <code>netstat -ab</code> in a console, it gives me countless entries like this, even after closing the application:</p>
<pre><code>TCP    127.0.0.1:65000        kubernetes:59996       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:59997       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:59998       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:59999       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60000       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60001       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60002       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60003       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60004       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60005       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60006       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60007       TIME_WAIT
TCP    127.0.0.1:65000        kubernetes:60009       TIME_WAIT
</code></pre>
<p>Also, my PC sometimes gets a lot of stutter some time after I exit the application. I assume this is due to Windows cleaning up my leaked port usage?</p>
<p>So I wonder, what am I doing wrong here?</p>
",https://stackoverflow.com/questions/71070443/why-do-my-tcp-clients-fail-to-connect-to-my-server-with-a-socket-exception-when,"['c#', 'sockets', '.net-core', 'port']",618,True,0,1
Creating and Disposing a LibVLC instance object multiple times causes program to exit?,"<p>C# console application built in VS2022 using LibVLCSharp 3.6.6 and LibVLC for Windows 3.0.17.4.</p>
<p>I'm stress-testing a RTSP client class that uses libVLCSharp to stream from an IOT camera. In order to check for memory leaks, I've been creating my client object, running the stream for a few seconds, then ending the stream and disposing of the client and all of the VLC components (the VLC instance, the media player and the media) before starting all over again.</p>
<p>I quickly found that around the 60-70th loop, the program inexplicably ends after trying to create the LibVLC() object. No crash, no freezing, no exceptions, just immediately exits the program after this line.</p>
<p><code>_VlcInstance = new LibVLC();</code></p>
<p>The only note it gives is this:</p>
<pre><code>RTSPTest.exe exited with code 3.
</code></pre>
<p>I couldn't find what VLC's exit code 3 is in the documentation and I don't think this is a memory leak issue. Been watching my memory as this has run and garbage collection appears to be doing its job nicely. Turning on verbosity also doesn't help, it gives no indication that anything is wrong and provides the same lines that comes with creating a LibVLC() object every time, right up until it ends suddenly.</p>
<p>Does anyone have a better idea of what's going on behind the scenes? The end-use for this client requires a lot of client objects to be created and discarded similar to this, so a sudden exit with no warnings or explanation could be a huge issue down the line.</p>
",https://stackoverflow.com/questions/73748843/creating-and-disposing-a-libvlc-instance-object-multiple-times-causes-program-to,"['c#', 'libvlc', 'libvlcsharp']",143,False,0,0
Where is the good promotion place to affect next-gen comm?,"<p>I'm working on forward error correction (FEC) / error correction code (ECC) technology
I hope this helps the next generation of communication systems with high reliability and low complexity.
One might think the easy way is to submit it to paper work.</p>
<p>But the problem is that professional researchers still keep the same level of verification twenty years ago. In the face of higher-speed applications in the future, stress testing should be strengthened. Everyone ignores the need to improve the verification level because it will affect the number of publications.</p>
<p>For the comparison with various technologies (turbo codes, ldpc, polar, Reed-solomon, BCH, convolutional code) of the current 5G-FEC, hard decoding should be used and the number of verifications must reach a level exceeding terabyte so that the speed of use will not fluctuate. However, based on historical factors, after finding the fact of soft decoding, soft decoding let the verification of the error correction code be lengthy. Take the movie as an example, Black Panthers know they have to take off their suits and compete with each other, but the field is still doing research with old ideas, so submitting paper work is not a good way to promote.</p>
<p>I made a video explaining these.   <a href=""https://youtu.be/xU6uBvKo8T8"" rel=""nofollow noreferrer"">https://youtu.be/xU6uBvKo8T8</a></p>
<p>Does anyone have an idea how to break the status?<br />
Of course, I have sent mail to the author of polar code.
But he is still missing the verification topics.</p>
",https://stackoverflow.com/questions/74530153/where-is-the-good-promotion-place-to-affect-next-gen-comm,"['telecommunication', 'error-correction', 'information-theory', 'forwarderrorcorrection']",7,False,0,0
How to Stress a Docker Prometheus Node exporter container,"<p>I'm trying to build an observability platform to be visualised on Grafana. I'm running all my docker containers which includes prometheus node exporter on AWS EC2 and I would like to stress this Prom Node exporter docker container. AWS EC2 does not display memory metrics on Grafana so in order to get the Mem metrics I spinned up a Docker Prometheus Node exporter on my AWS EC2 instance.</p>
<p>I have set up the Memory Utilisation for the Prometheus Node exporter and the defined threshold is 70% (If it exceeds 71% then it triggers an alert) so the purpose of the stress testing is to make the container exceed the defined threshold.</p>
<p>I installed pumba which is a stress command for docker containers and tried using the command below but this didn't work because I'm on a MacOS and the OS is does not support Cgroup.</p>
<pre><code>pumba stress -d 1m container_name
</code></pre>
<p>I later tried out several commands for stressing docker containers on my AWS EC2 but there was no visible changes seen on my Grafana widget for Prom-Node exporter memory utilisation running on the AWS EC2 instance.</p>
<pre><code>docker run --rm -it progrium/stress --cpu 2 --io 1 --vm 2 --vm-bytes 128M --timeout 10s
</code></pre>
<p>So I decided to run the same command on my localPC (I set up the containers -Prometheus, Grafana, node exporter etc) on my local Pc and tried stressing then I got. the visualisation which is attached below. My main is to achieve the same visualisation on my AWS EC2</p>
<p><a href=""https://i.stack.imgur.com/7iCHm.png"" rel=""nofollow noreferrer""><img src=""https://i.stack.imgur.com/7iCHm.png"" alt=""enter image description here"" /></a> Please do you have any idea how I stress a specific docker container?</p>
",https://stackoverflow.com/questions/73137746/how-to-stress-a-docker-prometheus-node-exporter-container,"['linux', 'amazon-web-services', 'docker', 'amazon-ec2', 'prometheus-node-exporter']",210,False,0,0
Create a good k6 load test with graphql/subscription,"<p>After struggling against k6, with the help of the answers to my other question, i was able (i think) to test subscriptions with websocket.</p>
<p>Now i'm trying to desing a good test. Our app is a Single Page App that mainly uses websocket subscriptions instead of queries. The problem is that i'm a bit lost on how to define how many subscriptiosn i should test in order to detect if my app will be able to support around 800 users working simultaneously in real time. (I started testing it first with a top of 150 VUs).</p>
<p>For now, i'm running the test with 3 commonly used subscriptions in our user path, but:</p>
<ol>
<li><p>Should I try more subscriptions, trying to cover the most used ones in our user path, or should I keep it simple and pick the 3 most used ones and add a timeout between them?</p>
</li>
<li><p>Is this a correct approach for load/stress testing with GRAPHQL subscriptions/websocket? I'm not sure if i'm being too cautious about this, but i'm afraid of giving a false status about our situation.</p>
</li>
<li><p>I am a bit confused on how to interpret the results screen, especially on how I can infer if we will be able to give a good experience to our users. Should I take the avg and p(95) of ws_ping as a reference for this?</p>
</li>
</ol>
<p>k6 screen result
<a href=""https://i.stack.imgur.com/vX89g.png"" rel=""nofollow noreferrer""><img src=""https://i.stack.imgur.com/vX89g.png"" alt=""enter image description here"" /></a></p>
<p>As a reference, here is part of the code i'm using to perform the test</p>
<p>Thanks in advance!</p>
<p>main.js</p>
<pre><code>import { Httpx } from 'https://jslib.k6.io/httpx/0.0.5/index.js';

const session = new Httpx({
  baseURL: `https://${enviorment}`
});

const wsUri = `wss://${enviorment}/v1/graphql`;

const pauseMin = 2;
const pauseMax = 6;

export const options = {
  stages: [
    { duration: '30s', target: 50 },
    { duration: '30s', target: 100 },
    { duration: '30s', target: 150 },
    { duration: '120s', target: 150 },
    { duration: '60s', target: 50 },
    { duration: '30s', target: 0 },
  ]
};

export default function () {
  session.addHeader('Content-Type', 'application/json');
  todas(keys.profesorFirebaseKey, keys.desafioCursoKey, keys.nivelCurso, keys.profesorKey)
}
</code></pre>
<p>todas.js:</p>
<pre><code>import ws from 'k6/ws';
import {fail,check} from 'k6'
import exec from 'k6/execution';

export function todas(id, desafioCursoKey, nivel, profesorKey) {

  const queryList = [`];

  let ArraySubscribePayload = []

  for (let i = 0; i &lt; 3; i++) {
    let subscribePayload = {
      id: String(1 + 3 * i),
      payload: {
        extensions: {},
        query: queryList[i],
        variables: {},
      },
      type: &quot;start&quot;,
    }
    ArraySubscribePayload.push(subscribePayload)
  }

  const initPayload = {
    payload: {
      headers: {
        xxxxxxxxxxxxx
      },
      lazy: true,
    },
    type: &quot;connection_init&quot;,
  };

  const res = ws.connect(wsUri, initPayload, function (socket) {

    socket.on('open', function () {

      socket.setInterval(function timeout() {

        socket.send(JSON.stringify(initPayload));

        socket.send(JSON.stringify(ArraySubscribePayload[0]));

        socket.setTimeout(function timeout() {
         
          socket.send(JSON.stringify(ArraySubscribePayload[1]));
          socket.send(JSON.stringify(ArraySubscribePayload[2]));
          ArraySubscribePayload[1].id = String(parseInt(ArraySubscribePayload[1].id) + 1)
          ArraySubscribePayload[2].id = String(parseInt(ArraySubscribePayload[2].id) + 1)
        }, 3000);

        ArraySubscribePayload[0].id = String(parseInt(ArraySubscribePayload[0].id) + 1)
        socket.ping();
      }, 7000);
    })
  });
}
</code></pre>
",https://stackoverflow.com/questions/70992153/create-a-good-k6-load-test-with-graphql-subscription,"['graphql', 'load-testing', 'stress-testing', 'graphql-subscriptions', 'k6']",517,False,1,0
FileReader Page Memory Leak,"<p>We are building a React SPA. The app allows you to submit a form with photos many times in a row. We started stress testing the app and realized that it stops working on iOS (iPad Mini) after several dozens of file uploads. Profiled the memory in Safari and found that what WebKit calls <a href=""https://webkit.org/blog/6425/memory-debugging-with-web-inspector/"" rel=""nofollow noreferrer"">Page Memory</a> keeps climbing whenever we use <code>FileReader.readAsArrayBuffer</code>, but it never seems to be released. At some point, the App becomes very slow (it never crashes, but connections to local IndexDB start to drop).</p>
<p>Here is the barebones application created with create-react-app:</p>
<pre><code>import FileReaderHandler from './FileReaderHandler';
import { useRef } from 'react';

function App() {
    const fileReaderHandler = useRef(new FileReaderHandler());

    const onSelectFiles = async (event) =&gt; {
        event.stopPropagation();
        event.preventDefault();
        const files = Array.from(event.target.files);
        for (const file of files) {
            await fileReaderHandler.current.readFile(file);
        }
    }

    return (
        &lt;div&gt;
            &lt;input multiple type=&quot;file&quot; accept=&quot;image/*&quot;
                onChange={event =&gt; onSelectFiles(event)}
                onClick={event =&gt; event.target.value = null}
            /&gt;
        &lt;/div&gt;
    );
}

export default App;
</code></pre>
<p><a href=""https://i.stack.imgur.com/7XVdC.png"" rel=""nofollow noreferrer""><img src=""https://i.stack.imgur.com/7XVdC.png"" alt=""App UI"" /></a></p>
<p>Here is the FileReaderHandler implementation:</p>
<pre><code>export default class FileReaderHandler {
    async readFile(file, fileReader) {
        return new Promise((resolve, reject) =&gt; {
            const fileReader = new FileReader();
            
            function resolveHandler () {
                cleanHandlers()
                resolve(fileReader.result)
            }

            function cleanHandlers () {
                fileReader.removeEventListener(&quot;loadend&quot;, resolveHandler);
            }

            fileReader.addEventListener(&quot;loadend&quot;, resolveHandler);

            fileReader.readAsArrayBuffer(file);
        })
    }
}
</code></pre>
<p>Here is the Safari timeline of me continuously selecting the same 10 images:
<a href=""https://i.stack.imgur.com/Ca49A.png"" rel=""nofollow noreferrer""><img src=""https://i.stack.imgur.com/Ca49A.png"" alt=""Safari memory timeline"" /></a></p>
<p>It makes no difference if the code handling the reading is in the component itself or in the class. I also tried using a singleton FileReader passed into the function from component, without much luck. Read online that one possible reason memory leaks can occur is due to not cleaning up event handlers, but as you can see in the code above - I do remove the listener.</p>
<p>I tried a version of this code without the Promise just in case the anonymous function somehow holds on to the references, but also no luck.</p>
<p>Finally, I also tried <code>URL.createObjectURL/revokeObjectURL</code>, and it has a similar effect on memory.</p>
<p>Refreshing the page is not an option, because the app needs to work offline and there are background processes running.</p>
<p>From what I understand from the WebKit link above, the memory is not on Heap, but rather some internal cache. Can someone with better understanding of WebKit memory management explain what is happening and how to prevent this?</p>
",https://stackoverflow.com/questions/72663368/filereader-page-memory-leak,"['javascript', 'memory-management', 'memory-leaks', 'webkit', 'html5-filesystem']",295,True,0,1
Spring/Tomcat closes the connection on exception,"<p>Recently I was debugging a problem that a client reported when they were stress-testing a spring application running in tomcat. They ran the tests, but most of the requests immediately resulted in an exception, causing a <code>500</code> response code.</p>
<p>At some point the client was no longer able to open new connections because the ephemeral port range was exhausted - all ports were in a <code>TIME_WAIT</code> state. The reason appears to be that when an exception occurs the connection is closed by either spring or tomcat.</p>
<p>I created a simple spring boot 2.7.6 application with a single controller, with two endpoints - <code>/5xx</code> which throws ResponseStatusException(HttpStatus.INTERNAL_SERVER_ERROR), and <code>/4xx</code>
to throw ResponseStatusException(HttpStatus.BAD_REQUEST).</p>
<p>After requesting these endpoints, the response contains a <code>Connection: close</code> header, and looking at the local ports with <code>ss</code> the ports in <code>TIME_WAIT</code> get increased.</p>
<p>Further, requesting an endpoint that doesn't exist - doesn't close the connection and simply returns 404.</p>
<p>When I switched to undertow, instead of tomcat - the connection wasn't closed for all endpoints.</p>
<p>When I switched to spring boot 3 and tomcat - the connection isn't closed as well for all three endpoints.</p>
<p>So, my questions are:</p>
<ul>
<li>What is the correct behavior? I'm thinking that the connection shouldn't be closed.</li>
<li>Does anyone know if this is a spring or tomcat feature and is there a configuration for this?</li>
</ul>
<p><strong>EDIT #1</strong>:</p>
<p>OS details:</p>
<ul>
<li>Ubuntu 22.04.1 LTS</li>
<li>Linux 5.15.0-56-generic #62-Ubuntu SMP Tue Nov 22 19:54:14 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux</li>
<li>net.ipv4.ip_local_port_range = 1024   60999</li>
</ul>
<p>Tomcat:</p>
<ul>
<li>embedded tomcat v9.0.69 with default configuration provided by spring boot</li>
<li>the situation improves by increasing <code>maxConnections</code> to 30000 (default is 10000)</li>
</ul>
",https://stackoverflow.com/questions/74841842/spring-tomcat-closes-the-connection-on-exception,"['spring', 'tomcat']",86,False,0,0
star-schema - fact table modeling,"<p>I need to model a star schema for some business needs about liquidity stress testing.</p>
<p>i will try to find an analogy example.
let's say we have deals about financing/financial securities etc</p>
<p>in the fact table,
at a given date, this deals have the real value of X euros
but will have a variation in time. thus we have some projection values.</p>
<p>my concern is about how to represent this projection values for this deals, and more specifically what granularity to choose.</p>
<p>( the example below is oversimplification of the fact table -- and yes it's Dimension Id's  that
are used otherwise )</p>
<p>Method 1 : as many metric columns as projection values calculated</p>
<div class=""s-table-container"">
<table class=""s-table"">
<thead>
<tr>
<th style=""text-align: left;"">AsofDate</th>
<th style=""text-align: center;"">DealId</th>
<th style=""text-align: right;"">0Day</th>
<th style=""text-align: right;"">1Day</th>
<th style=""text-align: right;"">7Days</th>
<th style=""text-align: right;"">1Month</th>
</tr>
</thead>
<tbody>
<tr>
<td style=""text-align: left;"">2022-01-01</td>
<td style=""text-align: center;"">financingDeal1</td>
<td style=""text-align: right;"">100</td>
<td style=""text-align: right;"">99</td>
<td style=""text-align: right;"">98</td>
<td style=""text-align: right;"">85</td>
</tr>
<tr>
<td style=""text-align: left;"">2022-01-01</td>
<td style=""text-align: center;"">financingDeal2</td>
<td style=""text-align: right;"">150</td>
<td style=""text-align: right;"">150</td>
<td style=""text-align: right;"">120</td>
<td style=""text-align: right;"">120</td>
</tr>
<tr>
<td style=""text-align: left;"">2022-01-01</td>
<td style=""text-align: center;"">financingDeal3</td>
<td style=""text-align: right;"">100</td>
<td style=""text-align: right;"">99</td>
<td style=""text-align: right;"">98</td>
<td style=""text-align: right;"">85</td>
</tr>
<tr>
<td style=""text-align: left;"">2022-01-01</td>
<td style=""text-align: center;"">financingDeal4</td>
<td style=""text-align: right;"">100</td>
<td style=""text-align: right;"">99</td>
<td style=""text-align: right;"">98</td>
<td style=""text-align: right;"">85</td>
</tr>
</tbody>
</table>
</div>
<p>Method 2 : add a granularity : a row is not anymore only a deal on a given that. it's a deal in a given date and it's projection in the next few days/months</p>
<div class=""s-table-container"">
<table class=""s-table"">
<thead>
<tr>
<th style=""text-align: left;"">AsofDate</th>
<th style=""text-align: center;"">DealId</th>
<th style=""text-align: right;"">projection</th>
<th style=""text-align: right;"">value</th>
</tr>
</thead>
<tbody>
<tr>
<td style=""text-align: left;"">2022-01-01</td>
<td style=""text-align: center;"">financingDeal1</td>
<td style=""text-align: right;"">0Day - actual</td>
<td style=""text-align: right;"">100</td>
</tr>
<tr>
<td style=""text-align: left;"">2022-01-01</td>
<td style=""text-align: center;"">financingDeal1</td>
<td style=""text-align: right;"">1Day</td>
<td style=""text-align: right;"">99</td>
</tr>
<tr>
<td style=""text-align: left;"">2022-01-01</td>
<td style=""text-align: center;"">financingDeal1</td>
<td style=""text-align: right;"">7Days</td>
<td style=""text-align: right;"">99</td>
</tr>
<tr>
<td style=""text-align: left;"">2022-01-01</td>
<td style=""text-align: center;"">financingDeal1</td>
<td style=""text-align: right;"">1Month</td>
<td style=""text-align: right;"">85</td>
</tr>
</tbody>
</table>
</div>
<p>from where i see it :
for method 1, the main inconvenient is if in the futur, we have a new projection value for 3 months, we will need to add a column in the ETL/ in the OLAP cube for '3months'</p>
<p>for method 2:
we will have as many rows as (deals * projections) and we do have 11 projections so it's 11 rows for each deal and we do have 1Million+ of them.</p>
<p>what is your opinion on this topic?</p>
<p>Thanks for your consideration</p>
",https://stackoverflow.com/questions/72713430/star-schema-fact-table-modeling,"['data-modeling', 'data-warehouse', 'star-schema']",49,False,0,0
How to handle useEffect being called twice in strictMode for things that should only be run once?,"<p>I understand that React calls useEffect twice in strict mode, this question is about asking what the correct way of handling it is.</p>
<p>I have recently hit an issue with <code>React</code> <code>useEffect</code> being called twice in <code>strictMode</code>. I'd like to keep strict mode to avoid issues, but I don't see any good way of making sure some specific effects are only run once. This is in a <code>next.js</code> environment, where I specifically want the code to only run in the client once.</p>
<p>For example, given the following component:</p>
<pre><code>import React, { useState, useEffect } from &quot;react&quot;;
import ky from &quot;ky&quot;;

const NeedsToRunOnce: React.FC = () =&gt; {
  const [loading, setLoading] = useState(false);

  const doSomethingThatOnlyShouldHappenOnce = (data: any) =&gt; {
    // Do something with the loaded data that should only happen once
    console.log(`This log appears twice!`, data);
  };

  useEffect(() =&gt; {
    if (loading) return;
    console.log(&quot;This happens twice despite trying to set loading to true&quot;);
    setLoading(true);
    const fetchData = async () =&gt; {
      const data = await ky.get(&quot;/data/json_data_2000.json&quot;).json();
      setLoading(false);
      doSomethingThatOnlyShouldHappenOnce(data);
    };
    fetchData();
  }, []);
  return &lt;div&gt;&lt;/div&gt;;
};

export default NeedsToRunOnce;
</code></pre>
<p><code>useEffect</code> will be called twice, and both times <code>loading</code> will still be false. This is because <code>strictMode</code> makes it be called twice with the same state. The request will go through twice, and call <code>doSomethingThatOnlyShouldHappenOnce</code> twice. All the <code>console.log</code> calls above will appear twice in the console.</p>
<p>Since I can't modify the state to let my component know that the request has already started happening, how can I stop the get request from happening twice and then calling code that I <em>only</em> want to be called once? (For context, I am initialising an external library with the data loaded in the <code>useEffect</code>, and this library should only be initialised once).</p>
<p>I found a <a href=""https://github.com/facebook/react/issues/24455"" rel=""nofollow noreferrer"">GitHub issue about this</a> where Dan Abramov says:</p>
<blockquote>
<p>Usually you’d want to have some kind of cleanup for your effect. It should either cancel your fetch or make sure you ignore its result by setting a variable inside your effect. Once you do this, there should be no actual difference in behavior.</p>
</blockquote>
<blockquote>
<p>If you cancel your fetch in effect cleanup, only one request will complete in development. However, it also doesn’t matter if another request fires. It’s being ignored anyway, and the stress-testing only happens in development.</p>
</blockquote>
<p>While I agree in principle, this is tedious in practice. I've already hit two different use cases in my application where I have some library call or request that needs to only be done once in the client.</p>
<p>What's a reliable way of making sure that a piece of code in <code>useEffect</code> is only run once here? Using a state to keep track of it having already been run doesn't help, since <code>react</code> calls the component twice with the same state in a row.</p>
",https://stackoverflow.com/questions/72204660/how-to-handle-useeffect-being-called-twice-in-strictmode-for-things-that-should,"['javascript', 'reactjs', 'typescript', 'next.js']",2261,True,1,1
AWS Checking StateMachines/StepFunctions concurrent runs,"<p>I am having a lot of issues handling concurrent runs of a StateMachine (Step Function) that does have a GlueJob task in it.</p>
<p>The state machine is initiated by a Lambda that gets trigger by a FIFO SQS queue.</p>
<p>The lambda gets the message, checks how many of state machine instances are running and if this number is below the GlueJob concurrent runs threshold, it starts the State Machine.</p>
<p>The problem I am having is that this check fails most of the time. The state machine starts although there is not enough concurrency available for my GlueJob. Obviously, the message the SQS queue passes to lambda gets processed, so if the state machine fails for this reason, that message is gone forever (unless I catch the exception and send back a new message to the queue).</p>
<p>I believe this behavior is due to the speed messages gets processed by my lambda (although it's a FIFO queue, so 1 message at a time), and the fact that my checker cannot keep up.</p>
<p>I have implemented some time.sleep() here and there to see if things get better, but no substantial improvement.</p>
<p>I would like to ask you if you have ever had issues like this one and how you got them programmatically solved.</p>
<p>Thanks in advance!</p>
<p>This is my checker:</p>
<pre class=""lang-py prettyprint-override""><code>def get_running_workflows(sf_client, sfn_arn, cnt=0, next_token=None):

    if next_token:
        response = sf_client.list_executions(stateMachineArn=sfn_arn, statusFilter='RUNNING', nextToken=next_token)
    else:
        response = sf_client.list_executions(stateMachineArn=sfn_arn, statusFilter='RUNNING')

    cnt += len(response['executions'])

    if cnt &gt; 0 and 'nextToken' in response:
        return get_running_workflows(cnt, response['nextToken'])

    return cnt
</code></pre>
<p><strong>EDIT 24-01-2022:</strong></p>
<p>Here's the implementation based @Mark-B answer.</p>
<p>I've created a DynamoDB table as follows:</p>
<pre class=""lang-yaml prettyprint-override""><code>resources:
  Resources:
    concurrencyHandler:
      Type: AWS::DynamoDB::Table
      Properties:
        TableName: concurrencyHandler
        AttributeDefinitions:
          - AttributeName: concurrencyProcess
            AttributeType: S
        KeySchema:
          - AttributeName: concurrencyProcess
            KeyType: HASH
        BillingMode: PAY_PER_REQUEST
</code></pre>
<p>Primary Key is used to store the process name that I am handling the concurrency for. This helps if I want to reuse the same table to handle concurrency for other processes.</p>
<p>Then, in the lambda that is consuming the messages from SQS and triggers my Step Function, I have written an atomic counter with a conditional check as follow:</p>
<pre class=""lang-py prettyprint-override""><code>    try:
        response = concurrency_table.update_item(
            Key={'concurrencyProcess': processName},
            ExpressionAttributeNames={
                '#C': 'concurrencyCount',
                '#O': step_function_name,
            },
            ExpressionAttributeValues={
                ':start': 0,
                ':inc': 1,
                ':limit': int(glue_max_concurrent_runs),
                ':time': datetime.datetime.utcnow().isoformat()
            },
            UpdateExpression=&quot;SET #C = if_not_exists(#C, :start) + :inc, #O = :time&quot;,
            ConditionExpression=&quot;#C &lt; :limit AND attribute_not_exists(#O)&quot;,
            ReturnValues=&quot;UPDATED_NEW&quot;
        )

    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] != 'ConditionalCheckFailedException':
            raise
        else:
            raise Exception(&quot;Either max Concurrency number reached or &quot;
                            &quot;Stepfunction is requesting second lock&quot;)
</code></pre>
<p>Any time a new message arrives, this code will increment the counter. If the counter is &gt;= than limit, it will fail. So the message will be sent back to the queue and processed again later.</p>
<p>Obviously, I have set up a clean-up, final step in my StepFunction workflow, one if the workflow is succesful, one if the workflow should fail for any reason. Here's the .yaml:</p>
<pre class=""lang-yaml prettyprint-override""><code>  UpdateConcurrencyCountSuccess:
    Type: Task
    Resource: arn:aws:states:::dynamodb:updateItem
    Parameters:
      TableName: ${self:custom.concurrencyTable}
      Key:
        concurrencyProcess:
          S: {processName}
      ExpressionAttributeNames:
        '#C': concurrencyCount
        '#O.$': $$.Execution.Name
      ExpressionAttributeValues:
        :dec:
          N: '1'
      UpdateExpression: &quot;SET #C = #C - :dec REMOVE #O&quot;
      ConditionExpression: attribute_exists(#O)
      ReturnValues: UPDATED_NEW
    Retry:
      - ErrorEquals:
          - DynamoDB.ConditionalCheckFailedException
        MaxAttempts: 0
      - ErrorEquals:
          - States.ALL
        IntervalSeconds: 5
        MaxAttempts: 20
        BackoffRate: 1.4
    ResultPath: $.update_concurrency_success 
    End: true
    Catch:
      - ErrorEquals:
          - States.ALL
        Next: PublishToSNSFailed
        ResultPath: $.error

  # Update Concurrency Count table, failed flow
  UpdateConcurrencyCountFail:
    Type: Task
    Resource: arn:aws:states:::dynamodb:updateItem
    Parameters:
      TableName: ${self:custom.concurrencyTable}
      Key:
        concurrencyProcess:
          S: {processName}
      ExpressionAttributeNames:
        '#C': concurrencyCount
        '#O.$': $$.Execution.Name
      ExpressionAttributeValues:
        :dec:
          N: '1'
      UpdateExpression: &quot;SET #C = #C - :dec REMOVE #O&quot;
      ConditionExpression: attribute_exists(#O)
      ReturnValues: UPDATED_NEW
    Retry:
      - ErrorEquals:
          - DynamoDB.ConditionalCheckFailedException
        MaxAttempts: 0
      - ErrorEquals:
          - States.ALL
        IntervalSeconds: 5
        MaxAttempts: 20
        BackoffRate: 1.4
    ResultPath: $.update_concurrency_failed     
    Next: ExecutionFailed
    Catch:
      - ErrorEquals:
          - States.ALL
        Next: ExecutionFailed
        ResultPath: $.error
</code></pre>
<p>Note how the attribute <code>#O</code> in DynamoDB is the Step Function Name, and takes as value the starting-execution-time. The name of the step function execution is generated inside my trigger-lambda (it is univocal). Within my workflow I can access the name via <code>$$Execution.Name</code> so to remove that attribute in the clean up process.</p>
<p>Having the step function univocal name as attribute can help debug in case (although very remote) an execution starts but fails to decrease the counter.</p>
<p><strong>EDIT 02-02-2022:</strong></p>
<p>After stress-testing this solution, I've come to the conclusion that it fails to keep the count of state machines running: when the max is set to 10,  somehow an 11th lock is given.</p>
<p>On top of, I must say, Glue having some sort of invisible &quot;state&quot; after &quot;SUCCESS&quot; which was leading to less concurrency than locks available (for instance: 0 glue concurrency available, but 2/3 free locks).</p>
<p>Hopefully AWS can implement some concurrency control and a direct trigger from SQS to StepFunction</p>
",https://stackoverflow.com/questions/70813239/aws-checking-statemachines-stepfunctions-concurrent-runs,"['amazon-web-services', 'aws-step-functions']",720,True,4,1
