title,body,link,tags,view_count,is_answered,score,answer_count
Acceptance testing with memory datasource doesn&#39;t work,"<p>I created acceptance test of controller that using memory datasource.
Before the test start, I was trying to have clean database and only add 1 user credential to login by calling endpoint since the rest of test require authenticated access with access token.</p>

<p>I can get the user info in <code>getUser</code> after created the user record from <code>givenUser</code>.
However, the login request can't find the credential. It seems that the helpers and client are not sharing the same memory datasource.</p>

<p>I am not sure what configuration setup is wrong in this case.</p>

<p><strong>src/tests/fixtures/datasources/testdb.datasource.ts</strong></p>

<pre class=""lang-js prettyprint-override""><code>import {juggler} from '@loopback/repository';

export const testdb: juggler.DataSource = new juggler.DataSource({
  name: 'db',
  connector: 'memory',
});
</code></pre>

<p><strong>src/datasources/mongodb.datasource.ts</strong></p>

<pre class=""lang-js prettyprint-override""><code>import {inject} from '@loopback/core';
import {juggler} from '@loopback/repository';
import * as config from './mongodb.datasource.json';

export class MongodbDataSource extends juggler.DataSource {
  static dataSourceName = 'mongodb';

  constructor(
    @inject('datasources.config.mongodb', {optional: true})
    dsConfig: object = config,
  ) {
    super(dsConfig);
  }
}
</code></pre>

<p><strong>user.controller.acceptance.ts</strong></p>

<pre class=""lang-js prettyprint-override""><code>import {Client, expect, supertest} from '@loopback/testlab';
import {ApiApplication} from '../..';
import {setupApplication} from './test-helper';
import {givenEmptyDatabase, givenUser,getUser} from '../helpers/database.helpers';

describe('UserController', () =&gt; {
  let app: ApiApplication;
  let client: supertest.SuperTest&lt;supertest.Test&gt;;;
  let jwtToken: string;

  before('setupApplication', async () =&gt; {
    ({app, client} = await setupApplication());
  });
  before(givenEmptyDatabase);
  before(givenUser);
  before(getUser);
  before(async () =&gt; {
    const response = await client
      .post('/login')
      .send({username: 'user1', password: 'password'});
    jwtToken = response.body.token;
  });

  after(async () =&gt; {
    await app.stop();
  });

  it('invokes GET /info without authentication', async () =&gt; {
    const expectedError = {
      error: {
        statusCode: 401,
        name: 'UnauthorizedError',
        message: 'Unauthorized'
      }
    };
    const res = await client.get('/user/info').expect(401);
    expect(res.body).to.containEql(expectedError);
  });
</code></pre>

<p><strong>test-help.ts</strong></p>

<pre class=""lang-js prettyprint-override""><code>import {ApiApplication} from '../..';
import {
  createRestAppClient,
  givenHttpServerConfig,
  Client,
} from '@loopback/testlab';
import {testdb} from '../fixtures/datasources/testdb.datasource';

export async function setupApplication(): Promise&lt;AppWithClient&gt; {

  const app = new ApiApplication();
  await app.bind('datasources.config.mongodb').to({
    name: 'mongodb',
    connector: 'memory',
    debug: true,
  });

  await app.boot();
  await app.start();

  const client = createRestAppClient(app);

  return {app, client};
}

export interface AppWithClient {
  app: ApiApplication;
  client: Client;
}
</code></pre>

<p><strong>database.helper.ts</strong></p>

<pre class=""lang-js prettyprint-override""><code>import { UserRepository } from '../../repositories';
import { User } from '../../models';
import { testdb } from '../fixtures/datasources/testdb.datasource';

export async function givenEmptyDatabase() {
  let userRepo: UserRepository;

  userRepo = new UserRepository(testdb);

  userRepo.deleteAll();
}

export async function givenUser() {
  let userRepo = new UserRepository(testdb);
  const user = {
    username: 'user1',
    password: 'password',
    created_at: new Date('2019-08-08'),
    updated_at: new Date('2019-08-08'),
  }
  await userRepo.create(operator);
}

export async function getUser() {
  let userRepo = new UserRepository(testdb);
  const users = await userRepo.find();
  console.log(users);
}
</code></pre>
",https://stackoverflow.com/questions/57984578/acceptance-testing-with-memory-datasource-doesnt-work,"['debugging', 'testing', 'datasource', 'loopback', 'acceptance-testing']",365,True,1,1
Seeding data for acceptance testing of AWS Serverless application,"<p>I'm starting to teach myself serverless development, using AWS Lambda and the Serverless CLI. So far, all is going great. However, I've got a snag with acceptance testing.</p>

<p>What I'm doing so far is:</p>

<ul>
<li>Deploy stack to AWS with a generated stage name - I'm using the CI job ID for this</li>
<li>Run all the tests against this deployment</li>
<li>Remove the deployment</li>
<li>Deploy the stack to AWS with the ""Dev"" stage name</li>
</ul>

<p>This is fine, until I need some data.</p>

<p>Testing endpoints without data is easy - that's the default state. So I can test that <code>GET /users/badid</code> returns a 404.</p>

<p>What's the typical way of setting up test data for the tests? </p>

<p>In my normal development I do this by running a full stack - UI, services, databases - in a local docker compose stack and the tests can talk to them directly. Is that the process to follow here - Have the tests talk directly to the varied AWS data stores? If so, how do you handle multiple (DynamoDB) tables across different CF stacks, e.g. for testing the UI?</p>

<p>If that's not the normal way to do it, what is?</p>

<p>Also, is there a standard way to clear out data between tests? I can't safely test a search endpoint if the data isn't constant for that test, for example. (If data isn't cleared out then the data in the system will be dependent on the order the tests run in, which is bad)</p>

<p>Cheers</p>
",https://stackoverflow.com/questions/58439038/seeding-data-for-acceptance-testing-of-aws-serverless-application,"['amazon-web-services', 'aws-lambda', 'automated-tests', 'serverless', 'acceptance-testing']",290,False,0,1
How to mock couchbase for acceptance testing without embedded server or a new cluster just for that?,"<p>I'm doing acceptance testing for a microservice that connects to couchbase, elasticsearch and Kafka. I don't want to use any embedded instance or any test cluster. I want to be able to mock the couchbase calls, but at the same should test the N1QL queries that I've written.</p>

<p>For elasticsearch, I'm using Wiremock and mocking the REST calls. But for couchbase, I don't know the structure of the rest calls that the client makes to the database.</p>

<p>Also, I went ahead and wrote a test case. With that, I mocked all the REST calls that couchbase made, from the error message. But I see that couchbase connects to a netty channel, how do I mock the channel?</p>

<p>If not that, is there a better way to mock couchbase?</p>

<pre><code>RxComputationScheduler- 3|ERROR||||c.c.c.d.i.n.u.c.D.rejectedExecution|181|Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:796)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:336)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:329)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:739)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87)
at com.couchbase.client.core.endpoint.AbstractEndpoint$3.call(AbstractEndpoint.java:381)
at com.couchbase.client.core.endpoint.AbstractEndpoint$3.call(AbstractEndpoint.java:374)
at rx.internal.operators.SingleOperatorOnErrorResumeNext$2.onError(SingleOperatorOnErrorResumeNext.java:69)
at rx.internal.operators.SingleTimeout$TimeoutSingleSubscriber$OtherSubscriber.onError(SingleTimeout.java:133)
at rx.Single$1.call(Single.java:477)
at rx.Single$1.call(Single.java:473)
at rx.Single.subscribe(Single.java:1979)
at rx.Single$18.call(Single.java:2518)
at rx.Single$18.call(Single.java:2505)
at rx.internal.operators.SingleTimeout$TimeoutSingleSubscriber.call(SingleTimeout.java:110)
at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
</code></pre>

<p>This is the error that I'm getting.</p>
",https://stackoverflow.com/questions/54459867/how-to-mock-couchbase-for-acceptance-testing-without-embedded-server-or-a-new-cl,"['mocking', 'netty', 'couchbase', 'acceptance-testing', 'wiremock']",2193,True,3,3
&#39;chrome&#39; could not be found and loaded in phpStorm acceptance test,"<p>I am new to unit testing,
I am new to acceptance testing. I am using laravel 5.5.
The below is my <code>acceptance.suite.dist.yml</code></p>

<pre><code>class_name: AcceptanceTester
modules:
  enabled:
    - WebDriver
        url: 'http://amazon-ce.localhost.tom/'
        browser: chrome
    - Db
    - Filesystem
config:
  PhpBrowser:
   url: 'http://amazon-ce.localhost.tom/'
WebDriver:
   url: 'http://amazon-ce.localhost.tom/'
   browser: chrome
Db:
  dsn: 'mysql:dbname=myDb;host=127.0.0.1'
  user: 'root'
  password: ''
</code></pre>

<p>when I run the test, 
it throwing 
<code>
Module WebDriver url: 'http://amazon-ce.localhost.tom/' 
browser: 'chrome' could not be found and loaded</code></p>

<p>can anyone help on this.? </p>
",https://stackoverflow.com/questions/55157461/chrome-could-not-be-found-and-loaded-in-phpstorm-acceptance-test,"['laravel', 'selenium', 'acceptance-testing']",31,False,0,1
Alternative ways to display Scalatest results,"<p>I am thinking about setting up an automated acceptance testing suite with Scalatest and output the results into a simple web dashboard (say print each scenario text stacked with check marks or crosses next to them). Is there a built-in functionality I can use for alternative ways to display the ordinary output or would I need to output and parse the logs myself?</p>
",https://stackoverflow.com/questions/56790727/alternative-ways-to-display-scalatest-results,"['scala', 'scalatest']",1140,True,3,2
Codeception undefined index: ELEMENT error,"<p>i am using Codeception for acceptance testing. On my local pc i have no problems to run the tests with selenium and chromedriver. Until 1 week, it works also fine at my git, but now, there comes this error. </p>

<p>""[PHPUnit\Framework\Exception] Undefined index: ELEMENT""</p>

<p>local it is working fine without errors. </p>

<p>I already searched this error and found an tip, that i have to start selenium with ""-enablepassthrough false"". But this does not really works. </p>

<p>Before 1 week with all the same setup, it works fine. </p>

<p>this is my acceptance.suite.yml:</p>

<pre><code>actor: AcceptanceTester
modules:
    enabled:
        - WebDriver:
            url: 'https://website.com' 
            host: 'selenium__standalone-chrome'
            port: 4444
            browser: chrome 
            window_size: 1920x1080 
</code></pre>
",https://stackoverflow.com/questions/56556709/codeception-undefined-index-element-error,"['php', 'selenium', 'webdriver', 'codeception']",2826,True,4,3
How to mock couchbase for acceptance testing without embedded server or a new cluster just for that?,"<p>I'm doing acceptance testing for a microservice that connects to couchbase, elasticsearch and Kafka. I don't want to use any embedded instance or any test cluster. I want to be able to mock the couchbase calls, but at the same should test the N1QL queries that I've written.</p>

<p>For elasticsearch, I'm using Wiremock and mocking the REST calls. But for couchbase, I don't know the structure of the rest calls that the client makes to the database.</p>

<p>Also, I went ahead and wrote a test case. With that, I mocked all the REST calls that couchbase made, from the error message. But I see that couchbase connects to a netty channel, how do I mock the channel?</p>

<p>If not that, is there a better way to mock couchbase?</p>

<pre><code>RxComputationScheduler- 3|ERROR||||c.c.c.d.i.n.u.c.D.rejectedExecution|181|Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:796)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:336)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:329)
at com.couchbase.client.deps.io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:739)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:760)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:428)
at com.couchbase.client.deps.io.netty.util.concurrent.DefaultPromise.setFailure(DefaultPromise.java:113)
at com.couchbase.client.deps.io.netty.channel.DefaultChannelPromise.setFailure(DefaultChannelPromise.java:87)
at com.couchbase.client.core.endpoint.AbstractEndpoint$3.call(AbstractEndpoint.java:381)
at com.couchbase.client.core.endpoint.AbstractEndpoint$3.call(AbstractEndpoint.java:374)
at rx.internal.operators.SingleOperatorOnErrorResumeNext$2.onError(SingleOperatorOnErrorResumeNext.java:69)
at rx.internal.operators.SingleTimeout$TimeoutSingleSubscriber$OtherSubscriber.onError(SingleTimeout.java:133)
at rx.Single$1.call(Single.java:477)
at rx.Single$1.call(Single.java:473)
at rx.Single.subscribe(Single.java:1979)
at rx.Single$18.call(Single.java:2518)
at rx.Single$18.call(Single.java:2505)
at rx.internal.operators.SingleTimeout$TimeoutSingleSubscriber.call(SingleTimeout.java:110)
at rx.internal.schedulers.EventLoopsScheduler$EventLoopWorker$2.call(EventLoopsScheduler.java:189)
at rx.internal.schedulers.ScheduledAction.run(ScheduledAction.java:55)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
</code></pre>

<p>This is the error that I'm getting.</p>
",https://stackoverflow.com/questions/54459867/how-to-mock-couchbase-for-acceptance-testing-without-embedded-server-or-a-new-cl,"['mocking', 'netty', 'couchbase', 'acceptance-testing', 'wiremock']",2193,True,3,3
Strange caching problem for API and DB on Windows Server 2012,"<p>Has anyone else had issues when trying to update a website in an application pool and the changes don't seem to be reflected at all when navigating to the URLs.</p>

<p>I'm a dev working in a development to user acceptance testing to then production environments. I'm currently trying to upload a asp net core api and react app to a Windows Server 2012 server. The API communicates with an isolated sql sever. I have a dev-sql db and a uat-sql db.</p>

<p>Are there any Caching features that I might be overlooking that are causing these problems?</p>

<p>I've definitely updated appsettings.json &amp; the environment variables to reflect the UAT database and API but... it still acts as if it's looking at the DEV database and dev API.</p>

<p>Any suggestions? </p>

<p>Thanks in advance.</p>
",https://stackoverflow.com/questions/58642006/strange-caching-problem-for-api-and-db-on-windows-server-2012,"['windows', 'asp.net-core', 'iis', 'windows-server-2012', 'windows-server']",27,False,0,0
Keeping Existing Sessions When Upgrading to Rails 5.2 (with Redis Session Store),"<p>I have recently finished a big upgrade of a Rails application. I took the App incrementally from <code>Rails 4.2.8</code> to <code>Rails 5.2.3</code>. Everything has gone smoothly, all our automated tests are passing, and we've begun user acceptance testing.</p>

<p>So far the only issue that has been found is that logged in users are being signed out by this upgrade. We previously had the <code>redis-rails</code> gem (version <code>4.0.0</code>) installed to enable using Redis for our <code>session_store</code> and now we're using the built-in Redis store that comes with <code>Rails 5.2</code>.</p>

<p>I have our configuration set up to use the same Redis connection and the same session_store key value as before, and I can't figure out why sessions are still being reset. It seems like perhaps <code>redis-rails 4.0.0</code> was not properly setting the session_store key value?</p>

<p>Has anyone else dealt with this and successfully retained existing sessions when upgrading from <code>redis-rails</code> to <code>Rails 5.2</code>?</p>

<p><em>Edit:</em>
Other possibly-related gems that have been updated include <code>ruby 2.3.4 =&gt; 2.5.5</code>, <code>devise 4.0.3 =&gt; 4.6.2</code>, &amp; <code>redis 3.2.2 =&gt; 4.1.2</code></p>
",https://stackoverflow.com/questions/56638657/keeping-existing-sessions-when-upgrading-to-rails-5-2-with-redis-session-store,"['ruby-on-rails', 'session', 'ruby-on-rails-5.2', 'session-store', 'redis-rails']",1661,True,1,1
Hive - Is it a good fit for building a datawarehouse?,"<p>So like most Enterprise companies, we have built a data warehouse in Hadoop, with user queries supported in Hive, and now after a few months and user acceptance testing everyone is a little surprised about how it is not like a standard (Oracle/Netezza) database when used by end-users for ad-hoc data analysis. 
While I understand that this is probably a very stupid way of doing projects (we should have researched the use cases and best fit technologies before building the product), and I know the basic technical aspects of how Hadoop differs from single node machines... I would still want to understand if using Hadoop/Hive makes sense for data warehouses in any scenario? 
For instance, </p>

<ul>
<li>Are there always trade-offs in query performance or can they be optimized with configuration changes, horizontal scaling of hardware?</li>
<li>Can it ever be as fast as something like Netezza - which uses non-commodity hardware but functions on a similar architecture?</li>
<li>Where is Hadoop great and absolutely defeats everything else in comparison?</li>
</ul>
",https://stackoverflow.com/questions/54241078/hive-is-it-a-good-fit-for-building-a-datawarehouse,"['hadoop', 'hive', 'data-warehouse']",211,True,0,3
How to prevent session loss after calling visit(uri)?,"<p>Fairly new to capybara and cucumber (and testing in general) here.</p>

<p>I am doing some black box acceptance testing and I need to test a page that can only be reached by typing in the url.</p>

<p>That page requires authentication so when the page is reached through a call to visit() I am redirected to the login page.</p>

<p>The problem is after logging in the app does not redirect me back to that page but goes to the homepage instead so I can't just fill in the login form and move on.</p>

<p>I have tried logging in first and then performing a call to that page but it seems the session is cleared after any call to visit().</p>

<p>How can I keep the session after a call to visit, or solve this problem in any other way?</p>

<p>Ideally I would like to log in once and visit a bunch of pages only reachable by typing the uri, without having to log in again and again. Is that possible?</p>

<p>I am only using the chrome selenium driver but at some point will use a headless driver.</p>

<p>Thanks!</p>
",https://stackoverflow.com/questions/54160011/how-to-prevent-session-loss-after-calling-visituri,"['cucumber', 'capybara']",166,True,1,2
Seeding data for acceptance testing of AWS Serverless application,"<p>I'm starting to teach myself serverless development, using AWS Lambda and the Serverless CLI. So far, all is going great. However, I've got a snag with acceptance testing.</p>

<p>What I'm doing so far is:</p>

<ul>
<li>Deploy stack to AWS with a generated stage name - I'm using the CI job ID for this</li>
<li>Run all the tests against this deployment</li>
<li>Remove the deployment</li>
<li>Deploy the stack to AWS with the ""Dev"" stage name</li>
</ul>

<p>This is fine, until I need some data.</p>

<p>Testing endpoints without data is easy - that's the default state. So I can test that <code>GET /users/badid</code> returns a 404.</p>

<p>What's the typical way of setting up test data for the tests? </p>

<p>In my normal development I do this by running a full stack - UI, services, databases - in a local docker compose stack and the tests can talk to them directly. Is that the process to follow here - Have the tests talk directly to the varied AWS data stores? If so, how do you handle multiple (DynamoDB) tables across different CF stacks, e.g. for testing the UI?</p>

<p>If that's not the normal way to do it, what is?</p>

<p>Also, is there a standard way to clear out data between tests? I can't safely test a search endpoint if the data isn't constant for that test, for example. (If data isn't cleared out then the data in the system will be dependent on the order the tests run in, which is bad)</p>

<p>Cheers</p>
",https://stackoverflow.com/questions/58439038/seeding-data-for-acceptance-testing-of-aws-serverless-application,"['amazon-web-services', 'aws-lambda', 'automated-tests', 'serverless', 'acceptance-testing']",290,False,0,1
Kubernetes 1.15.5 and romana 2.0.2 getting network errors when ANY pods added or removed,"<p>I have encountered some mysterious network errors in our kubernetes cluster. Although I originally encountered these errors using ingress, there are even more errors when I bypass our load balancer, bypass kube-proxy and bypass nginx-ingress. The most errors are present when going directly to services and straight to the pod IPs. I believe this is because the load balancer and nginx have some better error handling than the raw iptable routing.</p>

<p>To test the error I use apache benchmark from VM on same subnet, any concurrency level, no keep-alive, connect to the pod IP and use a high enough request number to give me time to either scale up or scale down a deployment. Odd thing is it doesn't matter at all which deployment I modify since it always causes the same sets of errors even when its not related to the pod I am modifying. ANY additions or removals of pods will trigger apache benchmark errors. Manual deletions, scaling up/down, auto-scaling all trigger errors. If there are no pod changes while the ab test is running then no errors get reported. Note keep-alive does seem to greatly reduce if not eliminate the errors, but I only tested that a handful of times and never saw an error.</p>

<p>Other than some bizarre iptable conflict, I really don't see how deleting pod A can effect network connections of pod B. Since the errors are brief and go away within seconds it seems more like a brief network outage.</p>

<p>Sample ab test: <code>ab -n 5000 -c 2 https://10.112.0.24/</code></p>

<p>Errors when using HTTPS:</p>

<pre><code>SSL handshake failed (5).
SSL read failed (5) - closing connection
</code></pre>

<p>Errors when using HTTP:</p>

<pre><code>apr_socket_recv: Connection reset by peer (104)
apr_socket_recv: Connection refused (111)
</code></pre>

<p>Example ab output. I ctl-C after encountering first errors:</p>

<pre><code>$ ab -n 5000 -c 2 https://10.112.0.24/
This is ApacheBench, Version 2.3 &lt;$Revision: 1826891 $&gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 10.112.0.24 (be patient)
Completed 500 requests
Completed 1000 requests
SSL read failed (5) - closing connection
Completed 1500 requests
^C

Server Software:        nginx
Server Hostname:        10.112.0.24
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256

Document Path:          /
Document Length:        2575 bytes

Concurrency Level:      2
Time taken for tests:   21.670 seconds
Complete requests:      1824
Failed requests:        2
   (Connect: 0, Receive: 0, Length: 1, Exceptions: 1)
Total transferred:      5142683 bytes
HTML transferred:       4694225 bytes
Requests per second:    84.17 [#/sec] (mean)
Time per request:       23.761 [ms] (mean)
Time per request:       11.881 [ms] (mean, across all concurrent requests)
Transfer rate:          231.75 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        5   15   9.8     12      82
Processing:     1    9   9.0      6     130
Waiting:        0    8   8.9      6     129
Total:          7   23  14.4     19     142

Percentage of the requests served within a certain time (ms)
  50%     19
  66%     24
  75%     28
  80%     30
  90%     40
  95%     54
  98%     66
  99%     79
 100%    142 (longest request)
</code></pre>

<p>Current sysctl settings that may be relevant:</p>

<pre><code>net.netfilter.nf_conntrack_tcp_be_liberal = 1
net.nf_conntrack_max = 131072
net.netfilter.nf_conntrack_buckets = 65536
net.netfilter.nf_conntrack_count = 1280
net.ipv4.ip_local_port_range = 27050    65500
</code></pre>

<p>I didn't see any conntrack ""full"" errors. Best I could tell there isn't packet loss. We recently upgraded from 1.14 and didn't notice the issue but I can't say for certain it wasn't there. I believe we will be forced to migrate away from romana soon since it doesn't seem to be maintained anymore and as we upgrade to kube 1.16.x we are encountering problems with it starting up.</p>

<p>I have searched the internet all day today looking for similar problems and the closest one that resembles our problem is <a href=""https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02"" rel=""nofollow noreferrer"">https://tech.xing.com/a-reason-for-unexplained-connection-timeouts-on-kubernetes-docker-abd041cf7e02</a> but I have no idea how to implement the iptable masquerade --random-fully option given we use romana and I read (<a href=""https://github.com/kubernetes/kubernetes/pull/78547#issuecomment-527578153"" rel=""nofollow noreferrer"">https://github.com/kubernetes/kubernetes/pull/78547#issuecomment-527578153</a>) that random-fully is the default for linux kernel 5 which we are using. Any ideas?</p>

<ul>
<li>kubernetes 1.15.5</li>
<li>romana 2.0.2</li>
<li>centos7</li>
<li>Linux kube-master01 5.0.7-1.el7.elrepo.x86_64 #1 SMP Fri Apr 5 18:07:52 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux</li>
</ul>

<p>====== Update Nov 5, 2019 ======</p>

<p>It has been suggested to test an alternate CNI. I chose calico since we used that in an older Debian based kube cluster. I rebuilt a VM with our most basic Centos 7 template (vSphere) so there is a little baggage coming from our customizations. I can't list everything we customized in our template but the most notable change is the kernel 5 upgrade <code>yum --enablerepo=elrepo-kernel -y install kernel-ml</code>.</p>

<p>After starting up the VM these are the minimal steps to install kubernetes and run the test:</p>

<pre><code>yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

yum -y install docker-ce-3:18.09.6-3.el7.x86_64

systemctl start docker

cat &lt;&lt;EOF &gt; /etc/yum.repos.d/kubernetes.repo
[kubernetes]
name=Kubernetes
baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-x86_64
enabled=1
gpgcheck=1
repo_gpgcheck=1
gpgkey=https://packages.cloud.google.com/yum/doc/yum-key.gpg https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
EOF

# Set SELinux in permissive mode (effectively disabling it)
setenforce 0
sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config

echo '1' &gt; /proc/sys/net/bridge/bridge-nf-call-iptables

yum install -y kubeadm-1.15.5-0 kubelet-1.15.5-0 kubectl-1.15.5-0

systemctl enable --now kubelet

kubeadm init --pod-network-cidr=192.168.0.0/16

mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config

kubectl taint nodes --all node-role.kubernetes.io/master-

kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml

cat &lt;&lt;EOF &gt; /tmp/test-deploy.yml
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: test
spec:
  selector:
    matchLabels:
      app: test
  replicas: 1
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
      - name: nginx
        image: nginxdemos/hello
        ports:
        - containerPort: 80
EOF

# wait for control plane to become healthy

kubectl apply -f /tmp/test-deploy.yml
</code></pre>

<p>Now the setup is ready and this is the ab test:</p>

<pre><code>$ docker run --rm jordi/ab -n 100 -c 1  http://192.168.4.4/
This is ApacheBench, Version 2.3 &lt;$Revision: 1826891 $&gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.4.4 (be patient)...apr_pollset_poll: The timeout specified has expired (70007)
Total of 11 requests completed
</code></pre>

<p>The ab test gives up after this error. If I decrease the number of requests to see avoid the timeout  this is what you would see:</p>

<pre><code>$ docker run --rm jordi/ab -n 10 -c 1  http://192.168.4.4/
This is ApacheBench, Version 2.3 &lt;$Revision: 1826891 $&gt;
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.4.4 (be patient).....done


Server Software:        nginx/1.13.8
Server Hostname:        192.168.4.4
Server Port:            80

Document Path:          /
Document Length:        7227 bytes

Concurrency Level:      1
Time taken for tests:   0.029 seconds
Complete requests:      10
Failed requests:        0
Total transferred:      74140 bytes
HTML transferred:       72270 bytes
Requests per second:    342.18 [#/sec] (mean)
Time per request:       2.922 [ms] (mean)
Time per request:       2.922 [ms] (mean, across all concurrent requests)
Transfer rate:          2477.50 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    1   0.8      1       3
Processing:     1    2   1.2      1       4
Waiting:        0    1   1.3      0       4
Total:          1    3   1.4      3       5

Percentage of the requests served within a certain time (ms)
  50%      3
  66%      3
  75%      4
  80%      5
  90%      5
  95%      5
  98%      5
  99%      5
 100%      5 (longest request)
</code></pre>

<p>This issue is technically different than the original issue I reported but this is a different CNI and there are still network issues. It does have the timeout error in common when I run the same test in the kube/romana cluster: run the ab test on the same node as the pod. Both encountered the same timeout error but in romana I could get a few thousand requests to finish before hitting the timeout. Calico encounters the timeout error before reaching a dozen requests.</p>

<p>Other variants or notes:
- net.netfilter.nf_conntrack_tcp_be_liberal=0/1 doesn't seem to make a difference
- higher <code>-n</code> values sometimes work but it is largely random.
- running the 'ab' test at low <code>-n</code> values several times in a row can sometimes trigger the timeout</p>

<p>At this point I am pretty sure it is some issue with our centos installation but I can't even guess what it could be. Are there any other limits, sysctl or other configs that could cause this?</p>

<p>====== Update Nov 6, 2019 ======</p>

<p>I observer that we had an older kernel installed in so I upgraded my kube/calico test VM with the same newer kernel 5.3.8-1.el7.elrepo.x86_64. After the update and a few reboots I can no longer reproduce the ""apr_pollset_poll: The timeout specified has expired (70007)"" timout errors.</p>

<p>Now that the timeout error is gone I was able to repeat the original test where I load test pod A and kill pod B on my vSphere VMs. On the romana environments the problem still existed but only when the load test is on a different host than where the pod A is located. If I run the test on the same host, no errors at all. Using Calico instead of romana, there are no load test errors on either host so the problem was gone. There may still be some setting to tweak that can help romana but I think this is ""strike 3"" for romana so I will start transitioning a full environment to Calico and do some acceptance testing there to ensure there are no hidden gotchas.</p>
",https://stackoverflow.com/questions/58637215/kubernetes-1-15-5-and-romana-2-0-2-getting-network-errors-when-any-pods-added-or,"['kubernetes', 'cni']",287,False,3,1
