@somesh, again your message is not helping all.
> On Sep 30, 2015, at 5:09 PM, Somesh Naidu <Somesh.Naidu@citrix.com> wrote:
>
>> that said we should still work towards a test set that can function as the
>
> So essentially, we are saying that the existing BVT isn't really a BVT and that we have
to come up with it. From a user/customer standpoint this doesn't inspire confidence.
>
Folks on this list, even though they talk as themselves, are users of ACS.They have built
businesses based on ACS and rely on it heavily. I could give you a long list. They have a
lot at stake in having a quality release of ACS. I am certain that they would not release
something that they would not trust or have not tested.
The issue is that folks test for there particular setup and that as a group we still don’t
have a complete view of the entire matrix that is possible with ACS (what Daan was alluding
to). And BVT is just an indication of a certain level of quality it by no means says that
all features of CloudStack work in every possible deployment.
> Anyway, is anybody willing to do this, as in, come up with a BVT/smoke/sanity?
It’s not a clear question.
Travis runs part of the simulator checks every single commits.
So does Jenkins, so do people with their own infra (not just Citrix, but actual users of ACS).
The issue with the particular test I was referring to (test_scale_vm) is that for some reason
it is not running in the current Travis runs. My personal view is that the current failure
of test_scale_vm is not symptomatic of a failure in ACS code, if I understood things correctly,
hence I don’t consider this a blocker.
> Or do we have to rely on Citrix to do this?
If you are talking about Citrix running their set of BVT, the question is not to rely on it
or not.
From my standpoint it would have been nice to see weekly (if not daily) status of all BVT
tests run by Citrix (to the form of what Raja is now currently sending).
>
> Daan - do we have a tracker for this request?
>
Not a clear question either. I don’t know what request you are talking about.
But for now, let’s leave this thread as is and wait for the next couple PRs to hit master
and we will see how BVT behaves.
> Regards,
> Somesh
>
> -----Original Message-----
> From: Daan Hoogland [mailto:daan.hoogland@gmail.com]
> Sent: Wednesday, September 30, 2015 5:44 AM
> To: dev
> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>
> I disagree, and let me carefully try to formulate why: BVT is useful and
> 'should' be run before any thing is declared good. The problem is the
> ability for the community at large to run it and to fix it for any kind of
> set-up. It should not be automatically treated as blockers but only if a
> full analysis points to a root cause that can be considered a blocker.
>
> that said we should still work towards a test set that can function as the
> you describe the BVT and be automatically adding blockers.
>
> regards,
>
> On Wed, Sep 30, 2015 at 11:28 AM, Rajani Karuturi <
> Rajani.Karuturi@citrix.com> wrote:
>
>> I agree with Raja. BVT test case failures should be treated as blockers.
>> Once the test issue is fixed, they might uncover new issues/regressions.
>>
>> ~Rajani
>>
>>
>>
>> On 30-Sep-2015, at 2:32 pm, Raja Pullela <raja.pullela@citrix.com> wrote:
>>
>>> The problem needs to be fixed - "have tested a solution that was
>> discussed with koushik" - we will have to create PR with that change. Once
>> the PR is pushed - we will have one less blocker/test failure.
>>>
>>> I agree this is not a cloudstack core code problem - but we should NOT
>> make that a criteria for priority of a bug - just my take. We should
>> include BVT failures in the Blocker criteria (whether the fix is in the
>> core functionality/test script). Otherwise, we will not hit 100% passrate
>> on BVTs.
>>>
>>> You are right - only a subset of these tests are being run on "travis"
>> otherwise we will catch more or some of these failures.
>>>
>>> -----Original Message-----
>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>> Sent: Wednesday, September 30, 2015 2:09 PM
>>> To: dev@cloudstack.apache.org
>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>
>>>
>>>> On Sep 30, 2015, at 10:28 AM, Raja Pullela <raja.pullela@citrix.com>
>> wrote:
>>>>
>>>> Yup... btw, I am on top of this and goal is to get to 100% passrate on
>> BVTs!
>>>>
>>>
>>> Yes I got it, and we all have the same goal.
>>>
>>> It does appear that not all smoke tests are run by Travis, otherwise we
>> would catch any errors on simulator for every PR.
>>>
>>> My point is that if a failure in your BVT is due to the test itself it
>> is not a cloudstack failure per se and hence not a blocker. Which all said
>> and done is not a problem because it should be fixed now if I am
>> understanding your assessment and Koushik report of this failure correctly.
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>>> Sent: Wednesday, September 30, 2015 1:48 PM
>>>> To: dev@cloudstack.apache.org
>>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>>
>>>>
>>>>> On Sep 30, 2015, at 10:15 AM, Raja Pullela <raja.pullela@citrix.com>
>> wrote:
>>>>>
>>>>> Sebastien, any BVT tests failing is a blocker and needs to be fixed -
>> which is the reason for Blocker. We already have a fix.
>>>>>
>>>>
>>>> Right, but that's a fix in the test itself, no an issue with the
>> cloudstack "core".
>>>>
>>>> In any case, great, one less blocker.
>>>>
>>>>> Koushik,
>>>>>
>>>>> modified the test "test_01_scale_vm" with required_hardware=false on
>> simulator setup and it works. So, I think we can let the second method
>> go. I will also test this on XS.
>>>>> Raja
>>>>>
>>>>>
>>>>> root@localhost:~/cloudstack# nosetests --with-marvin
>>>>> --marvin-config=/root/cloudstack/setup/dev/local1.cfg
>>>>> --zone=Sandbox-simulator --hypervisor=simulator -a
>>>>> tags=basic,required_hardware=false
>>>>> /root/cloudstack/test/integration/smoke/test_scale_vm.py
>>>>>
>>>>> ==== Marvin Init Started ====
>>>>>
>>>>> === Marvin Parse Config Successful ===
>>>>>
>>>>> === Marvin Setting TestData Successful===
>>>>>
>>>>> ==== Log Folder Path: /tmp//MarvinLogs//Sep_30_2015_08_06_32_AWNF1O.
>>>>> All logs will be available here ====
>>>>>
>>>>> === Marvin Init Logging Successful===
>>>>>
>>>>> ==== Marvin Init Successful ====
>>>>> ===final results are now copied to:
>>>>> /tmp//MarvinLogs/test_scale_vm_OPS7AD===
>>>>> root@localhost:~/cloudstack# cd /tmp//MarvinLogs/test_scale_vm_OPS7AD
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls
>>>>> failed_plus_exceptions.txt results.txt runinfo.txt
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# vi results.txt
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# ls -al total 48
>>>>> drwxr-xr-x 2 root root 4096 Sep 30 08:07 .
>>>>> drwxr-xr-x 8 root root 4096 Sep 30 08:06 ..
>>>>> -rw-r--r-- 1 root root 0 Sep 30 08:06 failed_plus_exceptions.txt
>>>>> -rw-r--r-- 1 root root 186 Sep 30 08:06 results.txt
>>>>> -rw-r--r-- 1 root root 36164 Sep 30 08:06 runinfo.txt
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD#
>>>>> root@localhost:/tmp/MarvinLogs/test_scale_vm_OPS7AD# cat results.txt
>>>>> Test scale virtual machine ... === TestName: test_01_scale_vm |
>>>>> Status
>>>>> : SUCCESS === ok
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> -
>>>>> Ran 1 test in 23.455s
>>>>>
>>>>> OK
>>>>>
>>>>> -----Original Message-----
>>>>> From: Sebastien Goasguen [mailto:runseb@gmail.com]
>>>>> Sent: Wednesday, September 30, 2015 1:04 PM
>>>>> To: dev@cloudstack.apache.org
>>>>> Subject: Re: [Blocker] test duplicated in test_scale_vm.py
>>>>>
>>>>>
>>>>>> On Sep 30, 2015, at 9:10 AM, Koushik Das <koushik.das@citrix.com>
>> wrote:
>>>>>>
>>>>>> Raja,
>>>>>> The earlier scale vm test couldn't run on simulator because of the
>> validation steps (IIRC it tries to connect to the real VM created during
>> test). Since it wasn't possible to run with simulator, the new test was
>> added to run just on the simulator. Check the required_hardware tag.
>>>>>>
>>>>>
>>>>> Ok so that's not really a blocker.
>>>>>
>>>>> Can we reduce the severity to major ?
>>>>> Or send a PR that will add this to the Travis run and run the correct
>> test for simulator ?
>>>>>
>>>>> Or make you modifications in your BVT settings, re-run and close the
>> issue ?
>>>>>
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Raja Pullela [mailto:raja.pullela@citrix.com]
>>>>>> Sent: Wednesday, 30 September 2015 12:12
>>>>>> To: CloudStack Dev (dev@cloudstack.apache.org)
>>>>>> Subject: [Blocker] test duplicated in test_scale_vm.py
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Filed this blocker on test case "test_scale_vm.py" -
>>>>>> https://issues.apache.org/jira/browse/CLOUDSTACK-8924
>>>>>> Fixing this will bring Simulator Basic and XS Basic to 100% passrate.
>>>>>>
>>>>>> Best,
>>>>>> Raja
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
>
> --
> Daan