Twister: test hierarchy, nomenclature and output #31090

PerMac · 2021-01-04T16:02:35Z

PerMac
Jan 4, 2021
Collaborator

Hi. As it was already seen by users (#30100), the nomenclature used in testing with Twister is confusing. The same "object" can be referred to by different names and there still exists some wording issues in Twister output. I created a small presentation (Zephyr test hierarchy and nomenclature.pdf ) showing the structure of tests and the names I suggest we can start using to avoid confusion. Most of the naming is already used in zephyr, however, not consistently. One of the currently most confusing is "test case" https://docs.zephyrproject.org/latest/guides/test/twister.html#test-cases. The presentation also explains the current output of Twister, what and how is counted, and what errors are still present there.
The goal of this discussion is to define/agree on the testing nomenclature used at different levels. Once it is done we can start fixing the documentation so the naming is consistent.
I think that we can also have here a good place to discuss what and how detailed we would like to have in the twister output.
In the presentation, I put in square brackets optional expansions of names. I think that in the documentation it is better to use the full name to reduce confusion. I tried to sort it out in a way that short names are unique for each "object".

nashif · 2021-01-18T15:07:55Z

nashif
Jan 18, 2021
Maintainer

@PerMac maybe attach the PDF in here instead of the external link

1 reply

PerMac Jan 18, 2021
Collaborator Author

I didn't know how to do this. It seems it can be done by just dragging and dropping a file into comment :)

PerMac · 2021-01-18T15:24:20Z

PerMac
Jan 18, 2021
Collaborator Author

Zephyr test hierarchy and nomenclature.pdf

0 replies

nashif · 2021-01-18T16:14:38Z

nashif
Jan 18, 2021
Maintainer

seems reasonable in general with the following comments:

some yaml configuration describe tests, not configurations or scenarios, most in samples:

common:
    tags: introduction
    integration_platforms:
      - native_posix
    harness: console
    harness_config:
      type: one_line
      regex:
        - "Hello World! (.*)"
tests:
  sample.basic.helloworld:
    tags: introduction

So we need a better way to describe those and keep things consistent. There are also tests that are not ztest based which follow the same model.

We need to reflect this in the generated junit files, so for example, for one test on 1 single platform: : twister -p native_posix -T tests/kernel/semaphore/ --platform-reports currently we get:

<?xml version="1.0"?>
<testsuites>
  <testsuite name="native_posix" time="2.017428" tests="23" failures="0" errors="0" skipped="0">
    <properties>
      <property name="version" value="zephyr-v2.4.0-3165-ge4f95863f98e"/>
    </properties>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.k_sem_define" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.k_sem_init" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_thread2thread" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_thread2isr" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_reset" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_count_get" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_give_from_isr" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_give_from_thread" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_no_wait" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_no_wait_fails" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_timeout_fails" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_timeout" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_timeout_forever" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_timeout_isr" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_take_multiple" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_give_take_from_isr" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.k_sem_correct_count_limit" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_multiple_threads_wait" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_measure_timeouts" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_measure_timeout_from_thread" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_multiple_take_and_timeouts" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_multi_take_timeout_diff_sem" time="2.017428"/>
    <testcase classname="kernel.semaphore" name="kernel.semaphore.sem_queue_mutual_exclusion" time="2.017428"/>
  </testsuite>
</testsuites>

testsuite, classname and name need to reflect whatever we have in the proposal, so testsuite would be the actual testsuite for example. When you run multiple testsuite on a platform, we need to end up with multiple testsuite in the XML file.

testsuite could be whatever we have in the source file, for example here:

        ztest_test_suite(test_semaphore,
                         ztest_user_unit_test(test_k_sem_define),
                         ztest_user_unit_test(test_k_sem_init),

so, test_semaphore. or semaphore or some other name we can come up with.

0 replies

LeiW000 · 2021-01-26T13:29:01Z

LeiW000
Jan 26, 2021
Collaborator

<testcase classname="kernel.semaphore" name="kernel.semaphore.sem_queue_mutual_exclusion" time="2.017428"/>

I still want to suggest to add "test_" back to the test case name in junit file. For example, kernel.semaphore.**test_**sem_queue_mutual_exclusion.

0 replies

erwango · 2021-01-28T10:06:56Z

erwango
Jan 28, 2021
Collaborator

The proposal seems fine, but I don't think the unit_test (or test case as called in the pdf), should be reported at high level.
If a configuration instance (a build) as a test case failed, it should be reported as failed. We can provide ways to get more details using additional commands, but I don't think twister should report this detail of failure. So on that particular aspect, the current way of working is fine.

I agree on other parts of the proposal.

0 replies

PerMac · 2021-01-28T15:32:10Z

PerMac
Jan 28, 2021
Collaborator Author

I saw @nashif using the name "test app" in the testing channel on Slack. Maybe it is a better name for "[test] configuration [instance]" from my proposal? IMO "test app" feels more intuitive, as something that can be executed on a device/qemu?

1 reply

erwango Jan 29, 2021
Collaborator

What about "test build" or "test binary" ? That's prosaic but unambiguous.

nashif · 2021-02-10T13:50:07Z

nashif
Feb 10, 2021
Maintainer

to move forward with this:

extract all of the definitions discussed here and the in the PDF and submit those to the documentation, preferably in a new section. Basically describe the hierarchy with new terminology cleanly. I suggest to do this in a forward looking way, meaning that we should not be forced to accommodate things that are suboptimal and not well-defined.
Map the current definitions as we have them in twister, testcase.yaml, ztest and junit and identify where changes need to be made. I know for a fact that TestSuite, TestCase and TestInstance in twister do NOT map correctly to any of the other areas. TestSuite in twister is actually not a Test Suite as used in junit or ztest for example, same for TestCase.. So we need to fix this across the board and eliminate confusion.
Start from ztest (and probably look at Unity as well, we might start using this at some point) and go up: ztest -> testcase.yaml -> twister -> junit (reporting in general)

0 replies

nashif · 2021-03-01T18:12:32Z

nashif
Mar 1, 2021
Maintainer

@PerMac was toying with output from twister based on recent proposed changes to terminology can have the following output for

twister -c --platform-report -p native_posix -T tests/kernel/sched/schedule_api/ -T tests/kernel/fifo/

Please give feedback.

report.xml.txt

1 reply

nashif Mar 2, 2021
Maintainer

another optimization:

report.xml.txt

PerMac · 2021-03-02T15:41:00Z

PerMac
Mar 2, 2021
Collaborator Author

I like the new output. The thing I caught is that the structure for samples don't match (BTW how it ended in the report???):
testsuite name="samples/hello_world/sample.basic.helloworld"
as compared to:
testsuite name="test_fifo_timeout"
I am in the middle of going through the PR with the changes. Haven't run anything by myself yet. Let me know if the PR is in a runnable state and I will set our internal CI to checkout on it so I can see how Jenkins is handling this format.

In addition, I would opt for adding .default to the scenario name if there is only module.submodule in the name. I believe it would make it cleaner, that the cases like below are on the same level, just differ by the scenario (for me it looks like .no_timeslicing is at lower level due to parent.child structure in objective languages)
<testcase classname="kernel.scheduler" name="test_bad_priorities" time="0"/>
and
<testcase classname="kernel.scheduler.no_timeslicing" name="test_bad_priorities" time="0"/>

1 reply

nashif Mar 2, 2021
Maintainer

(BTW how it ended in the report???):

oh, I added that to test the sample case.

The thing I caught is that the structure for samples don't match

we need to find a better way, right now I am just filling the gaps. One possibility is to autogenerate that from the sample name, i.e. test_hello_world or have the sample name as is. Another options is to add this is a key/value in the yaml file.

I am in the middle of going through the PR with the changes. Haven't run anything by myself yet. Let me know if the PR is in a runnable state and I will set our internal CI to checkout on it so I can see how Jenkins is handling this format.

it should be, however the changes leading to this output are still in my tree only.

In addition, I would opt for adding .default to the scenario name if there is only module.submodule in the name. I

fine with me.

PerMac · 2021-03-02T15:48:30Z

PerMac
Mar 2, 2021
Collaborator Author

The above relates already to the second file

0 replies

PerMac · 2021-03-02T15:54:22Z

PerMac
Mar 2, 2021
Collaborator Author

@nashif I think there still exists differentiation on sample.yaml and tests.yaml in you PR, right? Any reasons why keeping both of them? Maybe we can unify both names to tests.yaml? Those are treated the same in twister, no?

1 reply

nashif Mar 2, 2021
Maintainer

This will follow, I did not spend much time on this yet, should be easy and mostly involve renaming the files from sample.yaml to tests.yaml

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Twister: test hierarchy, nomenclature and output #31090

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 11 comments 5 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Navigation Menu

Twister: test hierarchy, nomenclature and output #31090

PerMac Jan 4, 2021 Collaborator

Replies: 11 comments · 5 replies

nashif Jan 18, 2021 Maintainer

PerMac Jan 18, 2021 Collaborator Author

PerMac Jan 18, 2021 Collaborator Author

nashif Jan 18, 2021 Maintainer

LeiW000 Jan 26, 2021 Collaborator

erwango Jan 28, 2021 Collaborator

PerMac Jan 28, 2021 Collaborator Author

erwango Jan 29, 2021 Collaborator

nashif Feb 10, 2021 Maintainer

nashif Mar 1, 2021 Maintainer

nashif Mar 2, 2021 Maintainer

PerMac Mar 2, 2021 Collaborator Author

nashif Mar 2, 2021 Maintainer

PerMac Mar 2, 2021 Collaborator Author

PerMac Mar 2, 2021 Collaborator Author

nashif Mar 2, 2021 Maintainer

PerMac
Jan 4, 2021
Collaborator

Replies: 11 comments 5 replies

nashif
Jan 18, 2021
Maintainer

PerMac Jan 18, 2021
Collaborator Author

PerMac
Jan 18, 2021
Collaborator Author

nashif
Jan 18, 2021
Maintainer

LeiW000
Jan 26, 2021
Collaborator

erwango
Jan 28, 2021
Collaborator

PerMac
Jan 28, 2021
Collaborator Author

erwango Jan 29, 2021
Collaborator

nashif
Feb 10, 2021
Maintainer

nashif
Mar 1, 2021
Maintainer

nashif Mar 2, 2021
Maintainer

PerMac
Mar 2, 2021
Collaborator Author

nashif Mar 2, 2021
Maintainer

PerMac
Mar 2, 2021
Collaborator Author

PerMac
Mar 2, 2021
Collaborator Author

nashif Mar 2, 2021
Maintainer