New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
twister test case selection numbers don't make any sense #30100
Comments
I will copy here a comment I made some time ago. This might answer some of your questions [tl,dr nomenclature used in reporting is confusing and different names are used for the same thing]. Don't take it as a final answer, it is just to explain how I saw this back then:). We will discuss it during the next tester wg meeting to see if we can agree on sorting this up. There might be some extra issues with counting, will verify this as well. I will create a PR fixing this. If you have some use cases which give wrong counting it would be beneficial if you add them here, it would be easier for me to debug on this faulty behavior.
|
Agree this is starting to get confusing and we are mixing old with new terminologies.Some language can be changed here and we probably can drop some of the detailed statistics which might lead to confusion. In a nutshell however and historically a test in sanitycheck lingua is:
Let's dissect this with an example:
|
I started a gh discussion to sort this issue and define a consistent nomenclature. |
@andrewboie Can you have a look at #31999 Does the output make more sense with it? |
Let's take this as an example:
First we have reported that 14202 tests were selected, with 298472 discarded. These two numbers have no relationship to anything that follows.
We then get told:
From a holistic point of view these numbers really don't make much sense. It would be better to report stuff in a way where the relationships were clearer, and we have an clear understanding what any given value means and what it might be a subset of.
The text was updated successfully, but these errors were encountered: