Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Twister mark successfully passed tests as failed #34571

Closed
evgeniy-paltsev opened this issue Apr 26, 2021 · 6 comments
Closed

Twister mark successfully passed tests as failed #34571

evgeniy-paltsev opened this issue Apr 26, 2021 · 6 comments
Assignees
Labels
area: Test Framework Issues related not to a particular test, but to the framework instead area: Twister Twister bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug

Comments

@evgeniy-paltsev
Copy link
Collaborator

Recently I've noticed that some of successfully passed tests are marked as failed.
As I can see it only happens with QEMU platforms (I've seen this issue on qemu_x86, qemu_arc_hs, qemu_arc_em, etc...)

Here is an example. Some tests are marked as FAILED Timeout (qemu) or FAILED unexpected eof (qemu 1.644s):

scripts/twister -v --platform qemu_x86 --report-dir /SCRATCH/arcjenkins2/slaves/ru20-custom-arcgnu3/workspace/arcoss_verification/zephyr_verification/sanity_check-6/jenkins-jobs/zephyr-sanity-check/../../logs
INFO    - Zephyr version: zephyr-v2.5.0-2577-g3eb05e54568e
INFO    - JOBS: 64
INFO    - Using 'zephyr' toolchain.
INFO    - Building initial testcase list...
INFO    - 1207 test scenarios (1152 configurations) selected, 635 configurations discarded due to filters.
INFO    - Adding tasks to the queue...
INFO    - Added initial list of jobs to queue
INFO    -   1/517 qemu_x86                  samples/subsys/logging/logger/sample.logger.rtt    SKIPPED (filter)
 [--snip--]
INFO    - 157/517 qemu_x86                  tests/subsys/settings/nvs/raw/system.settings.nvs  PASSED (qemu 2.215s)
INFO    - 158/517 qemu_x86                  tests/ztest/mock/testing.ztest.mock                FAILED unexpected eof (qemu 1.644s)
ERROR   - see: /SCRATCH/arcjenkins2/slaves/ru20-custom-arcgnu3/workspace/arcoss_verification/zephyr_verification/sanity_check-6/twister-out/qemu_x86/tests/ztest/mock/testing.ztest.mock/handler.log
INFO    - 159/517 qemu_x86                  tests/subsys/shell/shell/shell.min_cmds_select     PASSED (build)
INFO    - 160/517 qemu_x86                  tests/subsys/shell/shell_history/shell.history     FAILED unexpected eof (qemu 1.423s)
ERROR   - see: /SCRATCH/arcjenkins2/slaves/ru20-custom-arcgnu3/workspace/arcoss_verification/zephyr_verification/sanity_check-6/twister-out/qemu_x86/tests/subsys/shell/shell_history/shell.history/handler.log
INFO    - 161/517 qemu_x86                  tests/subsys/shell/shell/shell.min_cmds            PASSED (build)
INFO    - 162/517 qemu_x86                  tests/ztest/base/testing.ztest.verbose_0           FAILED Timeout (qemu)
ERROR   - see: /SCRATCH/arcjenkins2/slaves/ru20-custom-arcgnu3/workspace/arcoss_verification/zephyr_verification/sanity_check-6/twister-out/qemu_x86/tests/ztest/base/testing.ztest.verbose_0/handler.log
INFO    - 163/517 qemu_x86                  tests/subsys/logging/log_msg2/logging.log_msg2_no_overflow PASSED (qemu 2.149s)
INFO    - 164/517 qemu_x86                  tests/subsys/logging/log_msg2/logging.log_msg2_64b_timestamp PASSED (qemu 2.147s)

However we can see PROJECT EXECUTION SUCCESSFUL in the corresponding logs:

/SCRATCH/arcjenkins2/slaves/ru20-custom-arcgnu3/workspace/arcoss_verification/zephyr_verification/sanity_check-6/twister-out/qemu_x86/tests/ztest/mock/testing.ztest.mock/handler.log
--------------------
�[0mSeaBIOS (version zephyr-v1.0.0-0-g31d4e0e-dirty-20200714_234759-fv-az50-zephyr)
Booting from ROM..*** Booting Zephyr OS build zephyr-v2.5.0-2577-g3eb05e54568e  ***
Running test suite mock_framework_tests
===================================================================
START - test_parameter_tests
 PASS - test_parameter_tests in 0.1 seconds
===================================================================
START - test_return_value_tests
 PASS - test_return_value_tests in 0.1 seconds
===================================================================
START - test_multi_value_tests
 PASS - test_multi_value_tests in 0.1 seconds
===================================================================
START - test_return_data_tests
 PASS - test_return_data_tests in 0.1 seconds
===================================================================
Test suite mock_framework_tests succeeded
===================================================================
PROJECT EXECUTION SUCCESSFUL


/SCRATCH/arcjenkins2/slaves/ru20-custom-arcgnu3/workspace/arcoss_verification/zephyr_verification/sanity_check-6/twister-out/qemu_x86/tests/subsys/shell/shell_history/shell.history/handler.log
--------------------
�[0mSeaBIOS (version zephyr-v1.0.0-0-g31d4e0e-dirty-20200714_234759-fv-az50-zephyr)
Booting from ROM..*** Booting Zephyr OS build zephyr-v2.5.0-2577-g3eb05e54568e  ***
Running test suite shell_test_suite
===================================================================
START - test_history_add_get
 PASS - test_history_add_get in 0.1 seconds
===================================================================
START - test_history_purge
 PASS - test_history_purge in 0.1 seconds
===================================================================
START - test_history_get_up_and_down
 PASS - test_history_get_up_and_down in 0.1 seconds
===================================================================
START - test_too_long_line_not_stored
 PASS - test_too_long_line_not_stored in 0.1 seconds
===================================================================
START - test_no_duplicates_in_a_row
 PASS - test_no_duplicates_in_a_row in 0.1 seconds
===================================================================
START - test_storing_long_buffers
 PASS - test_storing_long_buffers in 0.1 seconds
===================================================================
Test suite shell_test_suite succeeded
===================================================================
PROJECT EXECUTION SUCCESSFUL


/SCRATCH/arcjenkins2/slaves/ru20-custom-arcgnu3/workspace/arcoss_verification/zephyr_verification/sanity_check-6/twister-out/qemu_x86/tests/ztest/base/testing.ztest.verbose_0/handler.log
--------------------
�[0mSeaBIOS (version zephyr-v1.0.0-0-g31d4e0e-dirty-20200714_234759-fv-az50-zephyr)
Booting from ROM..*** Booting Zephyr OS build zephyr-v2.5.0-2577-g3eb05e54568e  ***
Running test suite framework_tests
===================================================================
START - test_empty_test
 PASS - test_empty_test in 0.1 seconds
===================================================================
START - test_assert_tests
 PASS - test_assert_tests in 0.1 seconds
===================================================================
START - test_assert_mem_equal
 PASS - test_assert_mem_equal in 0.1 seconds
===================================================================
Test suite framework_tests succeeded
===================================================================
PROJECT EXECUTION SUCCESSFUL

NOTE: it doesn't reproduces all the time. The incorrectly marked tests differs between lauches.

Environment (please complete the following information):

  • OS: Linux
  • Toolchain: Zephyr SDK 0.12.3
  • Zephyr commit SHA: 3eb05e5 (it reproduces on older Zephyr commits as well)
  • West version: v0.10.1
  • cmake version: 3.16.3
  • make version: GNU Make 4.3
@evgeniy-paltsev evgeniy-paltsev added bug The issue is a bug, or the PR is fixing a bug area: Twister Twister area: Test Framework Issues related not to a particular test, but to the framework instead labels Apr 26, 2021
@evgeniy-paltsev
Copy link
Collaborator Author

I'm curious if someone have seen similar issues with twister?

@ioannisg ioannisg added the priority: medium Medium impact/importance bug label Apr 27, 2021
@hakehuang
Copy link
Collaborator

looks like the unexpected eof break the twister process to check result.

@nashif nashif added priority: low Low impact/importance bug and removed priority: medium Medium impact/importance bug labels May 26, 2021
@nashif
Copy link
Member

nashif commented May 26, 2021

I am still not able to reproduce this, lowering priority

@github-actions
Copy link

This issue has been marked as stale because it has been open (more than) 60 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this issue will automatically be closed in 14 days. Note, that you can always re-open a closed issue at any time.

@mengxianglinx
Copy link
Collaborator

Now PR #48848 can reproduce this issue, both in CI and run on local host.

@mengxianglinx mengxianglinx reopened this Aug 10, 2022
@github-actions github-actions bot removed the Stale label Aug 11, 2022
@nashif
Copy link
Member

nashif commented Aug 12, 2022

closing, as this is a duplicate of #49004

@nashif nashif closed this as completed Aug 12, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: Test Framework Issues related not to a particular test, but to the framework instead area: Twister Twister bug The issue is a bug, or the PR is fixing a bug priority: low Low impact/importance bug
Projects
None yet
Development

No branches or pull requests

5 participants