2020-08-06

CTest - утилита для прогона тестов

2020-08-06 12:11
vak: (Default)
Для запуска большого количества тестов имеет смысл задействовать все имеющиеся на компьютере процессоры. К сожалению, и Googletest, и Catch функционируют только как однопоточные приложения. Чтобы распараллелиться, нужны внешние средства. Утилита CTest, входящая в пакет CMake, может использоваться для этой цели. Она вполне неплохо работает автономно от CMake, надо только подготовить для неё список тестов. Кроме того, CTest умеет порождать отчёты в разных форматах.

К примеру, добавим к Googletest-Pipelines-Demo файл CTestTestfile.cmake со следующим содержимым:
add_test(fibonacci.check_valid ./demo --gtest_filter=fibonacci.check_valid)
add_test(fibonacci.check_overflow ./demo --gtest_filter=fibonacci.check_overflow)
Теперь можем запускать тесты следующим образом:
$ ctest
Test project /Users/vak/Project/TDD/Googletest-Pipelines-Demo
    Start 1: fibonacci.check_valid
1/2 Test #1: fibonacci.check_valid ............   Passed    0.01 sec
    Start 2: fibonacci.check_overflow
2/2 Test #2: fibonacci.check_overflow .........   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.03 sec
Или так:
$ ctest -V
Test project /Users/vak/Project/TDD/Googletest-Pipelines-Demo
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: fibonacci.check_valid

1: Test command: /Users/vak/Project/TDD/Googletest-Pipelines-Demo/demo "--gtest_filter=fibonacci.check_valid"
1: Test timeout computed to be: 10000000
1: Note: Google Test filter = fibonacci.check_valid
1: [==========] Running 1 test from 1 test suite.
1: [----------] Global test environment set-up.
1: [----------] 1 test from fibonacci
1: [ RUN      ] fibonacci.check_valid
1: [       OK ] fibonacci.check_valid (0 ms)
1: [----------] 1 test from fibonacci (0 ms total)
1: 
1: [----------] Global test environment tear-down
1: [==========] 1 test from 1 test suite ran. (0 ms total)
1: [  PASSED  ] 1 test.
1/2 Test #1: fibonacci.check_valid ............   Passed    0.00 sec
test 2
    Start 2: fibonacci.check_overflow

2: Test command: /Users/vak/Project/TDD/Googletest-Pipelines-Demo/demo "--gtest_filter=fibonacci.check_overflow"
2: Test timeout computed to be: 10000000
2: Note: Google Test filter = fibonacci.check_overflow
2: [==========] Running 1 test from 1 test suite.
2: [----------] Global test environment set-up.
2: [----------] 1 test from fibonacci
2: [ RUN      ] fibonacci.check_overflow
2: [       OK ] fibonacci.check_overflow (0 ms)
2: [----------] 1 test from fibonacci (0 ms total)
2: 
2: [----------] Global test environment tear-down
2: [==========] 1 test from 1 test suite ran. (0 ms total)
2: [  PASSED  ] 1 test.
2/2 Test #2: fibonacci.check_overflow .........   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.01 sec
Аналогичный пример для Catch: к Catch-Pipelines-Demo добавляем файл CTestTestfile.cmake:
add_test("Check valid input" ./demo -s --reporter=compact "Check valid input")
add_test("Check overflow" ./demo -s --reporter=compact "Check overflow")
Запускаем:
$ ctest
Test project /Users/vak/Project/TDD/Catch-Pipelines-Demo
    Start 1: Check valid input
1/2 Test #1: Check valid input ................   Passed    0.00 sec
    Start 2: Check overflow
2/2 Test #2: Check overflow ...................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.01 sec
С просмотром выдачи:
$ ctest -V
Test project /Users/vak/Project/TDD/Catch-Pipelines-Demo
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: Check valid input

1: Test command: /Users/vak/Project/TDD/Catch-Pipelines-Demo/demo "-s" "--reporter=compact" "Check valid input"
1: Test timeout computed to be: 10000000
1: demo_test.cpp:12: PASSED: fibonacci(0) == 0 for: 0 == 0
1: demo_test.cpp:13: PASSED: fibonacci(1) == 1 for: 1 == 1
1: demo_test.cpp:14: PASSED: fibonacci(2) == 1 for: 1 == 1
1: demo_test.cpp:15: PASSED: fibonacci(3) == 2 for: 2 == 2
1: demo_test.cpp:16: PASSED: fibonacci(4) == 3 for: 3 == 3
1: demo_test.cpp:17: PASSED: fibonacci(5) == 5 for: 5 == 5
1: demo_test.cpp:18: PASSED: fibonacci(10) == 55 for: 55 == 55
1: demo_test.cpp:19: PASSED: fibonacci(20) == 6765 for: 6765 (0x1a6d) == 6765 (0x1a6d)
1: demo_test.cpp:20: PASSED: fibonacci(30) == 832040 for: 832040 (0xcb228) == 832040 (0xcb228)
1: demo_test.cpp:21: PASSED: fibonacci(40) == 102334155 for: 102334155 (0x6197ecb)
1: demo_test.cpp:22: PASSED: fibonacci(50) == 12586269025 for: 12586269025 (0x2ee333961)
1: demo_test.cpp:23: PASSED: fibonacci(60) == 1548008755920 for: 1548008755920 (0x1686c8312d0)
1: demo_test.cpp:24: PASSED: fibonacci(70) == 190392490709135 for: 190392490709135 (0xad2934c6d08f)
1: demo_test.cpp:25: PASSED: fibonacci(80) == 23416728348467685 for: 23416728348467685 (0x533163ef0321e5)
1: demo_test.cpp:26: PASSED: fibonacci(90) == 2880067194370816120 for: 2880067194370816120 (0x27f80ddaa1ba7878)
1: demo_test.cpp:27: PASSED: fibonacci(93) == 12200160415121876738UL for: 12200160415121876738 (0xa94fad42221f2702)
1: Passed 1 test case with 16 assertions.
1: 
1/2 Test #1: Check valid input ................   Passed    0.00 sec
test 2
    Start 2: Check overflow

2: Test command: /Users/vak/Project/TDD/Catch-Pipelines-Demo/demo "-s" "--reporter=compact" "Check overflow"
2: Test timeout computed to be: 10000000
2: demo_test.cpp:35: PASSED: fibonacci(100)
2: Passed 1 test case with 1 assertion.
2: 
2/2 Test #2: Check overflow ...................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 2

Total Test time (real) =   0.01 sec
Файлы CTestTestfile.cmake можно создавать автоматически, запрашивая список тестов с помощью соответствующих флагов, и обрабатывая самодельным фильтром. Для Googletest:
$ ./demo --gtest_list_tests
fibonacci.
  check_valid
  check_overflow
Для Catch:
$ ./demo --list-test-names-only
Check valid input
Check overflow
По работе мне приходится много раз за день запускать сотни тестов. Многопроцессорность в разы ускоряет прогон:
$ ctest
Test project /Users/vak/Project/mla/build
        Start   1: simulator.c_api_test
  1/429 Test   #1: simulator.c_api_test ..........................................   Passed    0.01 sec
[...]
        Start 429: pytests/cond_test
429/429 Test #429: pytests/cond_test .............................................   Passed    0.86 sec

100% tests passed, 0 tests failed out of 429

Total Test time (real) =  17.34 sec

$ ctest -j12
Test project /Users/vak/Project/mla/build
[...]
429/429 Test #423: pytests/pool_test_negative ....................................   Passed    3.42 sec

100% tests passed, 0 tests failed out of 429

Total Test time (real) =   3.45 sec
Как видите, с флагом -j12 (на двенадцати ядрах Intel Core i9) дело продвигается в пять раз быстрее (17.34sec / 3.45sec).

Старая шутка

2020-08-06 21:54
vak: (Default)
 яндекс херня