I have been working on an internal project based on PostgreSQL for a while, and from time to time, I need to run some specific test cases to verify my changes. Here, I want to shared a tip to run a specific regression TAP test quickly, especially, when you are focusing on a particular bug and you know which test case can help verify the fix. A details document about the regression test can be found at Running the Tests.
2. Regression test
PostgreSQL provides a comprehensive set of regression tests to verify the SQL implementation embedded in PostgreSQL as well as the extended capabilities of PostgreSQL. Whenever you make some changes, you should run these existing test cases to make sure your change doesn’t break any existing features. Other than these regression tests, there are some special features using a test framework call TAP test. For example, kerberos, ssl, recovery etc.
If you want to run these tests, you have to make sure the option --enable-tap-tests has been configured. for example, ./configure --prefix=$HOME/pgapp --enable-tap-tests --enable-debug CFLAGS="-g3 -O0 -fno-omit-frame-pointer"
You can run the TAP test using either make check or make installcheck, but compared with those non-TAP tests, the different is that these TAP tests will always start a test server even you run make installcheck. Because of this different, some tests may take a longer time than you expected, and even worse, if some test cases failed in the middle then the entire test will stop, and your test cases may never get the chance to run. For example, I changed somethings related to the recovery features, and those changes suppose to be tested by test cases 021_row_visibility.pl and 025_stuck_on_old_timeline.pl, but whenever I run make check or make installcheck, it ends up with something like below.
t/001_stream_rep.pl .................. ok t/002_archiving.pl ................... ok t/003_recovery_targets.pl ............ ok t/004_timeline_switch.pl ............. ok t/005_replay_delay.pl ................ ok t/006_logical_decoding.pl ............ ok t/007_sync_rep.pl .................... ok t/008_fsm_truncation.pl .............. ok t/009_twophase.pl .................... ok t/010_logical_decoding_timelines.pl .. ok t/011_crash_recovery.pl .............. ok t/012_subtransactions.pl ............. ok t/013_crash_restart.pl ............... ok t/014_unlogged_reinit.pl ............. ok t/015_promotion_pages.pl ............. ok t/016_min_consistency.pl ............. ok t/017_shm.pl ......................... ok t/018_wal_optimize.pl ................ ok t/019_replslot_limit.pl .............. 11/20 Bailout called. Further testing stopped: pg_ctl start failed FAILED--Further testing stopped: pg_ctl start failed Makefile:23: recipe for target 'check' failed make: *** [check] Error 255
Now, 019_replslot_limit.pl always failed in the middle, but those test cases to verify my changes haven’t got the chance to run yet.
3. How to run a specific test?
To run a specific test cases, the key is to use a variable PROVE_TESTS provided by PostgreSQL. Details can be found at TAP Tests. This PROVE_TESTS variable allow to define a whitespace separated list of paths to run the specified subset of tests instead of the default t/*.pl. For example: in above case, you can run make check PROVE_TESTS='t/021_row_visibility.pl t/025_stuck_on_old_timeline.pl'. It will run these two test cases directly. The output is something like below.
recovery$ make check PROVE_TESTS='t/021_row_visibility.pl t/025_stuck_on_old_timeline.pl' make -C ../../../src/backend generated-headers make[1]: Entering directory '/home/sandbox/sharedsm/src/backend' make -C catalog distprep generated-header-symlinks make[2]: Entering directory '/home/sandbox/sharedsm/src/backend/catalog' make[2]: Nothing to be done for 'distprep'. make[2]: Nothing to be done for 'generated-header-symlinks'. make[2]: Leaving directory '/home/sandbox/sharedsm/src/backend/catalog' make -C utils distprep generated-header-symlinks make[2]: Entering directory '/home/sandbox/sharedsm/src/backend/utils' make[2]: Nothing to be done for 'distprep'. make[2]: Nothing to be done for 'generated-header-symlinks'. make[2]: Leaving directory '/home/sandbox/sharedsm/src/backend/utils' make[1]: Leaving directory '/home/sandbox/sharedsm/src/backend' rm -rf '/home/sandbox/sharedsm'/tmp_install /bin/mkdir -p '/home/sandbox/sharedsm'/tmp_install/log make -C '../../..' DESTDIR='/home/sandbox/sharedsm'/tmp_install install >'/home/sandbox/sharedsm'/tmp_install/log/install.log 2>&1 make -j1 checkprep >>'/home/sandbox/sharedsm'/tmp_install/log/install.log 2>&1 rm -rf '/home/sandbox/sharedsm/src/test/recovery'/tmp_check /bin/mkdir -p '/home/sandbox/sharedsm/src/test/recovery'/tmp_check cd . && TESTDIR='/home/sandbox/sharedsm/src/test/recovery' PATH="/home/sandbox/sharedsm/tmp_install/home/sandbox/pgapp/bin:$PATH" LD_LIBRARY_PATH="/home/sandbox/sharedsm/tmp_install/home/sandbox/pgapp/lib:$LD_LIBRARY_PATH" PGPORT='65432' PG_REGRESS='/home/sandbox/sharedsm/src/test/recovery/../../../src/test/regress/pg_regress' /usr/bin/prove -I ../../../src/test/perl/ -I . t/021_row_visibility.pl t/025_stuck_on_old_timeline.pl t/021_row_visibility.pl ......... ok t/025_stuck_on_old_timeline.pl .. ok All tests successful. Files=2, Tests=11, 13 wallclock secs ( 0.01 usr 0.00 sys + 1.73 cusr 4.03 csys = 5.77 CPU) Result: PASS
Of course, if you know the makefile very well, you can also do it on your own way. For example, by looking at the output, you can simply do in below steps to achieve the same results.
In this blog, I explained how to run a specific test case by using variable PROVE_TESTS for TAP test. You can also run the test manually to skip some tests either take too much time or may failed in the middle and block your test cases.
