Testing - Csound

Overview

Csound has a comprehensive testing framework covering unit tests, integration tests, and regression tests. Tests help ensure code quality, prevent regressions, and document expected behavior.

Test categories

Unit tests (tests/c)

Unit tests are written in C/C++ using the GoogleTest framework. They test individual functions and components in isolation. Location: tests/c/ Framework: GoogleTest (GTest) Purpose: Test compiler internals, API methods, and core functionality Examples:

csound_orc_compile_test.cpp - Compiler tests
csound_type_system_test.cpp - Type system tests
channel_tests.cpp - Channel API tests
csound_circular_buffer_test.cpp - Circular buffer tests

Integration tests (tests/commandline)

Integration tests use CSD (Csound Document) files executed by a Python test runner. They test the complete system with real Csound code. Location: tests/commandline/ Runner: Python script (test.py) Purpose: Test opcodes, compiler, and runtime behavior File format: .csd files with embedded Csound code

Regression tests (tests/regression)

A collection of tests for previously reported bugs to ensure they remain fixed. Location: tests/regression/ Purpose: Prevent regression of fixed bugs

Soak tests (tests/soak)

Large-scale tests that run most examples from the Csound manual. Location: tests/soak/ Purpose: Comprehensive testing using manual examples Features:

MD5 checksums for audio output comparison
Diff for text output comparison
Detects changes since previous run

Building tests

Enable test building

Tests are not built by default. Enable them with the BUILD_TESTS option:

cmake -DBUILD_TESTS=1 ..
make

Build requirements

GoogleTest: Required for unit tests (auto-installed with vcpkg)
Static library: Unit tests link against the static Csound library
Python: Required for integration test runner

Build with vcpkg

cmake -B build -S . -DUSE_VCPKG=1 -DBUILD_TESTS=1
cmake --build build

Running tests

All tests

Run both unit and integration tests:

cd build
make test
make csdtests

Unit tests only

Run GoogleTest unit tests:

cd build
make test

Or use CTest directly:

cd build
ctest

CTest options

Verbose output:

ctest --verbose

Run specific test:

ctest -R test_name_pattern

Parallel execution:

ctest -j8

Integration tests (CSD tests)

Run CSD integration tests:

cd build
make csdtests

Verbose output:

make csdtests-verbose

Manual execution

Run the Python test runner directly:

cd tests/commandline
python test.py --csound-executable=../../build/csound \
               --source-dir=. \
               --opcode7dir64=../../build

Test runner options

--csound-executable: Path to Csound binary
--source-dir: Test source directory
--opcode7dir64: Opcode plugin directory
--verbose: Enable verbose output
--runtime-environment: Runtime wrapper (e.g., wine, Node.js)

Platform-specific testing

Linux

cd build
make test
make csdtests

macOS

cd build
make test
make csdtests

Windows (MSVC)

cd build
cmake --build . --target RUN_TESTS --config Release
cmake --build . --target csdtests

MinGW with Wine

cd build
make test
make csdtests  # Uses Wine to execute Windows binaries

Emscripten

cd build
make test  # Uses Node.js to execute WebAssembly
make csdtests

Writing tests

Writing unit tests

Unit tests use GoogleTest. Add new test files to tests/c/CMakeLists.txt:

add_executable(unittests
    channel_tests.cpp
    csound_type_system_test.cpp
    your_new_test.cpp  # Add your test file
    # ...
)

Example unit test

#include <gtest/gtest.h>
#include "csound.h"

TEST(YourTestSuite, TestName) {
    CSOUND *csound = csoundCreate(NULL);
    ASSERT_NE(csound, nullptr);
    
    // Your test code here
    int result = csoundCompile(csound, argc, argv);
    EXPECT_EQ(result, 0);
    
    csoundDestroy(csound);
}

TEST(YourTestSuite, AnotherTest) {
    // Another test case
    EXPECT_TRUE(some_condition);
}

GoogleTest assertions

EXPECT_EQ(a, b) - Expect equality
EXPECT_NE(a, b) - Expect inequality
EXPECT_TRUE(condition) - Expect true
EXPECT_FALSE(condition) - Expect false
ASSERT_* variants - Fatal assertions (stop test on failure)

Writing integration tests

Integration tests are CSD files placed in tests/commandline/.

Example CSD test

<CsoundSynthesizer>
<CsOptions>
-n -d -m0
</CsOptions>
<CsInstruments>

sr = 44100
ksmps = 32
nchnls = 2
0dbfs = 1

instr 1
    aOut oscil 0.5, 440
    outch 1, aOut, 2, aOut
endin

</CsInstruments>
<CsScore>
i 1 0 0.1
e
</CsScore>
</CsoundSynthesizer>

Test organization

Organize tests by category:

tests/commandline/opcodes/ - Opcode tests
tests/commandline/errors/ - Error handling tests
tests/commandline/regression/ - Regression tests

Writing regression tests

When fixing a bug:

Create a minimal test case that reproduces the bug
Add the test to tests/regression/
Verify the test fails before the fix
Verify the test passes after the fix

This ensures the bug won’t reoccur in future versions.

Continuous integration testing

All tests run automatically on CI for every pull request. See the CI/CD guide for details.

CI test coverage

CI runs tests on:

Linux: Ubuntu with apt-get and vcpkg
macOS: Homebrew and installer builds
Windows: MSVC 64-bit and 32-bit, MinGW
Mobile: iOS and Android
Web: WebAssembly/Emscripten
Embedded: ARM Cortex-M7 bare metal

CI test commands

CI uses these commands to run tests:

# Unit tests
cmake --build build --target test

# Integration tests
cmake --build build --target csdtests

# Windows MSVC
cmake --build build --target RUN_TESTS --config Release

Test configuration

CMakeLists.txt for unit tests

Unit tests are configured in tests/c/CMakeLists.txt:

if(BUILD_TESTS)
    find_package(GTest CONFIG REQUIRED)
    
    add_executable(unittests
        channel_tests.cpp
        csound_orc_compile_test.cpp
        # ... more test files
    )
    
    target_compile_features(unittests PUBLIC cxx_std_17)
    
    target_link_libraries(unittests
        PRIVATE
            GTest::gtest_main
            ${CSOUNDLIB_STATIC}
    )
    
    include(GoogleTest)
    gtest_discover_tests(unittests)
endif()

CMakeLists.txt for integration tests

Integration tests are configured in tests/commandline/CMakeLists.txt:

# Common test arguments
set(COMMON_TEST_ARGS 
    --csound-executable=${CMAKE_BINARY_DIR}/csound 
    --source-dir=${CMAKE_CURRENT_SOURCE_DIR}
)

# Platform-specific configuration
if(MSVC)
    set(CSOUND_EXECUTABLE_PATH ${CMAKE_BINARY_DIR}/${CMAKE_BUILD_TYPE}/csound)
    add_custom_target(csdtests
        ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test.py
        --csound-executable=${CSOUND_EXECUTABLE_PATH}
        --opcode7dir64=${CMAKE_BINARY_DIR}/${CMAKE_BUILD_TYPE}
        --source-dir=${CMAKE_CURRENT_SOURCE_DIR}
    )
else()
    add_custom_target(csdtests
        ${Python_EXECUTABLE} ${CMAKE_CURRENT_SOURCE_DIR}/test.py
        ${COMMON_TEST_ARGS}
        --opcode7dir64=${CMAKE_BINARY_DIR}
    )
endif()

Debugging test failures

Run single unit test

cd build
./unittests --gtest_filter="TestSuiteName.TestName"

Run with verbose output

cd build
./unittests --gtest_filter="TestSuiteName.*" --gtest_verbose

Debug CSD test

Run a specific CSD file directly:

cd build
./csound ../tests/commandline/test_file.csd

With debug output:

./csound -v ../tests/commandline/test_file.csd

Address sanitizer

On macOS in Debug mode, address sanitizer is enabled by default:

cmake -DCMAKE_BUILD_TYPE=Debug -DUSE_ASA=1 ..
make
make test

This helps detect memory leaks and buffer overflows.

Best practices

Test isolation

Each test should be independent
Clean up resources (destroy Csound instances)
Don’t rely on test execution order

Test coverage

Test both success and failure paths
Test edge cases and boundary conditions
Test error handling

Test maintenance

Keep tests simple and focused
Use descriptive test names
Update tests when behavior changes
Remove obsolete tests

Performance

Keep unit tests fast (milliseconds)
Use minimal CSD files for integration tests
Avoid unnecessary file I/O

Troubleshooting

GoogleTest not found

Enable vcpkg to automatically install GoogleTest:

cmake -DUSE_VCPKG=1 -DBUILD_TESTS=1 ..

Static library not built

Unit tests require the static library:

cmake -DBUILD_STATIC_LIBRARY=1 -DBUILD_TESTS=1 ..

Python not found

Ensure Python is in your PATH and CMake can find it:

which python
python --version

Test timeout

Increase CTest timeout:

ctest --timeout 300

Opcode directory not found

Set the opcode directory manually:

export OPCODE7DIR64=./build
make csdtests

​Overview

​Test categories

​Unit tests (tests/c)

​Integration tests (tests/commandline)

​Regression tests (tests/regression)

​Soak tests (tests/soak)

​Building tests

​Enable test building

​Build requirements

​Build with vcpkg

​Running tests

​All tests

​Unit tests only

​CTest options

​Integration tests (CSD tests)

​Manual execution

​Test runner options

​Platform-specific testing

​Linux

​macOS

​Windows (MSVC)

​MinGW with Wine

​Emscripten

​Writing tests

​Writing unit tests

​Example unit test

​GoogleTest assertions

​Writing integration tests

​Example CSD test

​Test organization

​Writing regression tests

​Continuous integration testing

​CI test coverage

​CI test commands

​Test configuration

​CMakeLists.txt for unit tests

​CMakeLists.txt for integration tests

​Debugging test failures

​Run single unit test

​Run with verbose output

​Debug CSD test

​Address sanitizer

​Best practices

​Test isolation

​Test coverage

​Test maintenance

​Performance

​Troubleshooting

​GoogleTest not found

​Static library not built

​Python not found

​Test timeout

​Opcode directory not found