Over on https://bugs.debian.org/1058571, Noodles reports that the test suite needs to run serially when tpm2daemon uses swtpm. I confirmed this, noting the error message:

Executing: '…/build/../tests/tpm2dtests/start_sw_tpm.sh'
Child 2184896 returned: ((command ("…/build/../tests/tpm2dtests/start_sw_tpm.sh")) (status 1) (stdout ) (stderr swtpm: Could not open TCP socket: Address already in use
cat: swtpm.pid: No such file or directory
)) 
("…/build/../tests/tpm2dtests/start_sw_tpm.sh") failed: swtpm: Could not open TCP socket: Address already in use
cat: swtpm.pid: No such file or directory

I think the right approach would be to connect swtpm to a unix-domain socket, and pass the socket through into the tests, rather than relying on TCP-based sockets. Unix-domain sockets have better access control properties as well,

As noodles notes:

It should be possible to use Unix sockets instead, but AFAICT that's
going to require patching tpm2d/intel-tss.h to set the appropriate
parameters on TCTI setup, and hacking up the Scheme test files to pass
those through to the tests.