It looks like the S/MIME settings (disable/prefer) are ignored:
a) Regardless of being disabled, decryption seems to work and pinentry shows up for the smime key
b) Regardless of preference, openpgp keys are used for encryption (when security approval dialog is disabled)

a) To reproduce the disable issue:

1. Disable smime
2. Kill background processes and restart Outlook
3. Open a smime mail => pinentry shows up and mail is decrypted

b) To reproduce the preference issue:

1. Make sure you have both secret openpgp and smime certs (I tested with ted)
2. Deactivate "Always show security approval dialog"
3. Enable S/MIME and activate "Prefer S/MIME"
4. Kill background processes and restart Outlook (just to be sure)
5. Send an encrypted/signed mail => always openpgp encrypted/signed

Note: I had both secret openpgp and smime keys of ted/edward in keyring and both accounts in outlook added, if that matters