This is a follow-up of #725, where @mouse07410 asked about using PKCS#11 objects with TLS. Correct me if I'm wrong, but from my understanding it should already be possible to have a TLS server with the server's long-term key stored in a PKCS#11 device, now as the PK ops are tied to the Private_Key and Public_Key objects (which was introduced in #625). All you have to do is to create a PKCS11 key, e.g., a PKCS11_RSA_PrivateKey, and return it from Credentials_Manager::private_key_for(). TLS_Server will forward it to PK_Signer, which will select and sign with the PK signature op returned by PKCS11_RSA_PrivateKey::create_signature_op().

If you want more control over the handshake, e.g., you want do the ECDH key exchange ops in a PKCS#11 device, there is currently no way to do it. We are currently thinking about a new interface, let it be named TLS Adapter for now, for doing these operations in hardware. We will have to do these operations in a non-PKCS#11 compliant HSM, but we want to make it generic enough that PKCS#11 devices can be supported easily.

We will come up with a proposal for this use case probably in the next few weeks.

Update: Removed the chain validation part, of course it's already possible by overriding Credentials_Manager::verify_certificate_chain().

This pull request implements a Roughtime client API and CLI.

Roughtime is a authenticated time protocol developed by Google.

The protocol is under standardization, but there initial version is available one multiple public servers, including from Google and Cloulflare.

This PR implements this initial version since it's the one actually being used on the wild and the standard one is not finished yet.

• [x] UDP client
• [x] Roughtime client
• [x] CLI
• [x] Complete test suite

Apple released Xcode-13. When Botan is built with it, it fails tests. Both master and release-2, in the same way. On all my Macs.

Branch release-2

Configuration

./configure.py --prefix=/opt/local --with-os-features=security_framework,apple_keychain,commoncrypto,getentropy --with-openmp --with-commoncrypto --with-openssl --with-boost --with-lzma --with-bzip2 --with-zlib --with-sqlite3 --with-python-version=2.7 --with-sphinx --with-pdf --cc-abi-flags='-march=native -O3 -I/opt/local/include' 2>&1 | tee conf-out.txt

Configuration output:

$ cat conf-out.txt 
   INFO: ./configure.py invoked with options "--prefix=/opt/local --with-os-features=security_framework,apple_keychain,commoncrypto,getentropy --with-openmp --with-commoncrypto --with-openssl --with-boost --with-lzma --with-bzip2 --with-zlib --with-sqlite3 --with-python-version=2.7 --with-sphinx --with-pdf --cc-abi-flags=-march=native -O3 -I/opt/local/include"
   INFO: Configuring to build Botan 2.18.1 (revision git:b420a4545b0f9219a88c209e4c8c2474d519dfac)
   INFO: Running under 3.9.7 (default, Sep  1 2021, 12:35:15) [Clang 12.0.5 (clang-1205.0.22.9)]
   INFO: Implicit --cc-bin=clang++ due to environment variable CXX
   INFO: Implicit --cxxflags=-std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk due to environment variable CXXFLAGS
   INFO: Autodetected platform information: OS="Darwin" machine="x86_64" proc="i386"
   INFO: Guessing target OS is darwin (use --os to set)
   INFO: Guessing target processor is a x86_64 (use --cpu to set)
   INFO: Using /etc/ssl/cert.pem as system certificate store
   INFO: Auto-detected compiler version 4.0
   INFO: Auto-detected compiler arch x86_64
   INFO: Target is clang:4.0-macos-x86_64
   INFO: Assuming target x86_64 is little endian
   INFO: Skipping (incompatible CPU): aes_armv8 aes_power8 sha1_armv8 sha2_32_armv8 sm4_armv8
   INFO: Skipping (incompatible OS): certstor_system_windows proc_walk win32_stats
   INFO: Skipping (requires external dependency): tpm
   INFO: Enabling use of external dependency boost
   INFO: Enabling use of external dependency bzip2
   INFO: Enabling use of external dependency commoncrypto
   INFO: Enabling use of external dependency lzma
   INFO: Enabling use of external dependency openssl
   INFO: Enabling use of external dependency sqlite3
   INFO: Enabling use of external dependency zlib
   INFO: Loading modules: adler32 aead aes aes_ni aes_vperm aont argon2 aria asio asn1 auto_rng base base32 base58 base64 bcrypt bcrypt_pbkdf bigint blake2 block blowfish boost bzip2 camellia cascade cast128 cast256 cbc cbc_mac ccm cecpq1 certstor_flatfile certstor_sql certstor_sqlite3 certstor_system certstor_system_macos cfb chacha chacha20poly1305 chacha_avx2 chacha_rng chacha_simd32 checksum cmac comb4p commoncrypto compression cpuid crc24 crc32 cryptobox ctr curve25519 des dev_random dh dl_algo dl_group dlies dsa dyn_load eax ec_group ecc_key ecdh ecdsa ecgdsa ecies eckcdsa ed25519 elgamal eme_oaep eme_pkcs1 eme_raw emsa1 emsa_pkcs1 emsa_pssr emsa_raw emsa_x931 entropy fd_unix ffi filters fpe_fe1 gcm getentropy ghash ghash_cpu ghash_vperm gmac gost_28147 gost_3410 gost_3411 hash hash_id hex hkdf hmac hmac_drbg hotp http_util idea idea_sse2 iso9796 kasumi kdf kdf1 kdf1_iso18033 kdf2 keccak keypair lion locking_allocator lzma mac mce mceies md4 md5 mdx_hash mem_pool mgf1 misty1 mode_pad modes mp newhope nist_keywrap noekeon noekeon_simd numbertheory ocb ofb openssl par_hash passhash9 pbes2 pbkdf pbkdf1 pbkdf2 pem pgp_s2k pk_pad pkcs11 poly1305 poly_dbl prf_tls prf_x942 processor_rng psk_db pubkey rc4 rdrand_rng rdseed rfc3394 rfc6979 rmd160 rng roughtime rsa salsa20 scrypt seed serpent serpent_avx2 serpent_simd sessions_sql sessions_sqlite3 sha1 sha1_sse2 sha1_x86 sha2_32 sha2_32_bmi2 sha2_32_x86 sha2_64 sha2_64_bmi2 sha3 sha3_bmi2 shacal2 shacal2_avx2 shacal2_simd shacal2_x86 shake shake_cipher simd simd_avx2 siphash siv skein sm2 sm3 sm4 socket sodium sp800_108 sp800_56a sp800_56c sqlite3 srp6 stateful_rng stream streebog system_rng thread_utils threefish_512 threefish_512_avx2 tiger tls tls_10 tls_cbc tss twofish utils uuid whirlpool x509 x919_mac xmss xtea xts zlib
   INFO: Using symlink to link files into build dir (use --link-method to change)
   INFO: Botan 2.18.1 (revision git:b420a4545b0f9219a88c209e4c8c2474d519dfac) (unreleased undated) build setup is complete

Build output: make-out.txt.gz

Tests output

.  .  .  .  .
1416a6f128a2567fdf10079d74d2f64aaa8e2834216c698118f69109580b0f61c6fc53fdd578276e4f6b1e8fb1e5cd04a2450620c1dca97c517dc81ecfbd3776fbb75b2f211ddef474304929e0a2ef57121ba873a145e7cec15d3af0605f6e9cbc84ff70e4072f9e694557c302e2c2bb3db14bd52707b47890731e0cf6181d297d012967c3fd561f905b8a4ba23487]
xmss_verify_invalid:
XMSS/SHA2_10_256 verify invalid signature ran 28 tests in 28.65 msec all ok
XMSS/SHA2_10_512 verify invalid signature ran 28 tests in 74.90 msec all ok
XMSS/SHA2_16_256 verify invalid signature ran 28 tests in 37.11 msec all ok
XMSS/SHA2_16_512 verify invalid signature ran 28 tests in 94.26 msec all ok
XMSS/SHA2_20_256 verify invalid signature ran 28 tests in 41.33 msec all ok
XMSS/SHA2_20_512 verify invalid signature ran 28 tests in 88.67 msec all ok
XMSS/SHAKE_10_256 verify invalid signature ran 28 tests in 25.33 msec all ok
XMSS/SHAKE_10_512 verify invalid signature ran 28 tests in 95.61 msec all ok
XMSS/SHAKE_16_256 verify invalid signature ran 28 tests in 53.11 msec all ok
XMSS/SHAKE_16_512 verify invalid signature ran 28 tests in 118.49 msec all ok
XMSS/SHAKE_20_256 verify invalid signature ran 28 tests in 31.91 msec all ok
XMSS/SHAKE_20_512 verify invalid signature ran 28 tests in 129.48 msec all ok
Tests complete ran 2858301 tests in 18.89 sec 17713 tests failed

Full output: make-out.txt.gz

Branch master

Configuration

./configure.py --prefix=/opt/local --with-os-features=security_framework,apple_keychain,commoncrypto,getentropy --with-commoncrypto --with-openssl --with-boost --with-lzma --with-bzip2 --with-zlib --with-sqlite3 --with-python-version=3.9 --with-sphinx --with-pdf --system-cert-bundle=/opt/local/share/curl/curl-ca-bundle.crt --cc-abi-flags='-march=native -O3 -I/opt/local/include' 2>&1 | tee conf-out.txt

Output:

$ cat conf-out.txt 
   INFO: ./configure.py invoked with options "--prefix=/opt/local --with-os-features=security_framework,apple_keychain,commoncrypto,getentropy --with-commoncrypto --with-openssl --with-boost --with-lzma --with-bzip2 --with-zlib --with-sqlite3 --with-python-version=3.9 --with-sphinx --with-pdf --system-cert-bundle=/opt/local/share/curl/curl-ca-bundle.crt --cc-abi-flags=-march=native -O3 -I/opt/local/include"
   INFO: Configuring to build Botan 3.0.0-alpha0 (revision git:20e87b077c113744600510c431af1396663260a0)
   INFO: Running under 3.9.7 (default, Sep  1 2021, 12:35:15) [Clang 12.0.5 (clang-1205.0.22.9)]
   INFO: Implicit --cc-bin=clang++ due to environment variable CXX
   INFO: Implicit --cxxflags=-std=gnu++17 -O3 -march=native -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk due to environment variable CXXFLAGS
   INFO: Autodetected platform information: OS="Darwin" machine="x86_64" proc="i386"
   INFO: Guessing target OS is darwin (use --os to set)
   INFO: Guessing target processor is a x86_64 (use --cpu to set)
   INFO: Auto-detected compiler version 4.0
   INFO: Auto-detected compiler arch x86_64
   INFO: Target is clang:4.0-macos-x86_64
   INFO: Assuming target x86_64 is little endian
   INFO: Skipping (incompatible CPU): aes_armv8 aes_power8 sha1_armv8 sha2_32_armv8 shacal2_armv8 sm4_armv8
   INFO: Skipping (incompatible OS): certstor_system_windows win32_stats
   INFO: Skipping (requires external dependency): tpm
   INFO: Enabling use of external dependency boost
   INFO: Enabling use of external dependency bzip2
   INFO: Enabling use of external dependency commoncrypto
   INFO: Enabling use of external dependency lzma
   INFO: Enabling use of external dependency openssl
   INFO: Enabling use of external dependency sqlite3
   INFO: Enabling use of external dependency zlib
   INFO: Loading modules: adler32 aead aes aes_ni aes_vperm argon2 argon2fmt aria asio asn1 auto_rng base base32 base58 base64 bcrypt bcrypt_pbkdf bigint blake2 blake2mac block blowfish boost bzip2 camellia cascade cast128 cbc ccm cecpq1 certstor_flatfile certstor_sql certstor_sqlite3 certstor_system certstor_system_macos cfb chacha chacha20poly1305 chacha_avx2 chacha_rng chacha_simd32 checksum cmac comb4p commoncrypto compression cpuid crc24 crc32 cryptobox ctr curve25519 des dh dl_algo dl_group dlies dsa dyn_load eax ec_group ec_h2c ecc_key ecdh ecdsa ecgdsa ecies eckcdsa ed25519 elgamal eme_oaep eme_pkcs1 eme_raw emsa1 emsa_pkcs1 emsa_pssr emsa_raw emsa_x931 entropy fd_unix ffi filters fpe_fe1 gcm getentropy ghash ghash_cpu ghash_vperm gmac gost_28147 gost_3410 gost_3411 hash hash_id hex hkdf hmac hmac_drbg hotp http_util idea idea_sse2 iso9796 kdf kdf1 kdf1_iso18033 kdf2 keccak keypair lion locking_allocator lzma mac mce md4 md5 mdx_hash mem_pool mgf1 mode_pad modes mp newhope nist_keywrap noekeon noekeon_simd numbertheory ocb ofb openssl par_hash passhash9 pbes2 pbkdf pbkdf2 pem pgp_s2k pk_pad pkcs11 poly1305 poly_dbl prf_tls prf_x942 processor_rng psk_db pubkey rc4 rdseed rfc3394 rfc6979 rmd160 rng roughtime rsa salsa20 scrypt seed serpent serpent_avx2 serpent_simd sessions_sql sessions_sqlite3 sha1 sha1_sse2 sha1_x86 sha2_32 sha2_32_bmi2 sha2_32_x86 sha2_64 sha2_64_bmi2 sha3 sha3_bmi2 shacal2 shacal2_avx2 shacal2_simd shacal2_x86 shake shake_cipher simd simd_avx2 siphash siv skein sm2 sm3 sm4 socket sodium sp800_108 sp800_56a sp800_56c sqlite3 srp6 stateful_rng stream streebog system_rng thread_utils threefish_512 threefish_512_avx2 tls tls_cbc tss twofish utils uuid whirlpool x509 x919_mac xmss xts zlib
   INFO: Using symlink to link files into build dir (use --link-method to change)
   INFO: Botan 3.0.0-alpha0 (revision git:20e87b077c113744600510c431af1396663260a0) (unreleased undated) build setup is complete

Build output: make-out.txt.gz

Tests

.  .  .  .  .
c81ecfbd3776fbb75b2f211ddef474304929e0a2ef57121ba873a145e7cec15d3af0605f6e9cbc84ff70e4072f9e694557c302e2c2bb3db14bd52707b47890731e0cf6181d297d012967c3fd561f905b8a4ba23487 
xmss_verify_invalid:
XMSS/SHA2_10_256 verify invalid signature ran 28 tests in 34.19 msec all ok
XMSS/SHA2_10_512 verify invalid signature ran 28 tests in 69.04 msec all ok
XMSS/SHA2_16_256 verify invalid signature ran 28 tests in 34.38 msec all ok
XMSS/SHA2_16_512 verify invalid signature ran 28 tests in 94.06 msec all ok
XMSS/SHA2_20_256 verify invalid signature ran 28 tests in 40.87 msec all ok
XMSS/SHA2_20_512 verify invalid signature ran 28 tests in 79.71 msec all ok
XMSS/SHAKE_10_256 verify invalid signature ran 28 tests in 30.20 msec all ok
XMSS/SHAKE_10_512 verify invalid signature ran 28 tests in 83.43 msec all ok
XMSS/SHAKE_16_256 verify invalid signature ran 28 tests in 46.80 msec all ok
XMSS/SHAKE_16_512 verify invalid signature ran 28 tests in 130.14 msec all ok
XMSS/SHAKE_20_256 verify invalid signature ran 28 tests in 29.96 msec all ok
XMSS/SHAKE_20_512 verify invalid signature ran 28 tests in 121.47 msec all ok
Tests complete ran 2850756 tests in 18.28 sec 17713 tests failed (in entropy hash hash_rep mac newhope stream xmss_sign xmss_verify)

Full output: test-out.txt.gz

We'd like to contribute to Botan by starting the implementation of the TLS 1.3 protocol support. The motivation is that our use-cases require the TLS 1.3 from the client side and therefore we could support the development in that topic. The plan is to start in May (or even earlier with rough design and development plan).

The rough plan:

• design draft, to agree the direction of the development with the community,
• refactoring of the existing TLS code, i.e. splitting the common code and the protocol version specific, introducing pimpl idiom to have generic interfaces and version specific implementation, etc.,
• verification of refactored code to keep the quality on high level and avoid any regression,
• start the development of TLS 1.3 protocol,
• next steps...

The important thing would be to select proper start point (version) for the development. It seems that the release 2 (with the latest 2.18) is already mature and development on master branch is probably meant for upcoming 3.x release. Is there any rough plan, when first 3.x version will be released, as we would like to pick up some 'official tag' for the development?

Please, comment whether this topic sounds interesting and implementing the TLS 1.3 support would be valuable here?

Has anybody experienced this? I am trying to build Botan (version 2.10.0) in Windows 10, with mingw32, don't know if I use the correct flags. This is what I use:

configure.py --prefix=$BUILD_DIR --with-external- includedir=$OPENSSL_PREFIX/include --with-external-libdir=$OPENSSL_PREFIX/lib --os=mingw --cpu=i386 --minimized-build --enable- modules=rsa,dsa,ecdsa,ed25519,hmac,hmac_drbg,mode_pad,bigint,filters,block,auto_rng,x509,cbc,dh --with-openssl

Very thankful for any help with this

Hello!

Looking at guide, doing things as said: python configure.py --cc=msvc --os=windows --link-method=copy --cpu=i386 --disable-modules=pmull

then "nmake". And then accepting hoard of errors, like: build\include\botan/types.h(89) : error C2143: синтаксическая ошибка: отсутствие ";" перед "=" and so on.

When doing so: python configure.py --cc=gcc --os=mingw --link-method=copy --cpu=i386 --disable-modules=pmull and then while doing make accepting this: fatal error: botan/alg_id.h: No such file or directory #include <botan/alg_id.h>

Can anybody please give some hints to make things going? Totally newbie in cpp, help is badly needed.

Limited TLS 1.3 Client

Pull Request Dependencies

• The RFC 8448 based test uses the Test::Result consolidation constructor from this pull request. For simplicity the required commit is duplicated here and should be removed after the consolidation constructor is finalized and merged.

Follow-up Pull Requests

The number of open pull requests that are based on this are getting somewhat out of hand. Below is a list to keep track of the dependencies:

• THIS ONE: https://github.com/randombit/botan/pull/2922
  • https://github.com/randombit/botan/pull/2957
  • https://github.com/randombit/botan/pull/2974
  • https://github.com/randombit/botan/pull/2977
  • https://github.com/randombit/botan/pull/2983
  • https://github.com/randombit/botan/pull/2988
  • https://github.com/randombit/botan/pull/2989
  • INFORMATIVE: https://github.com/randombit/botan/pull/2981
  • INFORMATIVE: https://github.com/randombit/botan/discussions/2954

Potential Future Work

This is a non-exhaustive list of potential improvements or further development. Note that those points are not tackled by the follow-up PRs. I'll leave this here for future reference.

• [ ] Overhaul of TLS_Data_Reader
  • Some of its methods leave room for performance improvements
  • The interface could be easier to use
  • More invariant checks (e.g. assert !has_remaining() on destruction).
  • Also a "sub-reader" concept: i.e. a new ::get_tls_length_value_as_reader() could return a sub-reader that can then be passed into a sub-parser (e.g. Client Hello parsing its extensions)
  • Use it consistently for parsing
• [ ] Allow disabling TLS 1.2 at compile time
• [ ] std::span to avoid copying
  • C++17 doesn't actually provide it, but a custom implementation should be straight forward
• [ ] Separate TLS 1.2 and 1.3 cipher suites
  • TLS 1.3 suites are more focussed (some protocol parameters are negotiated differently)
  • the Ciphersuite class provides methods that are strictly not defined for TLS 1.3 (potential cause for bugs)
  • cipher suite handling code becomes more convoluted that necessary (e.g. Policy::ciphersuite_list(), generation via tls_suite_info.py)

This PR adds limited TLS 1.3 support. Available functionality:

• functional TLS 1.3 client
  • automatically downgrades to TLS 1.2 if allowed and required by the server
  • usable via Botan-CLI tls_client
• additional extensions as required per RFC 8446:
  • Key_Share
  • Supported_Versions
  • Cookie
  • Signature Algorithms
  • Signature Algorithms Certificate
  • Negotiated Groups
• additional extensions (not strictly required by RFC 8446)
  • Certificate Status Request (OCSP stapling)
  • Record_Size_Limit (RFC 8449)
  • Application Layer Protocol Notification (RFC 7301)
  • Server Name Indication
• Utility and Misc
  • ./botan tls_client --debug to print raw TLS traffic
  • Fixed_Output_RNG can optionally fall back to another RNG when its pool is empty
• Tests integrated or performed
  • New infrastructure has unit tests
  • Integration tests using the test vectors from RFC 8448
  • BoGo tests that do not depend on not-yet-implemented protocol features (see here for a list of issues found by BoGo)
  • Handshake and data exchange works using ./botan tls_client with the big players (google.com, cloudflare.com, ...)

Limitations:

• TLS 1.3 support cannot be compiled without enabling TLS 1.2 (TLS 1.2 can be disabled in the TLS policy though)
• no TLS 1.3 compatible server
• no client authentication (added in: https://github.com/randombit/botan/pull/2957)
• no session resumption via Tickets (added in: https://github.com/randombit/botan/pull/2974)
• no Pre-Shared-Keys (technical foundation added in: https://github.com/randombit/botan/pull/2974)
• no 0-RTT support (early data) (technical foundation added in: https://github.com/randombit/botan/pull/2974)
• no DTLS
• no fuzzing of the TLS 1.3 implementation (basic fuzz target for parser: https://github.com/randombit/botan/pull/2977)

Implementation Overview

To make this large PR a bit more approachable, we'll provide an overview to the newly introduced components and their responsibilities. Note that this work-in-progress pull request strives towards a usable implementation of "Minimal Viable Product" as outlined in the ToDo-List of this issue comment.

TODO

• [x] server certificate validation
  • [x] OCSP stapling
• [x] Hello Retry Request
• [x] handle CLOSE_NOTIFY properly (difference between TLS 1.2 and 1.3)
• [x] protocol version downgrade
• [x] Key Update
• [x] Key Material Export (RFC 8446 7.5 / RFC 5705)
• [x] review TLS callbacks invocations
• [x] bogo test suite integration
• [x] check occurrences of extensions as indicated in the table in RFC 8446 4.2
• [x] configure ClientHello settings (e.g. ALPN)
• [x] honor the record size limit extension
• ~~Session Resumption (PSK w/o 0-RTT)~~ not part of this PR

Intro

The descriptions mostly refer to the components in the tls13 module, as the TLS 1.2 implementation remains unchanged as far as possible. As initiated in the refactoring by Elektrobit Automotive GmbH, the user-facing interface of TLS::Client and TLS::Server also remains unchanged by means of a Pimpl-pattern.

We nevertheless anticipate some API changes in the public API. Please refer to this issue comment for further details.

Components

Channel

The Channel acts as "composition root" of the entire TLS 1.3 implementation. It implements the library's public APIs via a Pimpl-construction as detailed in this issue comment. This class implements the client/server agnostic parts of the protocol. Hence, orchestrating the Record_Layer and Handshake_Layer to transform received bytes from the network into handshake messages to be interpreted by the Client (and later Server) implementations or application data to be passed to the using code. Furthermore, it takes care of handling TLS alerts.

Record_Layer

This layer implements the record layer level of the protocol. It parses raw data from the wire to individual records and decrypts protected records using Cipher_State. This corresponds to tasks performed by the Channel (Channel_Impl_12) and the Record_Header class in the 1.2 implementation. When sending data it add record headers and encryption to the records.

Handshake_Layer

This class takes care of parsing and marshalling of Handshake_Messages as well as equipping them with the appropriate handshake protocol headers. As this class handles the byte-representation of all TLS handshake messages (i.e. before parsing or after serialization), it is responsible for updating the Transcript_Hash_State appropriately.

In TLS 1.2 the task of the Handshake_Layer were mostly implemented in the Handshake_IO class. Note that Handshake_Layer does not perform any handshake state validations. It simply parses/serializes messages as they come in from the wire or the downstream protocol implementation.

Cipher_State

This class implments the Key Schedule mechanism. Most importantly, it derives and holds traffic secrets and provides interfaces for the Record_Layer protect and deprotect records. The Client advances the Cipher_State through the Key Schedule "state machine".

Transcript_Hash_State

This class keeps track of the transcript hash while sending/receiving messages in the Handshake_Layer. Client (and later Server) consult the Transcript_Hash_State for relevant hash-data when updating the Cipher_State appropriately.

Client

The Client's main responsibility continues to lie in process_handshake_message, implemented via individual handle-methods for each specific message type. Hence, the client's TLS state machine is implemented here. During message processing, it updates the state of other components:

• advancing the Key Schedule in Cipher_State
• consult the Transcript_Hash_State
• manage expected next messages in Handshake_Transitions

Handshake_Transitions

This class aids the Client implementation in validating state transitions. It is merely an extraction of the confirm_transition_to and set_expected_next functionality from TLS 1.2's Handshake_State.

Handshake_State

HandshakeState manages the incoming and outgoing messages and keeps a record of them for later reference. Before storing the messages it filters them regarding the Outbound_Message and Inbound_Message variants. Hence, a misuse of Handshake_State will fail to compile (e.g. when a client implementation tries to send a Server_Hello_13). Similarly, the compiler can check that Client implements handle() methods for exactly the relevant handshake messages.

Handshake_Messages

The class hierarchy of Handshake_Messages implements the specifics of individual handshake messages. In the TLS 1.3 implementation we don't rely on the actual polymorphic nature of Handshake_Message but replace it with std::variant constructions to specifically define which handshake message types are expected where.

Each handshake message takes care of "local" protocol semantic checks. I.e. all validations that can be performed without contextual information is happening right in the parsing code. The same holds true for handshake message extensions (e.g. Key_Share, Supported_Groups, ...). Protocol logic that is tied to specific messages or extensions is implemented locally in those classes. For instance, Key_Share implements Diffie-Hellman in its exchange method and Client_Hello_13::retry() implements necessary updates and validations for handling "hello retry requests".

This is a different paradigm compared to the messages in TLS 1.2, where many messages where mere "Data Transfer Objects" without much logic. Hence, messages derive a _12 and _13 sub-class and share the parsing code in the parent class where possible. This is also useful in situations where the interface and usage of the same message type differs between TLS 1.2 and TLS 1.3, but the wire representation is unchanged.

Just trying out ECIES:

#include <botan/auto_rng.h>
#include <botan/ecies.h>
#include <botan/ecdh.h>
#include <iostream>

int main()
{
    try
    {
        Botan::AutoSeeded_RNG rng;
        Botan::ECDH_PrivateKey private_key(rng, Botan::EC_Group("secp521r1"));
        Botan::ECDH_PrivateKey private_key2(rng, Botan::EC_Group("secp521r1"));
        Botan::ECIES_System_Params params(private_key.domain(), "KDF1-18033(SHA-512)",
                                          "AES-256/CBC", 32, "HMAC(SHA-512)", 20,
                                          Botan::PointGFp::Compression_Type::COMPRESSED, Botan::ECIES_Flags::CHECK_MODE);
        Botan::ECIES_Encryptor enc(private_key, params, rng);
        enc.set_other_key(private_key2.public_point());
        std::string plain("some random data");
        std::vector<uint8_t> data(plain.data(), plain.data() + plain.size());
        auto ret = enc.encrypt(data, rng);
    }
    catch (const Botan::Invalid_State &e)
    {
        std::cout << e.what() << std::endl;
    }
    catch (const std::exception &e)
    {
        std::cout << e.what() << std::endl;
    }
}

output: Invalid state: state().empty() == false was false in finish:src/lib/modes/cbc/cbc.cpp

tried with 2.11.0 and 2.12.0 releases, results are the same. not sure if i did something wrong? i dont find ECIES example in the docs, so i tried to grab code from test_ecies.cpp

it still throws even if i change dem_algo_spec to other unauthenticated cipher modes, so i think its an issue in the ECIES_Encryptor implementation?

Only install.py is using this. It is not used during configure and build time. This allows for a clean separation of destdir and prefix in the style of autotools.

Fixes #996

I've tried the latest src from github, compiled fine on VS2013 with cpu=x64. But when trying to run a simple program that creates an RSA key, program hangs forever. Same thing happens trying to run botan.exe and botan_test.exe. Not sure if its an issue with AutoSeeded_RNG or what.

Code is pretty simple:

For 1.11.16

#include <fstream>
#include <iostream>
#include <string>
#include <vector>
#include <cstring>
#include <memory>

#include <botan/botan.h>
#include <botan/rsa.h>
#include <botan/x509cert.h>
#include <botan/x509self.h>
#include <botan/pem.h>
#include <botan/oids.h>

using namespace std;

int main()
{
    try
    {
        cout << "Hello Botan!" << endl;
    cout << "Generate key.." << endl;
    AutoSeeded_RNG rng;
        RSA_PrivateKey rsaPrivate(rng, 1024);

        // just testing some stuff
        vector<Botan::byte> publicBytes  = X509::BER_encode(rsaPrivate);
    vector<Botan::byte> privateBytes = PKCS8::BER_encode(rsaPrivate, rng, "passphrase");

    ofstream pub("public.pem");
    pub << X509::PEM_encode(rsaPrivate);

    getchar();

     }
     catch(std::exception& e)
     {
        cerr << "Error: " << e.what() << "\n";
     }

     return 0;
}

The implementation is divided into two parts: a low level and a high level API. Both interfaces are implemented against the PKCS#11 standard in version 2.4.

The implemented low level API provides access to all functions described in the PKCS#11 standard. The advantage of using the low level API instead of the C-style PKCS#11 API directly is a C++ interface that provides features like RAII, exceptions and automatic memory management. The low level API is implemented in src/lib/prov/pkcs11/p11.h and src/lib/prov/pkcs11/p11.cpp.

The high level API provides access to the most commonly used PKCS#11 functionality in an object oriented manner. Functionality of the high level API includes:

• Loading/unloading of PKCS#11 modules
• Initialization of token
• Change of PIN/PUK
• Session management
• Random number generation
• Enumeration of objects on the token (certificates, public keys, private keys)
• Import/export/deletion of certificates
• Generation/import/export/deletion of RSA and EC public and private keys
• Encryption/decryption using RSA with support for OAEP and PKCS1-v1_5 (and raw)
• Signature generation/verification using RSA with support for PSS and PKCS1-v1_5
• Signature generation/verification using ECDSA
• Key derivation using ECDH

Tests were written for the low level API in test_pkcs11_low_level.cpp and for the high level API in test_pkcs11_high_level.cpp. These tests need to know against which PKCS#11 module the tests should be executed. For this we've extended the test command line tool with a parameter --pkcs11-lib= where the user can specify the path to a PKCS#11 module. The tests assume that the PIN is set to “123456” and that the SO_PIN is set to “12345678”. Maybe we can drop the low level tests because 99% of the tests are testing only the PKCS#11 module and token. These tests can be executed manually for example by executing botan-test pkcs11-manage or all together by executing botan-test pkcs11. The PKCS#11 tests are not part of the normal test suite because of the dependency on the PKCS#11 module and potentially on a token reader and token. The implementation is tested against SoftHSMv2. Testing against SoftHSMv2 has the advantage that it is a software implementation and therefore to execute the tests no token reader and token is required. Furthermore, because it's open source, we can inspect the code in case there are errors or unexpected behavior. All tests execute successfully against SoftHSMv2 2.1.0.

Furthermore configure.py is extended with a new command --with-external-includedir. This must point to a directory containing the PKCS#11 header files. The header files can be obtained from here

Okay, there's a bit to unpack here, I guess. First off: this is partially me playing with C++20, so no hard feelings if we end up tuning down the concepts-fanciness a bit.

C++20 Strong Types

In strong_type.h there's a sketchy Strong Type wrapper. Turns out that C++20 finally makes this somewhat usable. Usage:

// declare a strong type (`struct SessionID_` is needed to make this
// a unique type among all the other byte vector based types)
using SessionID = Strong<std::vector<uint8_t>, struct SessionID_>

SessionID sid{unlock(rng.random_vec(32))};
// I'm hoping this will eventually be:
// auto sid = rng.random_range<SessionID>(32);

// Strong Types try hard to act like their underlying type
std::cout << hex_encode(sid);

With that, constructors that take two byte vectors are much more ergonomic. No more mixing up parameters. Overloading constructors (despite both overloads being byte vectors):

using SessionID = Strong<std::vector<uint8_t>, struct SessionID_>
using SessionTicket = Strong<std::vector<uint8_t>, struct SessionTicket_>

// A Session_Handle must be constructed with an ID, a Ticket or both
// but it cannot be constructed without either of them.
struct Session_Handle {
  Session_Handle(SessionID id);
  Session_Handle(SessionTicket ticket);
  Session_Handle(SessionID id, SessionTicket ticket);
};

class Session

This should eventually be a dumb data-sink that can (un)marshal itself (and perhaps that knows how to encrypt/decrypt itself). In particular it shouldn't have a notion about its own ID or ticket anymore. These are the realm of the Session_Manager.

struct Session_Handle

A descriptor for a session. As in the example above: It contains either a (TLS 1.2-ish) ID, a ticket (TLS 1.2 or 1.3) or both. The Session_Manager emits a Session_Handle for new sessions and spits them out again given a Session_Handle. It can also store a Session along with an existing session handle.

class Session_Manager

I put some documentation in the header, so won't repeat that here. Main point: There is a sharp distinction between APIs useful for the server and the client:

Servers create Session objects and pass them to Session_Manager::save(Session) -> Session_Handle. Hence, the Session_Manager is in full control of both the Session_ID and the Session_Ticket. Servers will retrieve Session objects via Session_Manager::retrieve(Session_Handle) -> Session or via Session_Manager::choose_from_offered_tickets (that's still somewhat rough). Again, the Session_Manager has full control over tickets and IDs and how to retrieve sessions from them.

Clients on the other hand always receive a Session_Handle from the server (that's just how TLS works). They use Session_Manager::save(Session, Session_Handle) to associate their sessions with the handles from the server. Usually, clients will then use Session_Manager::find(Server_Information) -> std::vector<std::pair<Session, Session_Handle>> to retrieve potential sessions for resumption.

Note that clients may retrieve multiple sessions for resumption. TLS 1.3 allows clients to provide several tickets (as PSK identities) at once. The Session_Manager can opt to always return a single Session, Session_Handle pair, of course.

Session deletion (e.g. after usage by the client/server) is done with Session_Manager::remove(Session_Handle). For session handles that contain a ticket only, this is likely a no-op. But smart Session_Manager implementations might even keep a burn list to avoid reuse of legitimate tickets. That's up to the using application.

class Session_Manager_Stateless

A strawman manager for a stateless server. It generates random Session IDs and creates tickets via Session::encrypt(). The sessions are not persisted and Session_Manager::retrieve(Session_ID) will always fail.

class Session_Manager_In_Memory

Like before, this keeps sessions in-memory in a std::map<>. Note that the SessionID strong-type acts as a key! This implementation also supports session tickets by simply composing with Session_Manager_Stateless internally and forwarding the necessary requests to it.

What's next

First: Thanks for reading through those thoughts. 😄

A few open questions:

• Strong types: yay? or nay?
• Should we go as far and separate the Session_Manager into a server and a client part. Now that the API seems almost fully disjoint (apart from ::remove()).
• Required technical understanding of the application developer: is this taking it too far?
  • Generation of valid SessionIDs and Tickets
  • Lifetime handling
  • Choosing from a number of offered tickets
  • Balancing how many tickets a TLS 1.3 client should offer at once
  • ...
• ...

Happy to throw most of this over board or iterate on the ideas.

In EMSA1 we always now set parameters to empty since that is the convention for DSA, ECDSA, etc. Previously if some other algorithm (eg RSA) was used, the parameters would be set. However the sig_algo_and_padding_ok check already prevented one from using EMSA1 padding with RSA, so there is no change to functionality as a result.

Hello,

This question seems to be a continuation of my question https://github.com/randombit/botan/issues/3078. I am still struggling with creating of PKCS#7 CMS SignedData document.

My .p7b document now looks almost the same as the .p7b created with the OpenSSL PKCS7_sign() function. The signature is the difference. My testing host accepts the document generated with OpenSSL, but it does not accept the document generated by myself using Botan. I am getting "failed to verify cms: crypto/rsa: verification error"

For a signature generation, I used different emsa options: EMSA4(SHA-256), EMSA3(SHA-256), EMSA1(SHA-256), EMSA3(Raw). And with all of them, I got failed.

One of the possible problems is that the data I have to sign is already hashed 256, as CMS standard required. Another problem may be that I can not define a proper signature algorithm.

Does anyone tried to do something like this and maybe has some recommendations? Below is the snippet showing the signing part.

/* Get SHA-256 from input data */
auto DataDigest = Botan::SHA_256().process(std::vector<uint8_t>(in.begin(), in.end()));

/* Build Signature Attributes */
encoder.encode( Botan::OID( Names::OIDstringPKCS7 ) );
Botan::Attribute content_type(Botan::OID( Names::OIDstringPKCS9ContentType ), encoder.get_contents_unlocked());
encoder.encode(DataDigest, OCTET_STRING);
Botan::Attribute message_digest(Botan::OID( Names::OIDstringPKCS9MessageDigest ), encoder.get_contents_unlocked());
encoder.start_cons( Botan::ASN1_Tag( 0 ), Botan::ASN1_Tag( Botan::ASN1_Tag::CONTEXT_SPECIFIC ))
	.encode(content_type)
	.encode(message_digest)
.end_cons();
Botan::secure_vector<uint8_t> DataDigestDer = encoder.get_contents();

/* Sign data */
Botan::PK_Signer signer(*rsa, rng, "EMSA4(SHA-256)", Botan::IEEE_1363);
signer.update(DataDigestDer);
std::vector<uint8_t> signature = signer.signature(rng);

Thanks in advance.

Pull Request Dependencies

• https://github.com/randombit/botan/pull/3053

Description

This adds facilities for PSK-based session resumption to the TLS 1.3 server implementation. The resumption mechanism of TLS 1.3 differs substantially from previous protocol versions: Therefore, this changes a few aspects of the Session_Manager and future consolidation work on this will likely be necessary.

New Public API

TLS::Server::send_new_session_tickets(const size_t tickets)

When called on TLS 1.3 server channel that finished its handshake, this sends "tickets" NewSessionTicket messages to the peer. Note that applications may call this at any time after the handshake was completed and as often as they need (up to 2^16 times -- then the ticket nonce depletes). On any other channel object (i.e. TLS 1.2 server), this throws.

TLS::Server::new_session_ticket_supported()

Returns false if the channel object is not capable of TLS 1.3 session tickets (e.g. because it is a TLS 1.2 server or the TLS 1.3 client did not advertise support for any PSK key exchange mode). If ::send_new_session_ticket() would likely succeed, this will return true.

TLS::Policy::new_session_tickets_upon_handshake_success()

Once a TLS 1.3 server successfully established a TLS connection with a compatible client, it will automatically send as many session tickets as defined by this policy. This is a convenience function to enable session resumption support without the using application needing to call ::send_new_session_ticket(). By default the TLS 1.2 implementation offers session resumption, so this returns 1 by default: I.e. clients will receive a single session ticket after a successful handshake.

TLS::Policy::session_ticket_lifetime() now returns std::chrono::seconds

It used to be uint32_t, I figured the chrono strong type is more convenient.

TLS::Session_Manager::save() now returns a session ticket

Previously, ::save() just saved the passed Session. Now it may return an opaque ticket for the New Session Ticket message. This gives full control of the ticket's content to application-defined Session_Manager implementations. Basically, just as RFC 8446 suggests.

The value of the ticket to be used as the PSK identity [...] is an opaque label. It MAY be either a database lookup key or a self-encrypted and self-authenticated value.

TLS::Session_Manager::choose_from_offered_tickets()

Is ::save()'s counter-part that gets a list of all client-offered ticket identities allowing applications to choose which PSK they would like to go with. This method is exclusively called by TLS 1.3.

Future Work: Session Management Consolidation

TLS 1.2 sessions are identified either by a session_id or the server's SNI. Optionally, stateless servers may pass a single opaque session-ticket to the client. With TLS 1.3 sessions are exclusively identified by their PSK identity: an opaque value that is either a database handle or a self-contained ticket. The strong association with Session IDs and SNI was a dropped with TLS 1.3.

Currently, these mechanisms don't co-exist well in Botan. Here is a proposal to consolidate them as much as possible by giving the Session_Manager more control over the ticket generation and simplifying the Session object. I believe we should first review/merge this pull request before opening a dedicated PR for such a consolidation.

class Session_Manager
   {
   struct SessionHandle
      {
      std::array<uint8_t, 32> id;
      std::vector<uint8_t> ticket;
      };

   /**
    * Save a session, return the session's identifier. Used by the server
    * to create a new session.
    * 
    * @return std::nullopt if not persisted, otherwise
    *         the session's ID and/or 
    *         an opaque ticket for the session
    * 
    * TLS 1.2 might use both handles, TLS 1.3 will use either the ticket (if set)
    * or the ID as the session's PSK identifier.
    */
   std::optional<SessionHandle> save(Session session);

   /**
    * Save a session passed from the server. Used by the client to
    * persist sessions generated by the server.
    */
   bool save(const Session& session, SessionHandle handle);

   /**
    * `Ticket` contains the PSK identifier which an implementation can
    * use to find and/or unpack the session information to be returned.
    * This will be called by TLS 1.3 servers only.
    *
    * Note that this might use `::find_by_id()` internally, when implemented
    * as a session database, or `::unpack_from_ticket()` for a stateless
    * implementation.
    */
   std::optional<std::pair<Session, uint16_t>>
      choose_from_offered_tickets(const std::vector<Ticket>& tickets,
                                  const std::string& hash_function);

   std::optional<Session> unpack_from_ticket(std::span<const uint8_t> ticket);
   std::optional<Session> find_by_id(std::span<const uint8_t, 32>& session_id);
   std::optional<Session> find_by_server_info(const Server_Information& info);
   };

I'm unsure about the removal of sessions. My best guess: the TLS implementation should not explicitly "remove" sessions, but rather just inform the Session_Manager that a session identifier was used() in a handshake attempt. Applications could then decide how to proceed with "used" sessions. Alternatively, the implementation could guarantee that accessor methods (find_by_..., unpack_from_...., choose_from_...) will be called exactly once for a handshake attempt. The `Session_Manager could then just assume sessions were used when it handed them out.

Note that the Session_Manager implementations provided by default are currently not well covered with unit tests. I suggest, this consolidation would be a good moment to improve on that.

Lot's of text, I hope my brain dump helps in the discussion.