As of ScreenOS version 6.3r5, the truncation lengths of HMAC-SHA-256 in IPsec have been changed to 128 bits to comply with RFC4868. To allow interoperability with earlier ScreenOS versions, "set envar hmac-sha256-96=yes" is introduced.
Problem or Goal:
For IKE Phase 2 negotiations, the early ScreenOS implemented authentication algorithm HMAC-SHA2-256 with the truncation lengths of 96 bits. This truncation length does not comply with RFC4868. To comply with RFC4868, the truncation lengths have been changed to 128 bits after ScreenOS version 6.3r5. The change in truncation lengths after ScreenOS version 6.3r5 causes it not to interoperate with earlier ScreenOS versions which use HMAC-SHA-256 with 96-bit truncation lengths. The change affects both IKEv1 and IKEv2.
This will allow Phase 2 negotiation to complete between an HMAC-SHA-256-128 negotiator and non-standard HMAC-SHA-256-96 negotiator, but receiving ESP packets will be dropped due to invalid ESP payload lengths.
Based on this, I understood that since ScreenOS version 6.3r5, the truncation lengths of HMAC-SHA-256 in IPsec have been changed to 128 bits. Is that default value ?
1. Do we have any way for checking the current configured bits ?
2. Is there a possibiltiy to change this value in case 96 bits is configured at this moment ?
3. I was looking the command to change this value and I found "set envar hmac-sha256-96=yes". Should I change 96 to 128. If so, shall I need to restart the box after ?
Re: HMAC-SHA256 backward compatibility to 128 bits
I have checked few other VPN flow basic and it seems size=96 is not Truncation in your flow basic output, it's length of data with padding that will be encrypted as per encryption algorithm-CBC etc.
Later 156 is the amount of data with the ESP headers.
1: First you see actual data size (without L2 headers)
****** 8831165.0: <Trust/ethernet0/0> packet received ****** <--- 1447 is actual data size
2: Later you will see the same output as per your debug, after packet processing.
in my debugs I see "(vn2) doing ESP encryption and size =1456 or (vn2) doing ESP encryption and size =48" this means it's not Truncation size.
As per the RFC4868:
The HMAC-SHA-256+ algorithms each produce an nnn-bit value, where nnn corresponds to the output bit length of the algorithm, e.g., HMAC- SHA-nnn. For use as an authenticator, this nnn-bit value can be truncated as described in [HMAC]. When used as a data origin authentication and integrity verification algorithm in ESP, AH, IKE, or IKEv2, a truncated value using the first nnn/2 bits -- exactly half the algorithm output size -- MUST be supported. No other authenticator value lengths are supported by this specification.
Please check "get envar" output and if you don't see set envar hmac-sha256-96=yes then it's default without backward compatibility.