I did a test in the lab on two SRX100B with IPSEC proposal esp/hmac-sha-256-96/aes-256-cbc and I got
Changing st0.0 MTU to 1400B indeed helps avoiding fragmentation. Path MTU discovery kicks in in ths situation. As this mechanism not always work I guess the best way would be to use both, decreased mtu on st0.0 and tcp mss adjust.
As interesting as it was it didn't help a bit with your original problem. Do you maybe have any screens enabled on VPN zone? Or maybe you are also NATing the traffic? Btw. for my tests I used FTP.