Hi,
________________________________________________________________________________________________
I agree we Can use separate Vlan on same physical infrastructure but wouldn't that will increase the risk(Split brain)?.
________________________________________________________________________________________________
IMO, the risk is same as using 2 physical switches/network but a single WAN. The single WAN or the common device connecting the WAN are the single points of failure which can cause split brain. See Figure 8.
Based on the requirements [more specifically Internet connectivity] it seems a mixed-mode HA is more appropriate. See Figure 7-8. This would probably need Z-flow traffic.
_________________________________________________________________________________________________
Layer 2 Link between Site A & B goes down now site B should connect to Site A via internet (can it be a L2TP?).
_________________________________________________________________________________________________
I understand the same WAN link is used for hosts on siteA to connect to hosts on SiteB on same vlan, for instance when SRXA is down hosts on siteA will reach reth interface on SRXB over the same WAN link carrying control/fabric for HA. There are different protocols that can enable L2 connectivity between DCs over Internet like L2TP, L2 over GRE, EVPN/VXLAN or MPLSoverGRE for example.
However, when the WAN link is down the SRXs will be in split brain, so anything that goes through the SRX for L3 would most probably go through the local SRX.
Hope this helps.
Cheers,
Ashvin