SRX Services Gateway
SRX Services Gateway

Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 09:01 AM

Having some issues with an SRX dropping the return traffic because it thinks it is a new flow and doesn't belong to any existing sessions and says "packet dropped, first pak not syn".

 

security flow trace appears to have matching flow data but the return traffic gets dropped.

 

SRX is trying to connect to a remote secondary identity management server across an IPsec tunnel that is terminated on the SRX itself. This connection to the identity mangement server is sourced from a revenue port. This same issue occurs with an SRX trying to download threat intel feeds from a policy enforcer server across the same IPsec tunnel. Any traffic sourced from inside the firewall on the same subnet works, it is only traffic sourced from the SRX itself.

 

 

I have included the output of the security flow trace debut basic-datapath as an attachment

 

 

Return dropped


Jun 25 13:57:43 13:57:43.722769:CID-0:RT:  ge-0/0/2.0:10.254.255.130/9443->10.254.254.254/59093, tcp, flag 12 syn ack
 
Jun 25 13:57:43 13:57:43.722834:CID-0:RT: find flow: table 0x4ec03d8, hash 5292(0xffff), sa 10.254.255.130, da 10.254.254.254, sp 9443, dp 59093, proto 6, tok 7, conn-tag 0x00000000
 
Jun 25 13:57:43 13:57:43.722848:CID-0:RT:  no session found, start first path. in_tunnel - 0x0, from_cp_flag - 0
 
Jun 25 13:57:43 13:57:43.722848:CID-0:RT:  packet dropped, first pak not syn
 
Jun 25 13:57:43 13:57:43.722848:CID-0:RT:flow_initiate_first_path: first pak no session

Attachments

21 REPLIES 21
Highlighted
SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 09:19 AM

Hi 

 

Could you please share the SRX configuration that you have in place currently? 

 

Regards,

HS

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 09:20 AM

Hi

 

 

 

ge-0/0/2.0:10.254.255.130/9443->10.254.254.254/59093, tcp, flag 12 syn ack

 

How does your flow trace configuration look like?

If you have packet filters enabled in both directions (request and response), that should cover both, the syn and syn-acks.

Is there a NAT in the traffic context?

 

Cheers

Pooja

 Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too!!!

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 09:30 AM

There is an asymmetric routing in your network. Outgoing traffic from SRX is going via st0.2 (ge-0/0/3 exit interface) tunnel interface but the return traffic is not coming via tunnel.
Return traffic is received on another interfaces ge-0/0/2.0. That is why SRX is dropping the packets. Please check your routing. Return traffic should come via tunnel interface.

 

routed (x_dst_ip 10.254.255.130) from junos-host (.local..0 in 0) to st0.2, Next-hop: 10.254.255.130
going into tunnel 67108908 (nsp_tunnel=0x851eb98)
ge-0/0/2.0:10.254.255.130/9443->10.254.254.254/60169, tcp, flag 10

 

 

 

 

Thanks,
Nellikka
JNCIE x3 (SEC #321; SP #2839; ENT #790)
Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too!!!
SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 09:33 AM

Pooja,

 

enabling no-syn-check seems like a bad idea for a production firewall since that disables it globally. However, I thought about using a firewall filter to match the specific flow of traffic and then put that traffic in packet-mode. However, I wasn't sure which interface to apply it too.

 

 

 

trace config currently inactive:

 

safesys@Alpharetta-SRX340-01> show configuration security flow traceoptions 
##
## inactive: security flow traceoptions
##
file secFlowDebug size 20m files 5;
flag basic-datapath;
packet-filter filter2 {
    source-prefix 66.194.109.124/32;
    destination-prefix 172.127.49.85/32;
}
packet-filter filter1 {
    source-prefix 172.127.49.85/32;
    destination-prefix 66.194.109.124/32;
}
SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 09:34 AM

I'll see if I can sanitize it and exclude any uncessary bits. It is quite long.

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 09:43 AM

Nellikka,

 

That was my initial thought as well and thought it was odd.  That interface is the interface it was sourced on..... 10.254.254.254 is configured on ge-0/0/2.0. Is it possible the logs are showing the return on the st0.2 interface that is bound to ge-0/0/3.0 and then should be routed to 10.254.254.254 ge-0/0/2.0 inteface? 

 

And to reinterate, this is only happening when the SRX is the source address. I can do a secflow trace for working traffic to compare. It wouldn't be sourced from the SRX and process on a different zone to zone rule instead of junos-host.

 

 

 

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 09:48 AM

Pooja,

 

Here is the matching syn for the flow you referenced:

 

Jun 25 13:57:43 13:57:43.687824:CID-0:RT:  .local..0:10.254.254.254/59093->10.254.255.130/9443, tcp, flag 2 syn
ge-0/0/2.0:10.254.255.130/9443->10.254.254.254/59093, tcp, flag 12 syn ack
SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 09:57 AM

 

If you know the "ideal" path this traffic context/flow should be taking, adding that flow into selective packet mode might be an option.

 

But I feel like it's too early in the troubleshooting here to make such a major change.

 

I did not notice the attached traces earlier on, I will respond back shortly after reviewing it.

 

Cheers

Pooja

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too!!!

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 10:07 AM

Hi 

 

 

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 10:14 AM

 

Something like this:

show | display set| match 10.254.254.254

show | display set | match 10.254.255.130

And, show the ike + ipsec configuration hierarchies, of course after sanitizing sensitive information?

 

Cheers

Pooja

Please Mark My Solution Accepted if it Helped, Kudos are Appreciated too!!!

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 12:06 PM

Pooja,

Here is the requested info.

 

 

user@SRX340-01> show configuration | display set | match 10.254.254.254  
set system syslog host 10.2.45.31 source-address 10.254.254.254
set security log source-address 10.254.254.254
set security address-book global address srx-FW-LAN-Interface 10.254.254.254/32
set interfaces ge-0/0/2 unit 0 family inet address 10.254.254.254/24

user@SRX340-01> show configuration | display set | match 10.254.255.130    
set services user-identification identity-management connection secondary address 10.254.255.130

user@SRX340-01> show route 10.254.255.130 

inet.0: 142 destinations, 180 routes (142 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

10.254.255.0/24    *[Static/2] 6d 15:17:43
                    >  via st0.2
                    [Static/5] 6d 15:17:42
                    >  via st0.2

mgmt_junos.inet.0: 3 destinations, 3 routes (3 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

0.0.0.0/0          *[Static/5] 6d 15:20:13
                    >  to 10.254.254.1 via fxp0.0



user@srx-SRX340-01> show configuration security ike 
proposal WUS_Sonicwall_Proposal {
    authentication-method pre-shared-keys;
    dh-group group1;
    authentication-algorithm sha1;
    encryption-algorithm aes-128-cbc;
    lifetime-seconds 28800;
}

policy Beta_WUS_Sonicwall {
    mode main;
    proposals WUS_Sonicwall_Proposal;
    pre-shared-key ascii-text "****************************"; ## SECRET-DATA
}

gateway 68_xxx_xxx_189_Beta_WUS_Sonicw {
    ike-policy Beta_WUS_Sonicwall;
    address 68.xxx.xxx.189;
    dead-peer-detection {
        always-send;
        interval 15;
        threshold 5;
    }
    nat-keepalive 1;
    local-identity inet 12.xxx.xxx.34;
    remote-identity inet 68.xxx.xxx.189;
    external-interface ge-0/0/3.0;
}                                       


user@srx-SRX340-01> show configuration security ipsec     
proposal WUS_Sonicwall_Proposal {
    protocol esp;
    authentication-algorithm hmac-sha1-96;
    encryption-algorithm aes-128-cbc;
    lifetime-seconds 28800;
}
policy Beta_WUS_Sonicwall {
    perfect-forward-secrecy {
        keys group2;
    }
    proposals WUS_Sonicwall_Proposal;
}

vpn 68_xxx_xxx_189_Beta_WUS_Sonicw {
    bind-interface st0.2;
    ike {
        gateway 68_xxx_xxx_189_Beta_WUS_Sonicw;
        ipsec-policy Beta_WUS_Sonicwall;
    }
    traffic-selector servers-to-WUSserver {
        local-ip 10.254.254.0/24;
        remote-ip 10.254.255.0/24;
    }
 }

 
 
 
user@SRX340-01> show configuration interfaces 

ge-0/0/2 {
    description "Trust interface";
    unit 0 {
        family inet {
            address 10.254.254.254/24;
        }
    }
}
ge-0/0/3 {
    description "untrust Interface";
    unit 0 {
        family inet {
            address 12.xxx.xxx.34/27 {
                primary;
                preferred;
            }
        }
    }
}

fxp0 {
    unit 0 {
        family inet {
            address 10.254.254.230/24;
        }
    }
}
st0 {
    unit 2 {
        family inet;
    }
}





user@SRX340-01> show configuration routing-instances 
mgmt_junos {
    routing-options {
        static {
            route 0.0.0.0/0 next-hop 10.254.254.1;
        }
    }
}





safesys@Alpharetta-SRX340-01> show configuration routing-options  
static {
    route 0.0.0.0/0 next-hop 12.xxx.xxx.33;
    route 10.254.255.0/24 {
        next-hop st0.2;
        preference 2;
    }



 

 

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 12:08 PM

 

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 12:22 PM

 

 

Can you test traffic through this very vpn which isn't sourced on the firwall itself?

Meaning non junos-host sourced traffic that transits the firewall, matches the traffic selector and leaves?

 

Cheers

Pooja

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 12:29 PM

Agreed, my bad. I thought your ge-0/0/2 interface was your untrust interface. 

 

Regards,

HS

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 12:36 PM

pmallya,

 

I've tested prior and it works for any other device on the same subnet. Are you wanting me to provide a secflow trace for the working flow?

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 12:36 PM

 

 

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-25-2019 01:32 PM

Hi 

 

 

 

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-26-2019 06:48 AM

I believe I may have found the issue this morning. As I was attempting to gather a sec flow trace for the working addresses in the same subnet as the SRX source I saw that the packet filters in my trace were not getting any matches.

 

This is where I was able to determine there was a WAN-Op device in front of the SRX and tunnels traffic through the IPsec tunnel.

 

So what I have been able to gather is that because the traffic is sourced directly on the SRX it traverses the IPsec tunnel without any intervention from the WAN-Op appliance. However, the return traffic from the server is traversing the WAN-Op appliance/tunnel and therefore causing the asymetrical routing issue. 

 

I have the guy's managing that appliance implementing some changes. I will report back once they are done and I can test again.

 

Thanks for everyone's help so far!

SRX Services Gateway

Re: Flow Session Lookup Fails for return traffic when sourced from the SRX

‎06-26-2019 06:51 AM

Hi