Routing

I have R2 as client directly connected to R4 (RR) and adjacency stuck in connect state. Not sure what the below logs mean and what should i look for. 

Logs on R2:

Jul 31 22:17:13.519268 bgp_reuse_scan: Starting scan
Jul 31 22:17:16.659338 bgp_connect_timeout: BGP_3895077211.172.27.255.4_Connect
Jul 31 22:17:16.659390 bgp_connect_start: peer 172.27.255.4 (Internal AS 3895077211)
Jul 31 22:17:16.659401 bgp_event: peer 172.27.255.4 (Internal AS 3895077211) old state Active event ConnectRetry new state Connect
Jul 31 22:17:16.659517 task_get_socket: domain AF_INET  type SOCK_STREAM  protocol 0  socket 59
Jul 31 22:17:16.659537 task_set_option_internal: task BGP_3895077211.172.27.255.4 socket 59 option RecvBuffer(0)
Jul 31 22:17:16.659546  value 16384
Jul 31 22:17:16.659558 task_set_option_internal: task BGP_3895077211.172.27.255.4 socket 59 option SendBuffer(1)
Jul 31 22:17:16.659566  value 16384
Jul 31 22:17:16.659577 task_set_socket: task BGP_3895077211.172.27.255.4 socket 59
Jul 31 22:17:16.659595 task_set_option_internal: task BGP_3895077211.172.27.255.4 socket 59 option NonBlocking(8)
Jul 31 22:17:16.659606  value 1
Jul 31 22:17:16.659617 task_set_option_internal: task BGP_3895077211.172.27.255.4 socket 59 option ReUsePort(38)
Jul 31 22:17:16.659626  value 1
Jul 31 22:17:16.659637 task_set_option_internal: task BGP_3895077211.172.27.255.4 socket 59 option PathMTUDiscovery(26)
Jul 31 22:17:16.659645  value 0
Jul 31 22:17:16.659658 task_set_option_internal: task BGP_3895077211.172.27.255.4 socket 59 option TOS(16)
Jul 31 22:17:16.659679  value 192
Jul 31 22:17:16.659691 task_set_option_internal: task BGP_3895077211.172.27.255.4 socket 59 option LridCheck(59)
Jul 31 22:17:16.659700  value 0
Jul 31 22:17:16.659716 task_addr_local: task BGP_3895077211.172.27.255.4 address 172.27.255.2
Jul 31 22:17:16.659741 task_connect: task BGP_3895077211.172.27.255.4+179 addr 172.27.255.4+179
Jul 31 22:17:16.659798 task_timer_reset: reset BGP_3895077211.172.27.255.4+179_Connect
Jul 31 22:17:16.659816 task_timer_set_oneshot_latest: timer BGP_3895077211.172.27.255.4+179_Connect interval set to 2:28
Jul 31 22:17:33.379368 bgp_reuse_scan: Starting scan
Jul 31 22:17:53.339267 bgp_reuse_scan: Starting scan
Jul 31 22:18:13.269227 bgp_reuse_scan: Starting scan

Logs on R4:

Jul 31 22:06:18.405030 task_process_events: connect ready for BGP_3895077211.172.27.255.2+179
Jul 31 22:06:18.405057 bgp_connect_complete: error connecting to 172.27.255.2 (Internal AS 3895077211): Socket is not connected
Jul 31 22:06:18.405062 bgp_close_socket: peer 172.27.255.2 (Internal AS 3895077211)
Jul 31 22:06:18.405067 task_close: close socket 58 task BGP_3895077211.172.27.255.2+179
Jul 31 22:06:18.405072 task_reset_socket: task BGP_3895077211.172.27.255.2+179 socket 58
Jul 31 22:06:18.405090 bgp_event: peer 172.27.255.2 (Internal AS 3895077211) old state Connect event OpenFail new state Idle
Jul 31 22:06:18.405174 bgp_event: peer 172.27.255.2 (Internal AS 3895077211) old state Idle event Start new state Connect
Jul 31 22:06:18.405179 bgp_connect_start: peer 172.27.255.2 (Internal AS 3895077211)
Jul 31 22:06:18.405182 bgp_event: peer 172.27.255.2 (Internal AS 3895077211) old state Connect event ConnectRetry new state Connect
Jul 31 22:06:18.405193 task_get_socket: domain AF_INET type SOCK_STREAM protocol 0 socket 58
Jul 31 22:06:18.405210 task_set_option_internal: task BGP_3895077211.172.27.255.2 socket 58 option RecvBuffer(0)
Jul 31 22:06:18.405215 value 16384
Jul 31 22:06:18.405231 task_set_option_internal: task BGP_3895077211.172.27.255.2 socket 58 option SendBuffer(1)
Jul 31 22:06:18.405234 value 16384
Jul 31 22:06:18.405239 task_set_socket: task BGP_3895077211.172.27.255.2 socket 58
Jul 31 22:06:18.405258 task_set_option_internal: task BGP_3895077211.172.27.255.2 socket 58 option NonBlocking(8)
Jul 31 22:06:18.405263 value 1
Jul 31 22:06:18.405268 task_set_option_internal: task BGP_3895077211.172.27.255.2 socket 58 option ReUsePort(38)
Jul 31 22:06:18.405271 value 1
Jul 31 22:06:18.405276 task_set_option_internal: task BGP_3895077211.172.27.255.2 socket 58 option PathMTUDiscovery(26)
Jul 31 22:06:18.405280 value 0
Jul 31 22:06:18.405286 task_set_option_internal: task BGP_3895077211.172.27.255.2 socket 58 option TOS(16)
Jul 31 22:06:18.405289 value 192
Jul 31 22:06:18.405295 task_set_option_internal: task BGP_3895077211.172.27.255.2 socket 58 option LridCheck(59)
Jul 31 22:06:18.405298 value 0
Jul 31 22:06:18.405305 task_addr_local: task BGP_3895077211.172.27.255.2 address 172.27.255.4
Jul 31 22:06:18.405317 task_connect: task BGP_3895077211.172.27.255.2+179 addr 172.27.255.2+179
Jul 31 22:06:18.405337 task_timer_reset: reset BGP_3895077211.172.27.255.2+179_Connect
Jul 31 22:06:18.405346 task_timer_set_oneshot_latest: timer BGP_3895077211.172.27.255.2+179_Connect interval set to 1:04

root@R4# run show bgp summary 
Groups: 2 Peers: 4 Down peers: 3
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0               
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.27.255.1     3895077211         48         49       0       0       20:54 0/0/0/0              0/0/0/0
172.27.255.2     3895077211          0          0       0       0       21:06 Connect

root@R2# run show bgp summary 
Groups: 1 Peers: 2 Down peers: 2
Table          Tot Paths  Act Paths Suppressed    History Damp State    Pending
inet.0               
                       0          0          0          0          0          0
Peer                     AS      InPkt     OutPkt    OutQ   Flaps Last Up/Dwn State|#Active/Received/Accepted/Damped...
172.27.255.3     3895077211          0          0       0       0       21:38 Active
172.27.255.4     3895077211          0          0       0       0       21:38 Active

 Please suggest.

Thanks!!

2nd Try... Anyone ?

Hi,

Can you post the interface and bgp configurations for both sides?

root@R2# show protocols bgp 
traceoptions {
    file debug-bgp size 1g files 2;
    flag all;
}
group cluster-1 {
    type internal;
    local-address 172.27.255.2;
    neighbor 172.27.255.3;
    neighbor 172.27.255.4;
}

root@R4# show protocols bgp 
traceoptions {
    file debug-bgp size 1g files 2;
    flag all;
}
group cluster-1 {
    type internal;
    local-address 172.27.255.4;
    cluster 0.0.0.1;
    neighbor 172.27.255.1;
    neighbor 172.27.255.2;
    neighbor 172.27.255.5;
}
group INTERNAL {
    type internal;
    local-address 172.27.255.4;
    neighbor 172.27.255.3;
}

root@R2# run show configuration interfaces ge-0/0/1                   
unit 0 {
    family inet {
        address 172.27.0.5/24;
    }
}


root@R4# run show configuration interfaces ge-0/0/0 
unit 0 {
    family inet {
        address 172.27.0.6/24;
    }
}

root@R2# run show configuration interfaces lo0
unit 0 {
 family inet {
 address 172.27.255.2/32;
 }
}

root@R4# run show configuration interfaces lo0
unit 0 {
 family inet {
 address 172.27.255.4/32;
 }
}

root@R4# show routing-options
router-id 172.27.255.4;
autonomous-system 3895077211;

root@R2# show routing-options
router-id 172.27.255.2;
autonomous-system 3895077211;

Can you ping R4s loopback from R2s loopback?

Do you have any RE filters applied?

Can't ping R2 and R4 each other. No RE filter applied. FYI, i'm running this on vMX on Junosphere.

root@R2# run show route 172.27.255.4  

inet.0: 11 destinations, 12 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.27.255.4/32    *[OSPF/10] 5d 00:08:25, metric 1
                    > to 172.27.0.6 via ge-0/0/1.0

root@R4# run show route 172.27.255.2    

inet.0: 13 destinations, 16 routes (13 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.27.255.2/32    *[OSPF/10] 5d 00:07:23, metric 1
                    > to 172.27.0.5 via ge-0/0/0.0

 Weird thing is i can't ping to directly connected interface.

root@R2# run show route 172.27.0.6  

inet.0: 11 destinations, 12 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.27.0.0/24      *[Direct/0] 6d 02:09:24
                    > via ge-0/0/0.0
                    [Direct/0] 6d 02:09:24
                    > via ge-0/0/1.0

root@R2# run ping 172.27.0.6  
PING 172.27.0.6 (172.27.0.6): 56 data bytes
^C
--- 172.27.0.6 ping statistics ---
6 packets transmitted, 0 packets received, 100% packet loss

Thanks!!

You posted:

root@R2# run show route 172.27.255.4  

inet.0: 11 destinations, 12 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.27.255.4/32    *[OSPF/10] 5d 00:08:25, metric 1
                    > to 172.27.0.6 via ge-0/0/1.0

This shows a next hop of 172.27.0.6 via ge-0/0/1.0.

But then you posted:

root@R2# run show route 172.27.0.6  

inet.0: 11 destinations, 12 routes (11 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.27.0.0/24      *[Direct/0] 6d 02:09:24
                    > via ge-0/0/0.0
                    [Direct/0] 6d 02:09:24
                    > via ge-0/0/1.0

 This is showing that 172.27.0.0/24 is configured on two different interfaces on the same router. It's unable to reach 172.27.0.6 because R2 is pointing to ge-0/0/0.0 instead of ge-0/0/1.0.

You need to use two different subnets for those interfaces. You can use 172.27.0.0/30 for one interface and 172.27.0.4/30 for the other interface.

For the simplicity, i have changed loopback of R2 as 2.2.2.2/32 and R4 as 4.4.4.4/32. They are both directly connected and running ospf. Though i can't ping each other. 

root@R2# run show ospf neighbor    
Address          Interface              State     ID               Pri  Dead
172.27.0.6       ge-0/0/1.0             Full      4.4.4.4          128    31

root@R4# run show ospf neighbor    
Address          Interface              State     ID               Pri  Dead
172.27.0.5       ge-0/0/0.0             Full      2.2.2.2          128    34

root@R4# run show route 2.2.2.2 

inet.0: 10 destinations, 13 routes (10 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

2.2.2.2/32         *[OSPF/10] 00:18:51, metric 1
                    > to 172.27.0.5 via ge-0/0/0.0

root@R2# run show route 4.4.4.4 

inet.0: 8 destinations, 9 routes (8 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

4.4.4.4/32         *[OSPF/10] 00:19:21, metric 1
                    > to 172.27.0.6 via ge-0/0/1.0

root@R2# run ping 4.4.4.4
PING 4.4.4.4 (4.4.4.4): 56 data bytes
^C
--- 4.4.4.4 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

The issue was not with the loopbacks. On R2, what ip addresses do you have configured on ge-0/0/1.0 and ge-0/0/0.0? Are they on the same subnet? If so, change it so that they are on different subnets.

When i put them on different subnet, ospf doesn't come up. 

You have to change the IP of the interface on the peer routers as well.

Try these things:

Assuming that R2 ge-0/0/1.0 is connected to R4 ge-0/0/0.0:

- Change the IP on ge-0/0/1.0 on R2 to 172.27.0.5/30

- Change the IP on ge-0/0/0.0 on R4 to 172.27.0.6/30

It looks like R2 is connected to another router on ge-0/0/0.0:

- Change the IP on ge-0/0/0.0 on R2 to 172.27.0.1/30

- Change the IP of whatever R2 is connected to 172.27.0.2/30

So it should look like this:

Please see attached full topology. 

Following is my interface config for all router. issue is still persist.

root@R1# run show interfaces terse |match /30 
ge-0/0/0.0              up    up   inet     172.27.0.1/30   
ge-0/0/1.0              up    up   inet     172.27.0.14/30  
ae0.0                   up    up   inet     172.27.0.10/30  


root@R2# run show interfaces terse |match /30 
ge-0/0/0.0              up    up   inet     172.27.0.2/30   
ge-0/0/1.0              up    up   inet     172.27.0.5/30 

root@R3# run show interfaces terse |match /30 
ge-0/0/0.0              up    up   inet     172.27.0.17/30  
ge-0/0/1.0              up    up   inet     172.27.0.13/30  
ge-0/0/3.0              up    up   inet     172.27.0.26/30  


root@R4# run show interfaces terse |match /30 
ge-0/0/0.0              up    up   inet     172.27.0.6/30   
ge-0/0/1.0              up    up   inet     172.27.0.18/30  
ge-0/0/5.0              up    up   inet     172.27.0.21/30  
ae0.0                   up    up   inet     172.27.0.9/30   


root@R5# run show interfaces terse |match /30 
ge-0/0/0.0              up    up   inet     172.27.0.22/30  
ge-0/0/1.0              up    up   inet     172.27.0.25/30  

3895077211; ASN???

The current ASN allocation is based on 2-byte AS number, also known as a 16-bit AS1.
AS numbers range 0-65535 lower and upper are reserved leaving 1-65534. Of these 64512 to 65535 are considered for private internal use, the rest are assigned/reserved for public ASN. So for now use one of those private numbers. If you chose to go with 4-byte AS /32 bit ASN, you now have 4.2B+ numbers; the format is AS-dot notation format, something like 65534.65534. So for now stick with 2 byte using a number from the private range.

root@R4# = RR
cluster 0.0.0.1; ??? <=== I would first verify that regular loopback peering works before going to RR. Anyways...
If R2 is a client to R4 then only R4 should be the neighbor. So delete neighbor 172.27.255.3
Also make sure these neighbors, have only the RR as their neighbor sice they are clients
neighbor 172.27.255.1;
neighbor 172.27.255.5
Suggestion: For your lab, use router id like this convention, R4 - router-id 4.4.4.4, R2 - router-id 2.2.2.2; When you start viewing the ospf database, it will make a world of difference when you can just really easily see the router id which does not look like your interface id. Just a suggestion. (Would require that you must purge the OSPF database) -- Scrath - I see you have already comnpleted this step, Leaving it in place though.

172.27.0/24 is fine. You can use /27 so you have ranges .1 - .30. If you use /30 you will only have IP .1 and .2 as valid IPs.
Try those and see what happens.

@ lyndidon: Thanks for the suggestion:

I'm working on to see if loopback peering is working fine.

I have changed AS to 64515. R3 and R4 are RR and R1, R2, R5 are client. That's why i have IBGP peering from R4 to R3(Not in cluster). 

Thanks!!

You still can't ping? On R2, please do the following and post your results:

show ospf neighbor

show interfaces ge-0/0/1.0 terse

show interfaces ge-0/0/0.0 terse

show route 172.27.0.6

show route 172.27.0.1

ping 172.27.0.6 source 172.27.0.5 rapid

ping 172.27.0.1 source 172.27.0.2 rapid

ping 4.4.4.4 source 2.2.2.2 rapid

Don't change anything with BGP until you can actually ping. Let's fix the layer 3 issue before jumping into layer 7.

@pevangelista

Please see requested output:

root@R2> show ospf neighbor 
Address          Interface              State     ID               Pri  Dead
172.27.0.1       ge-0/0/0.0             Full      1.1.1.1          128    38
172.27.0.6       ge-0/0/1.0             Full      4.4.4.4          128    32

root@R2> show interfaces ge-0/0/1.0 terse 
Interface               Admin Link Proto    Local                 Remote
ge-0/0/1.0              up    up   inet     172.27.0.5/30   
                                   multiservice

root@R2> show interfaces ge-0/0/0.0 terse 
Interface               Admin Link Proto    Local                 Remote
ge-0/0/0.0              up    up   inet     172.27.0.2/30   
                                   multiservice

root@R2> show route 172.27.0.6 

inet.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.27.0.4/30      *[Direct/0] 10:25:02
                    > via ge-0/0/1.0

root@R2> show route 172.27.0.1 

inet.0: 17 destinations, 17 routes (17 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.27.0.0/30      *[Direct/0] 10:09:46
                    > via ge-0/0/0.0

root@R2> ping 172.27.0.6 source 172.27.0.5 rapid 
PING 172.27.0.6 (172.27.0.6): 56 data bytes
.....
--- 172.27.0.6 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

root@R2> ping 172.27.0.1 source 172.27.0.2 rapid 
PING 172.27.0.1 (172.27.0.1): 56 data bytes
.....
--- 172.27.0.1 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

root@R2> ping 4.4.4.4 source 2.2.2.2 rapid 
PING 4.4.4.4 (4.4.4.4): 56 data bytes
.....
--- 4.4.4.4 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

Ok so we have OSPF back up and you have specific routes. You said there were no RE filters. Are there any firewall filters applied to the interfaces? Can you post the output of the following:

On R2:

show configuration interfaces ge-0/0/1

show arp interface ge-0/0/1.0

show route forwarding-table 172.27.0.6

On R4:

show configuration interfaces ge-0/0/0

show arp interface ge-0/0/0.0

show route forwarding-table 172.27.0.5

@ pevangelista:

No RE filter. No firewall filter. Just plain ospf and bgp config.

root@R2> show configuration interfaces ge-0/0/1 
unit 0 {
    family inet {
        address 172.27.0.5/30;
    }
}

root@R2> show arp interface ge-0/0/1.0 
MAC Address       Address         Name                      Interface           Flags
00:05:86:71:33:00 172.27.0.6      172.27.0.6                ge-0/0/1.0          none

root@R2> show route forwarding-table destination 172.27.0.6 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
172.27.0.6/32      dest     0 0:5:86:71:33:0     ucst      558     8 ge-0/0/1.0

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct      521     1

root@R4# exit               
Exiting configuration mode

root@R4> show configuration interfaces ge-0/0/0 
unit 0 {
    family inet {
        address 172.27.0.6/30;
    }
}

root@R4> show arp interface ge-0/0/0.0 

root@R4> show route forwarding-table destination 172.27.0.5 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
172.27.0.5/32      dest     0 172.27.0.5         hold      555     5 ge-0/0/0.0

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct      521     1

The hold type is not good: 

Explanation: 

• hold: Next hop is waiting to be resolved into a unicast or multicast type.

 Just for the fun of it try to add static arp entry on R4 ge-0/0/0

root@R4# exit               
Exiting configuration mode

root@R4> show configuration interfaces ge-0/0/0 
unit 0 {
    family inet {
        address 172.27.0.6/30;
    }
}

root@R4> show arp interface ge-0/0/0.0 

root@R4> show route forwarding-table destination 172.27.0.5 
Routing table: default.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
172.27.0.5/32      dest     0 172.27.0.5         hold      555     5 ge-0/0/0.0

Routing table: __master.anon__.inet
Internet:
Destination        Type RtRef Next hop           Type Index    NhRef Netif
default            perm     0                    rjct      521     1

@Doods:

i tried to flap the ge-0/0/1 interface on R2 but didn't help. How can i make next-hop type to ucst ?

Thanks!!

Try to add static arp entry on R4 ge-0/0/0: 

set interfaces ge-0/0/0 unit 0 family inet address <R4-IP> arp <R2-IP>  mac <R2-facing-interface-mac>

@Doods :

Configured static arp but still can't ping R2 to R4.

root@R4# run show arp no-resolve interface ge-0/0/0 
MAC Address       Address         Interface     Flags
00:05:86:71:18:01 172.27.0.5      ge-0/0/0.0           permanent

root@R4# run ping 172.27.0.5 rapid 
PING 172.27.0.5 (172.27.0.5): 56 data bytes
.....
--- 172.27.0.5 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

root@R2# run show arp no-resolve interface ge-0/0/1     
MAC Address       Address         Interface     Flags
00:05:86:71:33:00 172.27.0.6      ge-0/0/1.0           none

root@R2# run ping 172.27.0.6 rapid 
PING 172.27.0.6 (172.27.0.6): 56 data bytes
.....
--- 172.27.0.6 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

What's the output of "show route 172.27.0.5" on R4?

@pevangelista

root@R4# run show route 172.27.0.5 

inet.0: 19 destinations, 19 routes (19 active, 0 holddown, 0 hidden)
+ = Active Route, - = Last Active, * = Both

172.27.0.4/30      *[Direct/0] 00:46:36
                    > via ge-0/0/0.0

Did anything change on the forwarding table on R4 when you added the static arp entry?

Issue a ping to 172.27.0.5 from R4, and do a monitor traffic interface ge-0/0/1 on R2.

I'm curious if R2 receives an arp request from R4 for 172.27.0.5 and if R2 sends an arp reply. Take off the static arp entry before you do this.

@pevangelista

 when i added static arp, next-hop type became ucst on R4. 

Removed static arp entry and monitored interface on R2 while R4 pinging. It seems like R2 is sending reply back.

root@R2# run monitor traffic interface ge-0/0/1  
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on ge-0/0/1, capture size 96 bytes

Reverse lookup for 224.0.0.5 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use <no-resolve> to avoid reverse lookups on IP addresses.

10:19:27.446000  In IP 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:28.049920 Out IP truncated-ip - 20 bytes missing! 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:36.746038  In IP 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:38.009929 Out IP truncated-ip - 20 bytes missing! 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:41.306398  In arp who-has 172.27.0.5 tell 172.27.0.6
10:19:41.306422 Out arp reply 172.27.0.5 is-at 00:05:86:71:18:01
10:19:46.486545  In IP 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:46.679947 Out IP truncated-ip - 20 bytes missing! 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:54.155897  In IP 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:56.019957 Out IP truncated-ip - 20 bytes missing! 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:56.306439  In arp who-has 172.27.0.5 tell 172.27.0.6
10:19:56.306461 Out arp reply 172.27.0.5 is-at 00:05:86:71:18:01
^C
12 packets received by filter
0 packets dropped by kernel

But, on R4 i can't see request coming in from R2 

root@R4# run monitor traffic interface ge-0/0/0                                                
verbose output suppressed, use <detail> or <extensive> for full protocol decode
Address resolution is ON. Use <no-resolve> to avoid any reverse lookup delay.
Address resolution timeout is 4s.
Listening on ge-0/0/0, capture size 96 bytes

Reverse lookup for 224.0.0.5 failed (check DNS reachability).
Other reverse lookup failures will not be reported.
Use <no-resolve> to avoid reverse lookups on IP addresses.

10:18:34.985199 Out IP truncated-ip - 20 bytes missing! 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:18:35.141415  In IP 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:18:42.865164 Out IP truncated-ip - 20 bytes missing! 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:18:43.891872  In IP 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:18:46.265122 Out arp who-has 172.27.0.5 tell 172.27.0.6
10:18:50.525114 Out IP truncated-ip - 20 bytes missing! 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:18:52.265930  In IP 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:00.045122 Out IP truncated-ip - 20 bytes missing! 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:01.553796  In IP 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:03.065103 Out arp who-has 172.27.0.5 tell 172.27.0.6
10:19:09.542082  In IP 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:09.695157 Out IP truncated-ip - 20 bytes missing! 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:18.295119 Out IP truncated-ip - 20 bytes missing! 172.27.0.6 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:19.141162  In IP 172.27.0.5 > 224.0.0.5: OSPFv2, Hello, length 60
10:19:19.465110 Out arp who-has 172.27.0.5 tell 172.27.0.6
^C
15 packets received by filter
0 packets dropped by kernel

That's odd considering you don't have any filters applied to the interfaces. Do you have any configuration groups? Can you post the output of:

(R4)

show interfaces ge-0/0/0

show configuration interfaces ge-0/0/0 | display inheritance

show configuration interfaces lo0.0 | display inheritance

(R2)

show interfaces ge-0/0/1

show configuration interfaces ge-0/0/1 | display inheritance

show configuration interfaces lo0.0 | display inheritance

No. i don't have any groups, RE filter or Firewall applied anywhere on the  vMX.

its just has plain config for IGP and BGP.

[edit]
root@R2# show interfaces ge-0/0/1 
unit 0 {
    family inet {
        address 172.27.0.5/30;
    }
}

[edit]
root@R2# run show configuration interfaces ge-0/0/1 | display inheritance 
unit 0 {
    family inet {
        address 172.27.0.5/30;
    }
}

[edit]
root@R2# run show configuration interfaces lo0.0 | display inheritance 
family inet {
    address 2.2.2.2/32;
}


root@R4# show interfaces ge-0/0/0  
unit 0 {
    family inet {
        address 172.27.0.6/30;
    }
}

[edit]
root@R4# show interfaces ge-0/0/0 |display inheritance 
unit 0 {
    family inet {
        address 172.27.0.6/30;
    }
}

root@R4# run show configuration interfaces ge-0/0/0 | display inheritance 
unit 0 {
    family inet {
        address 172.27.0.6/30;
    }
}

[edit]
root@R4# run show configuration interfaces lo0.0 | display inheritance 
family inet {
    address 4.4.4.4/32;
}

Hmmmm. It's weird because you're receiving the OSPF hellos on R4, and the ARP request from R4 is making it to R2. So you have a shared LAN segment, but R4 is not receiving the ARP reply from R2. Can you show the output of the following while outside of configuration mode:

(R4)

show interfaces ge-0/0/0 extensive

(R2)

show interfaces ge-0/0/1 extensive

Also, try sending a continuous ping from R2 to R4, and do "monitor interface ge-0/0/0" on R4.

Here is the ouput :

root@R2> show interfaces ge-0/0/1 extensive 
Physical interface: ge-0/0/1, Enabled, Physical link is Up
  Interface index: 138, SNMP ifIndex: 514, Generation: 141
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Flow control: Enabled
  Pad to minimum frame size: Disabled
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 00:05:86:71:18:01, Hardware address: 00:05:86:71:18:01
  Last flapped   : 2017-07-26 08:29:47 PDT (1w0d 23:02 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :             12012028                  104 bps
   Output bytes  :             16937596                  112 bps
   Input  packets:               212671                    0 pps
   Output packets:               244694                    0 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Dropped traffic statistics due to STP State:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Active alarms  : None
  Active defects : None
  MAC statistics:                      Receive         Transmit
    Total octets                             0                0
    Total packets                            0                0
    Unicast packets                          0                0
    Broadcast packets                        0                0
    Multicast packets                        0                0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
    Total errors                             0                0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                    17
    Input DA rejects                         0
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 0, CAM source filters: 0
  Autonegotiation information:
    Negotiation status: Incomplete
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort            95      950000000    95              0      low    none
    3 network-control         5       50000000     5              0      low    none
  Interface transmit statistics: Disabled

  Logical interface ge-0/0/1.0 (Index 331) (SNMP ifIndex 524) (Generation 142)
    Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
    Traffic statistics:
     Input  bytes  :              1068188
     Output bytes  :              1270404
     Input  packets:                15772
     Output packets:                16576
    Local statistics:
     Input  bytes  :              1067152
     Output bytes  :              1270404
     Input  packets:                15758
     Output packets:                16576
    Transit statistics:
     Input  bytes  :                 1036                    8 bps
     Output bytes  :                    0                    0 bps
     Input  packets:                   14                    0 pps
     Output packets:                    0                    0 pps
    Protocol inet, MTU: 1500, Generation: 157, Route table: 0
      Flags: Sendbcast-pkt-to-re
      Addresses, Flags: Is-Preferred Is-Primary
        Destination: 172.27.0.4/30, Local: 172.27.0.5, Broadcast: 172.27.0.7, Generation: 161
    Protocol multiservice, MTU: Unlimited, Generation: 158, Route table: 0
      Policer: Input: __default_arp_policer__


root@R4> show interfaces ge-0/0/0 extensive 
Physical interface: ge-0/0/0, Enabled, Physical link is Up
  Interface index: 137, SNMP ifIndex: 513, Generation: 140
  Link-level type: Ethernet, MTU: 1514, MRU: 1522, Speed: 1000mbps, BPDU Error: None, MAC-REWRITE Error: None, Loopback: Disabled, Source filtering: Disabled,
  Flow control: Enabled
  Pad to minimum frame size: Disabled
  Device flags   : Present Running
  Interface flags: SNMP-Traps Internal: 0x4000
  Link flags     : None
  CoS queues     : 8 supported, 8 maximum usable queues
  Hold-times     : Up 0 ms, Down 0 ms
  Current address: 00:05:86:71:33:00, Hardware address: 00:05:86:71:33:00
  Last flapped   : 2017-07-26 08:31:07 PDT (1w0d 23:01 ago)
  Statistics last cleared: Never
  Traffic statistics:
   Input  bytes  :              6326832                   56 bps
   Output bytes  :             14408660                  136 bps
   Input  packets:                78869                    0 pps
   Output packets:               212685                    0 pps
   IPv6 transit statistics:
    Input  bytes  :                   0
    Output bytes  :                   0
    Input  packets:                   0
    Output packets:                   0
  Dropped traffic statistics due to STP State:
   Input  bytes  :                    0
   Output bytes  :                    0
   Input  packets:                    0
   Output packets:                    0
  Input errors:
    Errors: 0, Drops: 0, Framing errors: 0, Runts: 0, Policed discards: 0, L3 incompletes: 0, L2 channel errors: 0, L2 mismatch timeouts: 0, FIFO errors: 0,
    Resource errors: 0
  Output errors:
    Carrier transitions: 1, Errors: 0, Drops: 0, Collisions: 0, Aged packets: 0, FIFO errors: 0, HS link CRC errors: 0, MTU errors: 0, Resource errors: 0
  Active alarms  : None
  Active defects : None
  MAC statistics:                      Receive         Transmit
    Total octets                             0                0
    Total packets                            0                0
    Unicast packets                          0                0
    Broadcast packets                        0                0
    Multicast packets                        0                0
    CRC/Align errors                         0                0
    FIFO errors                              0                0
    MAC control frames                       0                0
    MAC pause frames                         0                0
    Oversized frames                         0
    Jabber frames                            0
    Fragment frames                          0
    VLAN tagged frames                       0
    Code violations                          0
    Total errors                             0                0
  Filter statistics:
    Input packet count                       0
    Input packet rejects                165822
    Input DA rejects                    165750
    Input SA rejects                         0
    Output packet count                                       0
    Output packet pad count                                   0
    Output packet error count                                 0
    CAM destination filters: 0, CAM source filters: 0
  Autonegotiation information:
    Negotiation status: Incomplete
  Packet Forwarding Engine configuration:
    Destination slot: 0 (0x00)
  CoS information:
    Direction : Output
    CoS transmit queue               Bandwidth               Buffer Priority   Limit
                              %            bps     %           usec
    0 best-effort            95      950000000    95              0      low    none
    3 network-control         5       50000000     5              0      low    none
  Interface transmit statistics: Disabled

  Logical interface ge-0/0/0.0 (Index 330) (SNMP ifIndex 523) (Generation 147)
    Flags: Up SNMP-Traps 0x4004000 Encapsulation: ENET2
    Traffic statistics:
     Input  bytes  :               817476
     Output bytes  :              1210160
     Input  packets:                10187
     Output packets:                15708
    Local statistics:
     Input  bytes  :               817236
     Output bytes  :              1210064
     Input  packets:                10184
     Output packets:                15706
    Transit statistics:
     Input  bytes  :                  240                    0 bps
     Output bytes  :                   96                    0 bps
     Input  packets:                    3                    0 pps
     Output packets:                    2                    0 pps
    Protocol inet, MTU: 1500, Generation: 165, Route table: 0
      Flags: Sendbcast-pkt-to-re
      Addresses, Flags: Is-Preferred Is-Primary
        Destination: 172.27.0.4/30, Local: 172.27.0.6, Broadcast: 172.27.0.7, Generation: 173
    Protocol multiservice, MTU: Unlimited, Generation: 166, Route table: 0
      Policer: Input: __default_arp_policer__

R4                                Seconds: 25                  Time: 07:36:14
                                                          Delay: 1/0/23
Interface: ge-0/0/0, Enabled, Link is Up
Encapsulation: Ethernet, Speed: 1000mbps
Traffic statistics:                                              Current delta
  Input bytes:                   6328752 (80 bps)                        [240]
  Output bytes:                 14411832 (112 bps)                       [396]
  Input packets:                   78893 (0 pps)                           [3]
  Output packets:                 212724 (0 pps)                           [5]
Error statistics:
  Input errors:                        0                                   [0]
  Input drops:                         0                                   [0]
  Input framing errors:                0                                   [0]
  Carrier transitions:                 1                                   [0]
  Output errors:                       0                                   [0]
  Output drops:                        0                                   [0]

You are having a strange high amout of rejects: 

This happens when the Destination MAC address is different than the expected one.

  Filter statistics:
    Input packet count                       0
    Input packet rejects                165822
    Input DA rejects                    165750

@Doods :

How it is possible as R4 is not receiving arp reply from R2. I have configured static arp entry as well but that didn't help. How static arp entry can be unexpected at R4 ?

This is what this counter is counting, now we need to understand why those drops are happening. 

What are the protocols that are enabled on R4

show conf protocols

OSPF and BGP enabled on R4 and all other router in topology.

root@R4> show configuration protocols 
bgp {
    group cluster-1 {
        type internal;
        local-address 4.4.4.4;
        cluster 0.0.0.1;
        neighbor 1.1.1.1;
        neighbor 2.2.2.2;
        neighbor 5.5.5.5;
    }
    group INTERNAL {
        type internal;
        local-address 4.4.4.4;
        neighbor 3.3.3.3;
    }
}
ospf {
    area 0.0.0.0 {
        interface lo0.0 {
            passive;
        }
        interface ge-0/0/0.0 {
            interface-type p2p;
        }
        interface ge-0/0/1.0 {
            interface-type p2p;
        }
        interface ge-0/0/5.0 {
            interface-type p2p;
        }
        interface ae0.0;
    }
}

Are you able to ping other interfaces? Does any of these work:

(R4)

ping 172.27.0.17 source 172.27.0.18 rapid

ping 172.27.0.22 source 172.27.0.21 rapid

ping 172.27.0.10 source 172.27.0.9 rapid

(R2)

ping 172.27.0.1 source 172.27.0.2 rapid

I haven't used Junosphere before, but are you able to capture traffic on the link between R2 and R4?

@pevangelista:

Yes. Atleast one link on each router has connectivity issue with directly connected interface. So, i wanted to rule out one link and from that i can figur out other.

root@R4# run ping 172.27.0.17 source 172.27.0.18 rapid 
PING 172.27.0.17 (172.27.0.17): 56 data bytes
.....
--- 172.27.0.17 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

[edit]
root@R4# run ping 172.27.0.22 source 172.27.0.21 rapid 
PING 172.27.0.22 (172.27.0.22): 56 data bytes
.....
--- 172.27.0.22 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

[edit]
root@R4# run ping 172.27.0.10 source 172.27.0.9 rapid 
PING 172.27.0.10 (172.27.0.10): 56 data bytes
!!!!!
--- 172.27.0.10 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 79.976/139.008/160.430/30.386 ms

root@R2> ping 172.27.0.1 source 172.27.0.2 rapid 
PING 172.27.0.1 (172.27.0.1): 56 data bytes
.....
--- 172.27.0.1 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

Similar behaviour on  R1 :

root@R1> ping 172.27.0.9 source 172.27.0.10 rapid 
PING 172.27.0.9 (172.27.0.9): 56 data bytes
!!!!!
--- 172.27.0.9 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 37.564/69.413/78.672/15.942 ms

root@R1> ping 172.27.0.2 source 172.27.0.1 rapid 
PING 172.27.0.2 (172.27.0.2): 56 data bytes
.....
--- 172.27.0.2 ping statistics ---
5 packets transmitted, 0 packets received, 100% packet loss

root@R1> ping 172.27.0.13 source 172.27.0.14 rapid 
PING 172.27.0.13 (172.27.0.13): 56 data bytes
!!!!!
--- 172.27.0.13 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 1.318/1.542/1.814/0.202 ms

 I'm not sure if i can run packet capture on Junosphere. I never did traffic capture on Junosphere and not sure how can i do that.

Hi, 

What type of device is that?

>show version

@Doods

Its vMX running on Junosphere.

root@R2> show version 
Hostname: R2
Model: vmx
Junos: 14.1-20140130_ib_14_1_psd.0

Hi, 

Please share the following on R2 and R4: 

show arp no-resolve 

show chassis mac-addresses    

@Doods:

Looks like R4 didn't learn R2's mac-address

root@R2# run show arp no-resolve 
MAC Address       Address         Interface     Flags
56:68:28:3d:5c:5f 10.233.240.2    fxp0.0               none
56:68:28:3d:5c:5f 10.233.244.2    fxp0.0               none
56:68:28:41:89:cc 10.233.255.254  fxp0.0               none
00:05:86:71:32:00 172.27.0.1      ge-0/0/0.0           none
00:05:86:71:33:00 172.27.0.6      ge-0/0/1.0           none
Total entries: 5

[edit]
root@R2# run show chassis mac-addresses 
MAC address information:
  Public base address     00:05:86:71:18:00
  Public count            960
  Private base address    00:05:86:71:1b:c0
  Private count           64

root@R4# run show arp no-resolve 
MAC Address       Address         Interface     Flags
56:68:28:3d:5c:5f 10.233.240.2    fxp0.0               none
56:68:28:3d:5c:5f 10.233.244.2    fxp0.0               none
56:68:28:41:89:cc 10.233.255.254  fxp0.0               none
00:05:86:71:35:c0 172.27.0.10     ae0.0                none
00:05:86:71:63:00 172.27.0.17     ge-0/0/1.0           none
00:05:86:71:99:00 172.27.0.22     ge-0/0/5.0           none
Total entries: 6

[edit]
root@R4# run show chassis mac-addresses 
MAC address information:
  Public base address     00:05:86:71:33:00
  Public count            960
  Private base address    00:05:86:71:36:c0
  Private count           64