Hi,
I think this thread can be an appropriate context for a post about the issue we just ran into.
There are router vendors implementing OSPF so that the netmask is set to 0.0.0.0 in Hello on a point-to-point interface. Our example is Vyatta, apparently running Quagga as its OSPF process, so all Quaggas out there might be affected.
In its turn, JunOS (11.2 in our case) still tries to validate the Hello netmask if the interface is Ethernet notwithstanding its p2p mode in OSPF. By the way, this is against the behavior RECOMMENDED by Section 10.5 of RFC 2328 that "on point-to-point networks and on virtual links, the Network Mask in the received Hello Packet should be ignored."
The end result is that adjacency fails to establish due to netmask mismatch in JunOS, e.g.:
Feb 8 22:39:17.458702 OSPF rcvd Hello 10.7.8.6 -> 224.0.0.5 (reth0.1 IFL 142 area 0.0.0.0)
Feb 8 22:39:17.458739 Version 2, length 48, ID 10.1.200.1, area 0.0.0.0
Feb 8 22:39:17.458768 checksum 0x0, authtype 0
Feb 8 22:39:17.458799 mask 0.0.0.0, hello_ivl 10, opts 0x2, prio 1
Feb 8 22:39:17.458828 dead_ivl 40, DR 0.0.0.0, BDR 0.0.0.0
Feb 8 22:39:17.458856 neighbor 10.1.252.1
Feb 8 22:39:17.458893 OSPF packet ignored: netmask 0.0.0.0 mismatch from 10.7.8.6 on intf reth0.1 area 0.0.0.0
The workaround is just to fall back to broadcast mode on OSPF links backed by broadcast physical interfaces.
(No such problem on true p2p links such as VPN tunnels.)
Hoping this will save some of you some troubleshooting time some day!
Thanks!