+7 495 320-55-52
- Назад
- Телефоны
- +7 495 320-55-52
- Заказать звонок
info@mikrotik.moscow
г. Москва, ул. Бакунинская, 84
Пн-Пт: 09-00 до 18-00
Сб-Вс: выходной

Inbound mails not reaching the mail server

Inbound mails not reaching the mail server, RouterOS

dot02

Guest

19.12.2022 23:04:00

Hi all, I have a RB3011 with ROS7.5 running as an edge router. Behind it is a pfsense FW via a transit vlan (172.16.100.0/24), and behind that FW, a mail server on one of the PFsense’s legs (mailserver=172.16.15.11). The Mikrotik does the NAT-ing between the internal 172.16.0.0/12 networks and the public internet. It looks like this (well, it looks fancier in real life): (Internet)----Mikrotik-----Pfsense-----Mailserver In order to be completely honest, I wanted to be completely transparent by telling you that there is indeed a Pfsense FW in between, but please don’t point it to be the responsible device: this FW has been in place (with different upgrades of course) for 15+ years the mail server has been in place for 15+ years (also with upgrades of course) all that has changed now is the edge router as I am migrating from Cisco (+ WAN link behind it) to Mikrotik (+new WAN link behind it). the workaround I have put in place (see below) and packet captures (also see below) put the PFsense out of cause. Now to my issue. Oubound mails work perfectly (from our server to the outside world). However, we are unable to receive mails from outside with simple DST-NAT (with the old cisco it worked flawlessly), that is what makes me wonder). I am pretty sure there is an additional FW rule that needs to be set on the MT, but I can’t figure out which… As a temp workaround, I had to implement DST-NAT and SRC-NAT on the mikrotik. It works in terms of receiving emails, however, this is not a viable solution as the mail server sees all inbound mails coming from the MT’s LAN IP, which means that the IP-based ACL’s (for mitigating brute-force attacks) had to be deactivated and most spam filters are also useless now. Here’s what is currently in place: /ip firewall filter
add action=accept chain=forward comment="INBOUND SMTP" dst-address=172.16.15.11 dst-port=25 in-interface=WAN_ISP protocol=tcp

/ip firewall nat
add action=dst-nat chain=dstnat comment=SMTP dst-address=<PUBLIC_IP> dst-port=25 in-interface=WAN_ISP protocol=tcp to-addresses=172.16.15.11 to-ports=25
add action=src-nat chain=srcnat dst-address=172.16.15.11 dst-port=25 protocol=tcp to-addresses=172.16.100.2 the last line from the config above (src-nat) is the one I configured as a workaround, but which I need to get rid of asap in order to reactivate the ACL’s and spam filters. 172.16.100.2 is the Mikrotik IP on the transit VLAN to the Pfsense Now, here are 2 packet captures on the FW (on the 172.16.100.0/24 transit VLAN from the MT), both times for an inbound mail coming from the internet: without the SRC-NAT rule ( I replaced the external’s server IP with 1.1.1.1 as it is a client’s server…): 1.1.1.1.36610 > 172.16.15.11.25: tcp 0
172.16.15.11.25 > 1.1.1.1.36610: tcp 0
1.1.1.1.36610 > 172.16.15.11.25: tcp 0
172.16.15.11.25 > 11.1.1.1.36610: tcp 0
1.1.1.1.36610 > 172.16.15.11.25: tcp 0
1.1.1.1.36610 > 172.16.15.11.25: tcp 0
172.16.15.11.25 > 1.1.1.1.36610: tcp 0
1.1.1.1.36610 > 172.16.15.11.25: tcp 0
172.16.15.11.25 > 1.1.1.1.36610: tcp 0 with the SRC-NAT rule (172.16.100.2 is the MT’s Transit interface IP) 172.16.100.2.44218 > 172.16.15.11.25: tcp 0
172.16.15.11.25 > 172.16.100.2.44218: tcp 0
172.16.100.2.44218 > 172.16.15.11.25: tcp 0
172.16.15.11.25 > 172.16.100.2.44218: tcp 28
172.16.100.2.44218 > 172.16.15.11.25: tcp 0
172.16.100.2.44218 > 172.16.15.11.25: tcp 27
172.16.15.11.25 > 172.16.100.2.44218: tcp 73
172.16.100.2.44218 > 172.16.15.11.25: tcp 35
172.16.15.11.25 > 172.16.100.2.44218: tcp 8
172.16.100.2.44218 > 172.16.15.11.25: tcp 34
172.16.15.11.25 > 172.16.100.2.44218: tcp 8
172.16.100.2.44218 > 172.16.15.11.25: tcp 6
172.16.15.11.25 > 172.16.100.2.44218: tcp 15
172.16.100.2.44218 > 172.16.15.11.25: tcp 1440
172.16.100.2.44218 > 172.16.15.11.25: tcp 784
.... Could you help me out and help me find the rule which I guess is missing on the MT so that the mail reception could work? Thanks a lot! Denis

dot02

Guest

02.01.2023 23:38:00

I made some additional investigations, and found something interresting… the setup is as follows: Client is somewhere on the internet (another public IP) Test server behind the PFsense Captures are made on the PFsense, transit VLAN to the Mikrotik 1) Capture with DST-NAT + SRC-NAT on the Mikrotik: We see here that everything is working as expected. The client’s public IP is replaced by the MT’s IP address (thanks to SRC-NAT) on the transit VLAN, and the Pfsense sees the SYN, SYNACK, ACK handshake between the MT and the server (172.16.50.69). It is followed by a GET request from the MT, and the content of the webpage is loaded.

2) Capture with only DST-NAT on the Mikrotik: Here, what is really, really weird is that we see the SYN request from the client, followed by the SYN, ACK from the server (which validates routing is OK, otherwise the packet would have been dropped), but then, the MT never sends the final ACK to the server, so we get into TCP retransmits until the request times out:

Znevna

Guest

03.01.2023 06:35:00

From what you’ve said so far, you didn’t replace anything. You just added extra stuff to the existing setup (which was already double nat): you configured the existing pfSense to handle dual wan; you added the extra WAN to pfSense via that new MikroTik that also does NAT; Now somehow, something doesn’t work with these changes and there’s nothing else to blame than the new MikroTik router because it worked fine for 15 years in a different setup. Fishy. Show all the configs, not just 3 firewall rules, and try to make a more complete diagram of the setup. Also what is the purpose of the first firewall rule you’ve shown? the accept to 172.16.15.11 port 25 from wan rule in forward chain?

dot02

Guest

03.01.2023 09:58:00

Hi Znevna, thanks for your input, I might need to add some more details to give you more insights here indeed: you didn’t replace anything. You just added extra stuff to the existing setup (which was already double nat): Not exactly. The MT will eventually replace the Cisco router. The MT is the new device that was added to the infrastructure. Both routers need to stay online only during the migration, the Cisco will be decommissioned afterwards. The Pfsense does not do any NAT, just simple routing and firewalling. NAT is done on the edge routers only (Currently on the Cisco, to be replaced by the Mikrotik). you configured the existing pfSense to handle dual wan Yes, I unfortunately had to add a second gateway (to the Mikrotik) as a temporary solution because I cannot migrate all services in one shot. It is a slight config change on the pfsense indeed (new additional gateway + specifying this specific GW for selected rules) you added the extra WAN to pfSense via that new MikroTik that also does NAT Not really. The Pfsense doesn’t do any NAT. The Cisco router does, as also does the MT. something doesn’t work with these changes and there’s nothing else to blame than the new MikroTik router because it worked fine for 15 years in a different setup. Fishy. Well, what I meant was that on the Cisco, it works perfectly with only DST-NAT, and on the MT, I face the issue that the ACK packer is never received by the Pfsense. As you could read in my previous posts, it seemed to me that the Pfsense was responsible for this behaviour, but my packet captures clearly show that the ACK packet is not received by the Pfsense when only DST-NAT is configured on the MT. If I add SRC-NAT on the MT, I get the packets on the inbound Pfsense interface without the slightest config change on the Pfsense. I’m not blaming or badmouthing the Mikrotik here (I like MT!), but clearly there is something wrong if the SYN packets reach the pfsense but the ACK does not. All that we can conclude here is that it cannot be a routing issue. The link between the MT and the PFsense is a “directly connected” transit vlan. I will prepare a diagram in the next couple of minutes and upload it. Also what is the purpose of the first firewall rule you’ve shown? the accept to 172.16.15.11 port 25 from wan rule in forward chain? 172.16.15.11 is the mail server. The “INBOUND SMTP” firewall filter rule you are mentioning is needed to accept inbound traffic (mail sent from a distant SMTP server to my server) after it was been DSTNAT-ed. Initially, I configured this rule on the output chain, but it turns out it is the forward chain that needs to be used instead. The logic seems to be that despite the NAT-ing done by the MT, the device still considers that it is just forwarding the packet from src= to dst=. I created the exact same rule for my test LAMP server (172.16.50.59, port tcp/80) for my last test of course. I’ll prepare the diagram… Cheers!

dot02

Guest

03.01.2023 12:09:00

Here’s a quick drawing of the setup: https://data-discourse.cdn.mikrotik.com/original/3X/e/0/e02897d5d9c70e76bad2bc9940f0666d44ccb83a.png while looking into the captures again, I think I found the cause of the issue. This one is with sdst-nat and src-nat on the MT. From the dst field of this frame (this is the SYN,ACK frame) we can see that the frame is sent to the MT as it should (which is to be expected) https://data-discourse.cdn.mikrotik.com/original/3X/4/c/4ceaa25c6793a7c26b3b0ff4bd4c2e4715f5df40.png …but without the src-Nat on the MT, the pfsense sends the SYN,ACK frame to the Cisco instead of the MT: https://data-discourse.cdn.mikrotik.com/original/3X/9/7/97067e80e62d82b579f0dedbe41d423a04a68b55.png …so we have assymetric routing. I need to figure out why the pfsense sends it to the Cisco instead of the MT.

Znevna Guest	#6 0 03.01.2023 15:43:00 That diagram you’ve posted looks broken to me, i don’t understand how that could work reliably with the details provided. Could be that I just don’t understand networking that well, who knows. Maybe someone else will take a stab at it.

mkx

Guest

03.01.2023 17:26:00

That’s what I was trying to point out (without being a smartass). Check pfsense routing … go through whole config and try to see where that particular SYNACK packet weered towards Cisco. I’m guessing that pfsense is simply not properly configured to handle dual WAN with regard to incoming connections (i.e. it doesn’t mark connections via MT as such and then uses default route, which seems to be via Cisco, instead the correct route via MT).

dot02

Guest

03.01.2023 20:08:00

@Znevna : well it’s quite easy. VLAN100 is a transit VLAN. On the Pfsense, I have 2 gateways set. GW1 to the Cisco router and GW2 to the MT. Each rule has automatically GW1 set as the default route (that is default Pfsense behaviour), unless you specify it otherwise (on a rule-by-rule basis, or via policy-based routing). In my case, this enables me to migrate services one after the other from the old Cisco router to the new MT. Which part do you feel confused about? @mkx : Yes you are absolutely right. I know dual WAN can be tricky, especially for incoming connections. Thing is that I thought I had configured the PFsense properly, and some services work fine on the MT already (but these are outbound connections). It appears that I need additional config on the PFsense (setting the “reply-to” variable) which requires a specific config on the Interface first… I’ll give it a try tonight. But clearly we now know from the packet captures that it is the pfsense that doesn’t send back the packets as it should.

mkx

Guest

04.01.2023 10:24:00

It’s not tricky even for incoming connections if incoming connections for certain service are only allowed via single WAN, in this case static routing setup is enough. But it gets tricky if incoming connections are allowed via both (all) WANs … then the ultimate router (in your case that’s pfsense) needs to identify the used WAN on first ingress packet and then use same WAN for sending all replies. In MT that’s done using mangle rules (routing rules are not good enough), I’ve no idea what has to be done on pfsense. And according to your network layout it might be that pfsense actually can’t identify correct WAN because dst-address is already changed (DST NAT) and src-address doesn’t define WAN being used (SMTP client can use any of your WAN addresses to connect). If you used different physical interface to connect pfsense to both routers, you could use pfsense interface as differentiator between WANs, but not in this case. The only difference between packets arriving through both routers will be src MAC address of packets (it will either be mikrotik’s or cisco’s) … As I already wrote: you’re complicating your life by going into dual WAN config too deep as it’s only temporary. I know that it’d be nice to know actual remote IP address of SMTP client but since configuring src-nat on MT solves all other problems you have, you could simply go this way. After you get rid of Cisco-routed WAN (and/or set default route on pfsense towards MT), simply remove/disable all SRC-NATs created to solve the routing triangle problem.

dot02 Guest	#10 0 04.01.2023 14:37:00 I’ve solved it last night. It is working like a charm now. Even if it is a bit off-topic as it deals more with the PFsense config than the MT, I’d like to share my findings in case someone else is running into a similar issue. The issue in my setup (2 gateway IP’s on the same transit subnet) is, as you stated, that the Pfsense does not know where to route packets in certain cases. Indeed, as you also stated, a better way, especially for a production setup, would be to implement 2 separate WAN interfaces on the Pfsense, each one having a dedicated subnet to 1 router. Of course, in my case that was not possible as it was just a temporary setup while migrating from one router to another. If incoming packets (from the Internet) are DST-NATed on a router, it is not a problem for the Pfsense as it sees the packets coming from a “directly connected” IP. This even supersedes static routes. For incoming packets that are only SRC-NATed on one of the routers, the return packet is a problem because PFsense, by design, uses the default gateway which is specified on its WAN interface to send outgoing packets to. Specifying the gateways on the FW rules does not make the trick when you have only 1 WAN interface like I do. There is a workaround using the “reply-to” setting on PFsense’s rules, but I didn’t need to go that far. In the end, I defined the MT as the default route on PF’s Wan interface, and added some routes on the MT for services that needed to be handled by the Cisco router. I then removed the SRC-NAT on my MT, and migrated the services one after the other from Cisco to MT (I still have telephony to migrate, but short of that, I’m done). Since this morning, 90% of production is now running over the MT. Regarding your suggestion with SRC-NAT, one has to be very careful. I tried that at the beginning of this topic, but while it is suitable for certain types of service, you definitely can’t use it for SMTP for instance: Antispam filters will not work anymore since they will see all mails coming from the SRC-NAT IP. Same for dynamic access-lists needed to secure webmail and imap services against brute-force attacks (e.g. IP gets blocked for 30 days if you have 5 failed authentication attempts within 10 minutes). Thanks to all of you for your time and suggestions, it definitely helped me figuring out the issue! Cheers!

Читают тему

Главная Каталог 0 Корзина 0 Избранные Кабинет 0 Сравнение Акции Контакты Услуги Бренды Отзывы Компания Лицензии Документы Реквизиты Поиск Блог Обзоры