How to diagnose cross account EC2/RDS connection with vpc flow logs in AWS?

Question

In AWS, made 2 accounts.

• account 1: A | Deployed EC2
• account 2: B | Deployed RDS

Want to connect from account 1's EC2 to account 2's RDS, but doesn't work. So enabled VPC flow logs to diagnose where is the problem. The connection between these 2 accounts is using transit gateway. Both side have connected to self VPC with route table.

After did a connection test from A to B via db connect client, tried to find Network Interface ID from A, such as eni-XXXXXXXX belongs to the EC2 instance, in B's CloudWatch Logs Group, can't find its name.

I followed this guide to do, Using AWS VPC Flow Logs to Diagnose Intermittent RDS Connections. But these steps can't resolve my problem, I can't diagnose with it:

Determining your RDS Instance’s Elastic Network ID (ENI)

1. Use the AWS RDS Console to obtain the instance’s endpoint (this will be in the form [xxx].[region].rds.amazonaws.com).
2. Use dig or something similar to lookup the IP address of the endpoint. A CNAME is not sufficient, you’ll need the actual IP.
3. Use the AWS Network Interface Console to search for the IP Address you got, and obtain the Network Interface ID (this will be eni-XXXXXXXX) from the resulting record.

Querying Your Flow Logs

1. Use the VPC Console to locate your VPC. Go to the Flow Log tab, and click through to the CloudWatch Logs Group.
2. In the CloudWatch Logs Group, use the search box to find your Network Interface ID.
3. Click into the record you’ve found, and you’ll see all the traffic for your instance!

How can confirm the network traffic each other correctly?

I found an unknown port in the flow logs.

2 023910321035 eni-12309fdsaj0jf012 ACCOUNT_1_EC2_IP ACCOUNT_2_RDS_IP 45052 31337 6 3 180 1619490209 1619490213 REJECT OK
2 023910321035 eni-12309fdsaj0jf012 ACCOUNT_1_EC2_IP ACCOUNT_2_RDS_IP 45052 31337 6 3 180 1619490209 1619490213 REJECT OK

The port 31337 didn't open. Why it will connect this port?

Answer 1

1. Do you really need Transit Gateway? As suggested in the answer to your other question VPC Peering may be enough. If not tell us why.

2. In that other question you confirmed that your setup in the end worked and you could connect from EC2 to RDS. If you can't now what has changed?

3. As suggested in the other answer and also by @Tim here in the comment do this:

   • Spin up an EC2 instance in the same Subnet and Security Group as your RDS.
   • Use netcat to listen on the DB port (e.g. 3306 for mysql) and tcpdump to observe the traffic from the "other" EC2.

   Using that figure out if your connectivity works - do you see the incoming packets with tcpdump? Do you see the responses?