Network-common-problem

发表于 更新于 分类于 Waline：

mac 篇

常用命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
# 查看当前路由表
netstat -rn
----------------------------------------------------------------
Routing tables
Internet:
Destination        Gateway            Flags        Netif Expire
default            192.168.43.88      UGSc           en0
default            11.13.2.254        UGScI          en7
-----------------------------------------------------------------
#获取默认路由
route get 0.0.0.0
--------------------------------------------------------------------------------
   route to: default
destination: default
       mask: default
    gateway: 192.168.43.88
  interface: en0
      flags: <UP,GATEWAY,DONE,STATIC,PRCLONING>
 recvpipe  sendpipe  ssthresh  rtt,msec    rttvar  hopcount      mtu     expire
       0         0         0         0         0         0      1500         0
---------------------------------------------------------------------------------
#删除默认路由
sudo route -n delete default 192.168.43.88
#添加外网网关
sudo route add -net 0.0.0.0 192.168.43.88
#添加内网网关
sudo route add -net 11.8.129.0 11.13.2.254

Linux 篇

常见命令

1
2
3
4
5
6
7
8
9
10
#和网络有关的配置文件 
/etc/resolv.conf 
#查看网关设置 
grep GATEWAY /etc/sysconfig/network-scripts/ifcfg* 
#增加网关: 
route add default gw 192.168.40.1 
#重启网络 
service network restart 
#查看DNS解析 
grep hosts /etc/nsswitch.conf

分析

traceroute <ip>

网络测试、测量、管理、分析，官网

ICMP错误信息分析：

!H 不能到达主机

!N 不能到达网络

!P 不能到达的协议

!S 源路由失效

!F 需要分段

正常情况：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[root@environment-test1 ~]# traceroute 4.2.2.2
traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets
 1  gateway (192.168.1.1)  0.440 ms  0.594 ms  0.743 ms
 2  * * *
 3  121.33.196.105 (121.33.196.105)  4.352 ms  4.443 ms  4.521 ms
 4  183.56.31.37 (183.56.31.37)  7.290 ms 183.56.31.21 (183.56.31.21)  9.217 ms 183.56.31.13 (183.56.31.13)  6.755 ms
 5  153.176.37.59.broad.dg.gd.dynamic.163data.com.cn (59.37.176.153)  6.884 ms  6.993 ms  7.084 ms
 6  121.8.223.13 (121.8.223.13)  9.307 ms  5.848 ms 183.56.31.173 (183.56.31.173)  4.443 ms
 7  202.97.94.130 (202.97.94.130)  4.029 ms  4.165 ms 202.97.94.142 (202.97.94.142)  5.546 ms
 8  202.97.94.98 (202.97.94.98)  11.225 ms 202.97.94.118 (202.97.94.118)  6.177 ms  6.600 ms
 9  202.97.52.18 (202.97.52.18)  209.571 ms 202.97.52.142 (202.97.52.142)  206.772 ms 202.97.58.2 (202.97.58.2)  197.316 ms
10  195.50.126.217 (195.50.126.217)  213.784 ms  213.917 ms  211.676 ms
11  4.69.163.22 (4.69.163.22)  312.436 ms 4.69.141.230 (4.69.141.230)  214.040 ms  213.168 ms
12  b.resolvers.Level3.net (4.2.2.2)  209.348 ms  210.701 ms  210.588 ms

有问题的情况：

1
2
3
[root@lfadmin ~]# traceroute 4.2.2.2
traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets
 1  gateway (192.168.1.1)  0.751 ms !N  0.817 ms !N  1.326 ms !N

ifconfig <网卡名字>

netstat -r相似route

显示路由连接信息等

1
2
3
4
5
6
7
8
[root@environment-test1 ~]# netstat -r
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         gateway         0.0.0.0         UG        0 0          0 enp3s0
link-local      0.0.0.0         255.255.0.0     U         0 0          0 enp3s0
172.17.0.0      0.0.0.0         255.255.0.0     U         0 0          0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U         0 0          0 doc...ridge
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 enp3s0

host <域名> 相似nslookup <域名>

dns分析

1
2
3
4
[root@environment-test1 ~]#  host www.baidu.com
www.baidu.com is an alias for www.a.shifen.com.
www.a.shifen.com has address 14.215.177.38
www.a.shifen.com has address 14.215.177.39

nmcli查看设备状态

ip route show | column -t 查看路由

问题1 ：无法连外网，可以ping 路由器

提示

1
2
3
[root@lfadmin ~]# traceroute 4.2.2.2
traceroute to 4.2.2.2 (4.2.2.2), 30 hops max, 60 byte packets
 1  gateway (192.168.1.1)  0.751 ms !N  0.817 ms !N  1.326 ms !N

解决原因，是网络配置文件uuid冲突，导致不能上网，修改即可

执行uuidgen ens33生产新的830a6ae2-85fb-41e7-9e5d-60d084f56f5f替换配置文件里面的

执行nmcli con | sed -n '1,2p'进行验证

问题2: 在使用移动的专线时，映射出8088端口，一些手机能访问服务一些手机不能访问服务

解决：移动说时该ip下的一些域名被idc封堵了，需要解封，或者备案，备了案就不回封堵了。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
# 能访问的热点
➜  ~ telnet 27.176.159.182 8088
Trying 27.176.159.182...
Connected to 27.176.159.182.
Escape character is '^]'.
^CConnection closed by foreign host.
➜  ~ sudo tcptraceroute 27.176.159.182 8088
Password:
Sorry, try again.
Password:
Selected device en0, address 172.20.10.3, port 60864 for outgoing packets
Tracing the path to 27.176.159.182 on TCP port 8088 (radan-http), 30 hops max
 1  172.20.10.1  4.176 ms  13.755 ms  3.996 ms
 2  * 192.168.25.254 2240.375 ms  2457.379 ms
 3  * * *
 4  * * *
 5  172.31.5.1  55.430 ms * *
 6  139.203.67.153  114.910 ms
    139.203.67.157  84.705 ms *
 7  * * 202.97.66.86 48.711 ms
 8  221.183.187.29  66.619 ms * *
 9  * * *
10  * * *
11  * * *
12  223.87.26.174  41.827 ms  47.060 ms  63.440 ms
13  * * *
14  221.182.42.126  58.215 ms
    221.182.42.130  130.437 ms  60.580 ms
15  * * *
16  27.176.159.182  91.646 ms  41.202 ms  90.640 ms
17  27.176.159.182 [open]  58.467 ms  84.589 ms  40.736 ms
➜  ~
# 不能访问的热点
➜  ~ telnet 27.176.159.182 8088
Trying 27.176.159.182...
^C
➜  ~ sudo tcptraceroute 27.176.159.182 8088
Password:
Selected device en0, address 172.20.10.6, port 51541 for outgoing packets
Tracing the path to 27.176.159.182 on TCP port 8088 (radan-http), 30 hops max
 1  172.20.10.1  4.015 ms  2.467 ms  2.508 ms
 2  192.168.25.254  29.404 ms  19.509 ms  40.122 ms
 3  10.1.0.9  41.479 ms * 42.718 ms
 4  * * *
 5  172.31.4.1  41.928 ms * *
 6  139.203.67.145  56.212 ms * *
 7  * * *
 8  * * 221.183.95.209 27.317 ms
 9  * 221.183.90.82 85.681 ms *
10  * * *
11  * 223.87.26.29 56.235 ms  52.355 ms
12  223.87.27.250  29.739 ms  33.021 ms  38.172 ms
13  223.85.135.158  20.524 ms  35.416 ms  40.353 ms
14  221.182.42.130  42.606 ms  31.836 ms  39.801 ms
15  * * *
......
30  * * *
Destination not reached

使用ping时都能ping通，tcp进行测试时，结果如下，一些能访问，一些不能访问：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Query: tcp 27.176.159.182:8088

Location	ISP	TCP port check result
Canada, BC, Vancouver	Telus	Connection to 27.176.159.182:8088 successful
Canada, BC, Vancouver	Shaw	Connection to 27.176.159.182:8088 successful
USA, CA, Fremont	Hurricane	Connection to 27.176.159.182:8088 successful
USA, CA, Fremont	IT7 FMT2	Connection to 27.176.159.182:8088 successful
USA, CA, Fremont	Linode	Connection to 27.176.159.182:8088 failed
USA, CA, San Francisco	Digital Ocean	Connection to 27.176.159.182:8088 failed
USA, CA, Santa Clara	Hurricane	Connection to 27.176.159.182:8088 failed
USA, CA, Los Angeles	Cogent	Connection to 27.176.159.182:8088 successful
Australia, Sydney	Vultr	Connection to 27.176.159.182:8088 failed
Taiwan, Taichung	Google	Connection to 27.176.159.182:8088 failed
China, Guiyang	Huawei	Connection to 27.176.159.182:8088 successful
China, Beijing	Tencent	Connection to 27.176.159.182:8088 failed
China, Beijing	Huawei	Connection to 27.176.159.182:8088 successful
China, Shandong	China Unicom	Connection to 27.176.159.182:8088 successful
China, Jiangsu	China Telecom	Connection to 27.176.159.182:8088 failed
China, Jiangsu	China Mobile	Connection to 27.176.159.182:8088 failed
China, Qingdao	Aliyun	Connection to 27.176.159.182:8088 successful
China, Shanghai	Aliyun	Connection to 27.176.159.182:8088 failed
China, Shanghai	Huawei	Connection to 27.176.159.182:8088 successful
China, Shanghai	Tencent	Connection to 27.176.159.182:8088 failed

后面联系移动，移动给了一个公网测试ip，为了不影响原有网络，直接通过附加ip的方式配置：

  1. 登录FortiGate-100F

  2. 在网络->接口->物理接口，编辑移动出口2 (port1)

  3. 打开附加的ip地址的开关,新建，输入移动给的测试ip211.137.109.15/255.255.255.0，勾选ping，方便测试。

  4. 保存之后通过ping 211.137.109.15就可以看到ping成功了，如果不成功稍等几分钟。

  5. 配置虚拟ip服务映射，在策略&对象->虚拟IP->新建,新建一个test服务

    1
    2
    3
    4
    5
    6
    7
    8
    9
    名称： test
    接口： 移动出口2 (port1)
    外部IP地址/范围： 211.137.109.15
    映射到IPv4地址/范围：172.16.10.30
    端口转发：开
    协议：TCP
    端口映射类型：一对一
    外部服务端口：8099
    映射到IPv4端口：8099

  6. 配置转发规则也就是防火墙策略，在策略&对象->防火墙策略->新建,新建一个test策略，这个页面策略的优先级是从上到下，如果没有匹配到就会被最后一个隐式拒绝策略拒绝。

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    名称： test
    类型：标准
    流入接口：移动给出口2(port1)
    流出接口：lan
    源地址：all  #这里设置china可以限制为国内访问
    目标地址：test #这里是上一步建立虚拟ip映射的名称
    计划任务：always
    服务：ALL
    动作：接受
    检测模式：基于流
    启用NAT：开
    ...剩下的默认

  7. 配置好后可以在第五步看到有一个关联项，现在测试tcp 211.137.109.15:8088，结果发现还是部分可以部分不可以，由此可以判断可能不是移动ip被封了

继续排查，用电脑直接连接移动专线的光猫的lan2口，提供服务，经全国访问测试tcp 211.137.109.15:8088，发现都能访问，基本排除了移动的问题，基本问题可以定位到FortiGate-100F的问题。

点击网络->诊断程序->Debug Flow开启过滤，输入8088端口，点击开始debug flow，分别抓取可以访问的手机的流量和不可以访问的手机流量。

可以访问的如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
Trace ID,时间设置,消息
"vd-root:0 received a packet(proto=6, 171.218.234.126:50428->211.137.109.15:8099) tun_id=0.0.0.0 from port1. flag [S], seq 152754952, ack 0, win 65535"
"allocate a new session-0002e97a, tun_id=0.0.0.0"
"in-[port1], out-[]"
len=1
checking gnum-100000 policy-25
"find DNAT: IP-172.16.10.30, port-8099"
"matched policy-25, act=accept, vip=25, flag=100, sflag=2000000"
"result: skb_flags-02000000, vid-25, ret-matched, act-accept, flag-00000100"
"VIP-172.16.10.30:8099, outdev-port1"
DNAT 211.137.109.15:8099->172.16.10.30:8099
find a route: flag=00000000 gw-192.168.100.253 via lan
"in-[port1], out-[lan], skb_flags-020000c0, vid-25, app_id: 0, url_cat_id: 0"
"gnum-100004, use int hash, slot=2, len=4"
"checked gnum-100004 policy-6, ret-matched, act-accept"
"checked gnum-100004 policy-10, ret-no-match, act-accept"
"checked gnum-100004 policy-13, ret-matched, act-accept"
ret-matched
"gnum-4e22, check-ffffffbffc02b9e4"
"checked gnum-4e22 policy-6, ret-no-match, act-accept"
"checked gnum-4e22 policy-6, ret-no-match, act-accept"
"checked gnum-4e22 policy-6, ret-no-match, act-accept"
"gnum-4e22 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000"
"find SNAT: IP-192.168.100.254(from IPPOOL), port-50428"
"policy-13 is matched, act-accept"
"after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-13"
"after iprope_captive_check(): is_captive-0, ret-matched, act-accept, idx-13"
"in-[port1], out-[lan], skb_flags-020000c0, vid-25"
"gnum-100015, check-ffffffbffc02a8d0"
"checked gnum-100015 policy-1, ret-no-match, act-accept"
"checked gnum-100015 policy-4, ret-no-match, act-accept"
"gnum-100015 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000"
"after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
"in-[port1], out-[lan], skb_flags-020000c0, vid-25"
len=0
Allowed by Policy-13: SNAT
SNAT 171.218.234.126->192.168.100.254:50428

不可以访问的如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Trace ID,时间设置,消息
"vd-root:0 received a packet(proto=6, 39.144.139.31:10233->211.137.109.15:8099) tun_id=0.0.0.0 from port1. flag [S], seq 2898517655, ack 0, win 65535"
"allocate a new session-03ce447f, tun_id=0.0.0.0"
"in-[port1], out-[]"
len=1
checking gnum-100000 policy-25
"find DNAT: IP-172.16.10.30, port-8099"
"matched policy-25, act=accept, vip=25, flag=100, sflag=2000000"
"result: skb_flags-02000000, vid-25, ret-matched, act-accept, flag-00000100"
"VIP-172.16.10.30:8099, outdev-port1"
DNAT 211.137.109.15:8099->172.16.10.30:8099
find a route: flag=00000000 gw-172.16.10.30 via l2t.root
"in-[port1], out-[l2t.root], skb_flags-020000c0, vid-25, app_id: 0, url_cat_id: 0"
"gnum-100004, use int hash, slot=126, len=1"
"checked gnum-100004 policy-0, ret-matched, act-accept"
ret-matched
"policy-0 is matched, act-drop"
"after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0"
"after iprope_captive_check(): is_captive-0, ret-matched, act-drop, idx-0"
"in-[port1], out-[l2t.root], skb_flags-020000c0, vid-25"
"gnum-100015, check-ffffffbffc02a8d0"
"checked gnum-100015 policy-1, ret-no-match, act-accept"
"checked gnum-100015 policy-4, ret-no-match, act-accept"
"gnum-100015 check result: ret-no-match, act-accept, flag-00000000, flag2-00000000"
"after check: ret-no-match, act-accept, flag-00000000, flag2-00000000"
Denied by forward policy check (policy 0)

对比发现

1
2
3
4
#不可以访问的走到了l2t.root
find a route: flag=00000000 gw-172.16.10.30 via l2t.root
#可以访问的走到了lan
find a route: flag=00000000 gw-192.168.100.253 via lan

在FortiGate-100F的CLI命令行执行get router info routing-table all

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
FortiGate-100F # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       V - BGP VPNv4
       * - candidate default

Routing table for VRF=0
S*      0.0.0.0/0 [10/0] via 117.176.159.129, port1, [1/0]
                  [10/0] via 192.168.2.1, wan1, [1/0]
                  [10/0] via 223.85.227.129, wan2, [1/0]
S       8.8.8.8/32 [10/0] via 10.0.11.1, to-forti-hk, [1/0]
C       10.0.10.0/30 is directly connected, to-forti-hk2
C       10.0.10.2/32 is directly connected, to-forti-hk2
C       10.0.11.0/30 is directly connected, to-forti-hk
C       10.0.11.2/32 is directly connected, to-forti-hk
S       100.64.8.8/32 [10/0] via 10.0.11.1, to-forti-hk, [1/0]
C       117.176.159.128/25 is directly connected, port1
S       172.16.0.0/16 [10/0] via 192.168.100.253, lan, [1/0]
                      [10/0] is directly connected, l2t.root, [1/0]
S       172.16.200.0/29 [10/0] is directly connected, l2t.root, [1/0]
C       192.168.2.0/24 is directly connected, wan1
C       192.168.100.0/24 is directly connected, lan
C       211.137.109.0/24 is directly connected, port1
C       223.85.227.128/25 is directly connected, wan2

分析发现172.16.0.0有两个路由地址，一个lan一个l2t.root

在界面网络->静态路由里面也可以看到172.16.0.0有两个路由地址，优先级且一样，这就导致了流量随机命中一个路由。修改优先级或禁用错误的路由，解决了。

参考

CentOS7配置网卡为静态IP，如果你还学不会那真的没有办法了！