|
Hi,
I have done some more testing over the last day and I think it could be
something with the lvs setup.
For yesterday, I configured a second director on a separate server using the
same real servers but with a different vip and used round robin dns to split
the incoming between both directors. This is the first time we have
actually seen it go above 7.5 gb on the outgoing and it peaked yesterday at
around 450mbps per real server.
Early this morning, I only had one director running and we hit 300mbps per
server and the video was stopping every so often. Once I added the second
director into the dns, initially the connections through the first director
were still giving problems but the connections through the second director
were good with no pauses. Once about 15% of the connections dropped off of
the first director, the streaming came good on that one as well.
At the moment, both directors are running and the bandwidth running through
each server is just over 500mbps. ipvsadm is showing about 900 connections
per server for each director (1800 combined).
Looking at the graphs in hyperic, the cpu load on the directors has not gone
above about 5%.
What I have noticed though is the 5 min load average goes to about 0.8 - 1.2
when the director connections start coming on and the number of received
packets dropped per minute on the director start going up a lot.
Originally, I had the directors setup in active - passive configuration
using heartbeat to failover. I have also been running ipvs_sync between
them to keep them in sync in case of failover.
I am thinking that it may be some sysctl settings that are wrong for the
amount of traffic I am trying to push through the directors. Any help on
these settings would be greatly appreciated. Below are the current sysctl
setting. I have removed what I didn't think was at all related but if there
are some other settings that are missing then please let me know.
[root@lbe1 ~]# sysctl -a
net.ipv4.vs.nat_icmp_send = 0
net.ipv4.vs.sync_threshold = 3 50
net.ipv4.vs.expire_quiescent_template = 0
net.ipv4.vs.expire_nodest_conn = 0
net.ipv4.vs.cache_bypass = 0
net.ipv4.vs.secure_tcp = 0
net.ipv4.vs.drop_packet = 0
net.ipv4.vs.drop_entry = 0
net.ipv4.vs.am_droprate = 10
net.ipv4.vs.amemthresh = 1024
net.ipv4.conf.eth1.promote_secondaries = 0
net.ipv4.conf.eth1.force_igmp_version = 0
net.ipv4.conf.eth1.disable_policy = 0
net.ipv4.conf.eth1.disable_xfrm = 0
net.ipv4.conf.eth1.arp_accept = 0
net.ipv4.conf.eth1.arp_ignore = 0
net.ipv4.conf.eth1.arp_announce = 0
net.ipv4.conf.eth1.arp_filter = 0
net.ipv4.conf.eth1.tag = 0
net.ipv4.conf.eth1.log_martians = 0
net.ipv4.conf.eth1.bootp_relay = 0
net.ipv4.conf.eth1.medium_id = 0
net.ipv4.conf.eth1.proxy_arp = 0
net.ipv4.conf.eth1.accept_source_route = 0
net.ipv4.conf.eth1.send_redirects = 1
net.ipv4.conf.eth1.rp_filter = 0
net.ipv4.conf.eth1.shared_media = 1
net.ipv4.conf.eth1.secure_redirects = 1
net.ipv4.conf.eth1.accept_redirects = 1
net.ipv4.conf.eth1.mc_forwarding = 0
net.ipv4.conf.eth1.forwarding = 1
net.ipv4.conf.eth0.promote_secondaries = 0
net.ipv4.conf.eth0.force_igmp_version = 0
net.ipv4.conf.eth0.disable_policy = 0
net.ipv4.conf.eth0.disable_xfrm = 0
net.ipv4.conf.eth0.arp_accept = 0
net.ipv4.conf.eth0.arp_ignore = 0
net.ipv4.conf.eth0.arp_announce = 0
net.ipv4.conf.eth0.arp_filter = 0
net.ipv4.conf.eth0.tag = 0
net.ipv4.conf.eth0.log_martians = 0
net.ipv4.conf.eth0.bootp_relay = 0
net.ipv4.conf.eth0.medium_id = 0
net.ipv4.conf.eth0.proxy_arp = 0
net.ipv4.conf.eth0.accept_source_route = 0
net.ipv4.conf.eth0.send_redirects = 1
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.eth0.shared_media = 1
net.ipv4.conf.eth0.secure_redirects = 1
net.ipv4.conf.eth0.accept_redirects = 1
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.default.promote_secondaries = 0
net.ipv4.conf.default.force_igmp_version = 0
net.ipv4.conf.default.disable_policy = 0
net.ipv4.conf.default.disable_xfrm = 0
net.ipv4.conf.default.arp_accept = 0
net.ipv4.conf.default.arp_ignore = 0
net.ipv4.conf.default.arp_announce = 0
net.ipv4.conf.default.arp_filter = 0
net.ipv4.conf.default.tag = 0
net.ipv4.conf.default.log_martians = 0
net.ipv4.conf.default.bootp_relay = 0
net.ipv4.conf.default.medium_id = 0
net.ipv4.conf.default.proxy_arp = 0
net.ipv4.conf.default.accept_source_route = 0
net.ipv4.conf.default.send_redirects = 1
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.default.shared_media = 1
net.ipv4.conf.default.secure_redirects = 1
net.ipv4.conf.default.accept_redirects = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.all.promote_secondaries = 0
net.ipv4.conf.all.force_igmp_version = 0
net.ipv4.conf.all.disable_policy = 0
net.ipv4.conf.all.disable_xfrm = 0
net.ipv4.conf.all.arp_accept = 0
net.ipv4.conf.all.arp_ignore = 0
net.ipv4.conf.all.arp_announce = 0
net.ipv4.conf.all.arp_filter = 0
net.ipv4.conf.all.tag = 0
net.ipv4.conf.all.log_martians = 0
net.ipv4.conf.all.bootp_relay = 0
net.ipv4.conf.all.medium_id = 0
net.ipv4.conf.all.proxy_arp = 0
net.ipv4.conf.all.accept_source_route = 0
net.ipv4.conf.all.send_redirects = 1
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.all.shared_media = 1
net.ipv4.conf.all.secure_redirects = 1
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.neigh.eth1.base_reachable_time_ms = 30000
net.ipv4.neigh.eth1.retrans_time_ms = 1000
net.ipv4.neigh.eth1.locktime = 99
net.ipv4.neigh.eth1.proxy_delay = 79
net.ipv4.neigh.eth1.anycast_delay = 99
net.ipv4.neigh.eth1.proxy_qlen = 64
net.ipv4.neigh.eth1.unres_qlen = 3
net.ipv4.neigh.eth1.gc_stale_time = 60
net.ipv4.neigh.eth1.delay_first_probe_time = 5
net.ipv4.neigh.eth1.base_reachable_time = 30
net.ipv4.neigh.eth1.retrans_time = 99
net.ipv4.neigh.eth1.app_solicit = 0
net.ipv4.neigh.eth1.ucast_solicit = 3
net.ipv4.neigh.eth1.mcast_solicit = 3
net.ipv4.neigh.eth0.base_reachable_time_ms = 30000
net.ipv4.neigh.eth0.retrans_time_ms = 1000
net.ipv4.neigh.eth0.locktime = 99
net.ipv4.neigh.eth0.proxy_delay = 79
net.ipv4.neigh.eth0.anycast_delay = 99
net.ipv4.neigh.eth0.proxy_qlen = 64
net.ipv4.neigh.eth0.unres_qlen = 3
net.ipv4.neigh.eth0.gc_stale_time = 60
net.ipv4.neigh.eth0.delay_first_probe_time = 5
net.ipv4.neigh.eth0.base_reachable_time = 30
net.ipv4.neigh.eth0.retrans_time = 99
net.ipv4.neigh.eth0.app_solicit = 0
net.ipv4.neigh.eth0.ucast_solicit = 3
net.ipv4.neigh.eth0.mcast_solicit = 3
net.ipv4.neigh.default.base_reachable_time_ms = 30000
net.ipv4.neigh.default.retrans_time_ms = 1000
net.ipv4.neigh.default.gc_thresh3 = 1024
net.ipv4.neigh.default.gc_thresh2 = 512
net.ipv4.neigh.default.gc_thresh1 = 128
net.ipv4.neigh.default.gc_interval = 30
net.ipv4.neigh.default.locktime = 99
net.ipv4.neigh.default.proxy_delay = 79
net.ipv4.neigh.default.anycast_delay = 99
net.ipv4.neigh.default.proxy_qlen = 64
net.ipv4.neigh.default.unres_qlen = 3
net.ipv4.neigh.default.gc_stale_time = 60
net.ipv4.neigh.default.delay_first_probe_time = 5
net.ipv4.neigh.default.base_reachable_time = 30
net.ipv4.neigh.default.retrans_time = 99
net.ipv4.neigh.default.app_solicit = 0
net.ipv4.neigh.default.ucast_solicit = 3
net.ipv4.neigh.default.mcast_solicit = 3
net.ipv4.udp_wmem_min = 4096
net.ipv4.udp_rmem_min = 4096
net.ipv4.udp_mem = 772896 1030528 1545792
net.ipv4.cipso_rbm_strictvalid = 1
net.ipv4.cipso_rbm_optfmt = 0
net.ipv4.cipso_cache_bucket_size = 10
net.ipv4.cipso_cache_enable = 1
net.ipv4.tcp_slow_start_after_idle = 1
net.ipv4.tcp_dma_copybreak = 4096
net.ipv4.tcp_workaround_signed_windows = 0
net.ipv4.tcp_base_mss = 512
net.ipv4.tcp_mtu_probing = 0
net.ipv4.tcp_abc = 0
net.ipv4.tcp_congestion_control = bic
net.ipv4.tcp_tso_win_divisor = 3
net.ipv4.tcp_moderate_rcvbuf = 1
net.ipv4.tcp_no_metrics_save = 0
net.ipv4.ipfrag_max_dist = 64
net.ipv4.ipfrag_secret_interval = 600
net.ipv4.tcp_low_latency = 0
net.ipv4.tcp_frto = 0
net.ipv4.tcp_tw_reuse = 0
net.ipv4.icmp_ratemask = 6168
net.ipv4.icmp_ratelimit = 1000
net.ipv4.tcp_adv_win_scale = 2
net.ipv4.tcp_app_win = 31
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_mem = 196608 262144 393216
net.ipv4.tcp_dsack = 1
net.ipv4.tcp_ecn = 0
net.ipv4.tcp_reordering = 3
net.ipv4.tcp_fack = 1
net.ipv4.tcp_orphan_retries = 0
net.ipv4.inet_peer_gc_maxtime = 120
net.ipv4.inet_peer_gc_mintime = 10
net.ipv4.inet_peer_maxttl = 600
net.ipv4.inet_peer_minttl = 120
net.ipv4.inet_peer_threshold = 65664
net.ipv4.igmp_max_msf = 10
net.ipv4.igmp_max_memberships = 20
net.ipv4.route.rt_cache_rebuild_count = 4
net.ipv4.route.secret_interval = 600
net.ipv4.route.min_adv_mss = 256
net.ipv4.route.min_pmtu = 552
net.ipv4.route.mtu_expires = 600
net.ipv4.route.gc_elasticity = 8
net.ipv4.route.error_burst = 5000
net.ipv4.route.error_cost = 1000
net.ipv4.route.redirect_silence = 20480
net.ipv4.route.redirect_number = 9
net.ipv4.route.redirect_load = 20
net.ipv4.route.gc_interval = 60
net.ipv4.route.gc_timeout = 300
net.ipv4.route.gc_min_interval_ms = 500
net.ipv4.route.gc_min_interval = 0
net.ipv4.route.max_size = 4194304
net.ipv4.route.gc_thresh = 262144
net.ipv4.route.max_delay = 10
net.ipv4.route.min_delay = 2
net.ipv4.icmp_errors_use_inbound_ifaddr = 0
net.ipv4.icmp_ignore_bogus_error_responses = 1
net.ipv4.icmp_echo_ignore_broadcasts = 1
net.ipv4.icmp_echo_ignore_all = 0
net.ipv4.ip_local_port_range = 32768 61000
net.ipv4.tcp_max_syn_backlog = 1024
net.ipv4.tcp_rfc1337 = 0
net.ipv4.tcp_stdurg = 0
net.ipv4.tcp_abort_on_overflow = 0
net.ipv4.tcp_tw_recycle = 0
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_fin_timeout = 60
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.ipfrag_time = 30
net.ipv4.ip_dynaddr = 0
net.ipv4.ipfrag_low_thresh = 196608
net.ipv4.ipfrag_high_thresh = 262144
net.ipv4.tcp_max_tw_buckets = 180000
net.ipv4.tcp_max_orphans = 65536
net.ipv4.tcp_synack_retries = 5
net.ipv4.tcp_syn_retries = 5
net.ipv4.ip_nonlocal_bind = 0
net.ipv4.ip_no_pmtu_disc = 0
net.ipv4.ip_default_ttl = 64
net.ipv4.ip_forward = 1
net.ipv4.tcp_retrans_collapse = 1
net.ipv4.tcp_sack = 1
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.core.netdev_budget = 300
net.core.somaxconn = 128
net.core.xfrm_larval_drop = 0
net.core.xfrm_acq_expires = 30
net.core.xfrm_aevent_rseqth = 2
net.core.xfrm_aevent_etime = 10
net.core.optmem_max = 20480
net.core.message_burst = 10
net.core.message_cost = 5
net.core.netdev_max_backlog = 1000
net.core.dev_weight = 64
net.core.rmem_default = 129024
net.core.wmem_default = 129024
net.core.rmem_max = 33554432
net.core.wmem_max = 33554432
vm.vm_devzero_optimized = 1
vm.max_reclaims_in_progress = 0
vm.max_writeback_pages = 1024
vm.flush_mmap_pages = 1
vm.pagecache = 100
vm.min_slab_ratio = 5
vm.min_unmapped_ratio = 1
vm.zone_reclaim_interval = 30
vm.zone_reclaim_mode = 0
vm.swap_token_timeout = 300 0
vm.topdown_allocate_fast = 0
vm.legacy_va_layout = 0
vm.vfs_cache_pressure = 100
vm.block_dump = 0
vm.laptop_mode = 0
vm.max_map_count = 65536
vm.percpu_pagelist_fraction = 0
vm.min_free_kbytes = 11485
vm.drop_caches = 0
vm.lowmem_reserve_ratio = 256 256 32
vm.hugetlb_shm_group = 0
vm.nr_hugepages = 0
vm.swappiness = 60
vm.nr_pdflush_threads = 2
vm.dirty_expire_centisecs = 2999
vm.dirty_writeback_centisecs = 499
vm.mmap_min_addr = 4096
vm.dirty_ratio = 40
vm.dirty_background_ratio = 10
vm.page-cluster = 3
vm.overcommit_ratio = 50
vm.panic_on_oom = 0
vm.overcommit_memory = 0
kernel.vsyscall64 = 1
kernel.blk_iopoll = 1
kernel.max_lock_depth = 1024
kernel.compat-log = 1
kernel.hung_task_warnings = 10
kernel.hung_task_timeout_secs = 120
kernel.hung_task_check_count = 4194304
kernel.hung_task_panic = 0
kernel.softlockup_panic = 0
kernel.softlockup_thresh = 10
kernel.acpi_video_flags = 0
kernel.randomize_va_space = 1
kernel.bootloader_type = 113
kernel.panic_on_unrecovered_nmi = 0
kernel.unknown_nmi_panic = 0
kernel.ngroups_max = 65536
kernel.printk_ratelimit_burst = 10
kernel.printk_ratelimit = 5
kernel.panic_on_oops = 1
kernel.pid_max = 32768
kernel.overflowgid = 65534
kernel.overflowuid = 65534
kernel.pty.nr = 1
kernel.pty.max = 4096
kernel.random.uuid = c2b133d1-ab0b-4c02-abf8-3a3fe18711ad
kernel.random.boot_id = 09cf8e5d-99fa-4a37-8d3a-4c18335b437f
kernel.random.write_wakeup_threshold = 128
kernel.random.read_wakeup_threshold = 64
kernel.random.entropy_avail = 2574
kernel.random.poolsize = 4096
kernel.threads-max = 147456
kernel.cad_pid = 1
kernel.sysrq = 0
kernel.sem = 250 32000 32 128
kernel.msgmnb = 65536
kernel.msgmni = 16
kernel.msgmax = 65536
kernel.shmmni = 4096
kernel.shmall = 4294967296
kernel.shmmax = 68719476736
kernel.acct = 4 2 30
kernel.hotplug =
kernel.modprobe = /sbin/modprobe
kernel.printk = 6 4 1 7
kernel.ctrl-alt-del = 0
kernel.real-root-dev = 0
kernel.cap-bound = -257
kernel.tainted = 0
kernel.core_pattern = core
kernel.core_uses_pid = 1
kernel.print-fatal-signals = 0
kernel.exec-shield = 1
kernel.panic = 0
kernel.domainname = (none)
kernel.hostname = lbe1.istreamlive.net
kernel.version = #1 SMP Wed Jan 5 17:52:25 EST 2011
kernel.osrelease = 2.6.18-194.32.1.el5
kernel.ostype = Linux
kernel.sched_interactive = 2
Thanks,
Roger.
-----Original Message-----
From: Malcolm Turnbull
Sent: Thursday, February 17, 2011 1:13 AM
To: LinuxVirtualServer.org users mailing list.
Subject: Re: [lvs-users] Could LVS/DR cause a bottleneck with media servers?
On 16 February 2011 02:56, Roger Littin <roger@xxxxxxxxxxxxx> wrote:
> Hi,
>
> I have a set up using ldirectord in direct routing to 30 media servers.
>
> The director is a dell R210 4 core with 8GB memory and has 2 1Gbps network
> connections, 1 public & 1 private.
>
> Each real server is a Dell R610 16 core with 12GB memory and has the same
> network connections as above.
>
> All the servers are running centos 5.
>
> According to the data center, these servers are connected to pairs of 40Gb
> switches and there is ample capacity.
>
> What happens is when the connections per real server get to around 1000 –
> 1200 concurrent connections, the bandwidth outgoing per server wont go
> above about 250Mbps which relates to about 7.5Gbps across all servers. At
> that time is when the complaints start coming in about stream problems.
>
> I guess the question is could the director somehow be limiting the
> throughput on the real servers or is the dc not telling the truth about?
> The bandwidth going through the nics on the director is around 50 – 75Mbps
> in on the public nic and out on the private nic to the real servers.
>
> Before we started using lvs, we had 10 servers running with round robin
> dns and these would easily handle 900Mbps each at the same time.
>
> lidrectord config file.
>
> # Global Directives
> checktimeout=10
> checkinterval=5
> #fallback=127.0.0.1:80
> autoreload=yes
> callback="/etc/ha.d/syncsettings.sh"
> logfile="/var/log/ldirectord.log"
> #logfile="local0"
> #emailalert="admin@xxxxx"
> #emailalertfreq=3600
> #emailalertstatus=all
> quiescent=no
>
> virtual=147
> real=172.31.214.12 gate 100
> real=172.31.214.13 gate 100
> real=172.31.214.14 gate 100
> real=172.31.214.15 gate 100
> real=172.31.214.16 gate 100
> real=172.31.214.17 gate 100
> real=172.31.214.18 gate 100
> real=172.31.214.19 gate 100
> real=172.31.214.21 gate 100
> real=172.31.214.22 gate 100
> real=172.31.214.23 gate 100
> real=172.31.214.24 gate 100
> real=172.31.214.25 gate 100
> real=172.31.214.26 gate 100
> real=172.31.214.28 gate 100
> real=172.31.214.29 gate 100
> real=172.31.214.30 gate 100
> real=172.31.214.31 gate 100
> real=172.31.214.32 gate 100
> real=172.31.214.33 gate 100
> real=172.31.214.34 gate 100
> real=172.31.214.35 gate 100
> real=172.31.214.36 gate 100
> real=172.31.214.37 gate 100
> real=172.31.214.38 gate 100
> real=172.31.214.39 gate 100
> real=172.31.214.40 gate 100
> real=172.31.214.41 gate 100
> real=172.31.214.42 gate 100
> scheduler=wlc
> protocol=fwm
> persistent=60
> netmask=255.255.255.255
> service=http
> checkport=1935
> request="/"
> receive="Wowza Media Server 2"
>
> iptables
>
> *mangle
> :PREROUTING ACCEPT [438:421747]
> :INPUT ACCEPT [438:421747]
> :FORWARD ACCEPT [0:0]
> :OUTPUT ACCEPT [95:14749]
> :POSTROUTING ACCEPT [122:21354]
> -A PREROUTING –d *.*.*.147 -p tcp -m tcp --dport 80 -j MARK --set-mark
> 0x93
> -A PREROUTING -d *.*.*.147 -p tcp -m tcp --dport 443 -j MARK --set-mark
> 0x93
> -A PREROUTING -d *.*.*.147 -p tcp -m tcp --dport 554 -j MARK --set-mark
> 0x93
> -A PREROUTING -d *.*.*.147 -p tcp -m tcp --dport 1935 -j MARK --set-mark
> 0x93
> -A PREROUTING -d *.*.*.147 -p udp -m udp --dport 6970:9999 -j
> MARK --set-mark 0x93
> COMMIT
>
> the main port that is used is 1935.
>
> Thanks in advance.
>
> Regards,
>
> Roger.
> _______________________________________________
> Please read the documentation before posting - it's available at:
> http://www.linuxvirtualserver.org/
>
> LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
> Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
> or go to http://lists.graemef.net/mailman/listinfo/lvs-users
Roger,
My gut would say that your switch is getting saturated (not the director).
I wonder if you could get some of the servers to reply to a different
switch (local subnet) but still through the director i.e.
prove the director can can more load if the outgoing traffic is
through a different switch.
Also CPU load on the director is a pretty good indicator of stress.
We've had customers doing similar kind of load (but every setup is
different).
--
Regards,
Malcolm Turnbull.
Loadbalancer.org Ltd.
Phone: +44 (0)870 443 8779
http://www.loadbalancer.org/
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
_______________________________________________
Please read the documentation before posting - it's available at:
http://www.linuxvirtualserver.org/
LinuxVirtualServer.org mailing list - lvs-users@xxxxxxxxxxxxxxxxxxxxxx
Send requests to lvs-users-request@xxxxxxxxxxxxxxxxxxxxxx
or go to http://lists.graemef.net/mailman/listinfo/lvs-users
|
