[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Using state and routing inbound traffic

On Friday, Aug 5, 2005, at 15:18 US/Pacific, Karl O. Pinc wrote:

I was also hoping to get some comment from somebody who'd tried queueing inbound traffic from a WAN link using a 2 port box to see how successful they were in improving perceived bandwidth.

I have. It was a couple years ago and that configuration is no longer in use, so most of this is from memory...

The goal was to make Quake 3 playable in the face of web and P2P traffic. Quake 3 is an online First-Person Shooter game that uses UDP as a transport, and is sensitive to both latency and dropped packets. It includes a small onscreen network graph that reflects latency, jitter, lost packets, and the status of the client side game prediction engine in an easy-to-read format. It makes a decent network diagnostic tool, actually...

The servers I played on had code to compensate for latency, so the most important thing for my setup was getting _consistent_ latency. Lower latency was preferable but not required. On an otherwise unused 320k SDSL, latency was around 90ms.

altq on $int_if priq bandwidth 200Kb queue { q3_def, q3_pri }
queue q3_def priority 0 qlimit 1 priq(default)
queue q3_pri priority 7 qlimit 5

pass out on $int_if inet proto udp from any port 27959 >< 27964 to keep state queue q3_pri
pass in on $int_if inet proto udp from to any port 27959 >< 27964 keep state queue q3_pri

IIRC under full load the latency with this configuration was fairly steady around 200ms, and I could play decent games.

Note that the bandwidth limit is 2/3 line capacity. I always disabled this queue setup when I was done with the game, as keeping it active was counterproductive. I didn't spend much time tuning this limit though, so I don't know how much higher it could have gone and remained acceptable.

The low queue limits are important. For Quake 3, dropped packets are preferable to delayed ones; game engine interpolation can recover drops, but inconsistent latency throws everything off. The limit of 5 is just to keep something in the queue for the priority engine to use. For the other streams, I wanted them to drop quickly to more closely reflect a bandwidth limit. Delaying them would give TCP the impression that the link was simply high-latency, and the result would be bursty traffic that leads to inconsistent latency for every other stream on the pipe.

The problem with queue-based prioritization is that in order to have a choice of things to prioritize, everything must be queued. The mere act of queueing introduces latency, and for things like TCP, latency decouples simulated bandwidth from real line capacity.

Stock altq (before the pf merge) contained a simple token bucket rate-limiter for inbound flows. Were code written to combine it with pf's state logic, it could be used to force bandwidth of other streams below a certain limit only when states needing reserve bandwidth were active. That assumes fixed reserve sizes though. A bonus would be using smoothed historical flow statistics to predictively limit bandwidth of all flows dynamically. This would avoid the latency problem inherent in priority queueing. Implementing such a thing efficiently would be an interesting project.

I threatened to do some work on the token bucket once, but ran out of time to work on such a project. Sorry :(