In our latest weekly column devoted to Guild Wars 2, we take a look at the way that ArenaNet works to fix the things that are of concern to players and why they make the choices they do. Check out our thoughts before leaving your ideas in the comments.

I’m very happy that ArenaNet has been talking about tackling the issue of culling.  This issue has caused me more than a handful of deaths, both in PvP and PvE. There’s nothing more aggravating than running around behind enemy lines, trying to get the jump on your foe.  Just when you think your sneaking skills have carried over from Skyrim, BAM!  You’re being attacked by several enemies you can’t even see!  This issue is often exploited and there’s not much you can do about it.  Well, if you want I guess you could exploit it yourself. 

Read more of David North's Guild Wars 2: Fixin' Things.

I’m just a little weird.

Guild missions opening  new content sounds awsome!

FoTM lower levels taping with max level players is an awsome idea and welcome it!- not so happy with disabeling 'rez points' while in combat...:-/

revampin Orr -i welcome it...the zone shouldve of changed once you defeated Z.  imo. making it more event and hub friendly...just felt unfinished. although its an undead zone-once you finished your story-things should of phased changed...

I like the individual wvwvw things beign done...

and charr/engineers do need a bit of lovin'

Originally posted by adam_nox
Culling isn't a graphics loading issue.  No one sees the invisible players, because it's a networking ans sync issue.  The fix talked about in this article makes me think they have no clue what's even going on.

Guess they should hire you, to teach them about the game they made :)

I tried pvp for the first and probably last time yesterday!

I got to the enemy location, only saw 2 other players, which I killed, then out of nowhere, I was laying on the ground unable to do anything except die, because there were around a dozen other players all hammering me.

If this is pvp, then thanks but no thanks!

Originally posted by KingofHartz
Culling is one of those things I had not defined before this article. I saw it happening, I determined that it was part of the game. It was something I did not apreciate and it really turned me off. Perhaps fixing this will get me to play more, but you know what they say, once bitten twice shy.

Culling is a problem in a lot of newer mmos due to the strain of loading a lot of players and higher resolution textures.  Rift has this problem and handled it by turning pcs and npcs into glowing green cylinders until they load.  Tera seems to handle it by loading 2d grey outlines (much nicer looking than Rift's solution).

Another solution is to use low rez rendering and make the game look like crap, but that isn't a very popular solution either.

You can see the other adverse affect to no culling or placeholding in games that load it all at once like LotRO where you can get freeze frame lag.  It's possible for a zergball to freeze an area up so they can attack before other players clients can respond.

Loading outlines or placeholders isn't very helpful in pvp though because not every target is the same (you don't target a heavy mit class like you do a glass cannon).  Anet needs to come up with a better solution.  People that posture and talk like it's an easy fix or that developers are incompetent because they don't have it solved are talking big right out of their asses.  It's a really difficult problem to solve elegantly and one many games struggle with.

Originally posted by adam_nox
Culling isn't a graphics loading issue.  No one sees the invisible players, because it's a networking ans sync issue.  The fix talked about in this article makes me think they have no clue what's even going on.

You can still see spell effects (and be hit by them, etc) and hear character voices.  This suggests to me that it's not a network issue at all.  They just did absolutely nothing to preload quick placeholder graphics, or anything like that.  I think its because they're artistically obsessive about the look and feel of the game, and seem to prefer gameplay suffer, than have any rapid loading graphics that might look kinda lame for a bit, while the real characters and customizations load.

This sort of obsession with graphics over gameplay is yet another thing that's killing the massive, in massively multiplayer.

Originally posted by Alberel
Class balance and bugged/broken traits are still a higher priority IMO... there's never any mention of them and there are STILL so many of them untouched since beta.

They mentioned it in the 'Plans for 2013'' blog / video.

They said that some classes only have one viable build, whereas others have many, and that they wanted to fix it.

Try to keep up.

I'm new to GW2, so I haven't seen the culling issue yet.  From what is described (characters being invisible when they shouldn't be), it could be any of several things:

1)  The client knows where a character is and has everything loaded and ready to draw it, but doesn't think that the character would appear on the screen if it did draw it, so it skips it that frame.  And then does so again the next frame, and the one after, and the one after...

2)  The client knows where a character is and knows that it needs to draw the character, but it doesn't have everything loaded yet, so it can't draw the character.

3)  The client doesn't know that there is a character there that it needs to draw, so it doesn't.

-----

More generally, there is an enormous amount of fakery involved in 3D graphics.  If you tell the video card to draw everything that is kind of near you, you'll get terrible performance.  It's a bunch of extra work for the processor to get the uniforms ready to draw something, send it to the video card, send the rendering command, have the video card go through several pipeline stages processing data (well, Guild Wars 2 uses the very old DirectX 9.0c, so it necessarily skips several of the early pipeline stages), and then systematically have every single primitive either get destroyed by clipping or have all of its pixels produced by rasterization end up being covered up by something else.

A huge part of optimizing graphics engines is that, as soon as you can figure out that drawing some object will not affect a frame, you scrap it and move on to the next.  This could be because it's behind the camera, way off to the side of the camera, behind some other wall so that you can't see it, or whatever.

You can check in a bunch of different places with a bunch of different granularities, too.  You can toss out entire objects CPU side.  You can discard particular primitives in geometry shaders.  The fixed function clipping will usually discard a lot of primitives as being entirely on the wrong side of a clipping plane.  There is a fixed function option for back-face culling to discard some of the triangles.  You can manually discard particular pixels in pixel shaders, such as if you read a texture and get a pixel that should be completely transparent.  And then many pixels will fail a depth buffer test or some other test and get discarded for that.

The earlier that you can figure out that something can be discarded, the more work you can save.  The problem is that if you think it's safe to discard something and it should have appeared, your game can appear horribly broken in a huge variety of ways.  That can easily end up a lot worse than merely one object not appearing; sometimes other things that should have been safe to discard because they were behind it also don't get drawn.

It could be a fundamentally bad algorithm that simply wasn't thought through properly.  Or it could be a single typo in the source code somewhere.

-----

If the problem is slow loading times, then people with a fast SSD should never see the culling glitch.  Do you know if an SSD makes a difference?

The problem could still be taking too long to load things off of an SSD if the problem is that the algorithm to load things was botched.  It could be a case where something gets deleted from memory, but doesn't get flagged to check it again and see if it needs to be loaded again, so it doesn't get loaded again when it ought to.

There could also be issues with the server failing to tell the client that there is a character there.  The server can't tell you where everyone is; it tries to restrict it to only people who are nearby.  But if the algorithm to determine who needs to be told where a character is gets botched, that could cause a culling issue.

-----

The real problem is that there are a huge number of places where the problem could be.  That makes tracking down the bug in order to fix it into a major pain.  If you can reliably reproduce the glitch, then it becomes much easier to trace the source and fix it.  But when you've got Internet activity involved, that makes it hard to reliably reproduce a lot of things.  Does anyone understand the culling glitch well enough that they can reproduce it at will?

Originally posted by Vhaln

Originally posted by adam_nox
Culling isn't a graphics loading issue.  No one sees the invisible players, because it's a networking ans sync issue.  The fix talked about in this article makes me think they have no clue what's even going on.

You can still see spell effects (and be hit by them, etc) and hear character voices.  This suggests to me that it's not a network issue at all.  They just did absolutely nothing to preload quick placeholder graphics, or anything like that.  I think its because they're artistically obsessive about the look and feel of the game, and seem to prefer gameplay suffer, than have any rapid loading graphics that might look kinda lame for a bit, while the real characters and customizations load.

This sort of obsession with graphics over gameplay is yet another thing that's killing the massive, in massively multiplayer.

Preloading place-holders is a second rate fix.  It's not that bad looking in Tera, but absolutely horrid in Rift (green cylinders everywhere?).

I think it's because the issue is really fricken complicated, multi-faceted and doesn't have easy answers for every part of the problem.

@Quizzical - here are a couple posts explaining the problem and their work on the solutions.

https://forum-en.guildwars2.com/forum/pvp/wuvwuv/Update-on-Culling/first#post1137307

This link is in the first post, but if you feel like skipping towards the detailed explanation first, here it is: https://forum-en.guildwars2.com/forum/pvp/wuvwuv/The-real-problem-here-is-invisible-enemies-Give-their-algorithms-time-to-match-servers-properly/page/4#post356817

Originally posted by Torvaldr

Originally posted by Vhaln

Originally posted by adam_nox
Culling isn't a graphics loading issue.  No one sees the invisible players, because it's a networking ans sync issue.  The fix talked about in this article makes me think they have no clue what's even going on.

You can still see spell effects (and be hit by them, etc) and hear character voices.  This suggests to me that it's not a network issue at all.  They just did absolutely nothing to preload quick placeholder graphics, or anything like that.  I think its because they're artistically obsessive about the look and feel of the game, and seem to prefer gameplay suffer, than have any rapid loading graphics that might look kinda lame for a bit, while the real characters and customizations load.

This sort of obsession with graphics over gameplay is yet another thing that's killing the massive, in massively multiplayer.

Preloading place-holders is a second rate fix.  It's not that bad looking in Tera, but absolutely horrid in Rift (green cylinders everywhere?).

I think it's because the issue is really fricken complicated, multi-faceted and doesn't have easy answers for every part of the problem.

 

@Quizzical - here are a couple posts explaining the problem and their work on the solutions.

https://forum-en.guildwars2.com/forum/pvp/wuvwuv/Update-on-Culling/first#post1137307

This link is in the first post, but if you feel like skipping towards the detailed explanation first, here it is: https://forum-en.guildwars2.com/forum/pvp/wuvwuv/The-real-problem-here-is-invisible-enemies-Give-their-algorithms-time-to-match-servers-properly/page/4#post356817

Thanks for the links.  That explains a lot.

It sounds like it's just a bad algorithm and they know it and are trying to figure out what to do about it.  That's unfortunate but manageable if you catch it long before release, but is a nasty problem to discover after a game launches.  Changes that fix the immediate problem could easily break a bunch of things elsewhere in the game.  That's okay if it just means other things to fix in an early build where it doesn't matter so much if it takes a week to notice that it broke something else, but you don't want to risk pushing other game-breaking things live.

In one of your links, he said that faster client hardware doesn't fix the client loading problem.  That I find rather surprising.  Having to load a bunch of textures for each of a bunch of different players all at once and needing them immediately is a big problem if you're dealing with 5400 RPM laptop hard drives, or even relatively faster 7200 RPM desktop hard drives.  But SSDs should be able to grab the data very, very quickly.  Maybe they have to do some complex processing of textures after they load them or something.

There could also be problems with constantly having to load one character into video memory and delete another, but a video card with a ton of video memory should make it so that you can just load everything remotely nearby and leave it there so long as it stays remotely nearby.  Incidentally, deleting texture buffers from video memory is essentially instantaneous, but uploading new textures and generating mipmaps for them most certainly is not.

The usual fix for having to draw too many players is to sort them and draw the nearest ones and simply not draw the more distant ones.  A lot of games have done stuff like that, and not just with players.  You can have a thread that processes characters start by going through the list of characters that the client knows about and check to see which might plausibly appear on the screen (way off to the side or behind the camera definitely won't, so skip it).  If the number of characters to render is over some cap (which you could let the user decide client-side in graphical options), then you sort them and pick the nearest ones to the camera and don't draw the rest that frame.

A quick off-camera cull, then sort what's left, is very fast.  Even in an extreme case with had 500 players in the area that the client was aware of, the running time is probably going to be far under 1 millisecond, even on slower hardware.  You can do that all over again every single frame.

I haven't dealt with network code, but I'd assume that servers could do something analogous.  When there's an event that the server needs to tell players about, sort by which players are nearest to the event, and tell the nearest 20 or 50 or some such.  I could easily see that being an Internet bandwidth problem, but I'm skeptical that it would be a CPU speed problem unless they've got a horribly inefficient algorithm.  If you're sorting distances, you're not going to have many ties, so QuickSort is O(n ln(n)) and runs very fast.  The only operations involved are very fast (add, subtract, multiply, array lookups, set something equal to something else), so you don't get anything slow like transcendentals.  (You avoid taking square roots by sorting the squares of the distances.)  Maybe they're trying to get way too fancy and ask the cilent where the camera is looking or something.