Trending Games | Trove | Elder Scrolls Online | Darkest Dungeon | Guild Wars 2

  Network:  FPSguru RTSguru
Login:  Password:   Remember?  
Show Quick Gamelist Jump to Random Game
Members:2,769,262 Users Online:0
Games:720  Posts:6,186,904
Recent forum postsRSS
Active threads
Cloud view
List all forums
General Forums
Developers Corner General Discussion
Popular Game Forums
Click a status to find game forum
Game Forums
Click a letter to find game forum
A-C
2029 Online 2112: Revolution 2Moons 4Story 8BitMMO 9 Dragons A Mystical Land A Tale in the Desert III A3 ACE Online ARGO Online Aberoth Absolute Force Online Absolute Terror Achaea Adellion Aerrevan Aetolia, the Midnight Age Age of Armor Age of Conan Age of Empires Online Age of Mourning Age of Wulin Age of Wushu Aida Arenas Aika Aion Albion Online Alganon All Points Bulletin (APB) Allods Online Altis Gates Amazing World Anarchy Online Ancients of Fasaria Andromeda 5 Angels Online Anime Trumps Anmynor Anno Online Applo Arcane Hearts Arcane Legends ArchLord ArcheAge Archeblade Archlord X Asda 2 Asda Story Ashen Empires Asheron's Call Asheron's Call 2 Astera Online Astonia III Astro Empires Astro Lords: Oort CLoud Asura Force Atlantica Online Atriarch Aura Kingdom Aurora Blade Auto Assault Avatar Star Battle Dawn Battle Dawn Galaxies Battle for Graxia Battle of 3 Kingdoms Battle of the Immortals Battlecruiser Online Battlestar Galactica Online Battlestar Reloaded Beyond Protocol Black Aftermath Black Desert Black Gold Black Prophecy Black Prophecy Tactics: Nexus Conflict Blacklight Retribution Blade & Soul Blade Hunter Blade Wars Blazing Throne Bless Blitz 1941 Blood and Jade Bloodlines Champions Bounty Bay Online Brain Storm Brawl Busters. Brick-Force Bright Shadow Bullet Run Business Tycoon Online CTRacer Cabal Online Caesary Call of Camelot Call of Gods Call of Thrones Camelot Unchained Canaan Online Cardmon Hero Cartoon Universe CasinoRPG Castle Empire Castlot Celtic Heroes Champions Online Champions of Regnum Chaos Online Chrono Tales Citadel of Sorcery CitiesXL Citizen Zero City of Decay City of Heroes City of Steam City of Transformers City of Villains Civilization Online Clan Lord Clash of Clans Cloud Nine Club Penguin Colony of War Command & Conquer: Tiberium Alliances Company of Heroes Online Conquer Online Conquer Online 3 Continent of the Ninth (C9) Core Blaze Core Exiles Corum Online Craft of Gods Crimecraft Crimelife 2 Cronous Crota II Crusaders of Solaris Cultures Online Cyber Monster 2 Céiron Wars
D-F
D&D Online DC Universe DK Online DOTA DOTA 2 DUST 514 DV8: Exile Dalethaan Dance Groove Online Dark Age of Camelot Dark Ages Dark Legends Dark Orbit Dark Relic: Prelude Dark Solstice Dark and Light DarkEden Online DarkSpace Darkblood Online Darkest Dungeon Darkfall Darkfall: Unholy Wars Darkwind: War on Wheels Das Tal Dawn of Fantasy Dawntide DayZ Dead Earth Dead Frontier Deco Online Deepworld Defiance Deicide Online Dekaron Demons at the Horizon Desert Operations Destiny Diablo 3 Diamonin Digimon Battle Dino Storm Disciple Divergence Divina Divine Souls Dofus Dominus Online Dragon Ball Online Dragon Born Online Dragon Crusade Dragon Empires Dragon Eternity Dragon Nest Dragon Oath Dragon Pals Dragon Raja Dragon's Call Dragon's Call II Dragon's Prophet DragonSky DragonSoul Dragona Dragonica Dragons and Titans Dream of Mirror Online Dreamland Online Dreamlords: The Reawakening Drift City Duels Dungeon Blitz Dungeon Fighter Online Dungeon Overlord Dungeon Party Dungeon Rampage Dungeon Runners Dynastica Dynasty Warriors Online Dynasty of the Magi EIN (Epicus Incognitus) EVE Online Earth Eternal Earth and Beyond Earthrise Eclipse War Ecol Tactics Online Eden Eternal Edge of Space Einherjar - The Viking's Blood Elder Scrolls Online Eldevin Elf Online Elite: Dangerous Embers of Caerus Emil Chronicle Online Empire Empire & State Empire Craft Empire Universe 3 EmpireQuest Empires of Galldon End of Nations Endless Ages Endless Blue Moon Online Endless Online Entropia Universe EpicDuel Erebus: Travia Reborn Eredan Eternal Blade Eternal Lands Eternal Saga Ether Fields Ether Saga Online Eudemons Online EuroGangster EverEmber Online EverQuest Next EverQuest Online Adventures Evernight Everquest Everquest II Evony Exarch Exorace F.E.A.R. Online Face of Mankind Fairyland Online Fall of Rome Fallen Earth Fallen Sword Fallout Online Family Guy Online Fantage Fantasy Earth Zero Fantasy Realm Online Fantasy Tales Online Fantasy Worlds: Rhynn Faunasphere Faxion Online Ferentus Ferion Fiesta Online Final Fantasy XI Final Fantasy XIV: A Realm Reborn Firefall Fists of Fu Florensia Flyff Football Manager Live Football Superstars Force of Arms Forge Forsaken World Fortnite Fortuna Forum for Discussion of Everlight Freaky Creatures Free Realms Freesky Online Freeworld Fung Wan Online Furcadia Fury Fusion Fall
G-L
GalaXseeds Galactic Command Online Game of Thrones: Seven Kingdoms Gameglobe Gate To Heavens Gates of Andaron Gatheryn Gauntlet Gekkeiju Online Ghost Online Ghost Recon Online Gladiatus Glitch Global Agenda Global Soccer Gloria Victis Glory of Gods GoGoRacer Goal Line Blitz Gods and Heroes GodsWar Online Golemizer Golf Star GoonZu Online Graal Kingdoms Granado Espada Online Grand Chase Grand Fantasia Grepolis Grimlands Guild Wars Guild Wars 2 Guild Wars Factions Guild Wars Nightfall H1Z1 Habbo Hotel Hailan Rising HaloSphere2 Haven & Hearth Hawken Hearthstone: Heroes of Warcraft Helbreath Hellgate Hellgate: London Hello Kitty Online Hero Online Hero Zero Hero's Journey Hero: 108 Online HeroSmash Heroes & Generals Heroes in the Sky Heroes of Bestia Heroes of Gaia Heroes of Might and Magic Online Heroes of Thessalonica Heroes of Three Kingdoms Heroes of the Storm Hex Holic Online Hostile Space Hunter Blade Huxley Illutia Illyriad Immortals USA Imperator Imperian Inferno Legend Infestation: Survivor Stories Infinite Crisis Infinity Infinity Iris Online Iron Grip: Marauders Irth Worlds Island Forge Islands of War Istaria: Chronicles of the Gifted Jade Dynasty Jagged Alliance Online Juggernaut Jumpgate Jumpgate Evolution KAL Online Kakele Online Kaos War Karos Online Kartuga Kicks Online King of Kings 3 Kingdom Heroes Kingdom Under Fire II Kingdom of Drakkar Kingory Kings and Legends Kings of the Realm KingsRoad Kitsu Saga Kiwarriors Knight Age Knight Online Knights of Dream City Kothuria Kung Foo! Kunlun Online L.A.W. LEGO Universe La Tale Land of Chaos Online Landmark Lands of Hope: Phoenix Edition LastChaos League of Angels League of Legends - Clash of Fates Legend of Edda: Vengeance Legend of Golden Plume Legend of Katha Legend of Mir 2 Legend of Mir 3 Legendary Champions Lego Minifigures Online Life is Feudal Light of Nova Lime Odyssey Line of Defense Lineage Lineage Eternal: Twilight Resistance Lineage II Linkrealms Loong Online Lord of the Rings Online Lords Online Lost Saga Lucent Heart Lunia Lusternia: Age of Ascension Luvinia World
M-Q
MU Online Mabinogi Maestia: Rise of Keledus MagiKnights Magic Barrage Magic World Online Manga Fighter MapleStory Martial Heroes Marvel Heroes Marvel Super Hero Squad Online Marvel: Avengers Alliance MechWarrior Online Megaten Meridian 59 : Evolution Merlin MetalMercs Metaplace Metin 2 MicroVolts Midkemia Online Might & Magic Heroes: Kingdoms MilMo Minecraft Mini Fighter Minions of Mirth Ministry of War Monato Esprit Monkey King Online Monkey Quest Monster & Me Monster Madness Online MonsterMMORPG Moonlight Online: Tales of Eternal Blood Mordavia Mortal Online Mourning My Lands Myst Online: URU Live Myth Angels Online Myth War Myth War 2 Mytheon Mythic Saga Mythos N.E.O Online NIDA Online Nadirim Naviage: The Power of Capital Navy Field Need for Speed World Nemexia Neo's Land NeoSteam Neocron Nether Neverwinter Nexus: The Kingdom Of The Winds NinjaTrick NosTale Novus Aeterno Oberin Odin Quest Odyssey RPG Ogre Island Omerta 3 Online Boxing Manager Onverse Order & Chaos Online Order of Magic Original Blood Origins Return Origins of Malu Orion's Belt Otherland Forums OverSoul Overkings Oz Online Oz World Pandora Saga Pantheon: Rise of the Fallen Panzar Parabellum Parallel Kingdom Parfait Station Path of Exile Pathfinder Online Perfect World Perpetuum Online Phantasy Star Online 2 Phantasy Star Universe Phoenix Dynasty Online Phylon Pi Story Picaroon Pirate Galaxy Pirate Storm Pirate101 PirateKing Online Pirates of the Burning Sea Pirates of the Caribbean Online Pixie Hollow Planeshift Planet Arkadia Planet Calypso PlanetSide 2 Planetside Planets³ Playboy Manager Pocket Legends Pockie Ninja Pockie Pirates Pockie Saints PoxNora Prime World Prime: Battle for Dominus Priston Tale Priston Tale II Prius Online Project Blackout Project Powder Project Titan Forums Project Wiki Puzzle Pirates Quickhit Football
R-S
R2 Online RAN Online RF Online ROSE Online Rage of 3 Kingdoms Ragnarok Online Ragnarok Online II RaiderZ Rakion Rappelz RappelzSEA Ravenmarch Realm Fighter Realm of the Mad God Realm of the Titans Realms Online Reclamation Red Stone Red War: Edem's Curse Regnum Online Remnant Knights Renaissance Repulse Requiem: Memento Mori Rift RiotZone Rise Rise of Dragonian Era Rise of Empire Rise of the Tycoon Rising of King Risk Your Life Rivality Rockfree Rohan: Blood Feud Role Play Worlds Roll n Rock Roma Victor Romadoria Rosh Online Roto X Rubies of Eventide Ruin Online Rumble Fighter Runes of Magic Runescape Rust Rusty Hearts Ryzom S4 League SAGA SD Gundam Capsule Fighter Online SMITE SUN Sagramore Salem SaySayGirls Scarlet Blade Scions of Fate Seal Online: Evolution Second Life Secret of the Solstice Seed Serenia Fantasy Seven Seas Saga Seven Souls Online Sevencore Shadow of Legend Shadowbane Shadowrun Online Shaiya Shards Online Shattered Galaxy Sho Online Shot Online Shroud of the Avatar SideQuest Siege on Stars Sigonyth: Desert Eternity Silkroad Online Skyblade Skyforge SmashMuck Champions Smoo Online Soldier Front Soul Master Soul Order Online Soul of Guardian Space Heroes Universe Sparta: War of Empires Spellcasters Sphere Spiral Knights Spirit Tales Splash Fighters Squad Wars Star Citizen Star Sonata 2 Star Stable Star Supremacy Star Trek Online Star Trek: Infinite Space Star Wars Galaxies Star Wars: Clone Wars Adventures Star Wars: The Old Republic StarQuest Online Stargate Worlds Starlight Story Starpires State of Decay SteelWar Online Stone Age 2 Stormfall: Age of War Storybricks Stronghold Kingdoms Sudden Attack Supremacy 1914 Supreme Destiny Sword Girls Sword of Destiny: Rise of Aions SwordX Swords of Heavens Swordsman
T-Z
TERA TS Online Tabula Rasa Tactica Online Tales Runner Tales of Fantasy Tales of Pirates Tales of Pirates II Tales of Solaris Talisman Online Tamer Saga Tank Ace Tantra Online Tatsumaki: Land at War Terra Militaris TerraWorld Online Thang Online The 4th Coming The Agency The Aurora World The Black Watchmen The Chronicle The Chronicles of Spellborn The Crew The Division The Hammers End The Legend of Ares The Lost Titans The Matrix Online The Mighty Quest for Epic Loot The Missing Ink The Mummy Online The Myth of Soma The Pride of Taern The Realm Online The Repopulation The Secret World The Sims Online The Strategems The West Theralon There Therian Saga Thrones of Chaos Tibia Tibia Micro Edition Tiger Knight Titan Siege Titans of Time Toontown Online Top Speed Topia Online Torchlight Total Domination Transformers Universe Traveller AR Travia Online Travian Trials of Ascension Tribal Hero Tribal Wars Tribes Universe Trickster Online Trove Troy Online True Fantasy Live Online Turf Battles Twelve Sky Twelve Sky 2 Twilight War Tynon U.B. Funkeys UFO Online URDEAD Online Ultima Forever: Quest for the Avatar Ultima Online Ultima X: Odyssey Ultimate Naruto Ultimate Soccer Boss Uncharted Waters Online Undercover 2: Merc Wars Underlight Unification Wars Universe Online Utopia Valkyrie Sky Vampire Lord Online Vanguard: Saga of Heroes Vanquish Space Vector City Racers Vendetta Online Victory - Age of Racing Vindictus Virtonomics Vis Gladius Visions of Zosimos VoidExpanse Voyage Century Online W.E.L.L. Online WAR (Warhammer Online) WAR2 Glory WYD Global Wakfu War Thunder War of 2012 War of Angels War of Legends War of Mercenaries War of Thrones War of the Immortals WarFlow Waren Story Wargame1942 Warhammer 40,000: Eternal Crusade Warhammer 40K: Dark Millennium Online Warhammer Online: Wrath of Heroes Warkeepers Warrior Epic Wartune WebLords Wild West Online WildStar Wind of Luck WindSlayer 2 Wings of Destiny Wish Wizard101 Wizardry Online Wizards and Champions Wonder King Wonderland Online World Golf Tour World of Battles World of Darkness World of Heroes World of Kung Fu World of Pirates World of Speed World of Tanks World of Tanks Generals World of Warcraft World of Warplanes World of Warships World of the Living Dead WorldAlpha Wurm Online Xenocell Xiah Xsyon Xulu YS Online Yitien ZU Online Zentia Zero Online Zero Online: The Andromeda Crisis Zodiac Online Zombies Ate My Pizza eRepublik

MMORPG.com Discussion Forums

General Discussion

General Discussion 

Hardware  » Inside the second - changing how GPU's are benchmarked

8 posts found
  Zezda

Advanced Member

Joined: 2/27/09
Posts: 703

 
OP  3/27/13 11:24:17 PM#1

Hey everyone

 

I don't usually post much these days on the forums but I've not seen anyone mention this before and It's going to be quite important (if it isn't a big deal already).

Basically the gist of it is that since.. well.. just about ever graphics card performance has often been measured in FPS (Frames Per Second) by the people running the tests. While this is a great thing and can help show the differences between cards it doesn't really address the subjective aspect of gaming. I suppose it mostly started with people noticing 'micro-stutter' with multi-gpu setups but it also happens to a lesser extent for all cards regardless of configuration due to the way things rendered and then displayed.

I fully recommend giving these articles a thorough read and really take in the information provided. The thing that makes me excited about this direction is that if the journalists are going to start measuring performance in this way it gives us two things. First of all we can see the general performance of cards (FPS) and secondly we can see how stable/consistent the cards are. This puts a lot more focus on the software driving the graphics cards themselves and really it will force Nvidia and AMD to make sure that their products overall are better supported and are of a higher quality. The consistency part of that matters a lot more than people give credit; it's why I couldn't play Crysis 2 on three monitors even although my FPS never went below 30. It's why people have probably bought a card thinking it can hit the FPS needed but their in-game experience tells them something is wrong. By making sure the cards are more accurately benchmarked we, as the consumer, are getting a better picture of what we are buying into and at the end of the day that's good for us and the market as it will push the vendors to release better products as we become a bit more informed toward what we are purchasing.

Anyway, here are the articles I mentioned. Anyone who plays games on a PC should be reading this to understand the issue and what is being done about it.

http://techreport.com/review/21516/inside-the-second-a-new-look-at-game-benchmarking - First article

http://techreport.com/review/24553/inside-the-second-with-nvidia-frame-capture-tools - Most recent

I'm sure we all get our information from different websides but the two I have been most pleased with recently have been Anandtech and Techreport. Between both of them they usually cover most things I'm interested.

  Quizzical

Guide

Joined: 12/11/08
Posts: 13154

3/28/13 10:51:53 AM#2

Interesting stuff, but they're wandering way out into the weeds in data that is hard to even define, let alone measure.  The basic problem is, how do you define how long a frame is?  At what point do you say, this is the start of a frame and the time that the frame takes is from this point in this frame to the same point in the next frame?

In my game engine, there are a bunch of different things that you could point to.  Is it the moment that the video card finishes rendering a frame and is ready to send it to the monitor?  Is it when the rendering thread starts the display() function?  How about when it finishes the display() function?  When other threads inform the rendering thread that all of the CPU work is done and stuff just needs to be sent to the processor?  When the rendering thread starts handling the between-frames maintenance such as sending new textures to the video card?  How about when it ends that between-frames maintenance?  When the rendering thread informs another thread that it is done with a frame so the CPU-side work for the next frame can start?

I think that the best answer is probably when the game engine checks the current time to determine how far to increment the state of the game world before drawing the next frame.  The problem is that neither Fraps nor FCAT have any clue about that.  Fraps will tend to be closer to that value, as there's less other stuff between what Fraps can measure and what FCAT can measure.

Meanwhile, that's something that is completely trivial for a game engine to measure and record.  All that you'd need to do is to set up an array to store the times, a counter variable that keeps track of which frame is being timed, and a single line of code in the game engine to copy the frame time that it already has to determine for other purposes into that array.  And then you'd need some code to output the contents of the array to a file or some such when you're done measuring.

Actually, the exact source code that I use to output my current frame rate in my own game engine is:

System.out.println("Frame rate: " + 1000000000l / (timestamp - oldFrameTime)); ////

One line, and not even an excessively complicated line.  The "timestamp" and "oldFrameTime" variables are defined and used elsewhere, for reasons unrelated to computing my frame rate.  The "////" is only there to make it easy to find that particular line by the Eclipse find/replace function, since I comment it out and then enable it again a lot.  You do need to have the console available to read that, though, and it does scroll off pretty fast, making it inappropriate for measuring frame times over the course of a minute rather than a few seconds.

  Zezda

Advanced Member

Joined: 2/27/09
Posts: 703

 
OP  3/28/13 2:41:32 PM#3
Originally posted by Quizzical

Interesting stuff, but they're wandering way out into the weeds in data that is hard to even define, let alone measure.  The basic problem is, how do you define how long a frame is?  At what point do you say, this is the start of a frame and the time that the frame takes is from this point in this frame to the same point in the next frame?

In my game engine, there are a bunch of different things that you could point to.  Is it the moment that the video card finishes rendering a frame and is ready to send it to the monitor?  Is it when the rendering thread starts the display() function?  How about when it finishes the display() function?  When other threads inform the rendering thread that all of the CPU work is done and stuff just needs to be sent to the processor?  When the rendering thread starts handling the between-frames maintenance such as sending new textures to the video card?  How about when it ends that between-frames maintenance?  When the rendering thread informs another thread that it is done with a frame so the CPU-side work for the next frame can start?

I think that the best answer is probably when the game engine checks the current time to determine how far to increment the state of the game world before drawing the next frame.  The problem is that neither Fraps nor FCAT have any clue about that.  Fraps will tend to be closer to that value, as there's less other stuff between what Fraps can measure and what FCAT can measure.

Meanwhile, that's something that is completely trivial for a game engine to measure and record.  All that you'd need to do is to set up an array to store the times, a counter variable that keeps track of which frame is being timed, and a single line of code in the game engine to copy the frame time that it already has to determine for other purposes into that array.  And then you'd need some code to output the contents of the array to a file or some such when you're done measuring.

Actually, the exact source code that I use to output my current frame rate in my own game engine is:

System.out.println("Frame rate: " + 1000000000l / (timestamp - oldFrameTime)); ////

One line, and not even an excessively complicated line.  The "timestamp" and "oldFrameTime" variables are defined and used elsewhere, for reasons unrelated to computing my frame rate.  The "////" is only there to make it easy to find that particular line by the Eclipse find/replace function, since I comment it out and then enable it again a lot.  You do need to have the console available to read that, though, and it does scroll off pretty fast, making it inappropriate for measuring frame times over the course of a minute rather than a few seconds.

With you completely. In an ideal world we would be able to peer into the process at any point along the pipeline but we just can't with the tools available to us.

 

Still, any tool that helps us benchmark what is happening at any point along the line rather than having a single point of reference helps us build up a better picture not just of how the hardware performs but how the drivers and possibly even the game engine itself can influence things. All in all I think it's good that more attention is being paid to what they are measuring and the understanding that a lot can happen in one second.

 

[EDIT] One example, along the same vein as what you were talking about with the games processing time intervals, (If I'm correct) is Skyrim. From what I remember hearing the timing that the engine keeps can get messed up quite easily if you uncap the frame limiter and it can mess up the NPC's and AI in general as well. It certainly doesn't seem impossible that it could perhaps be messing with the engine in a more physical sense as well.

  Cleffy

Elite Member

Joined: 5/09/04
Posts: 5468

3/28/13 5:59:04 PM#4
Using something like VSync helps with microstutter since it controls the flow of frames.  How do you determine the time between frames?  You can always setup a wait system to render a frame every 16ms, and only blitter to the backbuffer once 16ms is reached.  To me designing a game around the experience is more important than reaching benchmark results.  Companies target getting their game running at a certain fps so artificially limiting it with a timer effectively controls a nice even flow of frames.  Then ofcourse also making sure they don't somehow overload the calculations done in any one frame by doing something like delaying the loading of a character a frame or two in order to display the frame ontime.
  Quizzical

Guide

Joined: 12/11/08
Posts: 13154

3/28/13 7:44:33 PM#5
Originally posted by Cleffy
Using something like VSync helps with microstutter since it controls the flow of frames.  How do you determine the time between frames?  You can always setup a wait system to render a frame every 16ms, and only blitter to the backbuffer once 16ms is reached.  To me designing a game around the experience is more important than reaching benchmark results.  Companies target getting their game running at a certain fps so artificially limiting it with a timer effectively controls a nice even flow of frames.  Then ofcourse also making sure they don't somehow overload the calculations done in any one frame by doing something like delaying the loading of a character a frame or two in order to display the frame ontime.

It's not that simple.

http://www.mmorpg.com/discussion2.cfm/thread/374453

  elohssa

Novice Member

Joined: 3/29/13
Posts: 41

3/29/13 12:07:58 PM#6
Originally posted by Zezda

Hey everyone

 

I don't usually post much these days on the forums but I've not seen anyone mention this before and It's going to be quite important (if it isn't a big deal already).

Basically the gist of it is that since.. well.. just about ever graphics card performance has often been measured in FPS (Frames Per Second) by the people running the tests. While this is a great thing and can help show the differences between cards it doesn't really address the subjective aspect of gaming. I suppose it mostly started with people noticing 'micro-stutter' with multi-gpu setups but it also happens to a lesser extent for all cards regardless of configuration due to the way things rendered and then displayed.

I fully recommend giving these articles a thorough read and really take in the information provided. The thing that makes me excited about this direction is that if the journalists are going to start measuring performance in this way it gives us two things. First of all we can see the general performance of cards (FPS) and secondly we can see how stable/consistent the cards are. This puts a lot more focus on the software driving the graphics cards themselves and really it will force Nvidia and AMD to make sure that their products overall are better supported and are of a higher quality. The consistency part of that matters a lot more than people give credit; it's why I couldn't play Crysis 2 on three monitors even although my FPS never went below 30. It's why people have probably bought a card thinking it can hit the FPS needed but their in-game experience tells them something is wrong. By making sure the cards are more accurately benchmarked we, as the consumer, are getting a better picture of what we are buying into and at the end of the day that's good for us and the market as it will push the vendors to release better products as we become a bit more informed toward what we are purchasing.

Anyway, here are the articles I mentioned. Anyone who plays games on a PC should be reading this to understand the issue and what is being done about it.

http://techreport.com/review/21516/inside-the-second-a-new-look-at-game-benchmarking - First article

http://techreport.com/review/24553/inside-the-second-with-nvidia-frame-capture-tools - Most recent

I'm sure we all get our information from different websides but the two I have been most pleased with recently have been Anandtech and Techreport. Between both of them they usually cover most things I'm interested.

Its a bogus testing method developed by nvidia to make their cards outbench AMD.  Thats not to say its worthless data, but there is a huge deal of bias with the tests. 

 

There is nothing really wrong with the old approach of min-max & avg fps.  Its the most simplistic, and the most meaningful to actual gameplay.

  Quizzical

Guide

Joined: 12/11/08
Posts: 13154

3/29/13 12:40:02 PM#7
Originally posted by elohssa

Its a bogus testing method developed by nvidia to make their cards outbench AMD.  Thats not to say its worthless data, but there is a huge deal of bias with the tests. 

There is nothing really wrong with the old approach of min-max & avg fps.  Its the most simplistic, and the most meaningful to actual gameplay.

Actually, there is a lot wrong with the old approach of min, max, and average frame rates.  It's unable to catch spikes in which one frame takes a very long time, and those are hugely disruptive to gameplay.

While Nvidia is likely pushing this approach because they figured out that they should start optimizing drivers for it before AMD did, it is meaningful information.

I think it's interesting that as measured by when frames actually complete, for a single card, both AMD and Nvidia provide very smooth frame rates almost everywhere.  The real problem is that the game engine thinks that some frames take much longer than others.  FRAPS is closer to being able to measure that than Nvidia's tools, but there are still several layers of noise between what FRAPS can measure and what the game engine thinks is going on internally.

How much the video drivers let the rendering thread queue up rendering commands before telling it to stop and wait has a considerable impact on spikes where the game engine thinks that one frame takes much longer.  There are also major issues with when the game engine decides to do extra, non-rendering work, such as buffering new data (most commonly textures) on the video card.  How drivers handle that could easily account for most or even all of the the spikes in the data.

Actually, I wonder if the spikes are caused by how buffering new data changes the amount of time in advance that the video drivers let the rendering thread queue up commands.  Ordinary rendering work such as passing uniforms or rendering commands to the video card or switching textures or vertex arrays will likely leave about the same amount of work in the queue at the end of each frame when the game engine thinks that it's time to start another.  Video drivers probably don't have a good handle on how much time sending new textures and creating mipmaps for them takes compared to ordinary rendering commands.

Recompiling shaders and relinking programs will also give you a huge spike in your frame rates, but that's generally done when the game is launched, or possibly when a user changes certain graphical settings or at loading screens.  (I have no idea how prevalent taking loading screens or graphical setting changes as a time to recompile shaders is, but those are the times when one could do that without it striking me as completely stupid.)  It's very unlikely that that will be done in the middle of a benchmark run.

And there's also the problem that a bad game engine can create unnecessary spikes.  If you finish one frame and want to pass 50 new textures to the video card and create mipmaps for them before starting the next frame, then you're going to get a spike in your frame rates and there's nothing AMD or Nvidia can do about it.  That's fine at loading screens, of course, but if you're going to do that while the player is playing the game, you have to spread it out and not do too much in a single frame.

  Ridelynn

Elite Member

Joined: 12/19/10
Posts: 3314

3/29/13 11:08:35 PM#8

I've been over canned benchmark tools for a long time now.

Once you get past the base specs (clock speed, core count, FLOPS/IOPS/MIPS/whatever, memory bandwidth, etc) the rest is mainly up to driver and architectural decisions, and vary widely game to game.

I like how some companies release a canned demo loop so you can "benchmark" your system, that does help some by eliminating a lot of variables, but that benchmark is just relative to that particular game - it doesn't say too much about your hardware over all, and can only loosely correlate to performance with other software.

The ultimate "benchmark" is how it looks and performs to you - the consumer. I think the best method may just be HardOCP's testing: they take a game and will give you an Apples to Apples comparison on different hardware, but they will also give you a "Best Performance" settings, which can vary widely on different cards, and even on different drivers. They also post FPS graphs over time - not perfect, but far better than just a Min/Max/Average - you can see where the big stalls occur, and how they line up to other hardware on the same software load. And their editors are good about putting in little things they may notice that may not show up on benchmark numbers (microstuddering, artifacting, clipping issues, etc).