CFD processing power

[ReubenG]

I stand to be corrected here, but doesn't running the 3 graphics cards as physics processors require the code to send specific instructions to the GPU and not the main CPUs? Given that most high end CFD / FEA problems are run on multi-processor servers, have the current CFD codes been written to exploit the physics processing capability of GPUs?

Chris, I don't think the intent of earlier posts was to discourage you from trying CFD because you're not learning it in the normal environment (i.e. university studying engineering / physics / maths at a post-grad level). I have some training as an FE analyst, and just learning how to generate models that the solver would run was a time-consuming , frustrating process, even though I had a few PhD students and post-docs to bug when I couldn't decipher the error messages! The point may be that having access to a great PC is less important than access to a good mentor when learning any kind of computational mechanics.

So, if you can convince some one who is already a competent analyst to help when you are stuck, go ahead. If you can't find support, I would find some where else to spend the money....

An aside question to any other CFD / FEA / computational mechanics people - do any of you run your codes in Vista? My personal experience is with Abaqus, which we run on WinXP, or Linux servers for big jobs. My mates in CFD all run Linux.

[Saribro]

I stand to be corrected here, but doesn't running the 3 graphics cards as physics processors require the code to send specific instructions to the GPU and not the main CPUs?

Quite right. There are some physics APIs that can be accelerated on videocards (Havok/AGEIA soon due to nVidia buying them), but these are mostly used in games and have to be coded for (just like 3D graphics has APIs such as Direct3D/OpenGL). Just stuffing the cards in and -saying- they're for physics won't actually make them do any physics-processing :).

[Conceptual]

I stand to be corrected here, but doesn't running the 3 graphics cards as physics processors require the code to send specific instructions to the GPU and not the main CPUs?

Quite right. There are some physics APIs that can be accelerated on videocards (Havok/AGEIA soon due to nVidia buying them), but these are mostly used in games and have to be coded for (just like 3D graphics has APIs such as Direct3D/OpenGL). Just stuffing the cards in and -saying- they're for physics won't actually make them do any physics-processing .

I'm going to do some research on how the CrossFire works, and specifically, how programs use it. I know it is able to do the stand-alone physics processing, and maybe it is a motherboard/chipset driver that automatically diverts the physics usage.

More to come.

Chris

[NDR008]

We got Ansys CFX running on some old SUN machines... it runs faster on my laptop 64bit dual core with 1 GB RAM... depends what you want to use the CFD for and to what detail anyway.

[shawness]

Conceptual: First off, I have to give you huge props for trying this - it's a tricky area and you're a brave man to give it a shot!

The big issue you have to consider with CFD is that it's the ultimate example of "If it's worth doing, it's worth doing properly." If you don't use a mesh of sufficient complexity, you end up with data that isn't just inaccurate, but plain wrong. This is because fluid systems are chaotic - a small change in one area causes big changes elsewhere in the system, so if you don't model an area of complex flow (wheel interior is the best example) with sufficient accuracy, it throws off your results for the rest of the car. This is why we will run the model of the whole car, even if we're just looking at airflow in one specific area, and for something as complex as a current F1 chassis, unless you're using at least 30m cells, you're not going to get worthwhile data. This means you need a lot of horsepower.

When it comes down to hardware, there are actually two separate requirements. The first is doing the CAD processing, the surface meshing and the volume meshing. This requires a really beefy machine to do properly - when I worked in CFD our machines had two dual-core Opterons with 16GB RAM and 1GB Quadro video cards (single - crossfire/SLI isn't supported by anything I'm aware of at a pro level), and it was, at times, still pretty slow going.

Once you're ready to solve, you have a different set of requirements - it's all memory and CPU. Our farm was made up of Core2Duo machines with 4GB of RAM, no video cards, no hard drives except in masters, strung together with regular gigabit ethernet. Core2Quad processors aren't worth bothering with right now because CFD is so dependent on mem-ops, and the FSB on C2Q isn't high enough to support all four cores doing their maximum memory throughput. In fact, they end up fighting over memory bandwidth, so a 2Ghz C2D with 4GB RAM will kick the ass of a 2Ghz C2Q machine with 8GB. For a small CFD farm that's going to produce usable results on (as random examples) a Caterham or Radical in under 24 hours, I'd say you'd need at least six C2D nodes, plus a master.

As far as software is concerned, OpenFOAM is becoming increasingly popular as the solver as it's free and can be tweaked at the code level because it's open source. A number of F1 teams are moving towards using it, or using it already. Operating system on nodes is invariably linux, and XP or linux on the workstations, depending on what you're using. I don't know of anyone who uses Vista.

[Conceptual]

Hmmm.

I found some interesting stuff on the Crossfire system.
http://ati.amd.com/technology/crossfire ... ssFire.pdf

I thank everyone for their contribution to this thread, but I think the questions at this point have been answered. Until the CFD software is written to take advantage of Crossfire, it is never going to be anything but a concept. Realistically, there isnt a single machine that I can build at home that is going to have the processing power that is necessary to do even basic shapes, and in the end I guess, the money isn't worth being spent in this fashion. All I need is a Media Server that runs AutoCAD 2007, and has good enough output to run my 1080i projector in my living room.

Thanks again guys for exploring this subject with me, and I hope that the CFD guys read that PDF on the Crossfire, because I think a few of you openFOAM guys may be asking for Crossfire support on their user forums.

Enjoy,

Chris

[Saribro]

The problem isn't just porting CFD-apps (or whatever) to work on GPUs. These kind of high-end math/simulation/... apps need double precision, IEEE compliant floating point processing, which, if I'm not mistaken, not a single current GPU supports. The hardware isn't quite there yet to be used for just anything, and that's normal, they -are- graphics chips :D.

[Conceptual]

The problem isn't just porting CFD-apps (or whatever) to work on GPUs. These kind of high-end math/simulation/... apps need double precision, IEEE compliant floating point processing, which, if I'm not mistaken, not a single current GPU supports. The hardware isn't quite there yet to be used for just anything, and that's normal, they -are- graphics chips .

If you read that white paper (http://ati.amd.com/technology/crossfire ... ssFire.pdf) you will see that they specifically mention fluid dynamics as what it does incredibly well, by their standards.

I was pretty impressed with what I read about it, and as I looked at the higher end cards, they actually have Dual-Core GPU's on the card which would allow for 8 128bit cores with 4GB of DDR4 RAM configuration, and a very fast pipeline (PCI-e x16) for bandwidth needs.

I still think it would work, I just don't think that I am in a position to make it work.

Maybe some of the professional CFD guys like SlimJim can get his company to build a <$4000 prototype to see if it could be done?

Thanks again,

Chris

[shawness]

If you read that white paper (http://ati.amd.com/technology/crossfire ... ssFire.pdf) you will see that they specifically mention fluid dynamics as what it does incredibly well, by their standards.

I was pretty impressed with what I read about it, and as I looked at the higher end cards, they actually have Dual-Core GPU's on the card which would allow for 8 128bit cores with 4GB of DDR4 RAM configuration, and a very fast pipeline (PCI-e x16) for bandwidth needs.

I still think it would work, I just don't think that I am in a position to make it work.

Maybe some of the professional CFD guys like SlimJim can get his company to build a <$4000 prototype to see if it could be done?

Thanks again,

Chris

I think using video cards for CFD processing is certainly possbile, but the reason it hasn't been some is simply return on investment. Our CFD nodes - Core2Duo chip, 4GB RAM, motherboard, PSU, cheapo 2U case, etc - came in circa £400 each. It wouldn't have made economic sense for us because it'd require more space (4u vs 2u if we wanted to install multiple modern video cards), more powerful (and expensive) PSU's, and pretty significant coding work for an uncertain return.

As you say, it might be that adding in video card processors could increase processing power - but on the other hand, it might not - we just don't know! The CPU's would still have to govern the work, and I'm not sure what the accuracy of GPU's that are designed mainly for rendering games is going to be. The white paper does mention fluids, but that's best-guess fluid simulation in games, where accuracy isn't a factor.

Ultimately, I guess it's just going to come down to someone with the money to blow on the research being willing to try coding it up and validating it!

[Saribro]

If you read that white paper (http://ati.amd.com/technology/crossfire ... ssFire.pdf) you will see that they specifically mention fluid dynamics as what it does incredibly well, by their standards.

It's not a white paper, it's advertisement :).
Also, notice how they only mention "Game Computing" and "Game Physics Processing".
GPUs can do a lot more than they used to, but not everything yet.

[Conceptual]

If you read that white paper (http://ati.amd.com/technology/crossfire ... ssFire.pdf) you will see that they specifically mention fluid dynamics as what it does incredibly well, by their standards.

I was pretty impressed with what I read about it, and as I looked at the higher end cards, they actually have Dual-Core GPU's on the card which would allow for 8 128bit cores with 4GB of DDR4 RAM configuration, and a very fast pipeline (PCI-e x16) for bandwidth needs.

I still think it would work, I just don't think that I am in a position to make it work.

Maybe some of the professional CFD guys like SlimJim can get his company to build a <$4000 prototype to see if it could be done?

Thanks again,

Chris

I think using video cards for CFD processing is certainly possbile, but the reason it hasn't been some is simply return on investment. Our CFD nodes - Core2Duo chip, 4GB RAM, motherboard, PSU, cheapo 2U case, etc - came in circa £400 each. It wouldn't have made economic sense for us because it'd require more space (4u vs 2u if we wanted to install multiple modern video cards), more powerful (and expensive) PSU's, and pretty significant coding work for an uncertain return.

As you say, it might be that adding in video card processors could increase processing power - but on the other hand, it might not - we just don't know! The CPU's would still have to govern the work, and I'm not sure what the accuracy of GPU's that are designed mainly for rendering games is going to be. The white paper does mention fluids, but that's best-guess fluid simulation in games, where accuracy isn't a factor.

Ultimately, I guess it's just going to come down to someone with the money to blow on the research being willing to try coding it up and validating it!

Haven't played any Online FPS yet, have you? Accuracy is EVERYTHING to those people, seriously!

Anyways, I think that a 550w PS comes in for under $20 shipped, and 4 cards at $80 each with a 4u rack that has the possibility of removing the need for 4 1u racks comes at a pretty good bargain in my eyes, especially if you are space limited.

Anyways, I think it might be VERY interesting to see if the OpenFOAM people can make an option to send the processing to the CrossFireX cards, and then do a time/accuracy comparison of the CrossFire and the conventional cluster processing.

I think that the answer may be very surprising... Who could I ask to get this comparison tested?

Chris

[slimjim8201]

Game physics and large scale simulation matrix solving are two completely different things. There is so much more going on within a CFD solver than with simple game physics. Even the most high end games employ relatively "simple" game physics.

[SoundMan]

Even the most high end games employ relatively "simple" game physics.

yep, that’s true, even the most advanced racing game simulations are nowhere near to reality, they really should be called ‘Approximations’, not ‘Simulations’.

[Carlos]

Games use a little CFD modeling water, fog effects and that sort of thing with a PPU ( Physics Processing Unit ) but this single chip ( X4 ) would not have the computational horsepower to do the kind of CFD we discuss on F1T IMO. Also most memory on a graphics card is devoted to visuals. How many video cards even have a PPU chip to handle physics?

PPU development has only gotten uderway in the last few years. I don't think PPU chips are on too many video cards yet. From what I can find out, there is a PPU card available. Here's a review. This is pretty new stuff that's just starting to be used.
http://www.digit-life.com/articles2/vid ... eview.html
http://www.devhardware.com/c/a/Video-Ca ... sing-Unit/

What about a compromise between CFD and graphics cards. This is a very cool game.

A free download of Plasma Pong that combines gameplay and a few CFD equations

It's not the sort of thing that's too useful to do CFD on a Formula 1 front wing. Future potential maybe. Definitely fun though.


http://plasma-pong.en.softonic.com/

[Conceptual]

Games use a little CFD modeling water, fog effects and that sort of thing with a PPU ( Physics Processing Unit ) but this single chip ( X4 ) would not have the computational horsepower to do the kind of CFD we discuss on F1T IMO. Also most memory on a graphics card is devoted to visuals. How many video cards even have a PPU chip to handle physics?

PPU development has only gotten uderway in the last few years. I don't think PPU chips are on too many video cards yet. From what I can find out, there is a PPU card available. Here's a review. This is pretty new stuff that's just starting to be used.
http://www.digit-life.com/articles2/vid ... eview.html
http://www.devhardware.com/c/a/Video-Ca ... sing-Unit/

What about a compromise between CFD and graphics cards. This is a very cool game.

A free download of Plasma Pong that combines gameplay and a few CFD equations

It's not the sort of thing that's too useful to do CFD on a Formula 1 front wing. Future potential maybe. Definitely fun though.


http://plasma-pong.en.softonic.com/

Carlos,

Did you read the Whitepaper linked a few posts up? It explains most of that in there. And that is exactly what does. I think you may have just validated my point more than I could myself. And BTW, those articles are 3 years old, but the last page of the article is dead on with the CrossFireX, It is a PPU that is a highly specialized chip that can handle physics better than any CPU.

Thanks!

Chris