Setting up an efficient CFD computer

[Mohammad]

What can I say... a little knowledge is a very bad thing. If you had much understanding of the subject area, you'd be embarassed at that post.

You cannot fully utilise those GPUs (ATI) for complex CFD due to numerous archiectural reasons.

Same with the Nvidia lot.

The Intel Larabee may be somewhat better, but I'm not holding my breath.

Instead of the hundred-fold (or more) increase over conventional applications run on the CPU, even on simplified algorithms with current GPU archs your talking less than 10x improvment, often less than 4x improvement (rel. Clovertown).

Nehalem has already decimated that 4x improvement.

Hi,

I was reading the conversation between you and Conceptual as I am interested in computer hardware in general. I have never done CFD so I am wondering: what is the nature of algorithms used in CFD? are they data-parallel algorithms akin to those used in graphics rendering? or are they characterized by heavy dependencies among their results?

Obviously, if the algorithms are data-parallel, then there is nothing preventing the use of the configuration that Conceptual mentioned, as modern GPUs are relatively cheap and they have tremendous fine-grained parallel processing power.

Of course, just because a particular piece of hardware has thousands of parallel processing units doesn't mean that any problem can be sped up by that amount. It depends on the nature of the problem one is trying to solve. That is why regular CPUs are still among us.

Anyway, regarding the use of supercomputers in F1, I understand that BMW has the largest non-military supercomputer in Europe, and the cool thing about it is that it is made of off-the-shelf Intel Xeons.

[kilcoo316]

I was reading the conversation between you and Conceptual as I am interested in computer hardware in general. I have never done CFD so I am wondering: what is the nature of algorithms used in CFD? are they data-parallel algorithms akin to those used in graphics rendering? or are they characterized by heavy dependencies among their results?

They can be run in heavily parallel scenarios (on large MIMD instruction set CPUs), with excellent scaling.

However, the pipelines within each useable processor of GP-GPUs does not lend itself to the operation very well at the moment.

Lets take the 4870 1GB for instance

There are 800 stream processors... but only 10 SIMD cores (10x80=800) [which are the ones needed] - and even then, the instruction set available within each SIMD core is limited so you have to do multiple passes to process your data.

There are also fetch issues with data - the local cache is extremely small (16 KB IIRC), and worse, it is not a common cache to all SIMDs - so more clocks are wasted running data requests (the bandwidth does mean there is no choking of the memory at least).


Of course, for simple solutions, like euler flows great results can be obtained:

http://www.eng.cam.ac.uk/~gp10006/resea ... _DRAFT.pdf


For fully viscous flows, incorporating turbulence models performance drops right off.

[kilcoo316]

Since this is now a seperate topic, I'll add a little more on general CPUs - and what to get, but I expect anyone that uses CFD already knows this.


If you've a choice between a dual socket Intel Harpertown/Clovertown and dual socket AMD Shanghai... pick Shangai, every time.

The two Intels stall out at 4 CPUs and won't scale much above that due to FSB saturation.


If you've enough money to get an Intel Gainestown - get it, forget about AMD.


Oh, and if your running in Windows XP, always set your processor affinity for your multiple processes, particularly on the old Intel Conroe based arch - cache flushes due to windows changing the tasks across CPUs will be adding anything between 5-15% to your job-time.

[Mohammad]

Cool, thank you for the reply and the paper link.

Regarding the CPUs, I got a little bit lost with all the core code names so I looked them up on Wikipedia. I now see your rationale: Harpertown/Clovertown use the FSB for communication whilst AMDs use their own interconnect (I think this has been the case since the original Phenoms). Gainestown does away with the FSB interconnect and uses a specialized link so the FSB is freed to do other tasks.

On my personal desktop I am currently running a Core 2 Quad 9450 with 4GB DDR2 1066 RAM. Quite a respectable performance (In fact, I called my machine "Quadro" in honor of the processor ).

The only thing that I wish was available is a processor with finer-grained processing power as I run simulations (not CFD) that contain a healthy amount of data-parallel operations. I looked into GPGPUs (ATi Stream) but I really felt uncomfortable with the whole development framework and the small stack size. The fact that a bug in a GPGPU program can cause a blue screen and prompt a restart of the machine doesn't help either.

I think a promising platform for simulations could be FPGAs as they offer nice hardware acceleration capabilities and there is actually some research being done on automatically translating code in functional languages to hardware blocks. Whilst it may not be possible to place all the calculations on the FPGA, I think it would prove to be a reasonable solution that offers the benefits of off-the-shelf components and the performance of specialized co-processors.

[pipex]

To add a little bit to the topic, SGI explored some years ago using FPGAs and reconfigurable computing to general purpose applications with a product called RASC. I don't know if this product continues being sold or not.

The problem is always the complexity created in trying to port parts of an algorithm to run in the FPGA, due to the limitations of the hardware. And it is really complex to create a good working prototype hardware description without high level tools. FPGAs don't know about floating point or simple mathematical operations, it's raw hardware.

Some information about it here:
http://www.sgi.com/pdfs/3721.pdf

[kilcoo316]

Never really heard of FPGAs... a quick look around found they aren't so well suited to 64-bit precision applications:

http://drops.dagstuhl.de/opus/volltexte ... er.732.pdf

The strength of FPGAs seems to lie off standard precision (i.e. non 32 or 64 bit) calculations.

[kilcoo316]

On my personal desktop I am currently running a Core 2 Quad 9450 with 4GB DDR2 1066 RAM. Quite a respectable performance (In fact, I called my machine "Quadro" in honor of the processor ).

If you have a CFD package, if you want, you can try the following (for an identical simuation):

2 core run on CPU0 and CPU1

2 core run on CPU0/1 and CPU2/3

4 core run on all CPUs


You'll be surprised at the differences in scaling - and the vast majority of it is due to the memory architecture.

[The FOZ]

You'll be surprised at the differences in scaling - and the vast majority of it is due to the memory architecture.

So with the Core i7, integrated memory controller and triple-channel memory is going to improve things for me a weeee bit?