CFD - Computational Fluid Dynamics, Motorsport, Formula 1

[connollyg]

Out of interest Slimjim8201, what is the maximum cell size model you can produce with CFDesign? Has it got a 64 bit platform yet?

Good question. We've had an X64 version for years. The largest I've tested on a PC was about 15 million elements. The PC had 8 GB of ram and a single, dual core 2.4 ghz xeon processor. I believe one of the young bucks tried something with even more than that a few months back. Theoretically, our limit is the operating system's limit, which in the case of X64, is a LOT. The problem then is the solve times. Almost all of our users are operating on a single node. That is they are designing and simulating on a single PC. Not a Cray or some crazy UNIX cluster. Our distributed computing capabilities will take a huge leap with the next version, but even then, most of our users will still be running on a single node.

slimjim, that seems to go against the current trend, there are some F1 teams using multi-core SMP machines (eg McLaren with 500+ core Itanium system), but most are going to clusters, with 1000 cores being a common number. Most are going for the 1U dual socket nodes giving 4 cores (dual-core) or 8 cores (quad-core).

CPU clock speed is important, however the real issue is memory per core and/or memory bandwidth. Some CFD codes work better on SMP machines, others on clusters and it can be as fine-grained as the different solvers for a single application may perform better on the different architectures.

SMP machines like the SGI Altix 4700 (Itanium based) allow extremely large amounts of memory per core (Theorectical limit is 128TB of RAM per machine with upto 1K cores per node)

Clusters using Intel or AMD are typically limited to 32GB per node which are mainly dual socket, ie max of 8 cores (today)

But memory bandwidth is an issue as well, thats why some of the CFD codes only run on 1 core per socket on Xeon (each dual core shares a memory bus) (each quad core is basically 2 * Dual-core on the same piece of silicon).

AMD's Barcelona chip will (now its finally out) out-perform Intels current offering, until Intel releases its Nehalem chip later this year, then it might come down to who has the fastest clock.

G

[slimjim8201]

Guys,

What makes you think 30-50 Million is good enough for F1? my understanding is that the top teams are talking 100 Million and above, and i know there has been talk of billion cell models!

G

As long as the computers get bigger, better, faster, the simulations will continue to grow. Judging by the complexity of modern cars and all of their aero bits and pieces, it wouldn't at all surprise me if teams are nearing 1 billion cells for some simulations

One of our developers did a calculation on what it would take to simulate a modern jet engine. With a mesh density of required size for a full system calculation, and a network computing grid that could spit out a solution in 8 hours, the costs in power ALONE to run the simulation would be like $15,000. ONE simulation. That doesn't even include the hardware investment and overhead to run the network.

I know the F1 teams are throwing ridiculous amounts of money at CFD these days. I'd be interested in seeing how much each team spends in CFD testing and how it's risen over the past 5 years. I'm sure it's exponential.

With the banning of winglets shortly, mesh counts may not rise as fast as they have been over the past few years.

[slimjim8201]

Out of interest Slimjim8201, what is the maximum cell size model you can produce with CFDesign? Has it got a 64 bit platform yet?

Good question. We've had an X64 version for years. The largest I've tested on a PC was about 15 million elements. The PC had 8 GB of ram and a single, dual core 2.4 ghz xeon processor. I believe one of the young bucks tried something with even more than that a few months back. Theoretically, our limit is the operating system's limit, which in the case of X64, is a LOT. The problem then is the solve times. Almost all of our users are operating on a single node. That is they are designing and simulating on a single PC. Not a Cray or some crazy UNIX cluster. Our distributed computing capabilities will take a huge leap with the next version, but even then, most of our users will still be running on a single node.

slimjim, that seems to go against the current trend, there are some F1 teams using multi-core SMP machines (eg McLaren with 500+ core Itanium system), but most are going to clusters, with 1000 cores being a common number. Most are going for the 1U dual socket nodes giving 4 cores (dual-core) or 8 cores (quad-core).

CPU clock speed is important, however the real issue is memory per core and/or memory bandwidth. Some CFD codes work better on SMP machines, others on clusters and it can be as fine-grained as the different solvers for a single application may perform better on the different architectures.

SMP machines like the SGI Altix 4700 (Itanium based) allow extremely large amounts of memory per core (Theorectical limit is 128TB of RAM per machine with upto 1K cores per node)

Clusters using Intel or AMD are typically limited to 32GB per node which are mainly dual socket, ie max of 8 cores (today)

But memory bandwidth is an issue as well, thats why some of the CFD codes only run on 1 core per socket on Xeon (each dual core shares a memory bus) (each quad core is basically 2 * Dual-core on the same piece of silicon).

AMD's Barcelona chip will (now its finally out) out-perform Intels current offering, until Intel releases its Nehalem chip later this year, then it might come down to who has the fastest clock.

G

Thanks for the info.

It's very difficult for 99% our user base to financially justify the purchase of such large scale computing grid. It should be interesting to see how the next gen processors and memory handling affects performance. Interesting as in, I cannot wait until I can upgrade my machines...

[connollyg]

Guys,

What makes you think 30-50 Million is good enough for F1? my understanding is that the top teams are talking 100 Million and above, and i know there has been talk of billion cell models!

G

As long as the computers get bigger, better, faster, the simulations will continue to grow. Judging by the complexity of modern cars and all of their aero bits and pieces, it wouldn't at all surprise me if teams are nearing 1 billion cells for some simulations

One of our developers did a calculation on what it would take to simulate a modern jet engine. With a mesh density of required size for a full system calculation, and a network computing grid that could spit out a solution in 8 hours, the costs in power ALONE to run the simulation would be like $15,000. ONE simulation. That doesn't even include the hardware investment and overhead to run the network.

I know the F1 teams are throwing ridiculous amounts of money at CFD these days. I'd be interested in seeing how much each team spends in CFD testing and how it's risen over the past 5 years. I'm sure it's exponential.

With the banning of winglets shortly, mesh counts may not rise as fast as they have been over the past few years.

have you seen this article? http://www.tgdaily.com/content/view/36905/135/ which includes this table. (NB this table is known to contain some errors)

Formula 1 Supercomputer Championship 2008 (by TFLOPS):

ING Renault F1 Team: Appro Xtreme-X2, 1024 sockets, 4096 cores (AMD QC Opteron)
Scuderia Ferrari Marlboro: Acer/IBM/Racksaver, 1000+ processor sockets (upgrading to QC Opteron)
BMW Sauber F1 Team: Dell, Albert2, 512 sockets, 2048 cores (Intel Xeon)
Vodafone McLaren Mercedes: Silicon Graphics Altix, 512 Sockets, 1024 cores (Intel Itanium 2)
Red Bull Renault: IBM, 512 sockets, 1024 core (upgrading to AMD QC Opteron)
Panasonic Toyota F1 Team: Fujitsu, 320 Sockets, 640 cores (Intel Itanium 2)
AT&T WilliamsF1 Toyota: Lenovo Unnamed, 332 Sockets, 664 cores (Intel Xeon 5100)
Honda F1 Racing: SGI Altix ICE, unknown number of socket/cores, water-cooled Quad-Core Intel Xeon
Scuderia Torro Rosso Ferrari: N/A (uses Red Bull infrastructure)
Super Aguri Honda: N/A (uses Honda F1 infrastructure)
Force India Ferrari: Rental system (unknown specifications)

Overall, AMD’s Opteron leads Intel’s Xeon, followed by Intel’s Itanium 2 processors:
AMD: 2560 Sockets, 9216 cores
Intel: 1826 Sockets, 4931 cores

[AeroGT3]

Guys,

What makes you think 30-50 Million is good enough for F1? my understanding is that the top teams are talking 100 Million and above, and i know there has been talk of billion cell models!

G

50 million is good enough for TWO F1 cars. Do a grid independence study on an F1 component or something. If you do a good hybrid mesh topology, you'll be surprised how well you can capture lift and drag with a pretty coarse mesh. 100M cells is excessive, unless you are REALLY trying to use CFD as a forensic tool to hammer down your exact drag values.

For all design work, 20-30M cells for a single car (and even less) will capture the trends well enough to make decisions.

[connollyg]

50 million is good enough for TWO F1 cars. Do a grid independence study on an F1 component or something. If you do a good hybrid mesh topology, you'll be surprised how well you can capture lift and drag with a pretty coarse mesh. 100M cells is excessive, unless you are REALLY trying to use CFD as a forensic tool to hammer down your exact drag values.

For all design work, 20-30M cells for a single car (and even less) will capture the trends well enough to make decisions.

AeroGT3, don't shoot the messenger, i am telling you what i have heard, and 20-30M cells are where the top teams were a couple of years ago, they are all looking at bigger numbers now. Of course its also confused by the fact that comparing cell counts between applications is also fraught with problems. I understand that applications like OpenFOAM gives comparable results at lower cell counts than some of the other codes. Here is a link to one of the CFD codes that is catering for models of that size http://www.deskeng.com/articles/aaahpe.htm

G

[kilcoo316]

50 million is good enough for TWO F1 cars. Do a grid independence study on an F1 component or something. If you do a good hybrid mesh topology, you'll be surprised how well you can capture lift and drag with a pretty coarse mesh. 100M cells is excessive, unless you are REALLY trying to use CFD as a forensic tool to hammer down your exact drag values.

For all design work, 20-30M cells for a single car (and even less) will capture the trends well enough to make decisions.

Sorry, I have to strongly disagree.



If your wall functions are good, and operate at y+ in excess of 100, you might get away with that. But, with things like elemental gaps, suspension element junctions, brake ducts etc - you have loads of details that require detail resolution.

If you don't capture one part right, the rest of the simulation isn't worth ---.




slimjim8201 - have you any plans to enable CFdesign to be used on CUDA platforms, or on ATI/AMDs GP-GPUs? That would give you a massive leg-up over your competitors in terms of time to simulate.

[slimjim8201]

As a design tool, this is the approach we take at CFdesign. For trend studies that help improve product performance, reduce development time and enable testing of what-if scenarios, there is no need for ultra dense mesh counts.

Guys,

What makes you think 30-50 Million is good enough for F1? my understanding is that the top teams are talking 100 Million and above, and i know there has been talk of billion cell models!

G

50 million is good enough for TWO F1 cars. Do a grid independence study on an F1 component or something. If you do a good hybrid mesh topology, you'll be surprised how well you can capture lift and drag with a pretty coarse mesh. 100M cells is excessive, unless you are REALLY trying to use CFD as a forensic tool to hammer down your exact drag values.

For all design work, 20-30M cells for a single car (and even less) will capture the trends well enough to make decisions.

[AeroGT3]

Sorry, I have to strongly disagree.

If your wall functions are good, and operate at y+ in excess of 100, you might get away with that.

No, not might. Will. Can and have in my case, using a y+ of 80. I did a two car case with less than 50M cells in a lead-follow situation. The losses the second car experiences are almost exactly what BMW Sauber presented at a conference, albeit on a different car than I used. But I had all the suspension detail and things you talk about.

With the right mesh methods it is possible. Obviously if you shoot for a y+ of 1 and don't use hexcore/structured wake meshes, you'll have 1 or 2 hundred million cells.

But, with things like elemental gaps, suspension element junctions, brake ducts etc - you have loads of details that require detail resolution.

If you do a grid independence study I think you'll find they don't need to be as dense as you think . . . I think it's the volume transition with the boundary layer mesh that drives cell sizes lower than they "need" to be to catch forces.

I think you are making statements that are far too general. What kind of CFD are we talking about? Are we trying to evaluate design A vs design B, where we need to capture trends but can handle 5-10% error on the magnitudes? Or a y+ of one mesh where we need to match experimental data down to 3 sig figs? I am pretty confident I can get you 95% of the results with 50M cells. But I agree that the last 5% will cost you an addition 100 million.

[kilcoo316]

No, not might. Will. Can and have in my case, using a y+ of 80. I did a two car case with less than 50M cells in a lead-follow situation. The losses the second car experiences are almost exactly what BMW Sauber presented at a conference, albeit on a different car than I used. But I had all the suspension detail and things you talk about.

On a different car? Were the L/D and -CL results similar?

Your result could be good, or very tenuous.

If you do a grid independence study I think you'll find they don't need to be as dense as you think . . . I think it's the volume transition with the boundary layer mesh that drives cell sizes lower than they "need" to be to catch forces.

Yeap, when you go above a discretion ratio of 1.3 your asking for truncation error.

I think you are making statements that are far too general. What kind of CFD are we talking about? Are we trying to evaluate design A vs design B, where we need to capture trends but can handle 5-10% error on the magnitudes? Or a y+ of one mesh where we need to match experimental data down to 3 sig figs? I am pretty confident I can get you 95% of the results with 50M cells. But I agree that the last 5% will cost you an addition 100 million.

With the design tolerances of F1 cars right now, I don't think a 5% margin for error would be good enough to be honest. Certainly 25 million is not enough.


Don't forget, if you get your front wing wrong, then your front suspension is wrong, your splitter is wrong, your rad ducts and sidepods, as well as your diffuser - are all wrong.


Its not like an aircraft, where the wing is more or less in isolation (bar the horseshoe at the root), and same for the tail-planes.

[connollyg]

SlimJim, you may be able to do lots with a small machine, but the F1 CFD community is obviously looking at a bigger picture, the following table is taken from http://www.tgdaily.com/content/view/36905/135/ plus a few items by me. I know that some of the teams have even more compute power than this e.g. McLaren.
Some of this complexity is about mesh size, some is about run times, with teams trying to get job turn-around (includes pre-processing, simulation & post-processing) down from weeks to days and more recently down to 12 hours or less.
I understand modelling transients is quite difficult!

Formula 1 Supercomputer Championship 2008:
S:=Sockets
C:=Cores

• ING Renault F1 Team: Appro Xtreme-X2
  • S: 1024
  • C: 4096
  • AMD QC Opteron

• Scuderia Ferrari Marlboro: Acer/IBM/Racksaver
  • S: 1000+
  • C: 2000+
  • AMD upgrading to QC Opteron

• BMW Sauber F1 Team: Dell (?) Albert2
  • S: 512
  • C: 2048
  • Intel Xeon

• Vodafone McLaren Mercedes: Silicon Graphics Altix
  • S: 512
  • C: 1024
  • Intel Itanium 2

• Red Bull Renault: IBM
  • S: 512
  • C:1024
  • AMD Upgrading to QC Opteron

• Panasonic Toyota F1 Team: Fujitsu
  • S:320
  • C: 640
  • Intel Itanium 2

• AT&T WilliamsF1 Toyota: Lenovo Unnamed
  • S: 332
  • C: 664
  • Intel Xeon (DC)

• Honda F1 Racing: SGI Altix ICE
  • S: 256
  • C: 1024
  • Intel Xeon (QC) Watercooled

• Scuderia Torro Rosso Ferrari:
  • S:
  • C:
  • N/A (uses Red Bull infrastructure)

• Super Aguri Honda:
  • S:
  • C:
  • N/A (uses Honda F1 infrastructure)

• Force India Ferrari:
  • S:
  • C:
  • Rental system (unknown specifications)

[slimjim8201]

SlimJim, you may be able to do lots with a small machine, but the F1 CFD community is obviously looking at a bigger picture......

Our customers really have nothing to do with Formula 1. 99.9% of our users have one or two analysis machines.

[AeroGT3]

Yeap, when you go above a discretion ratio of 1.3 your asking for truncation error.

That's complete garbage - software developer's talk. I've done countless grid studies that show refinement of 1.3 is not necessary.

With the design tolerances of F1 cars right now, I don't think a 5% margin for error would be good enough to be honest. Certainly 25 million is not enough.

Your missing my point. Let's say you run two different front wings. Wing A is 10% more CL than wing B. They have the same CD. Then you go over to the wind tunnel. Wing A does in fact have 10% more CL and they do in fact have identical CD. However the CD the wind tunnel predicts is 5% lower than the CFD. Nonetheless, the CFD predicted the relationship and relative performance of A to B PERFECTLY, and that is what CFD is meant to do as a design tool.

Going beyond the refinement in these two meshes is a waste of time, because doing so will get you no benefit whatsoever. The existing mesh already contrasts wing A and B accurately.

Its not like an aircraft, where the wing is more or less in isolation (bar the horseshoe at the root), and same for the tail-planes.

Actually, that isn't true of aircraft. The tail can change Cp at the nose by 15 or more percent. It is an equally coupled problem.

[Gecko]

I'm with AeroGT3 on this one. You can go on and refine the mesh as much as you please and you will not manage to get within a few percent of the actual drag figures. The fact remains that one of the biggest limiting issues of current simulations are the turbulence models which are more or less bad, but there isn't any one of them that would be actually applicable to every situation. Predicting the skin friction drag of a complex smooth shape with any certainty is practically impossible.

Luckily for automotive applications most of the drag is pressure based. The mesh refinement at the boundary is mostly about capturing the turbulent effects, though, and you can quickly hit diminishing returns.

It's therefore much better if you run many different simulations on a coarser mesh using different turbulence models and compare the trends with a reference configuration, rather than create a very refined mesh and trust the actual lift and drag results of a single calculation. Of course running many cases on refined meshes is always better, but you are often trying to solve a problem in a given amount of time with given computer resources.

[connollyg]

Gecko, AeroGT3,

I am no expert on this subject, but you guys seem to be saying that small models are sufficient, but my information is that F1 models are big and getting bigger.

So there appears to be a discrepancy here, can you explain why? Is it because you are talking about components (like a front wing) and i am talking about whole car models?

G