nash85 wrote:arjun.yadav
I'm interested to know why are you running 3.6 billion cells on only 670 cores? Thats 5 million per core. Whats your Reynolds number? I would guess that a DNS of a golf ball would take months to run using that many cores. Btw, which HPC facility are you using?
Cheers
Neil
sorry for late reply, did not visit site in between.
I used 600 cores for 3billion (the last calculation). And my calculation time for 5000 time steps is roughly 15 days. I am using SIMPLE algorithm so no courant number dependency.
Some of the results will be presented on 7th september at Kobe , where the k super computer is. (I am using k-supercomputer).
Here are few interesting things about this solver and the reason I can use only 600 to 700 cores for it.
1. I have written a memory efficient code that uses minimum memory required so that i can run large cases. Specially written for these purposes.
2. I designed a special Poisson equation solver that could be used with SIMPLE algorithm. It is basically FFT based direct method adapted for non uniform poisson equation for pressure correction.
3. I designed a new all to all algorithm that is very efficient (0.3 seconds to redistribute this 3 billion cell data among processors) This means FFT based solver works like charm.
4. With 3 sub iterations I take roughly 300 seconds per time step. Because it is implicit algorithm I guess it is the most efficient algorithm (implicit + efficiency of direct solvers).
5. I wrote a post processor for creating iso-surfaces of Q criteria etc. so that I do not need to load all the data into paraview and can do post processing easily (animation will be presented at k comp on 7th sept).
Some of the things newly developed and
patented for my company in the process:
1. O(3) 1 D search algorithm for linear search.
2. Moving solid bodies and marking of solid - fluid. A new algorithm that can do in 0.5 seconds for 1 billion of more mesh. This require 3D nearest neighbor search so is very difficult task to do otherwise. (uses 1)
2. A non uniform Poisson solver for pressure correction.
PS: I have
http://www.inavier.com for unstructured grid solver.
I am working on a parallel version of this solver from point of view of doing very large calculations. I learned a lot from this large calculation, so I am hoping to put all that into this version. I have parallel matrix solver running now. And i have designed a very efficient GGI interface algorithm too. Actually parallel version of inavier for navier stokes is also working now but i am going to rewrite the whole thing.
So just wait for 3-4 months, we will have a unstructured grid solver that will be designed for large calculations (100million or more).
.
