c# - What is the behaviour of await inside of a Parallel.ForEach() loop? -

i have computationally intensive program attempting parallelize, 1 of limiting steps i/o operation controlled phenomenally inefficient api have no control on have no choice use. imperative parallelization not increase number of i/o operations, or benefit disappear.

the layout this: have 2 classes, foo , bar, , in order calculate foo, involves no small quantity of calculations, must pass instance, or few instances, of bar import other file in extremely expensive i/o operation. require large number of both foo , bar instances , many of these bar instances used calculate more 1 foo instance. result, not want discard bar instances after calculate each foo , not want import them more once each. potentially of note, make matters more complicated api 32-bit, whereas program must 64-bit avoid memoryexception, handled locally hosted server communicate using wcf.

here proposed solution, extremely new parallelization , in particular unsure of how await handled inside of foreach loop w.r.t freeing processors:

concurrentdictionary<string, task<bar>> barlist = new concurrentdictionary<string, task<bar>>();  parallel.foreach(foolist, foo => {     if (!barlist.containskey(this.requiredbarname))     {         task<bar> importbar = task.run(() => import.barbyname(this.requiredbarname));         barlist.add(this.requiredbarname,importbar);     }     this.requiredbartask = barlist.trygetvalue(this.requiredbarname);     foo.calculatestuff(); }  // foo.calculatestuff() looks async public void calculatestuff() {     // stuff...     bar requiredbar = await this.requiredbartask;     // more stuff requiredbar } 

what happen when code runs await? threadpool pick different task, or processor idle? if arrange sort of waitall() outside of parallel.foreach() able parallelize through of efficiently? have better ideas of how might implement this?

edit provide mcve:

i cannot satisfy verifiable component of cannot give api , can't give of data api might access, attempt provide call out server.

the program can go infinitely deep in way processes things, easier think of parser of specific instructions client allowed build using gui set of "bricks". in way dataflow looks offer decent solution.

in example don't take care of circular references or 1 channel calculating channel has been called parallel.foreach() method; in code handled logic , concurrent lists check when various things have been called.

public abstract class class {     public string name {get;set;}     public float[] data {get;set;}      async public task calculatedata(isampleservice proxy){} }  public class channel : class {     public class[] childclasses {get;set;}      async public override task calculatedata(isampleservice proxy)     {         foreach(class childclass in childclasses)         {             // not real processing step anything. there class handle happens here, unnecessary post.             if(childclass.data==null) await childclass.calculatedata(proxy);             this.data = childclass.data;         }     } }  public class input : class {     async public override task calculatedata(isampleservice proxy)     {             this.data = await proxy.returndata(this.name);     } }  async public static task processdataforexport(channel[] channelarray) { channelfactory<isampleservice> factory = new channelfactory<isampleservice>(new netnamedpipebinding(), new endpointaddress(baseaddress));  isampleservice proxy = factory.createchannel();  parallel.foreach(channelarray, channel =>     {         channel.calculatedata();     }); // task.whenall() might better alternative parallel.foreach() here. } 

what happen when code runs await?

the same thing happens await statement: after having evaluated whatever expression or statement retrieves task awaited, method return. intents , purposes, that end of method.

will threadpool pick different task, or processor idle?

that depends on else going on. example, awaiting on? if it's computational task queued thread pool, , wasn't assigned thread pool thread, sure…the thread pool might pick , start working on it.

if you're waiting on i/o operation, won't keep processor busy, there may still other tasks in thread pool queue (such other ones parallel.foreach() call). give processor work on.

certainly, using await doesn't result in processer being idle. in fact, main reason using avoid (*). await statement causes current method return, let current thread proceed, means if otherwise there weren't enough threads keep processor busy, has do. :)

(*) (well, sort of…really, main reason avoid blocking current thread, has side-effect of there being more work available processer handle :) )

if arrange sort of waitall() outside of parallel.foreach() able parallelize through of efficiently? have better ideas of how might implement this?

i don't see enough useful detail in question answer that. frankly, while can't put finger on it, use of await parallel.foreach() delegate seems fishy me somehow. call await, delegate's method return.

hence, far parallel.foreach() knows, you're done item in enumeration, of course you're not. have finished elsewhere. @ least, seems hinder parallel class's ability know enough work it's doing schedule effectively.

but maybe that's okay. or maybe it's not great, best you're going achieve given framework you're tied to. hard say.

encourage provide mcve commenter scott chamberlain's asked for. if he's right , problem addressable through dataflow api, give him chance provide answer shows that.