python - Spark MLlib - trainImplicit warning -

-

i keep seeing these warnings when using trainimplicit:

warn tasksetmanager: stage 246 contains task of large size (208 kb). maximum recommended task size 100 kb.

and task size starts increase. tried call repartition on input rdd warnings same.

all these warnings come als iterations, flatmap , aggregate, instance origin of stage flatmap showing these warnings (w/ spark 1.3.0, shown in spark 1.3.1):

org.apache.spark.rdd.rdd.flatmap(rdd.scala:296) org.apache.spark.ml.recommendation.als$.org$apache$spark$ml$recommendation$als$$computefactors(als.scala:1065) org.apache.spark.ml.recommendation.als$$anonfun$train$3.apply(als.scala:530) org.apache.spark.ml.recommendation.als$$anonfun$train$3.apply(als.scala:527) scala.collection.immutable.range.foreach(range.scala:141) org.apache.spark.ml.recommendation.als$.train(als.scala:527) org.apache.spark.mllib.recommendation.als.run(als.scala:203)

and aggregate:

org.apache.spark.rdd.rdd.aggregate(rdd.scala:968) org.apache.spark.ml.recommendation.als$.computeyty(als.scala:1112) org.apache.spark.ml.recommendation.als$.org$apache$spark$ml$recommendation$als$$computefactors(als.scala:1064) org.apache.spark.ml.recommendation.als$$anonfun$train$3.apply(als.scala:538) org.apache.spark.ml.recommendation.als$$anonfun$train$3.apply(als.scala:527) scala.collection.immutable.range.foreach(range.scala:141) org.apache.spark.ml.recommendation.als$.train(als.scala:527) org.apache.spark.mllib.recommendation.als.run(als.scala:203)

similar problem described in apache spark mail lists - http://apache-spark-user-list.1001560.n3.nabble.com/large-task-size-td9539.html

i think can try play number of partitions (using repartition() method), depending of how many hosts, ram, cpus have.

try investigate steps via web ui, can see number of stages, memory usage on each stage, , data locality.

or never mind warnings unless works correctly , fast.

this notification hard-coded in spark (scheduler/tasksetmanager.scala)

      if (serializedtask.limit > tasksetmanager.task_size_to_warn_kb * 1024 &&           !emittedtasksizewarning) {         emittedtasksizewarning = true         logwarning(s"stage ${task.stageid} contains task of large size " +           s"(${serializedtask.limit / 1024} kb). maximum recommended task size " +           s"${tasksetmanager.task_size_to_warn_kb} kb.")       }

.

private[spark] object tasksetmanager {   // user warned if stages contain task has serialized size greater   // this.   val task_size_to_warn_kb = 100 }

Comments

Post a Comment

Popular posts from this blog

c++ - No viable overloaded operator for references a map -

-
i trying use map can make tag names reference number. when try , use it, in code error (6 in total every time reference map): src/main.cpp:25:45: error: no viable overloaded operator[] type 'std::map<std::string, std::string>' const char* idcs = node.child_value(tagmap[3]); this code: #include "pugi/pugixml.hpp" #include <iostream> #include <string> #include <map> int main() { pugi::xml_document doca, docb; std::map<std::string, pugi::xml_node> mapa, mapb; std::map<std::string, std::string> tagmap {{"1", "data"}, {"2", "entry"}, {"3", "id"}, {"4", "content"}}; if (!doca.load_file("a.xml") || !docb.load_file("b.xml")) { std::cout << "can't find input files"; return 1; } (auto& node: doca.child(tagmap[1]).children(tagmap[2])) { co
Read more

java - Custom OutputStreamAppender not run: LOGBACK: No context given for <MYAPPENDER> -

-
i wish write custom appender on basis of outputstreamappender . wrote following class package tests; import java.io.ioexception; import java.io.outputstream; import ch.qos.logback.core.outputstreamappender; public class myappender<e> extends outputstreamappender<e> { public myappender() { system.out.println("myappender created"); setoutputstream(new outputstream() { @override public void write(int b) throws ioexception { throw new unsupportedoperationexception(); } }); } } then wrote following runner: package tests; import org.slf4j.loggerfactory; import ch.qos.logback.classic.logger; public class runner { private static final logger log = (logger) loggerfactory.getlogger(runner.class); public static void main(string[] args) { //log.addappender(new myappender<iloggingevent>()); log.info("hello world"); } } finall
Read more

java - Cannot secure connection using TLS -

-
i'm trying establish secure socket connection between java client applet (built jdk 1.7.0_75-b13) , vc++ server application. as test vehicle, used vc++ client/server sample found in msdn forums, modified use schannel , able establish socket using cipher suite tls_rsa_with_aes_128_cbc_sha. works of tls 1.0/1.1/1.2. when try opening socket java applet same server application, connection rejected server reporting following: tls 1.0 acceptsecuritycontext failed: 0x80090327 tls 1.1 acceptsecuritycontext failed: 0x80090331 tls 1.2 acceptsecuritycontext failed: 0x80090331 this java code used create socket: debugprint("setting secure connection"); sslsocketfactory sslsocketfactory = (sslsocketfactory) sslsocketfactory.getdefault(); sslsocket sslsocket = (sslsocket) sslsocketfactory.createsocket("127.0.0.1", socketnumber); debugprint("starting handshake"); sslsocket.settcpnodelay(true); sslsocket.setsolinger(false, 0); sslsocket.s
Read more