scala - reduceByKey type mismatch -

-

i got list

res22: array[(string, list[(string, int)])] = array((door_182,list((in,1), (in,1))), (door_89,list((in,1), (in,1), (in,1))), (door_180,list((in,1), (in,1), (in,1), (in,1))), (door_83,list((in,1), (in,1), (in,1))), (door_177,list((in,1), (in,1))), (door_23,list((in,1), (in,1))), (door_128,list((in,1), (in,1))), (door_34,list((in,1), (in,1))), (door_18,list((in,1), (in,1))), (door_32,list((in,1))), (door_76,list((in,1), (in,1), (in,1))), (door_87,list((in,1), (in,1), (in,1))), (door_197,list((in,1), (in,1))), (door_133,list((in,1), (in,1))), (door_119,list((in,1), (in,1))), (door_113,list((in,1), (in,1), (in,1), (in,1), (in,1))), (door_155,list((in,1), (in,1), (in,1), (in,1), (in,1))), (door_168,list((in,1), (in,1), (in,1))), (door_115,list((in,1), (in,1))), (door_9,list((in,1), (in,1))),...

i tried sum number of in each door this:

scala> reduced.map(n => (n._1, n._2)).reducebykey((v1,v2) => v1 + v2.tostring).collect

i error:

<console>:32: error: type mismatch; found   : list[(string, int)] required: string           reduced.map(n => (n._1, n._2)).reducebykey((v1,v2) => v1 + v2).collect                                                                      ^

how can solve this?

you can in 2 steps: each key aggregate lists , sum values in each list:

val x = sc.parallelize(list(("door_182",list(("in",1), ("in",1))), ("door_89",list(("in",1), ("in",1), ("in",1))), ("door_180",list(("in",1), ("in",1), ("in",1), ("in",1))), ("door_83",list(("in",1), ("in",1), ("in",1))), ("door_177",list(("in",1), ("in",1))))) x.reducebykey(_ ::: _)   .map {     case (door, list) => (door, list.foldleft(0){       case (count1, (in2, count2)) => count1 + count2    })   }.collect() res3: array[(string, int)] = array((door_180,4), (door_83,3), (door_177,2), (door_182,2), (door_89,3))

or in single operation aggregatebykey avoiding memory allocation:

x.aggregatebykey(0)(   {     case (count, list) => count + list.foldleft(0){        case (count1, (in2, count2)) => count1 + count2}   },    _ +_    )   .collect()

Comments

Post a Comment

Popular posts from this blog

c++ - No viable overloaded operator for references a map -

-
i trying use map can make tag names reference number. when try , use it, in code error (6 in total every time reference map): src/main.cpp:25:45: error: no viable overloaded operator[] type 'std::map<std::string, std::string>' const char* idcs = node.child_value(tagmap[3]); this code: #include "pugi/pugixml.hpp" #include <iostream> #include <string> #include <map> int main() { pugi::xml_document doca, docb; std::map<std::string, pugi::xml_node> mapa, mapb; std::map<std::string, std::string> tagmap {{"1", "data"}, {"2", "entry"}, {"3", "id"}, {"4", "content"}}; if (!doca.load_file("a.xml") || !docb.load_file("b.xml")) { std::cout << "can't find input files"; return 1; } (auto& node: doca.child(tagmap[1]).children(tagmap[2])) { co
Read more

java - Custom OutputStreamAppender not run: LOGBACK: No context given for <MYAPPENDER> -

-
i wish write custom appender on basis of outputstreamappender . wrote following class package tests; import java.io.ioexception; import java.io.outputstream; import ch.qos.logback.core.outputstreamappender; public class myappender<e> extends outputstreamappender<e> { public myappender() { system.out.println("myappender created"); setoutputstream(new outputstream() { @override public void write(int b) throws ioexception { throw new unsupportedoperationexception(); } }); } } then wrote following runner: package tests; import org.slf4j.loggerfactory; import ch.qos.logback.classic.logger; public class runner { private static final logger log = (logger) loggerfactory.getlogger(runner.class); public static void main(string[] args) { //log.addappender(new myappender<iloggingevent>()); log.info("hello world"); } } finall
Read more

java - Cannot secure connection using TLS -

-
i'm trying establish secure socket connection between java client applet (built jdk 1.7.0_75-b13) , vc++ server application. as test vehicle, used vc++ client/server sample found in msdn forums, modified use schannel , able establish socket using cipher suite tls_rsa_with_aes_128_cbc_sha. works of tls 1.0/1.1/1.2. when try opening socket java applet same server application, connection rejected server reporting following: tls 1.0 acceptsecuritycontext failed: 0x80090327 tls 1.1 acceptsecuritycontext failed: 0x80090331 tls 1.2 acceptsecuritycontext failed: 0x80090331 this java code used create socket: debugprint("setting secure connection"); sslsocketfactory sslsocketfactory = (sslsocketfactory) sslsocketfactory.getdefault(); sslsocket sslsocket = (sslsocket) sslsocketfactory.createsocket("127.0.0.1", socketnumber); debugprint("starting handshake"); sslsocket.settcpnodelay(true); sslsocket.setsolinger(false, 0); sslsocket.s
Read more