Flume - Stream log file from Windows to HDFS in Linux -

-

how stream log file windows 7 hdfs in linux ?

flume in windows giving error

i have installed 'flume-node-0.9.3' on windows 7 (node 1) . 'flumenode' service running , localhost:35862 accessible.
in windows, log file located @ 'c:/logs/weblogic.log'
the flume agent in centos linux (node 2) running.

  1. in windows machine, java_home variable set "c:\program files\java\jre7"
  2. the java.exe file located @ "c:\program files\java\jre7\bin\java.exe"
  3. flume node installed @ " c:\program files\cloudera\flume 0.9.3"

here flume-src.conf file placed inside 'conf' folder of flume on windows 7 (node 1)

source_agent.sources = weblogic_server source_agent.sources.weblogic_server.type = exec source_agent.sources.weblogic_server.command = tail -f c:/logs/weblogic.log source_agent.sources.weblogic_server.batchsize = 1 source_agent.sources.weblogic_server.channels = memorychannel source_agent.sources.weblogic_server.interceptors = itime ihost itype  source_agent.sources.weblogic_server.interceptors.itime.type = timestamp  source_agent.sources.weblogic_server.interceptors.ihost.type = host source_agent.sources.weblogic_server.interceptors.ihost.useip = false source_agent.sources.weblogic_server.interceptors.ihost.hostheader = host  source_agent.sources.weblogic_server.interceptors.itype.type = static source_agent.sources.weblogic_server.interceptors.itype.key = log_type source_agent.sources.weblogic_server.interceptors.itype.value = apache_access_combined  source_agent.channels = memorychannel source_agent.channels.memorychannel.type = memory source_agent.channels.memorychannel.capacity = 100  source_agent.sinks = avro_sink source_agent.sinks.avro_sink.type = avro source_agent.sinks.avro_sink.channel = memorychannel source_agent.sinks.avro_sink.hostname = 10.10.201.40  source_agent.sinks.avro_sink.port = 41414

i tried run above mentioned file executing following command inside flume folder:

c:\program files\cloudera\flume 0.9.3>"c:\program files\java\jre7\bin\java.exe"  -xmx20m -dlog4j.configuration=file:///%cd%\conf\log4j.properties -cp "c:\program files\cloudera\flume 0.9.3\lib*" org.apache.flume.node.application  -f c:\program files\cloudera\flume 0.9.3\conf\flume-src.conf -n source_agent

but gives following message:

error: not find or load main class files\cloudera\flume

here trg-node.conf file running in centos (node 2). centos node working fine:

collector.sources = avroin collector.sources.avroin.type = avro collector.sources.avroin.bind = 0.0.0.0 collector.sources.avroin.port = 41414 collector.sources.avroin.channels = mc1 mc2  collector.channels = mc1 mc2 collector.channels.mc1.type = memory collector.channels.mc1.capacity = 100 collector.channels.mc2.type = memory collector.channels.mc2.capacity = 100  collector.sinks = hadoopout collector.sinks.hadoopout.type = hdfs collector.sinks.hadoopout.channel = mc2 collector.sinks.hadoopout.hdfs.path =/user/root collector.sinks.hadoopout.hdfs.calltimeout = 150000 collector.sinks.hadoopout.hdfs.filetype = datastream collector.sinks.hadoopout.hdfs.writeformat = text collector.sinks.hadoopout.hdfs.rollsize = 0 collector.sinks.hadoopout.hdfs.rollcount = 10000 collector.sinks.hadoopout.hdfs.rollinterval = 600

the problem due white space between program , files in path:

c:**program files**\cloudera\flume 0.9.3

consider installing flume in path without whitespaces, work charm.


Comments

Post a Comment

Popular posts from this blog

c++ - No viable overloaded operator for references a map -

-
i trying use map can make tag names reference number. when try , use it, in code error (6 in total every time reference map): src/main.cpp:25:45: error: no viable overloaded operator[] type 'std::map<std::string, std::string>' const char* idcs = node.child_value(tagmap[3]); this code: #include "pugi/pugixml.hpp" #include <iostream> #include <string> #include <map> int main() { pugi::xml_document doca, docb; std::map<std::string, pugi::xml_node> mapa, mapb; std::map<std::string, std::string> tagmap {{"1", "data"}, {"2", "entry"}, {"3", "id"}, {"4", "content"}}; if (!doca.load_file("a.xml") || !docb.load_file("b.xml")) { std::cout << "can't find input files"; return 1; } (auto& node: doca.child(tagmap[1]).children(tagmap[2])) { co
Read more

java - Custom OutputStreamAppender not run: LOGBACK: No context given for <MYAPPENDER> -

-
i wish write custom appender on basis of outputstreamappender . wrote following class package tests; import java.io.ioexception; import java.io.outputstream; import ch.qos.logback.core.outputstreamappender; public class myappender<e> extends outputstreamappender<e> { public myappender() { system.out.println("myappender created"); setoutputstream(new outputstream() { @override public void write(int b) throws ioexception { throw new unsupportedoperationexception(); } }); } } then wrote following runner: package tests; import org.slf4j.loggerfactory; import ch.qos.logback.classic.logger; public class runner { private static final logger log = (logger) loggerfactory.getlogger(runner.class); public static void main(string[] args) { //log.addappender(new myappender<iloggingevent>()); log.info("hello world"); } } finall
Read more

java - Cannot secure connection using TLS -

-
i'm trying establish secure socket connection between java client applet (built jdk 1.7.0_75-b13) , vc++ server application. as test vehicle, used vc++ client/server sample found in msdn forums, modified use schannel , able establish socket using cipher suite tls_rsa_with_aes_128_cbc_sha. works of tls 1.0/1.1/1.2. when try opening socket java applet same server application, connection rejected server reporting following: tls 1.0 acceptsecuritycontext failed: 0x80090327 tls 1.1 acceptsecuritycontext failed: 0x80090331 tls 1.2 acceptsecuritycontext failed: 0x80090331 this java code used create socket: debugprint("setting secure connection"); sslsocketfactory sslsocketfactory = (sslsocketfactory) sslsocketfactory.getdefault(); sslsocket sslsocket = (sslsocket) sslsocketfactory.createsocket("127.0.0.1", socketnumber); debugprint("starting handshake"); sslsocket.settcpnodelay(true); sslsocket.setsolinger(false, 0); sslsocket.s
Read more