bigdata - Data scheme Cassandra using various data types -

-

currently developing solution in field of time-series data. within these data have: id, value , timestamp. here comes: value might of type boolean, float or string. consider 3 approaches:

a) every data type distinct table, sensor values of type boolean table, sensor values of type string another. obvious disadvantage have know sensor.

b) meta-column describing data type plus values of type string. obvious disadvantage data conversion e.g. calculating max, avg , on.

c) having 3 columns of different type 1 value per record. disadvantage 500000 sensors firing every 100ms ... plenty of unused space.

as knowledge limited appreciated.

500000 sensors firing every 100ms

first thing, make sure partition properly, make sure don't exceed limit of 2 billion columns per partition.

create table sensordata (   stationid uuid,   datebucket text,   recorded timeuuid,   intvalue bigint,   strvalue text,   blnvalue boolean,    primary key ((stationid,datebucket),recorded));

with half-million every 100ms, that's 500 million in second. you'll want set datebucket granular...down second. next i'll insert data:

 stationid                            | datebucket          | recorded                             | blnvalue | intvalue | strvalue --------------------------------------+---------------------+--------------------------------------+----------+----------+----------  8b466f1d-8d6b-46fa-9f5b-8c4eb51aa40c | 2015-04-22t14:54:29 | 6338df40-e929-11e4-88c8-21b264d4c94d |     null |       59 |     null  8b466f1d-8d6b-46fa-9f5b-8c4eb51aa40c | 2015-04-22t14:54:29 | 633e0f60-e929-11e4-88c8-21b264d4c94d |     null |     null |       cd  8b466f1d-8d6b-46fa-9f5b-8c4eb51aa40c | 2015-04-22t14:54:29 | 6342f160-e929-11e4-88c8-21b264d4c94d |     true |     null |     null  3221b1d7-13b4-40d4-b41c-8d885c63494f | 2015-04-22t14:56:19 | a48bbdf0-e929-11e4-88c8-21b264d4c94d |    false |     null |     null

...plenty of unused space.

you might suprised. cql output of select * above, appears there null values on place. watch happens when use cassandra-cli tool view how data stored "under hood:"

rowkey: 3221b1d7-13b4-40d4-b41c-8d885c63494f:2015-04-22t14\:56\:19 => (name=a48bbdf0-e929-11e4-88c8-21b264d4c94d:, value=, timestamp=1429733297352000) => (name=a48bbdf0-e929-11e4-88c8-21b264d4c94d:blnvalue, value=00, timestamp=1429733297352000)

as can see, data (above) stored cql row stationid=3221b1d7-13b4-40d4-b41c-8d885c63494f , datebucket='2015-04-22t14:56:19' shows blnvalue has value of 00 (false). notice intvalue , strvalue not present. cassandra doesn't force null value rdbms does.

the obvious disadvantage data conversion e.g. calculating max, avg , on.

perhaps know this, did want mention cassandra cql not contain definitions max, avg or other data aggregation function. you'll either need client-side, or implement apache-spark perform olap-type queries.

be sure read through patrick mcfadin's getting started time series data modeling. contains suggestions on how solve time series problems this.


Comments

Post a Comment

Popular posts from this blog

c++ - No viable overloaded operator for references a map -

-
i trying use map can make tag names reference number. when try , use it, in code error (6 in total every time reference map): src/main.cpp:25:45: error: no viable overloaded operator[] type 'std::map<std::string, std::string>' const char* idcs = node.child_value(tagmap[3]); this code: #include "pugi/pugixml.hpp" #include <iostream> #include <string> #include <map> int main() { pugi::xml_document doca, docb; std::map<std::string, pugi::xml_node> mapa, mapb; std::map<std::string, std::string> tagmap {{"1", "data"}, {"2", "entry"}, {"3", "id"}, {"4", "content"}}; if (!doca.load_file("a.xml") || !docb.load_file("b.xml")) { std::cout << "can't find input files"; return 1; } (auto& node: doca.child(tagmap[1]).children(tagmap[2])) { co
Read more

java - Custom OutputStreamAppender not run: LOGBACK: No context given for <MYAPPENDER> -

-
i wish write custom appender on basis of outputstreamappender . wrote following class package tests; import java.io.ioexception; import java.io.outputstream; import ch.qos.logback.core.outputstreamappender; public class myappender<e> extends outputstreamappender<e> { public myappender() { system.out.println("myappender created"); setoutputstream(new outputstream() { @override public void write(int b) throws ioexception { throw new unsupportedoperationexception(); } }); } } then wrote following runner: package tests; import org.slf4j.loggerfactory; import ch.qos.logback.classic.logger; public class runner { private static final logger log = (logger) loggerfactory.getlogger(runner.class); public static void main(string[] args) { //log.addappender(new myappender<iloggingevent>()); log.info("hello world"); } } finall
Read more

java - Cannot secure connection using TLS -

-
i'm trying establish secure socket connection between java client applet (built jdk 1.7.0_75-b13) , vc++ server application. as test vehicle, used vc++ client/server sample found in msdn forums, modified use schannel , able establish socket using cipher suite tls_rsa_with_aes_128_cbc_sha. works of tls 1.0/1.1/1.2. when try opening socket java applet same server application, connection rejected server reporting following: tls 1.0 acceptsecuritycontext failed: 0x80090327 tls 1.1 acceptsecuritycontext failed: 0x80090331 tls 1.2 acceptsecuritycontext failed: 0x80090331 this java code used create socket: debugprint("setting secure connection"); sslsocketfactory sslsocketfactory = (sslsocketfactory) sslsocketfactory.getdefault(); sslsocket sslsocket = (sslsocket) sslsocketfactory.createsocket("127.0.0.1", socketnumber); debugprint("starting handshake"); sslsocket.settcpnodelay(true); sslsocket.setsolinger(false, 0); sslsocket.s
Read more