hadoop - Azkaban - HIve Job Failed -


i trying run hive job in azkaban

i able upload hive job hive job not getting executed properly.

error [hive-demo] [azkaban] failed build job executor job hive-demojob type     'hive' unrecognized. not construct job[{hive.query.01: drop table words;, hive.query.03: describe words;, hive.query.02: create table words (freq int, word string) row format delimited fields terminated ' ' stored textfile;, working.dir: /home/hduser/technology/azkaban/azkaban-executor/executions/44, azk.hive.action: execute.query, azkaban.job.attempt: 0, type: hive, hive.query.05: select * words limit   10;, hive.query.04: load data local inpath "res/input" table words;, user.to.proxy:   azkaban, hive.query.06: select freq, count(1) f2 words group freq sort f2 desc  limit 10;, parent = {azkaban.flow.flowid: hive-demo, azkaban.flow.execid: 44,  azkaban.flow.start.timezone: america/los_angeles, azkaban.flow.start.hour: 22,  azkaban.flow.start.second: 17, azkaban.flow.start.year: 2014,  azkaban.flow.start.milliseconds: 111, azkaban.flow.start.minute: 54,  azkaban.flow.start.timestamp: 2014-01-31t22:54:17.111-08:00, azkaban.flow.start.month: 01,  azkaban.flow.projectversion: 2, azkaban.flow.projectid: 4, azkaban.flow.uuid: 29a46293-cc95- 4c66-b2b1-51014205ce3b, azkaban.flow.start.day: 31, parent = {}}}] of type[hive]. 2014/02/01  12:24:17.295 +0530 error [hive-demo] [azkaban] failed build job type, skipping job    2014/02/01 12:24:17.295 +0530 error [hive-demo] [azkaban] job run failed! 2014/02/01 12:24:17.295 +0530 info [hive-demo] [azkaban] finishing job hive-demo @ 1391237657295  2014/02/01 12:24:17.382 +0530 info [flowrunnermanager] [azkaban] cleaning finished  2014/02/01 12:24:17.383 +0530 info [flowrunnermanager] [azkaban] cleaning execution 43  finished flows list. 2014/02/01 12:24:17.408 +0530 info [hive-demo] [azkaban] job  finished hive-demo status failed 2014/02/01 12:24:17.463 +0530 info [hive-demo]  [azkaban] finishing flow. awaiting termination 2014/02/01 12:24:17.463 +0530 info [hive- demo] [azkaban] setting flow status failed. 2014/02/01 12:24:17.464 +0530 info [hive- demo] [azkaban] flow set failed 2014/02/01 12:24:17.464 +0530 info [hive-demo]  [azkaban] setting end time flow 44 1391237657464 2014/02/01 12:24:17.552 +0530 info  [flowrunnermanager] [azkaban] flow 44 finished. adding finished flows  list. 2014/02/01 12:26:17.383 +0530 info [flowrunnermanager] [azkaban] cleaning  finished 2014/02/01 12:26:17.384 +0530 info [flowrunnermanager] [azkaban] cleaning execution  44 finished flows list.  ========================================================================

here content in plugin.properties

job.class=azkaban.jobtype.hiveutils.azkaban.hiveviaazkaban  hive.home=hive_home  hive.aux.jars.path=/usr/lib/hive/lib env.hive_home=$hive_home env.hive_aux_jars_path=/usr/lib/hive/lib  #hive.jvm.args=-dhive.querylog.location=. -dhive.exec.scratchdir=/tmp/hive-    ${user.to.proxy} -dhive.aux.jars.path=${hive.aux.jars.path} jobtype.global.jvm.args=${hive.jvm.args}

this private.properties jobtype.classpath=$hadoop_home/conf,$hadoop_home/lib/*,$hive_home/lib/*,$hive_home/conf,/usr /lib/hive/lib jobtype.class=azkaban.jobtype.hadoopjavajob

hive.aux.jars.path=/usr/lib/hive/lib env.hadoop_home=$hadoop_home env.hive_home=$hive_home env.hive_aux_jars_path=/usr/lib/hive/lib  hive.home=$hive_home  jobtype.global.classpath=${hive.classpath.items}   hive.classpath.items=./*,${hive.home}/lib/*,${hive.home}/conf/,${hive.aux.jars.path}/*,${had    oop.home}/*,${hadoop.home}/lib/*,${hadoop.home}/conf/

i have added plugin folder in executor folder only.

why not detecting job type. please help.

it's hard tell going on since neither share job nor logs, try following , let know if works. create file test.job following content:

type=hive azk.hive.action=execute.query hive.query=create table test(id int, name string) row format delimited fields terminated '\t'; user.to.proxy=hadoop

and submit described in 'creating flow' section. further details on different types of jobs azkaban (2.1) supports out of box see job types/plugin documentation.


-

Comments

Post a Comment

Popular posts from this blog

php - regexp cyrillic filename not matches -

i take list of files php glob(."*.pdf") . files have cyrillic names. example: ООО «Рога и копыта» then load company names database , try find them in file names: preg_match("/$firm_name/ui", $file_name, $match); i have problem: firms if $firm_name visually similar $file_name , preg_match says there no match. mb_detect_encoding both variables return utf-8. there problem in file name, or maybe there encoding or something, can not understand, ask help. upd: $file_name='ООО «Рога и копыта»'; preg_match("/... .Рога и копыта./ui", $file_name, $match); as result, ОО «Рога и копыта» , 1 О missing
Read more

c# - OpenXML hanging while writing elements -

Image
i have program pulls data database, caches file , exports data multiple formats (excel, excel 2003, csv). i'm using openxml sdk 2.0 excel work. these export processes run in parallel (using parallel.foreach ), , amount of data can pretty large - e.g. csvs 800mb. during these larger exports, i've noticed writing of xml documents hang. instance, if have 8 exporting in parallel, @ point "pause". hang around same point: //this.writer openxmlwriter created worksheetpart. this.writer.writeelement(new cell() { cellvalue = new cellvalue(value), datatype = cellvalues.string }); when happens, pause debugger (vs2013 in case) , notice threads blocking around same portion of code - bit deeper in openxml sdk - stem call openxmlwriter.writeelement . i dug through source using justdecompile didn't find answers. appears there intermediary stream in use writing isolated storage , is, reason, blocking. underlying stream each of these filestream . here screenshot ...
Read more

sql - Select Query has unexpected multiple records (MS Access) -

i may tired, can't seem figure out happening. have 2 queries. query 1 has 10 fields. query 2 has 5 fields. fields 1 through 3 same in both queries. trying write select query , simple add field 5 query2 query1 result should query1 additional field (query2.field5). i join first 3 fields both queries , choose select records in query1 , in query2 match. don't know sql, use query designer. expect same number of records have query 1 instead have 3 times or more. can tell me doing wrong. tried reversing joins still seem same records. thanks in advance. here query: the fields in common divisionname , supplierid = vendorid , commodityname . select [part revenue exposed query p1].divisionname, [part revenue exposed query p1].commodityname, [part revenue exposed query p1].suppliername, [part revenue exposed query p1].partnumber, [part revenue exposed query p1].partdescription, [part revenue exposed query p1].bucode, ...
Read more