Azkaban任务流编写

Azkaban任务流编写

在Azkaban中,一个project包含一个或多个flows,一个flow包含多个job。job是你想在azkaban中运行的一个进程,可以是Command,也可以是一个Hadoop任务。当然,如果你安装相关插件,也可以运行插件。一个job可以依赖于另一个job,这种多个job和它们的依赖组成的图表叫做flow。本文介绍如何在Azkaban上编写四类任务流:Command、Hive、Java、Hadoop。

1、Command任务编写

这里将模拟一个数据从采集到上传最后入库的整个过程的工作流。涉及的job如下:

create_dir.job:创建对应的目录
get_data1.job:获取数据1
get_data2.job:获取数据2
upload_to_hdfs.job:数据上传到hdfs
insert_to_hive.job:从hdfs入库到hive中
  • create_dir.job
  • type=command
    command=echo "create directory before get data"
    
  • get_data1.job
    type=command
    command=echo "get data from logserver"
    dependencies=create_dir
    

  • get_data2.job
    type=command
    command=echo "get data from ftp"
    dependencies=create_dir
    
  • upload_to_hdfs.job
    type=command
    command=echo "upload to hdfs"
    dependencies=get_data1,get_data2
    

完成后的目录如下

打包成demo.zip压缩包,并上传到Azkaban中,可以看到依赖图如下:

点击执行


在Job List里可以看到每个job的运行情况

点击Details可以看到每个job执行的日志

Job中的其他配置选项

  • 可以定义job依赖另一个flow
    type=flow
    flow.name=fisrt_flow
    
  • 可以配置多个command命令
    type=command
    command=echo "hello"
    command.1=echo "world"
    command.2=echo "azkaban"
    
  • 可以配置job失败重启次数,及间隔时间,比如,上述ftp获取日志,我可以配置重试12次,每隔5分钟一次
    type=command
    command=echo "retry test" 
    retries=12
    #单位毫秒
    retry.backoff=300000
    


2、Hive任务编写

Hive任务的编写比较简单,在新的目录下新建hive.job文件,内容如下

#定义类型
type=hive
#定义执行HiveSQL的用户
user.to.proxy=azkaban
#固定值
azk.hive.action=execute.query
hive.query.01=drop table words;
hive.query.02=create table words (freq int, word string) row format delimited fields terminated by '\t' stored as textfile;
hive.query.03=describe words;
hive.query.04=load data local inpath "res/input" into table words;
hive.query.05=select * from words limit 10;
hive.query.06=select freq, count(1) as f2 from words group by freq sort by f2 desc limit 10;

以上第四条语句涉及到数据文件,需要在同级目录下新建res文件夹,然后新建input文件,内容如下

11	and
10	the
9	to
9	in
9	of
9	is
9	CLAUDIUS
8	KING
8	this
8	we
7	what
7	us
7	GUILDENSTERN
6	And
5	d
4	ROSENCRANTZ
3	a
2	his
1	QUEEN
1	he

然后打包成zip文件即可上传到azkaban中运行

3、Java任务编写

Java任务比较简单,只需要在类里提供一个run方法即可,如果需要设定参数,着在构造方法中指定Props类,然后在job文件里配置好参数。

Java类如下

package com.dataeye.java;

import org.apache.log4j.Logger;

import azkaban.utils.Props;

public class JavaMain {

	private static final Logger logger = Logger.getLogger(JavaMain.class);

	private final int fileRows;
	private final int fileLine;
	
	public JavaMain(String name, Props props) throws Exception {
		this.fileRows = props.getInt("file.rows");
		this.fileLine = props.getInt("file.line");
	}
	
	public void run() throws Exception {
		logger.info(" ### this is JavaMain method ###");
		logger.info("fileRows value is ==> " + fileRows);
		logger.info("fileLine value is ==> " + fileLine);
	}
	
}

java.job文件如下

type=java
#指定类的全路径
job.class=com.dataeye.java.JavaMain
#指定执行jar包的路径
classpath=lib/*
#用户参数1
file.rows=10
#用户参数2
file.line=50

新建目录,把java.job拷贝到该目录下,然后新建lib文件夹,把以上java类打包成jar文件,放入lib目录下,打包成zip文件,上传到azkaban中。执行成功后的日志如下

31-08-2016 14:41:15 CST simple INFO - INFO Running job simple
31-08-2016 14:41:15 CST simple INFO - INFO Class name com.dataeye.java.JavaMain
31-08-2016 14:41:15 CST simple INFO - INFO Constructor found public com.dataeye.java.JavaMain(java.lang.String,azkaban.utils.Props) throws java.lang.Exception
31-08-2016 14:41:15 CST simple INFO - INFO Invoking method run
31-08-2016 14:41:15 CST simple INFO - INFO Proxy check failed, not proxying run.
31-08-2016 14:41:15 CST simple INFO - INFO  ### this is JavaMain method ###
31-08-2016 14:41:15 CST simple INFO - INFO fileRows value is ==> 10
31-08-2016 14:41:15 CST simple INFO - INFO fileLine value is ==> 50
31-08-2016 14:41:15 CST simple INFO - INFO Apparently there isn't a method[getJobGeneratedProperties] on object[com.dataeye.java.JavaMain@591f989e], using empty Props object instead.
31-08-2016 14:41:15 CST simple INFO - INFO Outputting generated properties to /home/hadoop/azkaban/azkaban-solo-server-3.0.0/executions/339/simple_output_6034902760752438337_tmp
31-08-2016 14:41:15 CST simple INFO - Process completed successfully in 0 seconds.
31-08-2016 14:41:15 CST simple INFO - Finishing job simple attempt: 0 at 1472625675501 with status SUCCEEDED

日志中已经打印出run方法中的参数值。

4、Hadoop任务编写

Hadoop相对以上三种类型会复杂一些,需要注意的地方如下

  • 必须继承 AbstractHadoopJob 类
    public class WordCount extends AbstractHadoopJob
    
  • 必须要有构造方法,参数是String和Props,且要调用super方法
    public WordCount(String name, Props props) {
    	super(name, props);
    	//other code	
    }
    
  • 必须提供run方法,且在run方法的最后调用super.run();
    public void run() throws Exception{
    //other code
    super.run();}
    

下面提供一个 WordCount 任务的例子

WordCount.java类

package com.dataeye.mr;

import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.FileInputFormat;
import org.apache.hadoop.mapred.FileOutputFormat;
import org.apache.hadoop.mapred.JobConf;
import org.apache.hadoop.mapred.TextInputFormat;
import org.apache.hadoop.mapred.TextOutputFormat;
import org.apache.log4j.Logger;

import azkaban.jobtype.javautils.AbstractHadoopJob;
import azkaban.utils.Props;

import com.dataeye.mr.maper.WordCountMap;
import com.dataeye.mr.reducer.WordCountReduce;

public class WordCount extends AbstractHadoopJob {

	private static final Logger logger = Logger.getLogger(WordCount.class);

	private final String inputPath;
	private final String outputPath;
	private boolean forceOutputOverrite;

	public WordCount(String name, Props props) {
		super(name, props);
		this.inputPath = props.getString("input.path");
		this.outputPath = props.getString("output.path");
		this.forceOutputOverrite = props.getBoolean("force.output.overwrite", false);
	}

	public void run() throws Exception {
		
		logger.info(String.format("Hadoop job, class is %s", new Object[] { getClass().getSimpleName() }));

		JobConf jobconf = getJobConf();
		jobconf.setJarByClass(WordCount.class);

		jobconf.setOutputKeyClass(Text.class);
		jobconf.setOutputValueClass(IntWritable.class);

		jobconf.setMapperClass(WordCountMap.class);
		jobconf.setReducerClass(WordCountReduce.class);

		jobconf.setInputFormat(TextInputFormat.class);
		jobconf.setOutputFormat(TextOutputFormat.class);

		FileInputFormat.addInputPath(jobconf, new Path(this.inputPath));
		FileOutputFormat.setOutputPath(jobconf, new Path(this.outputPath));

		if (this.forceOutputOverrite) {
			FileSystem fs = FileOutputFormat.getOutputPath(jobconf).getFileSystem(jobconf);
			fs.delete(FileOutputFormat.getOutputPath(jobconf), true);
		}
		
		super.run();
	}

}

WordCountMap.java类

package com.dataeye.mr.maper;

import java.io.IOException;
import java.util.StringTokenizer;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

public class WordCountMap extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
	private static final IntWritable one = new IntWritable(1);
	private Text word = new Text();

	private long numRecords = 0L;

	public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
		String line = value.toString();
		StringTokenizer tokenizer = new StringTokenizer(line);
		while (tokenizer.hasMoreTokens()) {
			this.word.set(tokenizer.nextToken());
			output.collect(this.word, one);
			reporter.incrCounter(Counters.INPUT_WORDS, 1L);
		}

		if (++this.numRecords % 100L == 0L)
			reporter.setStatus("Finished processing " + this.numRecords + " records " + "from the input file");
	}

	static enum Counters {
		INPUT_WORDS;
	}
}

WordCountReduce.java类

package com.dataeye.mr.reducer;

import java.io.IOException;
import java.util.Iterator;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reducer;
import org.apache.hadoop.mapred.Reporter;

public class WordCountReduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, IntWritable> {
	public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
		int sum = 0;
		while (values.hasNext()) {
			sum += ((IntWritable) values.next()).get();
		}
		output.collect(key, new IntWritable(sum));
	}
}

以下是 wc.job 配置文件

type=hadoopJava

job.class=com.dataeye.mr.WordCount

classpath=lib/*

force.output.overwrite=true

input.path=/tmp/azkaban/wordcountjavain

output.path=/tmp/azkaban/wordcountjavaout

注意/tmp/azkaban/wordcountjavain文件是必须先存在hdfs中的。

新增目录,把wc.job文件拷贝到该目录下,然后新增lib目录,把以上java代码打包成jar文件。最后压缩成zip文件,上传到azkaban上执行即可。

以上介绍了四类常用的azkaban任务的编写过程。其他任务类型可以参考Azkaban官网:Azkaban 3.0 Documentation

编辑于 2016-09-01

文章被以下专栏收录