原创

【Flume】flume中Avro Sink到Avro Source的性能测试,是否压缩,是否加密

版权声明:本文为博主原创文章,遵循 CC 4.0 BY-SA 版权协议,转载请附上原文出处链接和本声明。
本文链接:https://xxlcube.blog.csdn.net/article/details/44154367

从官方下载的flume-ng的源码里有单元测试的代码,本文就通过单元测试来体验下flume中avro的性能如何

本文中的一个event的body内容大小是1KB,读者可自行组织文本,达到1KB即可,方便测试时,性能的计算

先把我的代码贴出来

package flume.performance.test;

import org.apache.flume.Channel;
import org.apache.flume.Context;
import org.apache.flume.Event;
import org.apache.flume.EventDeliveryException;
import org.apache.flume.Transaction;
import org.apache.flume.channel.MemoryChannel;
import org.apache.flume.conf.Configurables;
import org.apache.flume.event.EventBuilder;
import org.apache.flume.sink.AvroSink;

import com.google.common.base.Charsets;

public class AvroTest {

	private static final String hostname = "192.168.11.176";
	private static final Integer port = 9520;
	private static int batchSize = 100;
	private static int batch = 1000;
	
	private static AvroSink sink;
	private static Channel channel;
	private static Context context;
	
	
	public static void setUp() {
		if (sink != null) {
			throw new RuntimeException("double setup");
		}
		sink = new AvroSink();
		channel = new MemoryChannel();
		context = new Context();

		context.put("hostname", hostname);
		context.put("port", String.valueOf(port));
		context.put("batch-size", String.valueOf(batchSize));
		context.put("connect-timeout", String.valueOf(2000L));
		context.put("request-timeout", String.valueOf(3000L));
	}
	
	public static void testWithNone() {
		dataTransport();
	}
	
	public static void testWithCompress() {
		context.put("compression-type", "deflate");
		dataTransport();
	}
	
	public static void testWithSSL() {
		context.put("ssl", "true");
		context.put("truststore", "/home/flume/keystore/chiwei.keystore");
		context.put("truststore-password", "123456");
		context.put("truststore-type", "JKS");
		dataTransport();
	}
	
	public static void testWithCompressAndSSL() {
		context.put("compression-type", "deflate");
		context.put("ssl", "true");
		context.put("truststore", "<span style="font-family: Arial, Helvetica, sans-serif;">/home/flume/keystore/chiwei.keystore</span>");
		context.put("truststore-password", "123456");
		context.put("truststore-type", "JKS");
		dataTransport();
	}
	
	public static void main(String args[]) {
		if(args.length<3) {
			System.out.println("Please Input 3 args: type[none,compress,ssl,comssl] batch(int) batchSize(int)");
			return;
		}
		String str = args[0];
		batchSize = Integer.parseInt(args[2]);
		batch = Integer.parseInt(args[1]);
		
		setUp();
		
		if("none".equals(str)) {
			testWithNone();
		}else if("compress".equals(str)) {
			testWithCompress();
		}else if("ssl".equals(str)) {
			testWithSSL();
		}else if("comssl".equals(str)) {
			testWithCompressAndSSL();
		}
		System.exit(0);
	}
	
	private static void dataTransport() {
		Configurables.configure(sink, context);
		Configurables.configure(channel, context);
		sink.setChannel(channel);
		Event event = EventBuilder
				.withBody("XXXXXX",Charsets.UTF_8);
		sink.start();
		long allBegin = System.currentTimeMillis();
		int a;
		for (a = 0; a < batch; a++) {
			Transaction transaction = channel.getTransaction();

			transaction.begin();
			int i;
			for (i = 0; i < batchSize; i++) {
				channel.put(event);
			}
			transaction.commit();
			transaction.close();
			try {
				sink.process();
			} catch (EventDeliveryException e) {
				e.printStackTrace();
			}
		}
		System.out.println("Sum Time = "
				+ (System.currentTimeMillis() - allBegin)+" ms");
		System.out.println("How many batch ? --> " + batch);
		System.out.println("How many events in one batch ? --> " + batchSize);
	}
	
}


请自行组织文本内容1KB。

在对avro进行测试的时候,分为四种情况测试

本人本机到远程flume是通过vpn连上的,网络可能是一个原因

下面将四中情况的测试结果数据展示下(1000批数据,每批100条,总共104M数据)

1、不加密,不压缩

148秒,性能=0.70M/S

2、不加密,压缩

171秒,性能=0.61M/S

3、加密,不压缩

168秒,性能=0.62M/S

4、加密,压缩

141秒,性能=0.73M/S


下面再给出一个同一内网中的测试数据,性能明显提升,看来我的VPN不给力啊:

1、不加密,不压缩

java -jar avroTest-1.0.jar none 1000 100

Sum Time = 79968 ms
How many batch ? --> 1000
How many events in one batch ? --> 100
1.3M/S

2、不加密,压缩

Sum Time = 76740 ms
How many batch ? --> 1000
How many events in one batch ? --> 100
1.35M/S

3、加密,不压缩

Sum Time = 88234 ms
How many batch ? --> 1000
How many events in one batch ? --> 100
1.18M/S

4、加密,压缩

Sum Time = 77825 ms
How many batch ? --> 1000
How many events in one batch ? --> 100
1.33M/S


注意:

以上测试前提是事务容量100,因为测试的可变参数太多,希望大家自己尝试调整参数进行测试。


以上测试结果数据,只限于本次测试,多次测试,不同网络环境的测试,对结果都是有影响的,在此只是为了提供一个测试的方法而已,望大家不吝指教!!!




文章最后发布于: 2015-03-09 15:26:50
展开阅读全文
0 个人打赏
私信求帮助

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 酷酷鲨 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览