post,get请求nginx记录日志kafka自动扫描程序

        kafka的介绍以及使用安装请查看博文http://blog.csdn.net/hmsiwtv/article/details/46960053

讲的很详细，基本一看就会，这边主要介绍一下nginx记录日志kafka扫描

    一  nginx主要配置

 

log_format ht-video  '$request|$msec|$http_x_forwarded_for|$http_user_agent|$request_body';
	server{
                server_name 192.168.50.42;

                location / {
                        root html;
                        index index.html;
                }
                location /ht{
                        #root   html;
                        #index  index.html index.htm;
                        if ( $request ~ "GET" ) {
                                #access_log logs/test.log tes;
                                access_log /htlog/test.log test;
                        }

                        client_max_body_size 8000M;
                        #error_page 405 =200 $1;
                }
        }

 然后可以发送get请求  http://192.168.50.42:8080/ht/p?data=11211111111

注意在nginx的html目录下边需要有ht目录 然后目录下边需要建一个p文件，里面随便输点内容，是为了能请求到对应的文件

 

二 修改Kafka文件

vi /usr/local/kafka_2.11-0.9.0.1/config/server.properties
   这个文件里要改下面5点:
broker.id=0
port=9092
host.name=192.168.50.42

log.dirs=/log
zookeeper.connect=192.168.50.42:2181

 

三、启动zookeeper和kafka

/usr/local/kafka_2.11-0.9.0.1/bin/zookeeper-server-start.sh -daemon /usr/local/kafka_2.11-0.9.0.1/config/zookeeper.properties
/usr/local/kafka_2.11-0.9.0.1/bin/kafka-server-start.sh -daemon /usr/local/kafka_2.11-0.9.0.1/config/server.properties

 

四，创建topic

 

输入命令创建一个叫test的topic：bin/kafka-console-producer.sh --broker-list 192.168.50.42:9092 --topic test

消费topic的命令：bin/kafka-console-consumer.sh --zookeeper 192.168.50.42:2181 --topic test --from-beginning

 


 
 

五，扫描Nginx日志发送到kafka

 

tail -n 0 -f  /htlog/ht-video.log | /usr/local/kafka_2.10-0.9.0.1/bin/kafka-console-producer.sh --broker-list 192.168.50.42:9092 --topic test

 

六，java客户端消费日志

 

public static List<String> getKafkaData(){
    	List<String> list=new ArrayList<String>();
    	Properties props = new Properties();
    	props.put("zookeeper.connect", "192.168.50.42:2181");
    	props.put("group.id", "group");
    	props.put("zookeeper.session.timeout.ms", "40000");
    	props.put("zookeeper.sync.time.ms", "200");
    	props.put("auto.commit.interval.ms", "1000");
    	ConsumerConnector consumer = kafka.consumer.Consumer.createJavaConsumerConnector(new ConsumerConfig(props));
        Map<String, Integer> topicCountMap = new HashMap<String, Integer>();
        topicCountMap.put("test", new Integer(1));
        Map<String, List<KafkaStream<byte[], byte[]>>> consumerMap = consumer.createMessageStreams(topicCountMap);
        KafkaStream<byte[], byte[]> stream = consumerMap.get(Kafkaproperties.topic).get(0);
        ConsumerIterator<byte[], byte[]> it = stream.iterator();
        String message="GGGG";
        while (it.hasNext()) {
        	message=new String(it.next().message());
            System.out.println("receive：" + message);
            list.add(message);
        }
    	return list;
    }