【zookeeper】zookeeper单机和集群环境的搭建

首先去官网下载zookeeper安装包,本文采用3.4.6stable版本

http://zookeeper.apache.org/doc/r3.4.6/


tar命令解压后的目录结构如下:

[root@com23 zookeeper-3.4.6]# ll
total 1612
drwxr-xr-x  2 1000 1000    4096 Feb 20  2014 bin
-rw-rw-r--  1 1000 1000   82446 Feb 20  2014 build.xml
-rw-rw-r--  1 1000 1000   80776 Feb 20  2014 CHANGES.txt
drwxr-xr-x  2 1000 1000    4096 Jan 22 11:39 conf
drwxr-xr-x 10 1000 1000    4096 Feb 20  2014 contrib
drwxr-xr-x  2 1000 1000    4096 Feb 20  2014 dist-maven
drwxr-xr-x  6 1000 1000    4096 Feb 20  2014 docs
-rw-rw-r--  1 1000 1000    1953 Feb 20  2014 ivysettings.xml
-rw-rw-r--  1 1000 1000    3375 Feb 20  2014 ivy.xml
drwxr-xr-x  4 1000 1000    4096 Feb 20  2014 lib
-rw-rw-r--  1 1000 1000   11358 Feb 20  2014 LICENSE.txt
-rw-rw-r--  1 1000 1000     170 Feb 20  2014 NOTICE.txt
-rw-rw-r--  1 1000 1000    1770 Feb 20  2014 README_packaging.txt
-rw-rw-r--  1 1000 1000    1585 Feb 20  2014 README.txt
drwxr-xr-x  5 1000 1000    4096 Feb 20  2014 recipes
drwxr-xr-x  8 1000 1000    4096 Feb 20  2014 src
-rw-rw-r--  1 1000 1000 1340305 Feb 20  2014 zookeeper-3.4.6.jar
-rw-rw-r--  1 1000 1000     836 Feb 20  2014 zookeeper-3.4.6.jar.asc
-rw-rw-r--  1 1000 1000      33 Feb 20  2014 zookeeper-3.4.6.jar.md5
-rw-rw-r--  1 1000 1000      41 Feb 20  2014 zookeeper-3.4.6.jar.sha1
-rw-r--r--  1 root root   46852 Jan 22 11:42 zookeeper.out
我们进入conf目录下新建zoo.cfg配置文件,文件名随便都行,但是默认是这个文件名,为什么呢?

看zkEnv.sh就知道了

if [ "x$ZOOCFG" = "x" ]
then
    ZOOCFG="zoo.cfg"
fi

1、单机模式

文件内容如下:

#the basic time unit in milliseconds used by zookeeper,it is used to do heartbeats and the min session timeout will be twice the tickTime
tickTime=2000
#the location to store the in-memory database snapshots and ,unless specified otherwise,the transaction log of updates to database
dataDir=/home/zookeeper/zkData
#the port to listen for client connections
clientPort=2181
至此,单机配置就结束了,下面可以启动了

bin/zkServer.sh start
启动完成后,通过如下命令连接上去:

bin/zkCli.sh -server serverIp:2181
当然你的客户端肯定要装个zk啦
[zk: localhost:2181(CONNECTED) 27] ll
ZooKeeper -server host:port cmd args
        connect host:port
        get path [watch]
        ls path [watch]
        set path data [version]
        rmr path
        delquota [-n|-b] path
        quit 
        printwatches on|off
        create [-s] [-e] path data acl
        stat path [watch]
        close 
        ls2 path [watch]
        history 
        listquota path
        setAcl path acl
        getAcl path
        sync path
        redo cmdno
        addauth scheme auth
        delete path [version]
        setquota -n|-b val path
至此,单机模式就结束了!


2、集群模式

修改配置文件如下:

#the basic time unit in milliseconds used by zookeeper,it is used to do heartbeats and the min session timeout will be twice the tickTime
tickTime=2000
#the location to store the in-memory database snapshots and ,unless specified otherwise,the transaction log of updates to database
dataDir=/home/zookeeper/zkData
#the port to listen for client connections
clientPort=2181
#timeouts zookeeper uses to limit the length of time the zookeeper servers in quorum have to connect to a leader
initLimit=5
#limits how far out of date a server can be from a leader
syncLimit=2
#With both of above timeouts, you specify the unit of time using tickTime. In this example, the timeout for initLimit is 5 ticks at 2000 milleseconds a tick, or 10 seconds.
server.1=192.168.11.176:2888:3888
server.2=192.168.11.177:2888:3888
server.3=192.168.11.178:2888:3888
The entries of the form server.X list the servers that make up the ZooKeeper service. When the server starts up, it knows which server it is by looking for the file myid in the data directory. That file has the contains the server number, in ASCII.
这句话的意思,server.X中的x是从dataDir路径下的myid文件来读取的,所以我们要创建相应的myid文件,内容分别是1,2,3


至此,集群的配置已经完成了!

下面到各个节点机器上去启动zookeeper

sh bin/zkServer.sh start
可以去看zookeeper.out的启动日志输出

查看节点状态

[root@com23 zookeeper-3.4.6]# sh bin/zkServer.sh status
JMX enabled by default
Using config: /home/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: leader
[root@com22 zookeeper-3.4.6]# sh bin/zkServer.sh status
JMX enabled by default
Using config: /home/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: follower
[root@com21 zookeeper-3.4.6]# sh bin/zkServer.sh status
JMX enabled by default
Using config: /home/zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
Mode: follower
三个节点,一个leader,两个follower

这里补充一点:zookeeper的集群必须得奇数个节点,因为zookeeper有个机制,当集群中大于半数的节点挂了,整个zookeeper将停止服务,如果说是偶数台的话,一半对一半,zookeeper很难判断,所以zookeeper的集群建议偶数台搭建。


下面另起一个客户端去连接zk集群:

[root@com20 zookeeper-3.4.6]# sh bin/zkCli.sh -server 192.168.11.178:2181
Connecting to 192.168.11.178:2181
2015-01-22 14:41:08,947 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
2015-01-22 14:41:08,975 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=com20.authentication
2015-01-22 14:41:08,975 [myid:] - INFO  [main:Environment@100] - Client environment:java.version=1.7.0_60
2015-01-22 14:41:08,989 [myid:] - INFO  [main:Environment@100] - Client environment:java.vendor=Oracle Corporation
2015-01-22 14:41:08,990 [myid:] - INFO  [main:Environment@100] - Client environment:java.home=/usr/java/jdk1.7.0_67/jre
2015-01-22 14:41:08,991 [myid:] - INFO  [main:Environment@100] - Client environment:java.class.path=/home/zookeeper/zookeeper-3.4.6/bin/../build/classes:/home/zookeeper/zookeeper-3.4.6/bin/../build/lib/*.jar:/home/zookeeper/zookeeper-3.4.6/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/zookeeper/zookeeper-3.4.6/bin/../lib/slf4j-api-1.6.1.jar:/home/zookeeper/zookeeper-3.4.6/bin/../lib/netty-3.7.0.Final.jar:/home/zookeeper/zookeeper-3.4.6/bin/../lib/log4j-1.2.16.jar:/home/zookeeper/zookeeper-3.4.6/bin/../lib/jline-0.9.94.jar:/home/zookeeper/zookeeper-3.4.6/bin/../zookeeper-3.4.6.jar:/home/zookeeper/zookeeper-3.4.6/bin/../src/java/lib/*.jar:/home/zookeeper/zookeeper-3.4.6/bin/../conf:.:/usr/java/jdk1.7.0_67/lib:/usr/java/jdk1.7.0_67/lib/tools.jar:/usr/java/jdk1.7.0_67/lib/dt.jar:/usr/java/jdk1.7.0_67/jre/lib:/usr/java/jdk1.7.0_67/jre/lib/charsets.jar:/usr/java/jdk1.7.0_67/jre/lib/rt.jar
2015-01-22 14:41:08,991 [myid:] - INFO  [main:Environment@100] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2015-01-22 14:41:08,992 [myid:] - INFO  [main:Environment@100] - Client environment:java.io.tmpdir=/tmp
2015-01-22 14:41:08,993 [myid:] - INFO  [main:Environment@100] - Client environment:java.compiler=<NA>
2015-01-22 14:41:08,993 [myid:] - INFO  [main:Environment@100] - Client environment:os.name=Linux
2015-01-22 14:41:08,994 [myid:] - INFO  [main:Environment@100] - Client environment:os.arch=amd64
2015-01-22 14:41:08,995 [myid:] - INFO  [main:Environment@100] - Client environment:os.version=2.6.18-308.el5
2015-01-22 14:41:08,995 [myid:] - INFO  [main:Environment@100] - Client environment:user.name=root
2015-01-22 14:41:08,996 [myid:] - INFO  [main:Environment@100] - Client environment:user.home=/root
2015-01-22 14:41:08,996 [myid:] - INFO  [main:Environment@100] - Client environment:user.dir=/home/zookeeper/zookeeper-3.4.6
2015-01-22 14:41:09,000 [myid:] - INFO  [main:ZooKeeper@438] - Initiating client connection, connectString=192.168.11.178:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@8afbefd
Welcome to ZooKeeper!
2015-01-22 14:41:09,187 [myid:] - INFO  [main-SendThread(192.168.11.178:2181):ClientCnxn$SendThread@975] - Opening socket connection to server 192.168.11.178/192.168.11.178:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2015-01-22 14:41:09,243 [myid:] - INFO  [main-SendThread(192.168.11.178:2181):ClientCnxn$SendThread@852] - Socket connection established to 192.168.11.178/192.168.11.178:2181, initiating session
[zk: 192.168.11.178:2181(CONNECTING) 0] 2015-01-22 14:41:09,481 [myid:] - INFO  [main-SendThread(192.168.11.178:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server 192.168.11.178/192.168.11.178:2181, sessionid = 0x34b1057cbfa0000, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null

[zk: 192.168.11.178:2181(CONNECTED) 0] 
[zk: 192.168.11.178:2181(CONNECTED) 1] ls /
[zookeeper, zk_test]
[zk: 192.168.11.178:2181(CONNECTED) 2] get /zk_test
my_data
cZxid = 0x5
ctime = Thu Jan 22 13:52:58 CST 2015
mZxid = 0x5
mtime = Thu Jan 22 13:52:58 CST 2015
pZxid = 0x5
cversion = 0
dataVersion = 0
aclVersion = 0
ephemeralOwner = 0x0
dataLength = 7
numChildren = 0
[zk: 192.168.11.178:2181(CONNECTED) 3] 
如果你的客户端只是去连zk集群,只需要解压zk安装包即可,使用命令直接可以连接,无需任何配置!


如遇任何问题,建议大家去官网参考,官网的永远是权威!

http://zookeeper.apache.org/doc/r3.4.6/zookeeperStarted.html


共勉!



发布了223 篇原创文章 · 获赞 209 · 访问量 72万+
展开阅读全文

kafka 集高可用测试

05-15

先搭建了3个kafka集群 3台都启动 并且正常发送消息后 测试 某一台挂了情况 ``` #当前机器在集群中的唯一标识,和zookeeper的myid性质一样 broker.id=0 #当前kafka对外提供服务的端口默认是9092 listeners=PLAINTEXT://192.168.1.252:9092 #这个是borker进行网络处理的线程数 num.network.threads=3 #这个是borker进行I/O处理的线程数 num.io.threads=8 #发送缓冲区buffer大小,数据不是一下子就发送的,先回存储到缓冲区了到达一定的大小后在发送,能提高性能 socket.send.buffer.bytes=102400 #kafka接收缓冲区大小,当数据到达一定大小后在序列化到磁盘 socket.receive.buffer.bytes=102400 #这个参数是向kafka请求消息或者向kafka发送消息的请请求的最大数,这个值不能超过java的堆栈大小 socket.request.max.bytes=104857600 ############################# Log Basics ############################# #消息存放的目录,这个目录可以配置为","逗号分割的表达式,上面的num.io.threads要大于这个目录的个数这个目录, #如果配置多个目录,新创建的topic他把消息持久化的地方是,当前以逗号分割的目录中,那个分区数最少就放那一个 log.dirs=/usr/local/kafka_2.11-1.1.0/data/kafka/logs #默认的分区数,一个topic默认1个分区数 num.partitions=3 # The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. # This value is recommended to be increased for installations with data dirs located in RAID array. num.recovery.threads.per.data.dir=1 ############################# Internal Topic Settings ############################# # The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state" # For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3. offsets.topic.replication.factor=1 # transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 #默认消息的最大持久化时间,168小时,7天 log.retention.hours=168 # A size-based retention policy for logs. Segments are pruned from the log unless the remaining # segments drop below log.retention.bytes. Functions independently of log.retention.hours. #log.retention.bytes=1073741824 #这个参数是:因为kafka的消息是以追加的形式落地到文件,当超过这个值的时候,kafka会新起一个文件 log.segment.bytes=1073741824 #每隔300000毫秒去检查上面配置的log失效时间(log.retention.hours=168 ),到目录查看是否有过期的消息如果有,删除 log.retention.check.interval.ms=300000 ############################# Zookeeper ############################# #设置zookeeper的连接端口 zookeeper.connect=192.168.1.252:2181,192.168.1.253:2181,192.168.1.254:2181 # 超时时间 zookeeper.connection.timeout.ms=6000 ############################# Group Coordinator Settings ############################# # The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance. # The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms. # The default value for this is 3 seconds. # We override this to 0 here as it makes for a better out-of-the-box experience for development and testing. # However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup. group.initial.rebalance.delay.ms=0 ``` 创建了 主题 ![图片说明](https://img-ask.csdn.net/upload/201805/15/1526363032_696232.png) 然后 发送消息 ![图片说明](https://img-ask.csdn.net/upload/201805/15/1526363075_21045.png) msg1 msg2 然后 关闭其中一台kafka ![图片说明](https://img-ask.csdn.net/upload/201805/15/1526363143_679462.png) 控制台打印 ``` 2018/05/15-13:48:03 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator- Marking the coordinator 192.168.1.254:9092 (id: 2147483645 rack: null) dead for group defaultGroup 2018/05/15-13:48:03 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator- Discovered coordinator 192.168.1.254:9092 (id: 2147483645 rack: null) for group defaultGroup. 2018/05/15-13:48:04 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator- Marking the coordinator 192.168.1.254:9092 (id: 2147483645 rack: null) dead for group defaultGroup ``` 然后继续发消息 msg3 控制台无响应 继续发送msg4 无响应 重启那台kafka 控制台打印 ``` "msg4" "msg3" 2018/05/15-13:52:11 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN org.apache.kafka.clients.consumer.internals.Fetcher- Received unknown topic or partition error in fetch for partition trading-1. The topic/partition may not exist or the user may not have Describe access to it 2018/05/15-13:52:11 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator- Marking the coordinator 192.168.1.254:9092 (id: 2147483645 rack: null) dead for group defaultGroup 2018/05/15-13:52:11 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] WARN org.apache.kafka.clients.consumer.internals.ConsumerCoordinator- Auto offset commit failed for group defaultGroup: Offset commit failed with a retriable exception. You should retry committing offsets. 2018/05/15-13:52:11 [org.springframework.kafka.KafkaListenerEndpointContainer#0-0-C-1] INFO org.apache.kafka.clients.consumer.internals.AbstractCoordinator- Discovered coordinator 192.168.1.254:9092 (id: 2147483645 rack: null) for group defaultGroup. ``` kafka 不是主从复制么 不是谁挂了 用另一个么 问答

zookeeper后,运行zkCli.sh资源无法定位

12-06

zoo服务器开着呢.. sudo netstat -anp | grep 2181 tcp6 0 0 :::2181 :::* LISTEN 3603/java tcp6 0 0 ::1:2181 ::1:34411 TIME_WAIT - [root@localhost bin]# sudo netstat -anp | grep 2182 tcp6 0 0 :::2182 :::* LISTEN 3624/java tcp6 0 0 10.35.162.95:56986 10.35.162.95:2182 ESTABLISHED 5583/java tcp6 0 0 10.35.162.95:2182 10.35.162.95:56986 ESTABLISHED 3624/java [root@localhost bin]# sudo netstat -anp | grep 2183 tcp6 0 0 :::2183 :::* LISTEN 3645/java 这是我的出错信息 2017-12-06 21:01:16,204 [myid:] - INFO [main:ZooKeeper@438] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@799f7e29 2017-12-06 21:01:16,254 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@975] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) Welcome to ZooKeeper! JLine support is enabled 2017-12-06 21:01:16,452 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@852] - Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session 2017-12-06 21:01:16,471 [myid:] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1235] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x1602b8fa3380004, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null 问答

没有更多推荐了,返回首页

©️2019 CSDN 皮肤主题: 酷酷鲨 设计师: CSDN官方博客

分享到微信朋友圈

×

扫一扫,手机浏览