SharEDITor

教你成为全栈工程师(Full Stack Developer) 二十九-在storm上运行python程序(修正)

全栈技术 教你成为全栈工程师(Full Stack Developer) 发表于 2016-05-16 16:54:21 阅读1277次


上一节部署python程序出现了一些错误,没能正确运行,本节基于稳定版storm做了修正,运行成功

请尊重原创,转载请注明来源网站www.shareditor.com以及原始链接地址

部署0.9.5版本的storm集群

 

streamparse最新稳定版是基于0.9.5版本的storm的,所以我们需要把storm集群的版本回退到0.9.5,方法如下:

wget http://apache.fayea.com/storm/apache-storm-0.9.5/apache-storm-0.9.5.tar.gz

解压后修改conf/storm.yaml文件,添加如下配置项:

storm.zookeeper.servers:
    - "127.0.0.1"
nimbus.seeds: ["127.0.0.1"]
supervisor.slots.ports:
    - 6700
    - 6701
    - 6702
    - 6703

启动,执行:

[root@centos7vm apache-storm-0.9.5]# ./bin/storm nimbus &
[root@centos7vm apache-storm-0.9.5]# ./bin/storm supervisor &
[root@centos7vm apache-storm-0.9.5]# ./bin/storm ui &

启动需要花费数秒钟时间,直到打开http://localhost:8080能正常显示web页说明启动正常

 

创建wordcount并修改配置

 

确定安装的streamparse版本是

[root@centos7vm tmp]# sparse --version
sparse 2.1.4

执行

[root@centos7vm tmp]# sparse quickstart wordcount

生成了wordcount目录,进入wordcount目录修改config.json文件
请尊重原创,转载请注明来源网站www.shareditor.com以及原始链接地址

把nimbus和workers配置成你的storm机器(填写别名或ip,需要提前建立好ssh host的无密码登陆),比如我的是:

{
    "library": "",
    "topology_specs": "topologies/",
    "virtualenv_specs": "virtualenvs/",
    "envs": {
        "prod": {
            "user": "",
            "nimbus": "centos",
            "workers": ["centos"],
            "log": {
                "path": "",
                "max_bytes": 1000000,
                "backup_count": 10,
                "level": "info"
            },
            "virtualenv_root": "/root/tmp/wordcount/virtualenvs/"
        }
    }
}

 

生成jar包

 

执行

[root@centos7vm wordcount]# sparse jar

会看到生成了_build/wordcount-0.0.1-SNAPSHOT-standalone.jar文件

 

部署任务

 

执行

[root@centos7vm wordcount]# storm jar _build/wordcount-0.0.1-SNAPSHOT-standalone.jar streamparse.commands.submit_topology topologies/wordcount.clj

 

查看日志

[root@centos7vm wordcount]# tail /data/apache-storm-0.9.5/logs/worker-6700.log
2016-05-16T08:53:38.579+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt elephant: 656
2016-05-16T08:53:38.580+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt dog: 690
2016-05-16T08:53:38.585+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt zebra: 659
2016-05-16T08:53:38.588+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt dog: 691
2016-05-16T08:53:38.595+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt elephant: 657
2016-05-16T08:53:38.596+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt dog: 692
2016-05-16T08:53:38.604+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt elephant: 658
2016-05-16T08:53:38.606+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt dog: 693
2016-05-16T08:53:38.635+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt zebra: 660
2016-05-16T08:53:38.640+0800 b.s.t.ShellBolt [INFO] ShellLog pid:74811, name:count-bolt dog: 694

看到如上日志,说明正常运行了