Hadoop: hadoop-0.20.2.tar.gz
HBase: hbase-0.90.4.tar.gz
Hbase의 append 지원 문제로 Hadoop은 최신 버전이 아닌 이전 버전인 0.20.2 버전을 사용함
hadoop 계정을 생성하고
java version jdk 1.6.0_26을 다운로드하여 설치함
1. 환경 설정
[server]
192.168.186.131 ubuntu
[client]
192.168.186.132 ubuntu2
192.168.186.133 ubuntu3
[계정]
hadoop:hadoop
[java 설치 폴더]
/usr/lib/jvm/java-6-sun
[Hadoop 설치 폴더]
/usr/local/hadoop (hadoop-0.20.2 이름을 hadoop으로 변경함)
[Hbase 설치 폴더]
/usr/local/hbase (hbase-0.90.4 이름을 hbase 로 변경함)
2. 경로 등록
/home 폴더 이동 (hadoop home)
vi .bashrc
맨 아래에 다름과 같이 추가함
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export HADOOP_HOME=/usr/local/hadoop
export PATH=$JAVA_HOME/bin:$HADOOP_HOME/bin:$PATH:$HOME/bin
3. HADOOP 설치
root@xxx$ tar -xvf hadoop-0.20.2.tar.gz
root@xxx$ chown -R hadoop:hadoop hadoop-0.20.2
root@xxx$ mv hadoop-0.20.2 hadoop
[configuration]
- hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export HADOOP_HOME=/usr/local/hadoop
- core-site.xml
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://ubuntu:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/local/hadoop/tmp/hadoop-${user.name}</value>
</property>
</configuration>
- hdfs-site.xml
<configutation>
<property>
<name>dfs.name.dir</name>
<value>/home/hadoop/hdfs/name</value>
<property>
<name>dfs.data.dir</name>
<value>/home/hadoop/hdfs/data</value>
<property>
<name>dfs.replication</name>
<value>3</value>
</property>
</configuration>
- mapred-site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>ubuntu:9001</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/home/hadoop/hdfs/mapreduce/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/home/hadoop/hdfs/mapreduce/local</value>
</configuration>
- masters, slaves 구성
[masters]
ubuntu
[slaves]
ubuntu2
ubuntu3
4. iptables 설정(방화벽설정과 비슷함)
[master에서 설정]
$ iptables -A INPUT -p tcp -m tcp --dport 9000 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 9001 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50010 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50020 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50030 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50060 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50070 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50075 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 60010 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 60020 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 60030 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 2181 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 3888 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 2888 -j ACCEPT
$ iptables-save <----Save
$ iptables -L <----Load
(재부팅시 load되지 않는다. 내부에 저장하지는 않음]
[slaves에서 설정]
$ iptables -A INPUT -p tcp -m tcp --dport 9000 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 9001 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50010 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50020 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50030 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50060 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50070 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 50075 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 2181 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 3888 -j ACCEPT
$ iptables -A INPUT -p tcp -m tcp --dport 2888 -j ACCEPT
$ iptables save <----Save
$ iptables -L <----Load
5. SSH 설정
# ssh-keygen -t rsa -P ""
# cat /home/hadoop/.ssh/id_rsa.pub >> /home/hadoop/.ssh/authorized_keys
# ssh localhost
ssh: connect to host localhost port 22: Connection refused
error 발생시
다음과 같이 처리한다.
root@jin-laptop:~# sudo apt-get install openssh-server
-----------------
HBase 설정
/usr/local 에 설치
root@xxx$ tar -vxf hbase-0.90.4.tar.gz
root@xxx$ mv hbase-0.90.4 hbase
root@xxx$ chown hadoop:hadoop hbase
1) regionservers 구성시
conf/regionservers 에 해당 slave 추가
2) 환경 설정
- conf/hbase.env.sh
export JAVA_HOME=/usr/lib/jvm/java-6-sun
export HBASE_CLASSPATH=/usr/local/hadoop/conf
- conf/hbase-site.xml
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://ubuntu:9000/hbase</value>
</property>
<property>
<name>hbase.master</name>
<value>ubuntu:60000</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>ture</value>
</property>
<property>
<name>hbase.zookeeper.quorum</name>
<name>ubuntu</name>
</description>
</property>
3) Hadoop 의 hdfs-site.xml 설정파일을 복제한다.(혹은 link 처리함)
링크 : $ ln /usr/local/hadoop/conf/hdfs-site.xml /usr/local/hbase/conf/
복제 : $ cp /usr/local/hadoop/conf/hdfs-site.xml /usr/local/hbase/conf/
4) 실행
$ bin/start-hbase.sh
jps 로 확인
[master]
NameNode
JobTracker
HRegionServer
HQuorumPeer
HMaster
SecondaryNameNode
Jps
$ bin/hbase shell