CDH5 ベータ1 のNFSv3ゲートウェイを試してみました

HDFSのNFSv3 Gateway機能を試してみる

以前の記事にも書きましたが、Hadoop 2.2.0の新機能の一つに「NFSv3に対応」という点があります。「CDH5 beta1を速攻で試す!」でCDH5b1のパッケージをみたように、NFSv3 GatewayはCDH5 b1にも含まれています。制約は多いようですが、モノは試しに使ってみましょう。

パッケージのインストール

[root@localhost ~]# yum -y install hadoop-hdfs-nfs3
Loaded plugins: fastestmirror, refresh-packagekit, security
Loading mirror speeds from cached hostfile
 * base: centos.ustc.edu.cn
 * updates: mirror.neu.edu.cn
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package hadoop-hdfs-nfs3.x86_64 0:2.2.0+cdh5.0.0+353-0.cdh5b1.p0.79.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package Arch Version Repository Size
================================================================================
Installing:
 hadoop-hdfs-nfs3
 x86_64 2.2.0+cdh5.0.0+353-0.cdh5b1.p0.79.el6 cloudera-cdh5 4.7 k

Transaction Summary
================================================================================
Install 1 Package(s)

Total download size: 4.7 k
Installed size: 5.2 k
Downloading Packages:
hadoop-hdfs-nfs3-2.2.0+cdh5.0.0+353-0.cdh5b1.p0.79.el6.x | 4.7 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
 Installing : hadoop-hdfs-nfs3-2.2.0+cdh5.0.0+353-0.cdh5b1.p0.79.el6.x86 1/1
Unable to send message to PackageKit
 Verifying : hadoop-hdfs-nfs3-2.2.0+cdh5.0.0+353-0.cdh5b1.p0.79.el6.x86 1/1

Installed:
 hadoop-hdfs-nfs3.x86_64 0:2.2.0+cdh5.0.0+353-0.cdh5b1.p0.79.el6

Complete!
[root@localhost ~]#

hdfs-site.xmlの変更

hdfs-site.xmlに以下の修正を加えます

<property>
 <name>dfs.nfs3.dump.dir</name>
 <value>/tmp/.hdfs-nfs</value>
</property>

portmapを開始します

[root@localhost ~]# hadoop portmap
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

13/11/14 10:05:55 INFO portmap.Portmap: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting Portmap
STARTUP_MSG: host = localhost.localdomain/127.0.0.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 2.2.0-cdh5.0.0-beta-1
STARTUP_MSG: classpath = /etc/hadoop/conf:/usr/lib/hadoop/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop/lib/commons-codec-1.4.jar:/usr/lib/hadoop/lib/servlet-api-2.5.jar:/usr/lib/hadoop/lib/xz-1.0.jar:/usr/lib/hadoop/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop/lib/commons-digester-1.8.jar:/usr/lib/hadoop/lib/jersey-json-1.9.jar:/usr/lib/hadoop/lib/activation-1.1.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/jettison-1.1.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop/lib/commons-lang-2.5.jar:/usr/lib/hadoop/lib/jetty-6.1.26.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-configuration-1.6.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop/lib/commons-el-1.0.jar:/usr/lib/hadoop/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/commons-math-2.1.jar:/usr/lib/hadoop/lib/avro-1.7.4.jar:/usr/lib/hadoop/lib/stax-api-1.0.1.jar:/usr/lib/hadoop/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop/lib/jersey-core-1.9.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.5.jar:/usr/lib/hadoop/lib/commons-io-2.1.jar:/usr/lib/hadoop/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop/lib/commons-net-3.1.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/hue-plugins-3.0.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/lib/jersey-server-1.9.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop/lib/slf4j-log4j12.jar:/usr/lib/hadoop/lib/jsch-0.1.42.jar:/usr/lib/hadoop/lib/jsr305-1.3.9.jar:/usr/lib/hadoop/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop/lib/guava-11.0.2.jar:/usr/lib/hadoop/lib/log4j-1.2.17.jar:/usr/lib/hadoop/lib/junit-4.8.2.jar:/usr/lib/hadoop/lib/asm-3.2.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop/.//hadoop-nfs.jar:/usr/lib/hadoop/.//hadoop-common-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-nfs-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-auth-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop/.//hadoop-common-2.2.0-cdh5.0.0-beta-1-tests.jar:/usr/lib/hadoop/.//hadoop-annotations-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/commons-codec-1.4.jar:/usr/lib/hadoop-hdfs/lib/servlet-api-2.5.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/xmlenc-0.52.jar:/usr/lib/hadoop-hdfs/lib/commons-lang-2.5.jar:/usr/lib/hadoop-hdfs/lib/jetty-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-hdfs/lib/commons-el-1.0.jar:/usr/lib/hadoop-hdfs/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-hdfs/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.9.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.1.jar:/usr/lib/hadoop-hdfs/lib/jsp-api-2.1.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.9.jar:/usr/lib/hadoop-hdfs/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/guava-11.0.2.jar:/usr/lib/hadoop-hdfs/lib/log4j-1.2.17.jar:/usr/lib/hadoop-hdfs/lib/asm-3.2.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-2.2.0-cdh5.0.0-beta-1-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/xz-1.0.jar:/usr/lib/hadoop-yarn/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-yarn/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-yarn/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/paranamer-2.3.jar:/usr/lib/hadoop-yarn/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.9.jar:/usr/lib/hadoop-yarn/lib/netty-3.6.2.Final.jar:/usr/lib/hadoop-yarn/lib/guice-3.0.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-3.0.jar:/usr/lib/hadoop-yarn/lib/avro-1.7.4.jar:/usr/lib/hadoop-yarn/lib/hamcrest-core-1.1.jar:/usr/lib/hadoop-yarn/lib/jersey-core-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-io-2.1.jar:/usr/lib/hadoop-yarn/lib/jersey-server-1.9.jar:/usr/lib/hadoop-yarn/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-yarn/lib/log4j-1.2.17.jar:/usr/lib/hadoop-yarn/lib/junit-4.10.jar:/usr/lib/hadoop-yarn/lib/asm-3.2.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-site.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-2.2.0-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/./:/usr/lib/hadoop-0.20-mapreduce/lib/commons-collections-3.2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-codec-1.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/servlet-api-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xz-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-1.7.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-httpclient-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-digester-1.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-json-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/activation-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jline-0.9.94.jar:/usr/lib/hadoop-0.20-mapreduce/lib/kfs-0.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jettison-1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/xmlenc-0.52.jar:/usr/lib/hadoop-0.20-mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-lang-2.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-mapper-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/mockito-all-1.8.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/junit-4.8.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/paranamer-2.3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-configuration-1.6.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-core-asl-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-beanutils-core-1.8.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hadoop-fairscheduler-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-el-1.0.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-runtime-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-api-2.2.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jetty-util-6.1.26.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jets3t-0.6.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/ant-contrib-1.0b3.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-math-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/stax-api-1.0.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/hsqldb-1.8.0.10.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jasper-compiler-5.5.23.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-cli-1.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/zookeeper-3.4.5-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-logging-1.1.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/avro-compiler-1.7.4.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-core-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/slf4j-api-1.7.5.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-io-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-xc-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-net-3.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsp-api-2.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jersey-server-1.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jackson-jaxrs-1.8.8.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsch-0.1.42.jar:/usr/lib/hadoop-0.20-mapreduce/lib/jsr305-1.3.9.jar:/usr/lib/hadoop-0.20-mapreduce/lib/commons-compress-1.4.1.jar:/usr/lib/hadoop-0.20-mapreduce/lib/guava-11.0.2.jar:/usr/lib/hadoop-0.20-mapreduce/lib/log4j-1.2.17.jar:/usr/lib/hadoop-0.20-mapreduce/lib/asm-3.2.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-examples.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-test.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-examples-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-tools-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-test-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-core.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-ant-2.2.0-mr1-cdh5.0.0-beta-1.jar:/usr/lib/hadoop-0.20-mapreduce/.//hadoop-tools.jar
STARTUP_MSG: build = git://github.sf.cloudera.com/CDH/cdh.git -r ee825cb06b23d3ab97cdd87e13cbbb630bd75b98; compiled by 'jenkins' on 2013-10-28T00:09Z
STARTUP_MSG: java = 1.7.0_45
************************************************************/
13/11/14 10:05:55 INFO portmap.Portmap: registered UNIX signal handlers for [TERM, HUP, INT]
13/11/14 10:05:56 INFO oncrpc.SimpleUdpServer: Started listening to UDP requests at port 111 for Rpc program: portmap at localhost:111 with workerCount 1
13/11/14 10:05:56 INFO oncrpc.SimpleTcpServer: Started listening to TCP requests at port 111 for Rpc program: portmap at localhost:111 with workerCount 1

NFS3サービスを開始

[root@localhost ~]# /etc/init.d/hadoop-hdfs-nfs3 start
Starting Hadoop HDFS NFS v3 service:                       [  OK  ]
starting nfs3, logging to /var/log/hadoop-hdfs/hadoop-hdfs-nfs3-localhost.localdomain.out
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

[root@localhost ~]#

確認

rpcinfoを実行して確認します。

[root@localhost ~]# rpcinfo -p localhost
(PortmapMapping-100005:1:6:4242)
(PortmapMapping-100000:2:17:111)
(PortmapMapping-100005:3:6:4242)
(PortmapMapping-100005:2:17:4242)
(PortmapMapping-100003:3:6:2049)
(PortmapMapping-100000:2:6:111)
(PortmapMapping-100005:3:17:4242)
(PortmapMapping-100005:1:17:4242)
(PortmapMapping-100005:2:6:4242)
 program vers proto port service
 100005 1 tcp 4242 mountd
 100000 2 udp 111 portmapper
 100005 3 tcp 4242 mountd
 100005 2 udp 4242 mountd
 100003 3 tcp 2049 nfs
 100000 2 tcp 111 portmapper
 100005 3 udp 4242 mountd
 100005 1 udp 4242 mountd
 100005 2 tcp 4242 mountd
[root@localhost ~]#

うまくいっているようです。

マウント可能かどうかを確認

続いてshowmount -e でマウント可能になっているかを確認します。
(/etc/exportsのような細かい設定はどうやるんだろう?でも、今回は気にしないことに、、)

[root@localhost ~]# showmount -e localhost
Export list for localhost:
/ *
[root@localhost ~]#

ルートディレクトリが全員に公開されていますね。いけそうです。

マウントする

ではマウントしてみましょう。現状はtcpのみのようなので、明示的に指定します。

[root@localhost ~]# mount -t nfs -o vers=3,proto=tcp localhost:/ /media
13/11/14 10:11:07 WARN portmap.RpcProgramPortmap: Warning, no mapping for key: 100024 1 17
13/11/14 10:11:07 WARN oncrpc.RpcProgram: Invalid RPC call version 4
13/11/14 10:11:07 WARN oncrpc.RpcProgram: Invalid RPC call version 4
13/11/14 10:11:07 WARN oncrpc.RpcProgram: Invalid RPC call version 4
13/11/14 10:11:07 WARN portmap.RpcProgramPortmap: Warning, no mapping for key: 100024 1 17
mount.nfs: rpc.statd is not running but is required for remote locking.
mount.nfs: Either use '-o nolock' to keep locks local, or start statd.
mount.nfs: an incorrect mount option was specified

エラーになってしまいました orz。

エラーメッセージによると、nolockオプションが必要なようです。(statdを動かせば解消できるのかもしれませんが、未検証)

[root@localhost ~]# mount -t nfs -o vers=3,proto=tcp,nolock localhost:/ /media
[root@localhost ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
rootfs 14764384 6642572 7971816 46% /
devtmpfs 754480 152 754328 1% /dev
tmpfs 764396 264 764132 1% /dev/shm
/dev/sda1 14764384 6642572 7971816 46% /
localhost:/ 14764416 6792576 7971840 47% /media
[root@localhost ~]#

マウントできました!

動作確認

まずは一覧から。lsコマンドの際、(シェルでは当たり前の機能ですが、hadoop fsコマンドは補完してくれないので、やっぱり使えると嬉しい)タブ補完もできます。

[root@localhost ~]# ls /media/user/training/movierating/
_logs part-m-00000 part-m-00001 part-m-00002 part-m-00003 _SUCCESS
[root@localhost ~]# cat /media/user/training/movierating/part-m-00000|more
1 1193 5
1 661 3
1 914 3
1 3408 4
1 2355 5
1 1197 3
1 1287 5
1 2804 5
1 594 4
1 919 4
1 595 5
1 938 4
1 2398 4
1 2918 4
1 1035 5
1 2791 4
1 2687 3
1 2018 4
1 3105 5
1 2797 4
1 2321 3
1 720 3
1 1270 5
--More--

現状はアプリからファイルの作成は推奨されていないようなので、コピーをしてみます。

[root@localhost ~]# cp /etc/passwd /media/user/training/
cp: cannot create regular file `/media/user/training/passwd': Permission denied
[root@localhost ~]#

失敗してしまいました。
ドキュメントによると、ログインしているユーザーのuser IDを使用して、NFSv3 GatewayがIDに対応したユーザー名でHDFSにアクセスするとのこと。rootユーザーの権限がないフォルダにコピーしようとしたのが原因のようです。なので別のユーザー(training)でやってみましょう。

OS標準のコマンドでも、hadoop fsコマンドでもアクセスできますね。

[training@localhost work]$ cp /etc/passwd /media/user/training
[training@localhost work]$ head -3 /media/user/training/passwd
root:x:0:0:root:/root:/bin/bash
bin:x:1:1:bin:/bin:/sbin/nologin
daemon:x:2:2:daemon:/sbin:/sbin/nologin
[training@localhost work]$ ls -l /media/user/training/passwd
-rw-r--r-- 1 training 2584148964 2336 Nov 14 10:19 /media/user/training/passwd
[training@localhost work]$ hadoop fs -ls /user/training/passwd
Found 1 items
-rw-r--r-- 1 training supergroup 2336 2013-11-14 10:19 /user/training/passwd
[training@localhost work]$

ただ、グループ名(supergroup)が/etc/groupにないためか、変な表示になっています。/etc/groupに追加しても改善しなかったので、これは後で確認しないと。

逆方向(HDFS->ローカルファイルシステムもバッチリ)

[training@localhost work]$ cp /media/user/training/xyz/part-r-00000 /tmp
[training@localhost work]$ ls -l /tmp/part-r-00000
-rw-r--r-- 1 training training 51872 Nov 14 10:20 /tmp/part-r-00000
[training@localhost work]$

おまけ

アプリの書き込みは非推奨と書かれていましたが、試しにやってみます。

[training@localhost work]$ echo "1234567890" > /media/user/training/memo.txt
[training@localhost work]$ cat /media/user/training/memo.txt
1234567890
[training@localhost work]$ echo "9876543210" > /media/user/training/memo.txt
bash: /media/user/training/memo.txt: Invalid argument
[training@localhost work]$ echo "9876543210" > /media/user/training/memo.txt2
[training@localhost work]$

ライトワンスなので、上書きはできません。しかし、

[training@localhost work]$ cat /etc/hosts >> /media/user/training/memo.txt2
[training@localhost work]$ cat /media/user/training/memo.txt2
9876543210
127.0.0.1 huetest
[training@localhost work]$

追記はできる!?fsckで見てみると、書き込み中になっているような感じ。

[training@localhost work]$ sudo -u hdfs hadoop fsck /user/training/memo.txt2 -files -blocks -locations
DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

Connecting to namenode via http://localhost:50070
FSCK started by hdfs (auth:SIMPLE) from /127.0.0.1 for path /user/training/memo.txt2 at Thu Nov 14 10:37:45 EST 2013
Status: HEALTHY
 Total size: 0 B (Total open files size: 30 B)
 Total dirs: 0
 Total files: 0
 Total symlinks: 0 (Files currently being written: 1)
 Total blocks (validated): 0 (Total open file blocks (not validated): 1)
 Minimally replicated blocks: 0
 Over-replicated blocks: 0
 Under-replicated blocks: 0
 Mis-replicated blocks: 0
 Default replication factor: 1
 Average block replication: 0.0
 Corrupt blocks: 0
 Missing replicas: 0
 Number of data-nodes: 1
 Number of racks: 1
FSCK ended at Thu Nov 14 10:37:45 EST 2013 in 1 milliseconds

The filesystem under path '/user/training/memo.txt2' is HEALTHY

まとめ

ちょっと試してみただけですが、この新しいNFSv3は普通に使えますね。
パフォーマンスやセキュリティがどうなのか、その他の制約はどうかということが気になりますが、手元の環境では全く問題なく動作させることができたので、皆さんも是非気軽に試してみて下さい。

Pocket

Leave a Reply

Your email address will not be published. Required fields are marked *

CAPTCHA


日本語が含まれない投稿は無視されますのでご注意ください。(スパム対策)