HDFS Handling
The Hadoop Distributed File System (HDFS) is a filesystem that is supported by the nodes in the Hadoop compute cluster. HDFS has numerous similarities to typical filesystems such as those of the *nix variety. The hadoop CLI utility provides a collection of commands that can be run from the local command line.
Gremlin provides methods to easily interact with both the local file system and HDFS from within the REPL. These methods are provided in the table below. Note that the global variables local and hdfs reference the local and HDFS file systems, respectively. Finally, pattern refers to a regular expression pattern and path refers to an explicit path to a file or directory.
| Method | Description |
|---|---|
ls() |
list all the contents in the home directory |
ls(pattern) |
list all the contents that match the pattern |
exists(path) |
return true or false on whether path exists |
rm(pattern) |
remove all paths that match the pattern |
rmr(pattern) |
recursively remove all paths that match the pattern |
copyToLocal(from,to) |
copy the HDFS path to the local filesystem |
copyFromLocal(from,to) |
copy the local path to HDFS |
mergeToLocal(from,to) |
merge the files at hdfs path to a single file on the local filesystem |
head(pattern,lines?) |
look at the top ?lines of the files that match pattern |
unzip(from,to) |
BZ2 unzip the path to another path |