Frequently used HDFS / Hadoop shell commands

  1. Hadoop version
  2. Contents of the root directory in HDFS
  3. Amount of space used and available on currently mounted filesystem
  4. Number of directories, files and bytes under the paths that match the specified file pattern
  5. DFS filesystem checking utility
  6. A cluster balancing utility
  7. Create a new directory named "data" below the /user/ranjank directory in HDFS.
  8. Add a sample text file from the local directory named "sample_data.csv" to the new directory you created in HDFS during the previous step.
  9. List the contents of this new directory in HDFS.
  10. Add the entire local directory called "examples" to the /user/ranjank/data directory in HDFS.
  11. How much space this directory occupies in HDFS.
  12. Delete a file “abc.txt" from the "examples" directory.
  13. Ensure this file is no longer in HDFS.
  14. Delete all files from the "example1" directory using a wildcard.
  15. Empty the trash
  16. Remove the entire "example" directory and all of its contents in HDFS.
  17. Add the abc.txt file from the local directory named "/home/hduser/examples/" to the hadoop directory you created in HDFS
  18. To view the contents of your text file abc.txt which is present in your data directory.
  19. Add the abc.txt file from "data" directory which is present in HDFS directory to the current directory in the local directory.
  20. cp is used to copy files between directories present in HDFS
  21. 'get' command can be used alternaively to ‘-copyToLocal’ command
  22. Display last kilobyte of the file "abc.txt" to stdout.
  23. Default file permissions are 666 in HDFS. Use '-chmod' command to change permissions of a file
  24. Use '-chown' to change owner name and group name simultaneously
  25. Use '-chgrp' command to change group name
  26. Move a directory from one location to other
  27. Default replication factor to a file is 3. Use '-setrep' command to change replication factor of a file
  28. Copy a directory from one node in the cluster to another. Use '-distcp' command to copy,
    overwrite option to overwrite in an existing files
    update command to synchronize both directories
  29. Command to make the name node leave safe mode

  30. List all the hadoop file system shell commands
  31. Last but not least, always ask for help!

Leave a Comment

Your email address will not be published. Required fields are marked *

Time limit is exhausted. Please reload CAPTCHA.

Fork me on GitHub