Java Program to Write File in HDFS

In this post we’ll see a Java program to write a file in HDFS. You can write a file in HDFS in two ways-

1. Create an object of FSDataOutputStream and use that object to write data to file.
2. You can use IOUtils class provided by Hadoop framework.

Writing a file in HDFS using FSDataOutputStream

package org.netjs;

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class HDFSFileWrite {

 public static void main(String[] args) {
  Configuration conf = new Configuration();
  FSDataInputStream in = null;
  FSDataOutputStream out = null;

  try {
   FileSystem fs = FileSystem.get(conf);
   // Input & Output file paths
   Path inFile = new Path(args[0]);
   Path outFile = new Path(args[1]);
   // check if file exists
   if (!fs.exists(inFile)) {
    System.out.println("Input file not found");
    throw new IOException("Input file not found");
   }
   if (fs.exists(outFile)) {
    System.out.println("Output file already exists");
    throw new IOException("Output file already exists");
   }
   in = fs.open(inFile);
   out = fs.create(outFile);

   byte buffer[] = new byte[256];

   int bytesRead = 0;
   while ((bytesRead = in.read(buffer)) > 0) {
    out.write(buffer, 0, bytesRead);
     }


  } catch (IOException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }finally {
   try {
    if(in != null) {    
     in.close();    
    }
    if(out != null) {
     out.close();
    }
   } catch (IOException e) {
    // TODO Auto-generated catch block
    e.printStackTrace();
   }
  }
 }
}

export HADOOP_CLASSPATH=<PATH TO .class FILE>

To run program - hadoop org.netjs.HDFSFileWrite /user/process/display.txt /user/process/writeFile.txt

Writing a file in HDFS using IOUtils class

import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IOUtils;

public class HDFSFileWrite {

 public static void main(String[] args) {
  Configuration conf = new Configuration();
  FSDataInputStream in = null;
  FSDataOutputStream out = null;

  try {
   FileSystem fs = FileSystem.get(conf);
   // Input & Output file paths
   Path inFile = new Path(args[0]);
   Path outFile = new Path(args[1]);
   // check if file exists
   if (!fs.exists(inFile)) {
    System.out.println("Input file not found");
    throw new IOException("Input file not found");
   }
   if (fs.exists(outFile)) {
    System.out.println("Output file already exists");
    throw new IOException("Output file already exists");
   }

   in = fs.open(inFile);
   out = fs.create(outFile);
   IOUtils.copyBytes(in, out, 512, false);


  } catch (IOException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
  }finally {
   IOUtils.closeStream(in);
   IOUtils.closeStream(out);
  }
 }
}

That's all for this topic Java Program to Write File in HDFS. If you have any doubt or any suggestions to make please drop a comment. Thanks!

Related Topics

1. Java Program to Read File in HDFS
2. File Write in HDFS - Hadoop Framework Internal Steps
3. HDFS Commands Reference List
4. What is SafeMode in Hadoop
5. NameNode, DataNode And Secondary NameNode in HDFS

You may also like -

• Installing Hadoop on a Single Node Cluster in Pseudo-Distributed Mode
• Word Count MapReduce Program in Hadoop
• Speculative Execution in Hadoop
• MapReduce Flow in YARN
• Capacity Scheduler in YARN
• How to Create Ubuntu Bootable USB
• What is Dependency Injection in Spring
• Writing File in Java

>>>Go to Hadoop Framework Page