Jun 24

window下开发hadoop2.6,服务端在linux报Could not locate executable null的解决办法 不指定

admin , 12:20 , Hadoop , 评论(0) , 引用(0) , 阅读(278) , Via 本站原创 | |
搜索
我已经获得阿里云幸运券,准备分享给您。请点击获取   window下开发hadoop2.6,服务端在linux报Could not locate executable null的解决办法
 在window下文件是空的,文件名有创建
2017-06-23 23:07:18,228 WARN  [main] util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-06-23 23:07:18,257 ERROR [main] util.Shell (Shell.java:getWinUtilsPath(396)) - Failed to locate the winutils binary in the hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
 at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:378)
 at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:393)
 at org.apache.hadoop.util.Shell.<clinit>(Shell.java:386)

查看hadoop源码Shell.java:
private static String checkHadoopHome() {
  String home = System.getProperty("hadoop.home.dir");
  if (home == null)
   home = System.getenv("HADOOP_HOME");
  try {
   if (home == null)
    throw new IOException(
      "HADOOP_HOME or hadoop.home.dir are not set.");
   if (home.startsWith("\"") && home.endsWith("\""))
   
 public static final String getQualifiedBinPath(String executable)
   throws IOException {
  String fullExeName = (new StringBuilder()).append(HADOOP_HOME_DIR)
    .append(File.separator).append("bin").append(File.separator)
    .append(executable).toString();
  File exeFile = new File(fullExeName);
  if (!exeFile.exists())
   throw new IOException((new StringBuilder()).append(
     "Could not locate executable ").append(fullExeName).append(
     " in the Hadoop binaries.").toString());
  else
   return exeFile.getCanonicalPath();
 } 
解决方法:
    在你的window环境下新建一个目录,比如我的目录是D:\mysoft\Hadoop2.6_home
 把hadoop的源码包dist 路径bin和lib下的winutils.exe 都拷贝到你刚刚创建的目录。
 右键我的电脑属性,添加环境变量HADOOP_HOME目录下,(注意:如果你的hadoop服务端是在window,则估计是覆盖吧)。

 重启电脑使环境变量生效。
 根据HADOOP的源代码查看得知,如果不想重启电脑,可以先临时在hadoop脚本中设置变量hadoop.home.dir 的值。
 如
 System.setProperty("hadoop.home.dir", "D:\\mysoft\\Hadoop2.6_home\\");
 且需要在FileSystem.get方法之前,否则无效。
 FileSystem fs2 = FileSystem.get(uri, conf2, user);//.get(uri, conf2, "hadoop");
自此Could not locate executable null\bin\winutils.exe in the Hadoop binaries.问题解决
----end by itlife365.com
Exception in thread "main" org.apache.hadoop.security.AccessControlException: Permission denied: user=Administrator, access=WRITE, inode="/user_ljs/hadoop/output":hadoop:supergroup:drwxr-xr-x
的错误:指定hadoop用户。


public static void main(String[] args) throws IOException, URISyntaxException, InterruptedException {
  //upload a local file to hdfs
  
  //get a configruation object
  Configuration conf = new Configuration();
  Configuration conf2 = new Configuration();
  //to set a parameter,figure out the file system is hdfs
  conf.set("fs.defaultFS", "hdfs://itlife365:9000/");//fs.defaultFS
  conf.set("dfs.replication", "3");
  // get a instance of Hdfs FileSystem
  
  //FileSystem fs = FileSystem.get(conf);//window下此次要注释,在hadoop.home.dir 设置前不能使用
  URI uri = new URI("hdfs://itlife365:9000");
  String user ="hadoop";
  System.setProperty("hadoop.home.dir", "D:\\mysoft\\Hadoop2.6_home\\");
  FileSystem fs2 = FileSystem.get(uri, conf2, user);//.get(uri, conf2, "hadoop");
  
  //open a file stream of the dest
  Path destFile=new Path("hdfs://itlife365:9000/test.txt");
  FSDataOutputStream os = fs2.create(destFile);
  
  //open a inputstream of the local source file
  FileInputStream is = new FileInputStream("E:\\ISPConfig.txt");
  fs2.copyFromLocalFile(new Path("E:\\ISPConfig.txt"), new Path("/user_ljs/hadoop/output/"));
 }


---end by itlife365.com
Tags: