5.1 - Could not locate executable nullbinwinutils.exe 2018-06-04 20:23:33 ERROR Shell:397 - Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable nullbinwinutils.exe in the Hadoop binaries. 16/02/26 18:29:33 INFO SparkContext: Created broadcast 0 from textFile at FrameDemo.scala:13 16/02/26 18:29:34 ERROR Shell: Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable nullbinwinutils.exe in the Hadoop binaries. 2018-06-04 20:23:33 ERROR Shell:397 - Failed to locate the winutils binary in the hadoop binary path java.io.IOException: Could not locate executable null bin winutils.exe in the Hadoop binaries. Windows binaries for Hadoop versions (built from the git commit ID used for the ASF relase) Shell 1.8k 2.3k kerberosandhadoop. Kerberos and Hadoop: The Madness beyond the Gate. @apache @hortonworks @hortonworks-spark Activity overview Contributed to apache/hadoop, steveloughran.
- Java 8 : We are going to use the Java 8 Function interface and the hot Lambda expressions.
- Maven 3 : Just to automate collecting the project dependencies.
- Eclipse : My usual IDE for Java/JavaEE developments.
- Download the executable winutils from the Hortonworks repository.
- Create a dummy directory where you place the downloaded executable winutils.exe. For example : C:SparkDevbin.
- Add the environment variable HADOOP_HOME which points to C:SparkDev. You have 2 choices :
- Windows > System Setting
- Eclipse > Your Class which can be run as a Java Application (containing the static main method) > Right Click > Run as > Run Configurations > Evironment Tab :
Create a Maven Project. Configure pom.xml as follows :
The benefit of creating a local Spark context is the possibility to run everything locally without being in need of deploying Spark Server separately as a master. This is very interesting while development phase. So here it is the basic configuration :
Hortonworks Winutils.exe Download
Now that we have an operational environment, let’s move to punchy examples of RDD Tranformations and Actions tutorial.