Apache FLUME Installation and Configuration in Windows 10

1. Download & install Java

2. Create a Junction Link.  (Needed as the Java Path contains spaces)

    C:\Windows\system32>mklink /J "C:\Program_Files" "C:\Program Files"
    Junction created for C:\Program_Files <<===>> C:\Program Files

3. Set Path and Classpath for Java


4. Download Flume

    Download apache-flume-1.7.0-bin.tar.gz

5. Extract using 7-Zip

 Move to C:\flume\apache-flume-1.7.0-bin directory

6. Set Path and Classpath for Flume


7. Download Windows binaries for Hadoop versions


8. Copy

To C:\hadoop\hadoop-2.6.0\bin

9. Set Path and Classpath for Hadoop


10. Edit log4j.properties file


11. Copy flume-env.ps1.template as flume-env.ps1.

    Add below configuration:
    $JAVA_OPTS="-Xms500m -Xmx1000m -Dcom.sun.management.jmxremote"

12. Copy flume-conf.properties.template as flume-conf.properties

13. — Flume Working Commands —

C:\> cd %FLUME_HOME%/bin
C:\flume\apache-flume-1.7.0-bin\bin> flume-ng agent –conf %FLUME_CONF% –conf-file %FLUME_CONF%/flume-conf.properties.template –name agent

14. Install HDInsight Emulator (Hadoop) on Windows 10

a. Install Microsoft Web Platform Installer 5.0
b. Search for HDInsight
c. Select Add -> Install -> I Accept -> Finish
d Format Namenode

        C:\hdp> hdfs namenode -format

e. Start Hadoop and Other Services

        C:\hdp> start_local_hdp_services

f. Verify

        c:\hdp> hdfs dfsadmin -report

g. Hadoop Sample Commands

C:\hdp\hadoop-> hdfs dfs -ls hdfs://lap-04-2312:8020/
C:\hdp\hadoop-> hdfs dfs -mkdir hdfs://lap-04-2312:8020/users
C:\hdp\hadoop-> hdfs dfs -mkdir hdfs://lap-04-2312:8020/users/hadoop
C:\hdp\hadoop-> hdfs dfs -mkdir hdfs://lap-04-2312:8020/users/hadoop/flume

f. Stop Hadoop and Other Services

        C:\hdp> stop_local_hdp_services

13. — Other Flume Commands —

C:\> cd %FLUME_HOME%/bin
C:\flume\apache-flume-1.7.0-bin\bin> flume-ng agent –conf %FLUME_CONF% –conf-file %FLUME_CONF%/seq_log.properties –name SeqLogAgent
C:\flume\apache-flume-1.7.0-bin\bin> flume-ng agent –conf %FLUME_CONF% –conf-file %FLUME_CONF%/seq_gen.properties –name SeqGenAgent
C:\flume\apache-flume-1.7.0-bin\bin> flume-ng agent –conf %FLUME_CONF% –conf-file %FLUME_CONF%/flume-conf.properties –name TwitterAgent


Note: Sample Configurations and Properties are attached.

flume-conf-properties   seq_gen-properties   seq_log-properties


  1. anbu

    Followed the steps but received the below error
    Resolve-Path : Cannot find path ‘C:\Projects\apache-flume-1.6.0-bin\bin\conf’ b
    ecause it does not exist.
    At C:\Projects\apache-flume-1.6.0-bin\bin\flume-ng.ps1:309 char:28
    + $fullPath = (Resolve-Path <<<< $path).Path
    + CategoryInfo : ObjectNotFound: (C:\Projects\apa….0-bin\bin\co
    nf:String) [Resolve-Path], ItemNotFoundException
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.Resol


    • tanmoy

      Check FLUME_HOME entry in your env variable. I had similar problem. After changing the env variable restart the system.


  2. Pingback: Flume – Messy Mass Data
  3. stalin lobo

    I’m getting this error. please help.

    Running FLUME agent :
    class: org.apache.flume.node.Application
    arguments: -n agent -f “F:\hadoop\flume\apache-flume-1.6.0\conf\flume-conf.properties.template”

    Start-Process : This command cannot be executed due to the error: The parameter is incorrect.
    At F:\hadoop\flume\apache-flume-1.6.0\bin\flume-ng.ps1:189 char:23
    + $x = Start-Process <<<< $javaPath -ArgumentList "$fullJavaCommand" -Wait -NoNewWindow
    + CategoryInfo : InvalidOperation: (:) [Start-Process], InvalidOperationException
    + FullyQualifiedErrorId : InvalidOperationException,Microsoft.PowerShell.Commands.StartProcessCommand


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.