With HDInsight

The HFactory Tools for HDInsight service automates the deployment of HFactory Tools on top of a HDInsight cluster. Just follow the below simple steps and be started in no time!

Prerequisites

A resource group

You need a new resource group to put your HDInsight cluster into:

Create a resource group

A virtual network

Create a virtual network in your resource group using the Resource Manager deployment method.

Then give the name of the network and its subnet:

Specify virtual network parameters

A network security group

Create a network security group using the Resource Manager deployment method and specify its name.

Refresh the resource group to access it when deployed:

Refresh the resource group

Add inbound rules to the network security group:

Add inbound rules

First HTTPS from everywhere or at least this four Azure ips to allow monitoring of the HDInsight cluster by Azure (168.61.49.99, 168.61.48.131, 23.99.5.239, 138.91.141.162):

Add https inbound rule

Then SSH to be able to connect to the VM and the head node of HDInsight:

Add ssh inbound rule

HBase HDInsight

Create the cluster

In your resource group create a new HDInsight cluster.

Select an available name:

Choose a name

Select the HBase type: Select HBase

Give the credentials:

Give credentials

Define the storage account and container:

Storage parameters

Define the cluster size:

Choose the cluster size

*** Important: use your previously created network *** Select your Network and subnet:

Use your network

Finally check that the resource group is the correct one and deploy the cluster.

HFactory Server VM

Get the quorum

Once the HDInsight cluster is deployed, go to the cluster dashboard using the address given in the azure portal:

Go to the dashboard

Connect to it using the credentials given during the installation:

Connect to the dashboard

Go into the HBase configuration by clicking on HBase first then Configs:

Go to the HBase configs

Use the search box to get the quorum property and copy this value that you will need to configure the HFactory VM:

Get the quorum

Create the VM

Select HFactory Tools for HDInsight from the Azure marketplace or use this direct link and click create:

VM creation

Provide a name, a user, a password or a SSH-key, a public ip:

Basic VM configuration

Most importantly use the same network and subnet than the HDInsight cluster:

Configure the network for the VM

SSH configuration

If you have configured an SSH access for all, simply connect to the VM with SSH. If you only have access to the namenode add the following lines to your $HOME/.ssh/config files:

Host HFTForHDI
  User          hfactory
  HostName      172.17.0.6
  ProxyCommand  ssh hfactory-ssh@hfactoryready-ssh.azurehdinsight.net nc %h %p 2> /dev/null
  IdentityFile ~/keys/id_rsa

You need to change the User to the username of your VM, Hostname to its private ip. You can drop the IDentityFile line if you are using your default key or the password method. Finally change ssh hfactory-ssh@hfactoryready-ssh.azurehdinsight.net to the SSH connection string that you can find here:

Get the ssh connection to HDInsight

VM configuration

Once SSH is properly configured connect to the VM and give the zookepper quorum when asked to.

You can contact Ubeeko contact@ubeeko.com for a license to copy to /usr/local/hfactory-server/conf/hfactory.license to upgrade from the developper license included in the VM.

You can now start the server with:

sudo systemctl start hfactory

You can also make the server start on boot or reboot with:

sudo systemctl enable hfactory

By default HFactory server listen on port 30100, you can change that by adding the hserver.listen.port property to the /usr/local/hfactory-server/conf/hserver.conf.

You can copy your applications into /usr/local/hfactory-server/apps and start using them.