Configure Hadoop and start cluster services using Ansible Playbook

 Today, we will learn how to configure hadoop cluster using ansible.

Setup:

We need 3 machines - Ansible controller node(192.168.1.7), Hadoop Namenode(192.168.1.9), Hadoop Datanode(192.168.1.8)

Steps:

namenode

install java 

install hadoop

create a directory 

hdfs-site file config 

<property>

<name>dfs.data.dir</name>

<value>/nn<value>

</property>

format directory with hadoop

core-site file config

<property>

<name>fs.default.name</name>

<value>hdfs://192.168.1.9:9001<value>

</property>

stop firewall

start service


slavenode

install java 

install hadoop

create a directory

hdfs-site file config

<property>

<name>dfs.data.dir</name>

<value>/dn<value>

</property>

core-site file config

<property>

<name>fs.default.name</name>

<value>hdfs://192.168.1.9:9001<value>

</property>

stop firewall

start service


The inventory file looks like this


The playbook looks like this





After running this inventory file by command

ansible-playbook hadoop.yml

The hadoop setup can be verified by command

hadoop dfsadmin -report


Thank You for reading.

I would like to thanks Vimal Daga Sir.

My Email — ayanchawlae@gmail.com






Location: 370-A, Shastri Nagar, Lajpat Nagar, Jalandhar, Punjab 144001, India

Comments

Post a Comment

Popular posts from this blog

Key Announcements at Microsoft Ignite

Image
World is going to be an Agentic World  Universal Interface + Reasoning Planning + Memory & Context = Agentic World Copilots are the way in future -  First -> Security -  Secure Future Initiative -  Security Updates url -  aka.ms/Ignite2024Security 1. Purview -  This new hacking event will be the largest of its kind, with an additional $4 million in potential awards for research into high-impact areas, specifically cloud and AI. Zero Day Quest will provide new opportunities for the security community to work hand in hand with Microsoft engineers and security researchers – bringing together the best minds in security to share, learn, and build community as we work to keep everyone safe.   aka.ms/ZeroDayQuest Second -> Copilot 3 Fold method for Copilot -  Adoption - More people should use copilot Extensibility - More extensions for the copilot ROI - How to measure the Roi for copilot Adoption -  2. Copilot Pages - Copilot is UI fo...
Read more

Ansible Handlers

Image
 In this post, you will learn about ansible handlers. Handlers are like if and else in ansible, suppose you have a configuration file in httpd server, and you want whenever you change something in configuration file, the httpd service must restart. In this case we will use handlers.  Components-: Controller node(192.168.1.7), Target node(192.168.1.8) The ansible-playbook looks like  It will only run the handlers portion only when notify will be changed. Target Node-  httpd is running on 8081. Playbook running when conf file is changed Playbook running second time You can view the webpage running on port 8081 Thank You  for reading. I would like to thanks  Vimal Daga  Sir. ( https://www.linkedin.com/in/vimaldaga/ ) My Email — ayanchawlae@gmail.com My LinkedIn —   https://www.linkedin.com/in/ayanchawlae/
Read more

Ansible Roles (Httpd + Haproxy)

Image
 Today we will create 2 ansible roles and then will run them combined to create a load balancer setup with 2 nodes connected to it. Lets first init the roles by ansibe galaxy After these commads t15.yml is the main file in which we have combined both the codes Lets create tasks in both the roles httpdserver -> tasks -> main.yml The main.yml file in httpdserver Similarly for loadbalancer role loadbalancer  -> tasks -> main.yml The main.yml for loadbalancer The config file of haproxy looks like We can see the working of loadbalancer Thank You  for reading. I would like to thanks  Vimal Daga  Sir. ( https://www.linkedin.com/in/vimaldaga/ ) My Email — ayanchawlae@gmail.com My LinkedIn —   https://www.linkedin.com/in/ayanchawlae/
Read more