How to increase or decrease Hadoop Datanode Partition Dynamically [Part 1]

Task #

1. Integrating LVM with Hadoop and providing Elasticity to DataNode Storage.
2. Increase or Decrease the Size of Static Partition in Linux.
3. Automating LVM Partition using Python-Script.

All the setup is going on the top of Local VM (Oracle Virtual Machine)

.

Create

Go with the default option.

Go with the default option.

Created100GB.

After create we have to attach to the OS.

We can see the volume attached successfully.

fdisk -l

fdisk /dev/sdb

After attaching a volume to OS we have to partition the volume.

n for the new partition.

p for primary partition.

w to save the partition

Check partition is created or not.

fdisk -l /dev/sdb

OR

lsblk

It reloads the drivers for this partition.

udevadm settle

Create PV (Physical Volume)

pvcreate /dev/sdb1

Create VG (Volume Group)

vgcreate hadoop /dev/sdb1

Create LV (Logical Volume)

As you see I created 3 LVs.

lvcreate — size 50G — name lv1 hadoop

lvcreate — size 39G — name lv2 hadoop

lvcreate — size 10G — name lv3 hadoop

Technically we don’t have exactly 100GB but we have a little bit lesser than 100GB.

vgdisplay hadoop

We allocate 99 GB to the LV. Now, are being left with almost 1GB VG.

mkfs.ext4 /dev/hadoop/lv1
mkfs.ext4 /dev/hadoop/lv2
mkfs.ext4 /dev/hadoop/lv3
mkdir part1
mkdir part2
mkdir part3
mount /dev/hadoop/lv1 part1/
mount /dev/hadoop/lv2 part2/
mount /dev/hadoop/lv3 part3/

Now, we can see that LVs are mounted to the folders. So we can easily use it.

df -hT

Now, We are attaching a new volume to the OS.

Why?

Because we are going to increase the volume dynamically without losing older data. By this we can achive Elasticity.

fdisk -l

fdisk  /dev/sdc

After attaching a volume to OS we have to partition the volume.

n for the new partition.

p for primary partition.

w to save the partition

fdisk -l

Create a PV from a new volume partition.

pvcreate /dev/sdc1

Extend the VG i.e hadoop.

vgextend hadoop /dev/sdc1

vgdisplay hadoop

Now, we can see that the VG is extended with 20GB.

pvscan

These are the volume partitions from where we are getting space or volume.

At last, we have to extend the LV.

lvextend — size 25G /dev/mapper/hadoop-lv3

lvdisplay /dev/mapper/hadoop-lv3

Now, we LV is updated to 25GB.

resize2fs /dev/mapper/hadoop-lv3 25G

Now, we have to resize the volume because we didn’t format an extra volume yet.

Also, it does not remove the older format that we had previously.

df -h

Now, we can clearly see that we increased the volume dynamically with the help of the LVM.

Hadoop doesn't have the elasticity feature. But by the conceptual knowledge, we overcome the challenge with the help of the Linux LVM concept.