SREngineered - Shubham Rasal

Create Simple Chat App Using UDP Protocol In Python

Shubham Rasal — Sun, 18 Apr 2021 02:31:03 GMT

In this blog, we are going to create a chat server based on UDP protocol in Python.

Here’s the problem statement:

🔅 Create your own Chat Servers, and establish a network to transfer data using Socket Programing by creating both Server and Client machines as Sender and Receiver both. Do this program using UDP data transfer protocol.

🔅 Use multi-threading concept to get and receive data parallelly from both the Server Sides. Observe the challenges that you face to achieve this using UDP.

Before jump into code, let’s understand

What is UDP?

In computer networking, the User Datagram Protocol is one of the core members of the Internet protocol suite. With UDP, computer applications can send messages, in this case, referred to as datagrams, to other hosts on an Internet Protocol network.

It is a connectionless protocol and not reliable. It is not used in other chatting apps like Facebook and WhatsApp. The communication speed is comparatively fast than the TCP protocol. It is used in online video surfing and gaming.

Creating Chat server[One sided connection]

In this setup, Server only receives messages and the client can only send messages. We can utilize the socket library in python.

In server, we have to write the below code, which will continue in receiving mode using while true.

import socket        

#AF_INET used for IPv4  
#SOCK_DGRAM used for UDP protocol  
s = socket.socket(socket.AF_INET , socket.SOCK_DGRAM )  
#binding IP and port   

s.bind(("192.168.225..242",2222))  
print("Server started ...192.168.225..242:2222")  
print("Waiting for Client response...")   
#recieving data from client  
while True:  
       print(s.recvfrom(1024))

In client program:

import socket  

#client program  

s = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)  

while True:  
       ip ,port = input("Enter server ip address and port number :\n").split()  
       m = input("Enter data to send server: ")  
       res = s.sendto(m.encode(),("192.168.225.242",2222))   
       if res:  
          print("\nsuccessfully send")

we can also see the working of it using the below image

You can see on the client end we really don’t need to establish a connection with the client. We can directly send data to the server.

But the problem here is the server can not respond back and the client can not receive any message at the same time.

Here’s we can take help of multi-threading, by which we can receive as well send messages simultaneously.

Both sides communicate using multithreading

In python, we have a threading module, with help we can add many threads to run our program. we can also pass different functions to different threads to run.

Let’s understand with one help. consider the human body is a process and our parts are different threads of it. they are performing their tasks simultaneously. The liver, heart, lungs, brain are the organs involved in the process and each process is running simultaneously.

# chatapp.py

import socket  
import threading  
import os  

s = socket.socket(socket.AF_INET , socket.SOCK_DGRAM )  
s.bind(("192.168.225.34",2222))  
print("\t\t\t====>  UDP CHAT APP  <=====")  
print("==============================================")  
nm = input("ENTER YOUR NAME : ")  
print("\nType 'quit' to exit.")  

ip,port = input("Enter IP address and Port number: ").split()  

def send():  
    while True:  
        ms = input(">> ")  
        if ms == "quit":  
            os._exit(1)  
        sm = "{}  : {}".format(nm,ms)  
        s.sendto(sm.encode() , (ip,int(port)))  

def rec():  
    while True:  
        msg = s.recvfrom(1024)  
        print("\t\t\t\t >> " +  msg[0].decode()  )  
        print(">> ")  
x1 = threading.Thread( target = send )  
x2 = threading.Thread( target = rec )  

x1.start()  
x2.start()

Here we are leveraging the power of threads to simultaneously send and receive the messages.

In the above code, I have two functions send() and rec() and we are starting them at the same time. The threading module gives us a Thread class and we can use that to make and start at the same time.

Type quit exiting the program.

UDP is not reliable protocol which does not check that other system is running or not because it is connectionless and does not have any acknowledgment.

Conclusion

We have successfully created a simple chat application using UDP protocol in python.

I would love to hear your thoughts and ideas about these topics. Don’t hesitate to share in the comment section below.

Also, you can always message me over LinkedIn as well.

More Ideas:

You can enhance this setup to send OS commands and get the output back, instead of logging in systems.

Thank you…

How to load variable dynamically according to os in ansible?

Shubham Rasal — Fri, 26 Mar 2021 20:54:26 GMT

Ansible 10

Problem Statement:

Create an Ansible Playbook which will dynamically load the variable file named the same as OS_name and just by using the variable names we can Configure our target node.
(Note: No need to use when keyword here.)

Let’s understand the use case of this problem. There will be many cases when some of the package names are different Linux OS distributions. Also, there will be cases that they are different file formats only such as deb or rpm.

To solve this problem, Ansible offers us one module named “include_vars” which loads variables from files dynamically within the task. This module can load the YAML or JSON variables dynamically from a file or directory, recursively during task runtime.

To detect on which operating system we are performing tasks, we can use the “setup” module. This module is automatically called by playbooks to gather useful variables about the remote hosts that can be used in playbook. This module is also available for windows target.

Let’s take an example of installing an Apache web server in CentOS and Ubuntu Linux. We do not want a hard-coded playbook installing a web server for a different distribution, OS, etc.

First, find out which variables setup module offers use by which we can detect remote host distribution, and accordingly we can load the variable file on the fly.

First check, can we connect both the system using the ping module.

Now, we have connectivity to both the servers, we have to get that variable name which gives us a remote host type variable. The setup module gives many variables under the “ansible_distribution” section. so let's see what variable it gives.

Now, we know the distribution name and major version, we will create variables files and give this distribution name and version name to that files. eg: Ubuntu-18.yml, CentOS-8.yml

Also, you can use the os_family variable which gives the os_family of remote hosts.

Create Ubuntu-18.yml file and add the below variables in that file

*# Ubuntu-18.yml*  
package_name: apache2  
service_name: apache2  
document_root: /var/www/html

Create CentOS-8.yml file and add below variables

# CentOS-8.yml package_name: httpd service_name: httpd document_root: /var/www/html/

*# CentOS-8.yml*  
package_name: httpd  
service_name: httpd  
document_root: /var/www/html/

Now we have to write a playbook that will install a webserver and read variables dynamically according to distribution type.

Let's first make sure, we are using selecting proper files using ansible facts. so for that, we will use debug module to create a final string.

---  
- name: "Install webserver"  
  hosts: webserver  
  tasks:  
  - name: "Test variables"  
    debug:  
      msg: "{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.yml"

Run this playbook, $ ansible-playbook main.yml

Now let’s write a playbook that will read the variables according to the OS type of remote host and install the webserver.

---  
- name: "Install webserver"  
  hosts: all  
  vars_files:  
     - "{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.yml"  

  tasks:  
    - name: "Install the web server"  
      package:  
              name: " {{ package_name }}"  
              state: present  

    - name: "Create document root directory"  
      file:   
          path: "{{document_root }}"  
          state: directory  
          recurse: yes

 - name: "Create index.htm page in document root"  
      copy:  
              content: " Welcome to {{ ansible_distribution }} server !! "  
              dest: "{{ document_root }}/index.html"  

    - name: "Start the service"  
      service:  
              name: "{{ service_name }}"  
              state: started

- name: "Test the servers"  
  hosts: localhost  
  tasks:  
          - name: "HealthCheck the servers"  
            uri:  
               url: "http://{{item}}"  
               return_content: yes  
            with_items: "{{ groups['webserver'] }}"  
            register: output  
            failed_when: '"Welcome" not in output.content'

The above playbook will install and test our web server is working properly or not.

$ ansible-playbook main.yml

We can test the installation is successful or not using the curl command.

Conclusion:

We have successfully made our playbook variables dynamically according to distribution and major version number without using the when keyword.

Thank you…

About the writer:
Shubham loves technology, challenges, continuous learning, and reinventing himself. He loves to share his knowledge and solve daily problems using automation.

Visit him: blog.shubhcodes.tech to know more.

Launch and Configure docker container using ansible-playbook

Shubham Rasal — Thu, 21 Jan 2021 17:33:24 GMT

ANSIBLE-9

Launch and Deploy python flask app on docker container using ansible

Even this is a very rare use case where we need to configure the container using ansible. Enabling ssh inside the container is not a good practice, but in some cases, we might need to do this.

My use case was to set up an ansible practical lab where I can use multiple hosts of different Linux instantly without wasting more resources. This gives me the power to launch each time fresh container and test my playbooks on multiple os distributions with diff versions.
One more use case will be, you have one server in development and you are deploying many microservices with multiple teams, then you can deploy microservices using container and give ssh access direct to the container to teams for troubleshooting instead of giving direct access to the server.

Problem statement

Write an ansible playbook to

Install docker-engine on the host node.
Launch Container and expose it
Update the inventory file with container IP dynamically
Configure deploy python app on the container

This article covers a step-step guide to solve our problem statement.

Write Ansible Playbook to install docker-engine.

Here, I am writing the playbook for RedHat or CentOS.

The above playbook will add the yum docker repo and install the docker-engine community version. To handle docker containers from ansible, the requirement is to install docker SDK, to install that we will install pip, and using pip we will download docker SDK. Lastly, start a docker service.

$ ansible-playbook docker-configure.yml

Create Dockerfile

We want to create a container in such a way that, we can connect the docker container using ssh public key authentication. Also, connect using ansible and configure the container.

Generate ssh key.

$ ssh-keygen -f ./mycontainerkey

The command above will create two files, private and public keys. We want to add a public key to the container.

Now let's write Dockerfile

Above Dockerfile taking ubuntu:latest as the base image. then we created one use with ‘docker’ username and add created ssh public key to authorized-keys and give docker user Sudo power inside the container.

As you can notice we are also using entrypoint.sh file, let’s create that also, which will start ssh service and create log files.

Now we have Dockerfile ready, it’s time to launch the docker container using ansible and update inventory dynamically according to container IP.

We want to deploy a python app container on the flask server, keeping in that mind, we will expose ports according to that.

As you can see I have defined a few variables,

dockerfile_folder: It is the folder where we have store Dockerfile, mycontainerkey.pub, and entrypoint.sh. we want to copy all these files to the host for creating an image.

docker_image_name: Give the image name that we are creating from Dockerfile

docker_container_name: Assign container name that docker will launch

patting_ssh_port: Assign port number of the server that will be exposed for ssh so the team can log in to the container.

patting_http_port: we want to deploy a python app container, I want the client to connect to port 80 of the docker host and they will be connected to the container.

In this file, we are copying Dockerfile and building the image, and then launching it using the ansible docker module.

After that, we update the inventory file using lineinfile by adding a docker container IP address. lineinfile module searches [containers] pattern and after this line, it adds a new line. (Make empty group [containers] inside your inventory file. )

$ ansible-playbook docker-container.yml

we can cross-check using the $ docker ps command

Let’s write a simple hello world python flask app for the demo.

//app.py

We want to deploy this application on the container we launched in the above steps. We will write one playbook which will deploy this application over the docker container.

The above playbook will copy the source code to /srv/ folder then install pip3. After that, it will install the required libraries using pip and finally run our flask app.

$ ansible-playbook deploymyapp.yml

Now you open your browser can test the application is working or not. We have exposed the container’s 8080 port to the host machine 80 port. so copy the host machine IP and paste it into the browser.

If you have successfully completed here… then You deserve a pat on the back.

Let me show you now complete directory structure, so you will get a complete idea.

You can find all the files in the repository at Playbook/Docker_container location. Don’t forget to star it, that keeps me motivated to solve challenges and to write about them.

[ShubhamRasal/ansible-playbooks
Contribute to ShubhamRasal/ansible-playbooks development by creating an account on GitHub.github.com](https://github.com/ShubhamRasal/ansible-playbooks/tree/master/Playbooks/Docker_container "https://github.com/ShubhamRasal/ansible-playbooks/tree/master/Playbooks/Docker_container")

If you have any doubts or something improvement needed in this blog, please feel free to reach out to me on LinkedIn.

I hope you learned something new and find ansible more interesting. Let me know your thoughts about ansible and how do plan to use ansible?

Thank you…

About the writer:
Shubham loves technology, challenges, is open to learning and reinventing himself. He loves to share his knowledge. He is passionate about constant improvements.
He writes blogs about Cloud Computing, Automation, DevOps, AWS, Infrastructure as code.
Visit his Medium home page to read more insights from him.

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

Create a role for setting up a load balancer and web server dynamically

Shubham Rasal — Mon, 11 Jan 2021 03:47:48 GMT

ANSIBLE-8

Ansible role for setting up load balancer using HAProxy and webserver using Apache software.

Introduction 🤓

In this article, we will write a simple role for configuring HAPorxy software for load balancing which will dynamically add backend servers.

I have also covered what is a load balancer and how to write the playbook for configuring a load balancer on AWS using a playbook.
You can refer to the blog to know more about load balancer and configuration.

[How to configure Load Balancer and webserver on AWS using Ansible Playbook?
Configure Haproxy dynamically when a new webserver gets added using ansible.medium.com](https://medium.com/faun/how-to-configure-load-balancer-and-webserver-on-aws-using-ansible-playbook-60c22c0355ed "https://medium.com/faun/how-to-configure-load-balancer-and-webserver-on-aws-using-ansible-playbook-60c22c0355ed")

In this blog, we will focus more on writing and using the ansible role. This is the continuation of the below blog which configures web servers. Now we want to add a load balancer that will dynamically register the webservers as backend servers.

[Ansible: Write Ansible role to configure apache webserver
Write Ansible role to configure apache webserver on Redhat and Ubuntu.medium.com](https://medium.com/faun/ansible-write-ansible-role-to-configure-apache-webserver-9c08aaf66528 "https://medium.com/faun/ansible-write-ansible-role-to-configure-apache-webserver-9c08aaf66528")

Action plan 🔥

Install Haproxy
Configure load balancer
Register backend servers that are mentioned in the inventory file.
Use HAProxy and Apache roles to configure the web server and also register the server to HAProxy dramatically.
Override default variables.

Let’s start the action ☄️

Initialize ansible role. Read more about “ansible-galaxy”

#create directory and change directory
$ mkdir roles
$ cd roles #create ansible role
$ ansible-galaxy init haproxy

This command will create a folder structure for a role.
First, decide the variables we are going to use

lb_port_number: This variable is used for configuring haproxy. Clients will connect to this port number of haproxy.

lb_inventory_group: This variable is the name of the backend server group name in ansible inventory.

lb_backend_port_number: This variable is the value of the backend server port number.

Now we have declared the variables, Now we will add a configuration jinja template that will read variables from the defaults/main.yml file.
Also, register all the servers mentioned in the backend group of inventory.

Now we have the configuration file ready, it’s time to write tasks for installing and transfer this template to a managed node.

we have to restart the haproxy only when if there is any change or we add a new node in haproxy, for that you can notice we have called the “Restart haproxy” handler in the template task. we have to write handlers in handlers/main.yml file.

That’s it… We have written the simple ansible role for configuring haproxy.

Now we will use this haproxy role and apache rule. In such a way that, first we will configure webservers then we will register those web servers in haproxy load balancer as backends. If we have a case that, we have a new webserver then it will simply configure and also register its backend server dynamically.

Use the roles

Create an inventory file that will have two groups for load balancer and backend servers.

I have three managed nodes. We want to configure two of them as a webserver and one as the load balancer.

Create one file and name it as ‘inventory’. Test the connection using the ping module.

Write a playbook that will use both the roles.

Now we want to override default variables used in roles. There are multiple ways to do that but we will use the group_vars directory.

Define webservers variable in group_vars/backend/main.yml file.

Define webservers variable in group_vars/load_balancer/main.yml file.

write some demo file to test our load balancer is working or not.

The above file print the IP address of the managed server, so that we can test it is balancing the load.

Again, let’s see what the will be directory structure?

Now it's time to run and test our playbook.

$ ansible-playbook webserver_lb.yml -i inventory

Output

You can find the above playbook on this GitHub repository. Bookmark or star it which helps to keep me motivated.

[ShubhamRasal/ansible-playbooks
You can find the above-used file here. check it out and don’t forget to mark a start. It helps me to maintain my motivation. Thank you | github.co](https://github.com/ShubhamRasal/ansible-playbooks/tree/master/Roles/myapache "https://github.com/ShubhamRasal/ansible-playbooks/tree/master/Roles/myapache")

If you have any doubts or something improvement needed in this blog, please feel free to reach out to me on my LinkedIn account.

I hope you learned something new and find ansible more interesting.
Let me know your thoughts about ansible and how do plan to use ansible?

Thank you…

Ansible: Write Ansible role to configure apache webserver

Shubham Rasal — Sun, 10 Jan 2021 14:40:29 GMT

ANSIBLE-7

Write Ansible role to configure apache webserver on Redhat and Ubuntu.

What is ansible?

Ansible is an open-source IT Configuration Management, Deployment, and Orchestration tool. Ansible helps us to gain large productivity in a variety of automation challenges. This tool is very simple to use and also powerful enough to automate complex multi-tier IT application environments.

Ansible offers a simple architecture that doesn’t require special software to be installed on nodes. It also provides a robust set of features and built-in modules which facilitate writing automation scripts.

[What is Ansible? How ansible is helping companies in automation?
Factories that failed to automate fell behind due to so much competition in the market. That’s why automation became…developer-shubham-rasal.medium.com](https://developer-shubham-rasal.medium.com/what-is-ansible-acb573304acd "https://developer-shubham-rasal.medium.com/what-is-ansible-acb573304acd")

In this article, we are going to write an Ansible role to configure the Apache webserver. The Apache HTTP server is the most widely-used web server in the world. It provides many powerful features including dynamically loadable modules, robust media support, and extensive integration with other popular software.

You can also refer to this article, In this article, I have explained how to write a playbook for configuring the webserver.

[How to configure apache webserver using ansible?
Configure Apache server using ansible-playbook | Shubham Rasaldeveloper-shubham-rasal.medium.com](https://developer-shubham-rasal.medium.com/how-to-configure-apache-webserver-using-ansible-4b7077d88505 "https://developer-shubham-rasal.medium.com/how-to-configure-apache-webserver-using-ansible-4b7077d88505")

In this guide, we are going to write an Ansible role to configure the Apache HTTP server.but,

What is an Ansible role?

Roles let you automatically load related vars_files, tasks, handlers, and other Ansible artifacts based on a known file structure. Once you group your content in roles, you can easily reuse them and share them with other users.

An Ansible role has a defined directory structure with seven main standard directories. You must include at least one of these directories in each role. You can omit any directories the role does not use.

You can read more about the roles and directory structure in detail. Ansible roles.

Let’s start the action

Initialize ansible role. Read more about “ansible-galaxy”

#create directory and change directory

$ mkdir roles
$ cd roles

#create ansible role

$ ansible-galaxy init myapache

This command will create a folder structure for a role.

I recommend you to read about the directory structure. Here the Ansible Role directory.

We want our role to configure apache for RedHat and Debian family. We can use this role to configure HTTPD for Redhat and apache2 for Debian Linux distribution.

What does this Role do?

Install httpd for RedHat or centos. Install apache2 for Debian.
Create a custom document root folder for the new Apache VirtualHost and set up a test page.
Enable the new Apache VirtualHost.

Let’s decide the variables that we will need.

we want to give access to the role user to override this variable so we will write these variables in the default/main.yml file. variables declared in this file can be overridden by role use at the time of using it.

Now we have declared the variable.

Apache Conf Template

The apache.conf.j2 the file is a Jinja 2 template file that configures a new Apache VirtualHost. The variables used within this template are defined in the /default/main.yml variable file.

#template/apache.conf.j2

Write tasks to configure httpd for the RedHat family.

The above tasks will configure httpd on RedHat distributions like RedHat Linux or Centos Linux.

Now let’s write tasks for configuring apache2 for Debian.

Now we have two task files. Now we need to use these files according to os type. We will include these files depending on the os type. So let’s include both the files (RedHat.yml and Debian.yml) in the main.yml file

We have also notified some handlers in the above tasks file. In the ansible role, we write handlers in handlers/main.yml file.

Now we have written all the necessary files. Let’s see the completed directory structure.

Now we can use this role, but before that, we have to tell ansible where you have stored your roles. You can add role location at ansible.cfg file.
You can directly add ansible roles path in /etc/ansible.cfg or you can also write ~/.ansible.cfg for user specific role.

now check your role is register and ready to use. using the below command.

$ ansible-galaxy role list

Let’s test this ansible role.

I have created two instances on AWS of ubuntu and RedHat Linux. Create an inventory file and add IP addresses and usernames to the inventory file.

Now add the ssh key.

$ ssh-add mykey.pem

Test that we can successfully connect to our managed nodes or not using the ping module.

$ ansible webserver -m ping -i inventory

Now write one playbook which will use our role.

Now we are ready to run and configure apache servers using an ansible role.

$ ansible-playbook mywebserver.yml -i inventory

Here’s a detailed output of the above playbook.

You can find the above playbook on this GitHub repository. Bookmark or star it which helps to keep me motivated.

[ShubhamRasal/ansible-playbooks
How to write an Ansible role to configure the apache server. github.com](https://github.com/ShubhamRasal/ansible-playbooks/tree/master/Roles/myapache "https://github.com/ShubhamRasal/ansible-playbooks/tree/master/Roles/myapache")

If you have any doubts or something improvement needed in this blog, please feel free to reach out to me on my LinkedIn account.

I hope you learned something new and find ansible more interesting.
Let me know your thoughts about ansible and how do plan to use ansible?

This article is part of configuring load balancer and webserver using ansible role. The next part you can read here…

[Create a role for setting up a load balancer and web server dynamically
Ansible role for setting up load balancer using HAProxy and webserver using Apache software.developer-shubham-rasal.medium.com](https://developer-shubham-rasal.medium.com/create-a-role-for-setting-up-a-load-balancer-and-web-server-dynamically-8f4e717eee30 "https://developer-shubham-rasal.medium.com/create-a-role-for-setting-up-a-load-balancer-and-web-server-dynamically-8f4e717eee30")

Thank you…

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

Why you should use Kubernetes?

Shubham Rasal — Fri, 01 Jan 2021 05:39:54 GMT

Kubernetes-1

Introduction to Kubernetes. Why we need Kubernetes? Use cases of Kubernetes.

Introduction 🤓

When you talk to IT guys about containers, I am sure the next topic of conversion will be on container management and orchestration.

Hey, but what is a container?

Linux containers are technologies that allow you to package and isolate applications with their entire runtime environment-all of the files necessary to run. This makes it easy to move the contained application between the environments(dev, staging/test, prod, etc) while retaining full functionality. Containers help reduce conflicts between your development and operations teams by separating areas of responsibility.
Now next question then should be,

What is container orchestration?

Container orchestration automates the deployment, management, scaling, and networking of containers. The companies that need to deploy and manage hundreds or thousands of containers and hosts can benefit from container orchestration.
Container orchestration automates and manages tasks such as:

Provisioning and deployment
Configuration and scheduling
Container availability
Load balancing and traffic routing
Scaling and removing containers

Now, knowing about containers and the need for orchestration leads to the next question

What is Kubernetes?🤯

Kubernetes (also known as k8s or “Kube”) is an open-source container orchestration platform that automates many of the manual processes involved in deploying, managing, and scaling containerized applications.

In other words, you can cluster together groups of hosts running Linux containers, and Kubernetes helps you easily and efficiently.

Here’s how Dan Kohn, executive director of the Cloud Native Computing Foundation (CNCF), in a podcast with Gordon Haff, explained it: “Containerization is this trend that’s taking over the world to allow people to run all kinds of different applications in a variety of different environments. When they do that, they need an orchestration solution in order to keep track of all of those containers and schedule them and orchestrate them. Kubernetes is an increasingly popular way to do that.”

History of Kubernetes

The name Kubernetes originates from Greek, meaning helmsman or pilot. Google open-sourced the Kubernetes project in 2014. Kubernetes combines over 15 years of Google’s experience running production workloads at scale with best-of-breed ideas and practices from the community.

Kubernetes was originally developed and designed by engineers at Google. Google was one of the early contributors to Linux container technology and has talked publicly about how everything at Google runs in containers. (This is the technology behind Google’s cloud services.)

Google generates more than 2 billion container deployments a week, all powered by its internal platform, Borg. Borg was the predecessor to Kubernetes, and the lessons learned from developing Borg over the years became the primary influence behind much of Kubernetes technology.

Fun fact: The 7 spokes in the Kubernetes logo refer to the project’s original name, “Project Seven of Nine.”

Why you need Kubernetes? 🔥

Containers are a good and easy way to bundle and run your applications. In a production environment, you need to manage the containers that run the applications and ensure that there is no downtime. For example, if a container goes down, another container needs to start.

You have to ssh to the server, then launch the container, again ssh to another server launch another one. maintain if any container fails… then again check and launch a new container.

Wouldn’t it be easier if this behavior was handled by a system?

That’s how Kubernetes comes to the rescue! Kubernetes provides us an interface to run distributed systems smoothly. It takes care of scaling and failover for your application, provides deployment patterns, and more.

Kubernetes provides you with:

Service discovery and load balancing: Kubernetes can expose a container using the DNS name or using their own IP address. If traffic to a container is high, Kubernetes is able to load balance and distribute the network traffic so that the deployment is stable.
Storage orchestration: Kubernetes allows you to automatically mount a storage system of your choice, such as local storage, public cloud providers, and more.
Automated rollouts and rollbacks: You can describe the desired state for your deployed containers using Kubernetes, and it can change the actual state to the desired state at a controlled rate. For example, you can automate Kubernetes to create new containers for your deployment, remove existing containers, and adopt all their resources to the new container.
Automatic bin packing: You provide Kubernetes with a cluster of nodes that it can use to run containerized tasks. You tell Kubernetes how much CPU and memory (RAM) each container needs. Kubernetes can fit containers onto your nodes to make the best use of your resources.
Self-healing: Kubernetes restarts containers that fail, replaces containers, kills containers that don’t respond to your user-defined health check, and doesn’t advertise them to clients until they are ready to serve.
Secret and configuration management: Kubernetes lets you store and manage sensitive information, such as passwords, OAuth tokens, and SSH keys. You can deploy and update secrets and application configuration without rebuilding your container images, and without exposing secrets in your stack configuration.

The Future of Kubernetes

According to the CNCF, Kubernetes is now the second-largest open source project in the world just behind Linux.

58% of respondents of the survey conducted by CNCF are using Kubernetes in production, while 42% are evaluating it for future use. In comparison, 40% of enterprise companies (5000+) are running Kubernetes in production.

In production, 40% of respondents are running 2–5 clusters, 1 cluster (22%), 6–10 clusters (14%), and more than 50 clusters (13% up from 9%).

As for which environment Kubernetes is being run in, 51% are using AWS (down from 57%), on-premise servers (37% down from 51%), Google Cloud Platform (32% down from 39%), Microsoft Azure (20% down from 23%), OpenStack (16% down from 22%), and VMware (15% up from 1%). The graph below illustrates where respondents are running Kubernetes vs. where they’re deploying containers.

CASE STUDY: Spotify

Challenge

Spotify is an audio streaming platform launched in 2008 has grown over 200 million monthly active uses across the world. They wanted to empower creators and enable really immersive listening experience for all of the consumer that Spotify have. Spotify is an early adopter of microservices and Docker. Spotify had containerized microservices running across its fleet of VMs with a homegrown container orchestration system called Helios. By late 2017, it became clear that “having a small team working on the features was just not as efficient as adopting something that was supported by a much bigger community,”

Solution

To solve this challenge Spotify uses Kubernetes. The migration which happens in parallel with Helios running go very smoothly as “Kubernetes fit very nicely as a compliment and it replaced Helios”. Spotify gets benefited from added velocity and reduced cost and also aligns with the rest of the industry on best practices and tools.

Impact

The biggest service currently running on Kubernetes takes about 10 million requests per second as an aggregate service and benefits greatly from autoscaling, says Site Reliability Engineer James Wen. Plus, he adds, “Before, teams would have to wait for an hour to create a new service and get an operational host to run it in production, but with Kubernetes, they can do that on the order of seconds and minutes.” In addition, with Kubernetes’s bin-packing and multi-tenancy capabilities, CPU utilization has improved on average two- to threefold.

Thank you.

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

How Ansible Tower helps to achieve automation?

Shubham Rasal — Tue, 29 Dec 2020 11:22:17 GMT

ANSIBLE-6

let’s discover the automation with ansible tower use cases

Before understanding what is ansible tower, let's understand what is automation and how an ansible and ansible tower helps us to achieve that.

What is automation?

Automation happens when one person meets a problem they never want to solve again.

I would like to define it in simpler words…
when someone doesn't want to repeat the same task again and again then That's where automation happens.

The world is automating and those who are successful in automation will win.

Let me describe it a little bit, we all know all manufacturing companies use automation in their companies. Automation has transformed factories. It gave manufacturing the ability to perform work faster, more efficiently, at high quality. Factories increased their productivity as well as quality.

Factories that failed to automate fell behind because of so much competition in the market.
That’s why automation is becoming essential for businesses to sustain.

This is not an option anymore

IT departments are also like modern factories that power today’s digital businesses. And let me make a bold statement here…

Just like today's manufacturing companies can not compete without automation same like that the IT companies who fail to use automation soon become out of the competition.

Why automation in IT?

We can achieve a few imperative points in IT organization which are,

Application delivery is fuel for growth
Automation simplifies process
Automation never sleeps
Don’t repeat the same task over and over
Speed up your workflow.

Now we have somewhat an idea of why we need automation, lets see how we do it?

Ansible

You need a tool that can act as a glue layer automating across services and applications no matter where they are. Here’s ansible comes to help you in that.

Yes, you can automate almost everything using ansible from the cloud, containers, network devices, chat applications, monitoring, configuring operating systems, deploying microservices, and much more.

Now if you are using ansible and your server goes down?

let's say we are using one server as a controller node for ansible and due to unexpected reasons our server goes down. then it causes a single point of failure and due to that, our entire automation fails.

To overcome this Redhat provides a cluster for ansible automation with more features called Ansible Tower

Ansible Tower

Ansible Tower (formerly ‘AWX’) is a web-based solution that makes Ansible even easier to use for IT teams of all kinds. It’s designed to be the hub for all of your automation tasks.

Ansible Tower is primarily RedHat official product which has the main purpose to ensure more security features, give control access to who can access what.

There are a lot of features of ansible such as

Inventory can be graphically managed with many cloud providers.
It can log all of your jobs
It also integrates well with LDAP
It also provides amazing browsable REST API
It has command-line tools available for easy integration in CI/CD such as Jenkins
Can create approval-based workflows
Integrated notification like slack, Hipchat
Schedule ansible jobs

Ansible Tower has allowed us to provide better operations and security to our clients. It has also increased our efficiency as a team.
— NASA

Ansible Tower Use Cases

We can create a template to create cloud infrastructure and provision instances on various clouds such as AWS, Azure, Alibaba, Oracle, etc
We can also create approval workflows such as,
- launch microservices in a development environment
- send a notification to slack for approval to deploying it into QA/Pre-prod
- once approved then deploy it to QA/Pre-prod
- Notify each step to slack or other platforms.
We can store credentials for different purposes such as cloud credentials, ssh key, git credentials, but in case there are not pre-defined credentials then you can also create customized credentials.
We can also schedule our tasks of executing the playbooks or jobs.
We can create a job workflow of multiple jobs such as provision VMs, then configuring VMs
We can also see performance, monitors tower, memory usage.
We can also make effective and efficient usage of dynamic inventory at very greater ease.

Thank you.

How to configure Load Balancer and webserver on AWS using Ansible Playbook?

Shubham Rasal — Thu, 24 Dec 2020 19:11:05 GMT

ANSIBLE-5

Configure Haproxy dynamically when a new webserver gets added using ansible.

Before starting how to configure the load balancer and web server let’s understand what is load balancer and webservers.

Load Balancer

The load balancer is software by which we can distribute some tasks to a set of resources to achieve more efficiency in the overall processing.

Why we need a load balancer?

Suppose you have a single server and it has standard hardware and software... and as you grow or suddenly traffic for your app increases. Now each server has some limitations for serving requests at a time, it can be hardware as well as software. Now you decided to add one more server and it comes with a new IP address and then you again need to add this IP as A record in the domain name provider such as godaddy.com or namecheap.com or AWS route 53.

Now you gain more and more clients and more web servers you start adding then it gets complicated… because we need to add these new IPs in the update the DNS provided by the domain name provider. But to reflect these changes in DNS it takes some time (hell lot of time)

Due to a lot of stuff and more complexity... we want something that will manage our servers and face the client's requests and then spread requests to our webservers and again respond back to clients. — — A Load Balancer.

we can use this load balancing in many places such as for web servers or you can use it in API servers or database servers.

So we are going to use “HAPROXY” Load Balancer because it free, open-source software that provides a high availability load balancer and proxy server for TCP and HTTP-based application that spreads requests across multiple servers. It is written in C and very fast and efficient and uses the Round Robin algorithm.

Let's understand the action plan first… before getting hands dirty.

Problem Statement:

Use Ansible playbook to Configure Reverse Proxy i.e. Haproxy and update its configuration file automatically on each time new Managed node (Configured With Apache Webserver) join the inventory.
And set up this on AWS cloud.

I have created a small video explaining how to configure load balancer using ansible-playbook?

I am assuming that you have knowledge about AWS. You can check the below article to launch instances using AWS CLI.

[What is AWS CLI? How to use AWS CLI?
Launch EC2 instance, create EBS and attach EBS volume to EC2 instance using aws CLI.developer-shubham-rasal.medium.com](https://developer-shubham-rasal.medium.com/what-is-aws-cli-how-to-use-aws-cli-6f1bdedabd2b "https://developer-shubham-rasal.medium.com/what-is-aws-cli-how-to-use-aws-cli-6f1bdedabd2b")

So let’s understand the architecture we need to solve this use case.

We have two security groups. One for the load balancer and the second for webservers. Create a security group for load balancer allowing ports 80 and 5000 port and 22 port for ssh. Create another security group and allow load balancer group in its inbound rule for port 5000 and 22 port for ssh. so that outsiders/clients can not directly connect to our webservers.

2. As you can imagine the architecture we want to create from the above image.. So create one instance as a load balancer and assign a load balancer security group. Create 2–3 instances and assign a webserver security group.

Now we have done with AWS setup… So let's start with Ansible and write some interesting playbooks. I am using my local machine as the controller node here.

You can get a brief idea about the setup from the below diagram.

We want to configure webservers using apache server and then add that new server entry in load balancer that way load balancer can also spread requests to the new node.

Now we have a private key file of instances but our ansible should be able to connect to these instances...
The simplest method would be to add your own public keys to your EC2 instance and ignore the PEM file for all future logins

$ ssh-add keyfile.pem

now you are ready to connect using ssh as well as from ansible also.

so the next step is to add instances IP in the inventory file.

Check the connectivity using

$ ansible all -m ping

Now let’s see the folder structure I have created for configuring load balancers as well as web servers.

Here, we have load balancer directory will have load balancer specific files i.e configuration jinja template file and variables file, and then webservers directory will have its configuration template and variables file.

source code has simple index.php which echoes the IP address of the machine.. so we can identify easily that our setup is working or not.

now let’s see how to configure the webserver
You can read more in brief about configuring web server in the below article

This is the configuration file of the apache server where we are changing the document root and port for our project.

Now let’s see web servers vars.yml file

In this file, we are giving the project name, source code directory path, config file path, the port number where we want to deploy our webserver, and the document root of the project on the server.

also, we have to make an entry of the webserver after it is configured as a webserver into the load balancer file so the load balancer can use that webserver. for that we have to update its config file. To dynamically update its file again we will use the jinja template.

we need to add these jinja lines into haproxy.cfg file. It will read the IP addresses from the webserver group in an inventory file and automatically add all the entries in the load balancer.

Now let us see ansible-playbook.

The above playbook will download apache and PHP on webservers then transfer the .conf file and configure the webserver.

In the second play, it will configure a load balancer and also register the webserver into the load balancer.

To see the output, I recommend you to watch the video mentioned above.

You can find all the code and files in below Github repository.

[ShubhamRasal/ansible-playbooks
Ansible playbook to configure a load balancer and webservers. github.com](https://github.com/ShubhamRasal/ansible-playbooks/tree/master/load_balancer "https://github.com/ShubhamRasal/ansible-playbooks/tree/master/load_balancer")

If you have any doubts or something improvement needed in this blog, please feel free to reach out to me on my LinkedIn account.

I hope you learned something new and find ansible more interesting.
Let me know your thoughts about ansible and how do plan to use ansible?

Thank you.

👋 Join FAUN today and receive similar stories each week in your inbox! ️ Get your weekly dose of the must-read tech stories, news, and tutorials.

Follow us on Twitter 🐦 and Facebook 👥 and Instagram 📷 and join our Facebook and Linkedin Groups 💬

If this post was helpful, please click the clap 👏 button below a few times to show your support for the author! ⬇

How to configure apache webserver using ansible?

Shubham Rasal — Thu, 17 Dec 2020 20:26:21 GMT

ANSIBLE-4

Configure Apache server using ansible-playbook

Introduction

Redhat Ansible

Ansible is an open-source automation tool by which we can automate all cloud provisioning, configuration management, application deployment,intra-service orchestration, and many more IT needs.

Ansible is an open-source community project sponsored by Red Hat, it’s the simplest way to automate IT. Ansible is the only automation language that can be used across entire IT teams from systems and network administrators to developers and managers.
You can read more about ansible in the below article and how it helps in DevOps.

Apache Webserver

Apache is an open-source and free web server software that powers around 40% of websites around the world. The official name is Apache HTTP Server, and it’s maintained and developed by the Apache Software Foundation. It allows website owners to serve content on the web — hence the name “webserver.” It’s one of the oldest and most reliable web servers, with the first version released more than 20 years ago, in 1995.

What do we want to do?

Install an Apache web server on the managed node
Configure webserver. change the document root and port number.
Restart the webserver

but restarting HTTPD Service is not idempotence in nature and also consumes more resources so we have to suggest a way to rectify this and solve challenges in the Ansible playbook.

Action

Create entry of managed node in the inventory file.

You can refer to this article on how to create ssh key and use ssh public key authentication.

[Launching Docker Container Using Ansible
Launch Apache webserver docker container using ansibledeveloper-shubham-rasal.medium.com](https://developer-shubham-rasal.medium.com/launching-docker-container-using-ansible-372bbdca9165 "https://developer-shubham-rasal.medium.com/launching-docker-container-using-ansible-372bbdca9165")

Create a new group named “webserver” and add a managed node server IP address there.

Write an ansible-playbook

This is my directory structure.

vars.yml file has all the variables where webserver.yml is ansible-playbook. creatorsbyheart.conf.j2 is a jinja template for changing document root and port number. my site is source-code-directory.

here we have to mention the project or website name for creating the conf file in the configuration directory.
source_code_path is the source code directory location.
webserver_conf_file is the jinja template file location
and port_number and document_root_location for configuration the webserver on that location.

Now let’s take a look at the conf file for our website

here we are reading the port number and document root from the vars.yml file through ansible-playbooks.

Here we are configuring the apache server using ansible.

In ansible, If we restart service using ansible service module… it restarts the service again. i.e it is not ideompotenet in nature. because we only to restart the webserver only if anything is changed.

so for that, we have used handlers that only get notified when we change the conf file. This way we are restarting the webserver each time we run the playbook can be avoided.

I have configured only for the RedHat family you can change variables and software names according to your managed node os_family.

You can find the above playbook on this GitHub repository. Bookmark or star it for future use.

[ShubhamRasal/ansible-playbooks
Contribute to ShubhamRasal/ansible-playbooks development by creating an account on GitHub.github.com](https://github.com/ShubhamRasal/ansible-playbooks/blob/master/Docker/docker-configure.yml "https://github.com/ShubhamRasal/ansible-playbooks/blob/master/Docker/docker-configure.yml")

If you have any doubts or something improvement needed in this blog, please feel free to reach out to me on my LinkedIn account.

I hope you learned something new and find ansible more interesting.
Let me know your thoughts about ansible and how do plan to use ansible?

Thank you.

How to configure Hadoop Cluster using Ansible?

Shubham Rasal — Thu, 17 Dec 2020 08:15:43 GMT

How to configure Hadoop Cluster using Ansible?

Ansible -3

In this article, we will configure the Hadoop name node and data node using ansible. — Shubham Rasal

Introduction

We are going to write playbooks for installing Hadoop and configuring both the name node and data nodes.

I am assuming you have installed ansible and set the inventory file path.

[What is Big Data? How big companies manage Big Data?
How big MNC’s like Google, Facebook, Instagram, etc stores, manages, and manipulate Thousands of Terabytes of data with…developer-shubham-rasal.medium.com](https://developer-shubham-rasal.medium.com/what-is-big-data-how-big-companies-manage-big-data-21124d639d50 "https://developer-shubham-rasal.medium.com/what-is-big-data-how-big-companies-manage-big-data-21124d639d50")

In this article, I have given high-level details about what is Hadoop and why it is used? Check it if you want to know more about Hadoop.

so without delay start writing our playbooks.

Action

Action mode 🔥

We will use ssh public key authentication to configure managed/target nodes. The motivation to use public-key authentication over simple passwords is security. Public key authentication provides cryptographic strength that even extremely long passwords can not offer. With SSH we also don’t need to remember long passwords.

So let’s create one ssh key using the below command for that…
Go to the .ssh folder

$ cd .ssh$ ssh-keygen -t rsa -b 4096

Now it will generate two files private and public keys in the .ssh file.

Now we have to copy the public key to the managed node for ssh authentication. for that we will use the ssh-copy-id command or else you can use the SCP command to authorized files also.

$ssh-copy-id -i ansible_key.pub username@managed_node_ip

It will ask first the time user password and enter the password then you are ready to go. above command append your id to ssh authorized file of a managed node.

In my case, my managed node IP address is 192.168.225.182. You can check IP address using the $ ifconfig command

Now let’s test that we can connect manually to the managed node before moving to an ansible setup.

$ ssh username@hostname

using the above command you can test that we have successfully transferred the ssh public key or not. You can see that the managed node authorized_keys file has your generated public key.

Now we have set up ssh connection now we are ready to connect and configure the managed node using ansible.

Follow the same process for other nodes to get secure connections. In our case, we have two managed nodes one is the Hadoop name node and the data node.

Now create two groups in the inventory file and put IPs in those groups.

let’s test before moving forward do we have connectivity or not using the ping module.

yes… now we can move forward

Now we are all set to write the playbook … then let's start.

Now I have partitioned this task into two parts,
1. Installation
2. Configuration

The installation will be the same on both the machines and the configuration might have to change according to need.

This is a directory structure where hadoop-install.yml will have installation code and hadoop-configuration.yml will contains configuration code.

We have two directories 1. name node which will contain files for name node and 2. data node which will have data-node specific files.
and we have JDK and Hadoop rpm files which we are going to install.

Let's begin playbooks then installing Hadoop.
//hadoop-install.yml

The above playbook will detect the user directory and copy JDK and Hadoop and then if they are successfully transferred it will install them.

Now let’s run the above playbook

$ ansible-playbook hadoop-install.yml

As you can see now we have installed hadoop on both the name node and data node. Let’s confirm it on both the nodes that we have successfully installed or not.

yes, now we have installed hadoop… its time to configure the name node and data node.

now here's we have to declare few variables..

//vars.yml

This file now only one variable but you can add more and more variables that are common to both nodes here.

Configure Namenode:

Now we have a different vars.yml file for name node where we will store name node-specific variables.

we have to edit two files in name node 1. core-site.xml and hdfs-site.xml
we will make it dynamic using jinja templating.

above file will read name node IP from inventory file and port from global vars.yml file

above file is reading namenode_directory variable from we declared in namenode/vars.yml.

Configure Datanode:

//datanode/vars.yml

//datanode/core-site.xml

//datanode/hdfs-site.xml

now we have created templates and variables let's configure the nodes.

hadoop-configuration.yml

Here we are creating a directory and transferring templates and starting Hadoop service on both nodes according to their type

#to run only for name node
$ ansible-playbook hadoop-configuration.yml --tags namenode

#to run only for data node
$ ansible-playbook hadoop-configuration.yml --tags datanode

#to run full playbook
$ ansible-playbook hadoop-configuration.yml

And we are done… we have successfully completed the configuration setup for Hadoop using ansible.

You can check using the below output confirm.

You can find the above playbook on this GitHub repository. Bookmark or star it for future use.

[ShubhamRasal/ansible-playbooks
You can't perform that action at this time. You signed in with another tab or window. You signed out in another tab or…github.com](https://github.com/ShubhamRasal/ansible-playbooks/blob/master/Docker/docker-configure.yml "https://github.com/ShubhamRasal/ansible-playbooks/blob/master/Docker/docker-configure.yml")

If you have any doubts or something improvement needed in this blog, please feel free to reach out to me on my LinkedIn account.

I hope you learned something new and find ansible more interesting.
Let me know your thoughts about ansible and how do plan to use ansible?

Thank you.

About the writer:
Shubham loves technology, challenges, is open to learning and reinventing himself. He loves to share his knowledge. He is passionate about constant improvements.
He writes blogs about Cloud Computing, Automation, DevOps, AWS, Infrastructure as code.
Visit his Medium home page to read more insights from him.

Launching Docker Container Using Ansible

Shubham Rasal — Sun, 13 Dec 2020 21:03:57 GMT

Ansible -2

Launch Apache webserver docker container using ansible

We are going to launch the apache webserver docker container on top of a managed node with the help of Ansible.

Before moving to the action let's take a look at what is docker, ansible, and apache server so we will get a little idea about these technologies and the need for this integration. so without waiting let's get into the introduction.

Introduction 🤓

Redhat Ansible

Ansible is an open-source automation tool by which we can automate all cloud provisioning, configuration management, application deployment,intra-service orchestration, and many more IT needs.

Docker 🐳

Docker is an open-source project launched in 2013. It is a software platform for building applications based on containers — small and lightweight execution environment share make shared use of the operating system kernel but otherwise run in isolation from one another.

Docker enables you to separate your applications from your infrastructure so you can deliver software quickly.

Apache Webserver

Problem Statement 🤯

Write an Ansible PlayBook that does the following operations in the managed nodes:

Configure Docker.
Start and enable Docker services.
Pull the httpd (Apache) server image from the Docker Hub.
Run the docker container and expose it to the public.
Copy the html code in the document root directory and start the webserver.

Now we have a clear idea of what we want to do…
So let’s see step by step how we are going to do this

Action mode 🔥

So let’s create one ssh key using the below command for that…
Go to the .ssh folder

$ cd .ssh

$ ssh-keygen -t rsa -b 4096

Now it will generate two files private and public keys in the .ssh file.

Now we have to copy the public key to the managed node for ssh authentication. for that we will use the ssh-copy-id command or else you can use the SCP command to authorized files also.

$ssh-copy-id -i ansible_key.pub username@managed_node_ip

It will ask first the time user password and enter the password then you are ready to go. above command append your id to ssh authorized file of a managed node.

In my case, my managed node IP address is 192.168.225.182. You can check IP address using $ ifconfig command

Now let’s test that we can connect manually to the managed node before moving to an ansible setup.

$ ssh username@hostname

using the above command you can test that we have successfully transferred the ssh public key or not. You can see that the managed node authorized_keys file has your generated public key.

Now we have set up ssh connection now we are ready to connect and configure the managed node using ansible.

Now add managed node IP to your inventory file.
I have created one group called ‘docker’ and inside that group, I have added managed node IP. You can see the in the below image.

$ ansible docker --list-hosts

The above command shows the IP address list in the group name that we have added to the inventory file.

Now let's run the first command to ensure that ansible has a proper connection with the managed node.

$ ansible docker -m ping

If you see a green response and ping: pong then yes.. we are fully ready to start…

Install Docker

Configure yum repository

First, we have to add a yum repo for docker so we can download docker software from the yum repository.
Let’s check that the managed node has a yum repo for docker or not.

The below image is of the managed node command output. I will show both the screenshots of the managed node and controller node for better understanding.

You can see that it does not have any docker repo added.

Ansible has a yum_repository module by which we can add a yum repository in the managed node. But we want to configure yum only if the managed node is from the RedHat family.

Create an Ansible playbook with the below code with extension .yml

You can see in the above image that I have added ‘docker’ in hosts that group name that we have added in the inventory file. we have added the above code it's time to run the above code.

$ ansible-playbook docker-cofigure.yml

You can see the orange output means something has changed on the managed node. so we have configured the docker repo successfully. you can check in the below image.

Install docker

Now we have configured the yum repo then we can easily download docker software using the package module in ansible.

Before this let’s check that on the managed node we have already installed docker or not. As you see we our managed node does not have docker software.

To install software we can use the package module in ansible. so let's take the help of the package module to install docker. Add below code under yum repository task.

after this time to run the playbook again and test.

Ansible has idempotent so you can check that it does not configure yum again. and it has changed something in docker task and returned with orange output that means it has successfully installed docker software for us. let’s go and check on the managed node.

yes.. we have successfully installed docker on the managed node.

Now to perform an operation on docker we have to start the docker service. because it is by default inactive.

to start docker service we have a ‘service’ module in ansible. we will use the ‘service’ module to start the docker service.

add below code to our playbook file to start the docker service.

now let’s run the playbook again to see the effect.

now we have started docker using ansible let’s confirm it using systemctl command on the managed node.

Now we have installed docker and started docker successfully.. its time to launch the container… no we have to download docker SDK because it is a pre-requisite for our next module

so let’s write the code to download docker SDK before launching the docker container. First, install python pip3 using the package and then download docker SDK using pip

After running these tasks using ansible-playbook docker-configure.yml

You can see in above image it has successfully downloaded docker SDK using pip. You can check it on the managed node using the pip list command.

Now finally finished with docker installation totally. now we can launch as many containers as we want…

but before that, we have to copy our source code of the website to the managed node.
we have to copy somewhere in the managed node but let's make it a little organized and create one directory for our project and then copy source code to that directory.

so let’s declare one variable in the playbook, or you can create another file for the variable and then imports the variable from that file. but for simplicity, I am keeping it into the single playbook. The benefit of declaring a variable for another project I don’t need to go and manually update code, Just update the value for the variable.

Here “dev-creatorsbyheart” is my development version of the creatorsbyheart project. I want to also name this as a container name so I can differentiate between different environments of the project.
Let’s create a directory where we want to copy the source code.

Let's confirm that the directory is not already created. As you can see at the current location there is no directory of the project name.

Add below code at the end of the playbook.

Here we are reading the value of user_dir from ansible_facts. In our case, it is /root as we have login using root. Then at that location, we will create a new directory of the project name using the ‘file’ module in ansible.
Let’s run it and check it has created is successfully or not.

Now that we have created the directory time to copy source code to the directory created on a managed node in the above step.

for copying files/directory we have the ‘copy’ module in ansible. we will use that module to achieve our goal.
before that declare a variable for project files because the path varies for different projects.

Add below task for copy files.

Here we are copying files from the directory mentioned in the source_code_dir variable to the destination folder we created in the previous step.

Let’s run and check the ansible successfully copied to the proper destination or not.

Now the time is arrived to launch the container and deploy our website using the container. My need is to use apache webserver so I will use apache docker image. Before that let's check any container is running or not on the managed node.

No there is no container running now… so let’s run our webserver without wasting time…
Now there will be the time we have multiple docker containers and the port number will also vary. so create a new variable for the port number.

and add the below task also for launching the docker container. I have used only the variable for port numbers you can customize as per your need.

We are using an ansible ‘docker_container’ module for launching an httpd container. It will pull the image if it is not already downloaded in a managed node and launch the container. we are exposing its HTTP port and attaching the source code directory with the document root of the apache webserver. That way container can directly access files from the directory.

Let’s check the container is running or not on managed not.

Yes, it running and now we can connect to a webserver using managed node IP and the port we have mentioned in variables.

If you facing trouble with connecting to this URL, you might need to restart the firewall of a managed node or add port forwarding rules for the container.
In ansible we have ‘firewalld’ module for that.. you can use that as you need.

You can multiple things with ansible to know what we can achieve using Ansible I recommend you to read this blog.

You can find the above playbook on this GitHub repository. Bookmark or star it for future use.

If you have any doubts or something improvement needed in this blog, please feel free to reach out to me on my LinkedIn account.

I hope you learned something new and find ansible more interesting.
Let me know your thoughts about ansible and how do plan to use ansible?

Thank you.

Create High Availability Architecture with AWS CLI | Shubham Rasal

Shubham Rasal — Thu, 10 Dec 2020 20:08:39 GMT

AWS -11

Creating cloud architecture for the web app with low latency for resources

We are going to create the above architecture using AWS CLI. Let’s discuss architecture and how we are going to do it?
🔶The architecture includes-
◼️ Launch EC2 instance
◼ Create EBS volume
◼Mount EBS volume to EC2 instance
◼Create an S3 bucket
◼Upload static objects such as images, videos, or documents in the S3 bucket
◼Create a content delivery network (CDN) using aws CloudFront distribution for S3 bucket
◼️️ Configure instance as the webserver
◼Place CloudFront URL in an application for security and low latency
◼Deploy source code on EC2 instance webserver.

The final goal is to achieve all these tasks using AWS CLI without going to the website and manual clicks.

If you are very new to AWS CLI, I would like you to check out the basic introduction to AWS CLI

[Deploying Angular App to AWS S3 with CloudFront using AWS CLI
Deploy angular website on aws using aws CLIdeveloper-shbham-rasal.medium.com](https://developer-shubham-rasal.medium.com/deploying-angular-app-to-aws-s3-with-cloudfront-using-aws-cli-ace33350a950 "https://developer-shubham-rasal.medium.com/deploying-angular-app-to-aws-s3-with-cloudfront-using-aws-cli-ace33350a950")

Action Mode 🔥

I am assuming that you have created an IAM user that has EC2, CloudFront, and IAM access and configured AWS CLI.

Create Key Pair

To create keypair we have to use the ec2 service of AWS. by using

$ aws ec2 help

you will see all the subcommands under ec2.

To create a key pair and save it in the proper format(.pem), use the below command.

Note: Windows users use PowerShell for the above commands. To activate PowerShell on normal cmd. Enter $ PowerShell command. OR Win + R and enter PowerShell to open Powershell.

Create a Security Group

You may want to create a security group in a specific VPC. For that, we need a vpc id. Let’s find the VPC id using AWS CLI.

the above command will return the list of VPC’s and tags associated with VPC.
copy the vpc id you want to create SG in and save it somewhere.
Now let's create a security group.

Don't forget to edit the command according to your need. You can change your name, description, and vpc id.

Create security group rules

we want to add rules to the security group that we have created above.
If you copy-paste the security group id then skip the below command...
Let's find the security group name and id.

the above command will give you a list of security group names and ids.
copy the id that we want to add a new rule and paste it in the below command.

the above command will create a new rule for ssh.. you can always customize as per your need... you know that right?
You can check more examples of multiple rules here...

Check that rule added successfully or not using describe subcommand.

Create a new instance

You may want to launch an instance in a specific availability zone. let’s see with AZ are available in the region you specified while configuring.

We want to attach one more EBSvolume to our instance and for that, both should in the same availability zone. Fix your availability zone for that.
AWS CLI has very well described documentation for ec2.. let's take the help of that. It will give more examples and necessary information. You always need something different than I used here.. so go and use

$ aws ec2 help

To launch an instance we need ami id of the image that we want to launch..

$ aws ec2 describe-images

I have used the key pair and security group that we have created above. I choose the ap-south-1a availability zone and amazon Linux ami to launch this instance.
You can do much more customization to this command. Check this aws Documentation.(open)
copy instance id and save it somewhere.. we will see later where we need.

run instance output

Create EBS volume

Our goal is to create a new volume and attach it to an instance that we have just created. So we need to create it in the same Availability Zone.

Update the above command as per your need. (size in GiBs).
copy volume id and save it somewhere.

Attach EBS volume to the instance.

update the instance id and volume id in the above command that we have copied and save.. remember?

Create an S3 bucket

What is S3? I mean what google says…
Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services that provides object storage through a web service interface.

here we are using s3 APIs to create a new bucket with the name dev-creatorsbyheart.com

Update the s3 bucket policy so we can access it publically.

Create a policy

Create a new file and give the name s3_bucket_policy.txt and paste the below code. Don’t forget to change the bucket name in the “Resource” attribute

update the bucket policy using the about command and make objects available to the public for getting action.

Upload files to s3 bucket using s3 command

Check the image is visible or not using the image URL. https://s3.amazonaws.com//<object>

If it is working all then it is time to create CloudFront for s3

Create CloudFront distribution for S3 bucket

What is CloudFront? This is what google says…

Amazon CloudFront is a content delivery network offered by Amazon Web Services. Content delivery networks provide a globally-distributed network of proxy servers that cache content, such as web videos or other bulky media, more locally to consumers, thus improving access speed for downloading the content.

Create a new file containing the below code and name it cf_config.json

create CloudFront with the above configuration. Update the above file and replace a bucket name in the target origin name.

It will create CloudFront distribution for the s3 bucket. now we can also access s3 objects using the CloudFront URL.

If you are with me till here… give a pat on the back… good job.

Give this URL to the developer so they can use this URL to show resources in the source code. Using CloudFront URL resources can load fastly and we can achieve less latency.

Now you can do ssh and configure the instance as a web server and mount EBS volume to document root and deploy source code.

I hope you learned something new and find aws CLI interesting.
Let me know your thoughts about this article and how do plan to use the aws CLI tool?

Thank you

About the writer:

Shubham loves technology, challenges, is open to learning and reinventing himself. He loves to share his knowledge. He is passionate about constant improvements.
Visit his Medium home page to read more insights from him.

What is Ansible ? How ansible is helping companies in automation ?

Shubham Rasal — Tue, 01 Dec 2020 11:44:27 GMT

Ansible 1:

How we can use Ansible to automate IT and DevOps?

Ansible is a radically simple IT automation engine that automates cloud provisioning, configuration management, application deployment, intra-service orchestration, and many other IT needs.

Ansible is an absolutely free and open-source tool that is used for the above-mentioned purposes.

Designed for multi-tier deployments since day one, Ansible models your IT infrastructure by describing how all of your systems inter-relate, rather than just managing one system at a time.

It uses no agents and no additional custom security infrastructure, so it’s easy to deploy — and most importantly, it uses a very simple language (YAML, in the form of Ansible Playbooks) that allow you to describe your automation jobs in a way that approaches plain English.

What do we can automate using ansible?

Ansible functionality addresses key security and compliance use cases for service providers, including:

Standardizing and enforcing auditing and compliance.
Maintaining firewall compliance.
Centralizing all system-related changes.
Upgrading driver and firmware.
Automating remediation and patching of system vulnerabilities, such as Wannacry, Spectre, and Meltdown.
Detecting system vulnerabilities and needed remediation by gathering Ansible facts and system event logs and exporting the information to system monitoring tools.

Ansible for DevOps

Automation has transformed factories. It gave manufacturing the ability to perform work faster, more efficiently, at a higher quality. Factories increased productivity as well as quality.

Factories that failed to automate fell behind due to so much competition in the market. That's why automation became essential for businesses to sustain.

IT departments are the modern factories powering today’s digital businesses. And just as today’s factories can’t compete without automation, automation will soon become imperative for IT organizations. because

Application delivery is fuel for growth
Automation simplifies process and automation never sleeps
Don’t repeat the same task over and over
Speed up your workflow

You need a tool that can act as the glue layer automating across services and applications no matter where they are. Once one person on your team learns how to do something, they can capture their solution in an Ansible Playbook and enable everyone to use it

Simple to adopt, simple to use, simple to understand — Ansible is designed around the way people work and the way people work together across the entire organization:

1. Dev

Challenge: Dev spending too much time focusing on tooling required to deliver capabilities and not enough time focusing on results

Need: To respond and scale in pace with demand

How does Ansible automation help?

Accelerates feedback loop
Discover bugs sooner
Reduces risk of tribal knowledge
Faster, coordinated, and more reliable deployments

2. Ops

Challenge: Need technology that can be used across many different groups with many different skill sets

Need: Centrally govern and monitor disparate systems and workloads

How does Ansible automation help?

Reduce shadow IT
Reduce deployment time
Provision systems faster
Reduce risk of tribal knowledge
Deploy automated patching

3. QA/Security

Challenge: Tracking of what changed where and when

Need: Reduce risk of human error

How does Ansible automation help?

Establish identical QA, Dev, and Prod environments for faster, coordinated, and more reliable deployments
Establish security baselines
Increase visibility and accuracy for compliance requirements
Relieve the burden of traditional documentation by creating living, testable documentation

3. Business

Challenge: Getting to market faster

Need: Create a competitive advantage

How does Ansible automation help?

Align IT with the business
Increase time for innovation and strategy
Reduce costs of onboarding new team members
Increase cross-team collaboration

I hope you learned something new and find ansible interesting.
Let me know your thoughts about ansible and how do plan to use ansible?

Thank you.

About the writer:
Shubham loves technology, challenges, is open to learning and reinventing himself. He loves to share his knowledge. He is passionate about constant improvements.
Visit his Medium home page to read more insights from him.

How ISRO uses Machine learning?

Shubham Rasal — Tue, 20 Oct 2020 12:51:15 GMT

ML-1

What is AI?

Artificial intelligence (AI), is intelligence demonstrated by machines, unlike the natural intelligence displayed by humans.AI systems will typically demonstrate at least some of the following behaviors associated with human intelligence: planning, learning, reasoning, problem-solving, knowledge representation, perception, motion, and manipulation and, to a lesser extent, social intelligence and creativity.

Artificial intelligence (AI) refers to the simulation of human intelligence in machines that are programmed to think like humans and mimic their actions.

What is Machine Learning?

Machine learning is an application of artificial intelligence (AI) that provides systems the ability to automatically learn and improve from experience without being explicitly programmed. Machine learning focuses on the development of computer programs that can access data and use it to learn for themselves.
Huge historical data is feed to the machine learning model and by that machine learning does the prediction.

How ISRO uses Artificial Intelligence and Machine Learning?

➤Chandrayaan 2: AI-powered ‘Pragyan’ Rover

On 22 July 2019, ISRO launched Chandrayaan 2 spacecraft into an earth orbit as part of the second lunar mission.

In Chandrayaan 2, the Pragyan rover was AI-powered which can communicate only with the Lander, includes a piece of motion technology developed by IIT-Kanpur researchers that will help the rover manoeuvre on the surface of the moon and aid in landing. The algorithm will help the rover trace water and other minerals on the lunar surface, and also send pictures for research and examination.
The rover is a six-wheeled robotic vehicle and is capable of conduct in-situ payload experiments. It is powered by AI tools and frameworks, uses solar energy for its functioning, and can communicate only with the Lander. The Pragyan Rover payloads consist of Alpha Particle X-ray Spectrometer (APXS) and Laser Induced Breakdown Spectroscope (LIBS).

Chandrayaan-2 Pragyan shows how AI is helping space exploration.

➤Multi Object Tracking Radar (SDSC-SHAR)

The challenge of building a Space object tracking solution to build successful sustenance of satellites through the difficult terrain of open space with millions of unknown objects that could impact every ISRO sponsored mission.

The objective is to build Multi Object Tracking Radar. ISRO first developed Target identification using machine learning algorithms from MOTR radar data.
Radar data consists of Range, Azimuth, Elevation and Signal to Noise Ratio (SNR). From Range and SNR correlation target size can be classified. From SNR variation alone in a single-track duration, target nature can be established. Using Machine Learning algorithms, a model should be trained on radar tracked data (Range, Azimuth, Elevation and SNR). The trained model should identify a target nature (controlled or uncontrolled) and size. Using standard libraries in Python Machine Learning Algorithms have become realizable models.

➤Image Processing and Pattern Recognition (IIRS)

In 1980s and 1990s ISRO’s the challenge was to build efficient and cost neutral image processing and pattern recognition solution for upcoming missions for next decade. Hence, Unmanned Image Processing and Pattern Recognition (IIRS).

ISRO leveraged Artificial Neural networks (ANN) which is the generic name for a large class of machine learning algorithms, most of them are trained with an algorithm called back propagation. ISRO’s team used various path to explore various deep learning algorithms in various applications of earth observation data like; self-learning based classification, prediction, multi-sensor temporal data in crop/forest species identification, remote sensing time series data analysis.

➤AI-enabled monitoring system for forest conservation

The National Remote Sensing Centre (NRSC), which ISRO has designed and developed, is a monitoring system to observe forest cover change and combat deforestation by leveraging optical remote sensing, geographic information system, AI, and automation technologies.

ISRO creates a machine learning model that checks the imagery and detect small-scale deforestation and improve the frequency of reporting.
It also enables scientists to process satellite imagery faster and reduces the time frame for new reports from one year to one month. NSRC aims at preventing negative changes in the green cover and protection of wildlife.
The NRSC technology makes it possible for monitoring forest cover changes over small areas of one hectare by improving the resolution from 50 meters to 30 meters through optical remote sensing which provides insights into the smallest of deforestation activity.

➤Autonomously Navigating Robot for Space Mission (IISU)

ISRO’s challenge was to build and send unmanned robots to help fetch critical space information in multiple missions throughout the year.

A half Vyomnoid with Sensing and perception of surroundings with 3D vision and Dexterous manipulative abilities to carry out defined crew functions in an unmanned mission or assist crew in manned missions.
Design & Realization of FULL Vyomnoid with features that include full autonomy with 3D vision, dynamically controlled movement in zero ‘g’, Artificial Intelligence / Machine Learning enabled real time decision making with vision optimization and path planning algorithms.
ISRO leveraged Artificial Intelligence enabled Path Navigation algorithms to solve this.

➤MORE:

SRO has advised scientists and researchers to focus on building a generalized parameter extraction software based on artificial neural network (ANN) learning methods that utilize multidimensional approximation of ANN to map characteristics of microwave filters. A communication satellite contains a large number of microwave filters that are required to undergo extensive tuning after fabrication.
ISRO is currently planning to develop high-end propulsion technology to ensure cost-effective re-usable, recoverable, re-startable and reliable space launches with AI-based sensors equipped in propellants.

Sources:

ISRO Research paper: (open)
indiaai.gov.in (open)
Mint.com (open)
un-spider.org. (open)
sac.gov.in (open)

So this was an Overview of How ISRO is solving problems in every way possible because of AI and how our lives are evolving and getting better day by day with the help of AI, I hope you found this blog helpful….Your feedback is valuable and it will help me improve… clap if you like it.

Deploying Angular App to AWS S3 with CloudFront using AWS CLI

Shubham Rasal — Sun, 18 Oct 2020 18:12:11 GMT

aws -10

Deploy angular website on aws using aws CLI

Introduction 🤩

In this blog, we will see how to deploy the angular app on AWS fastest way using the AWS CLI tool. As you came here. so I assume that you are aware of AWS and Angular. so without wasting time on theory let's get into action.

Let’s plan before the real action

Create an S3 bucket
Create and update permission of bucket and make it public
Start static website hosting service of S3
Create CloudFront distribution for S3 bucket
Create an Angular project
Upload your code and assets to S3
Invalidate the CloudFront
Get CloudFront URL

Its simple isn’t it? yes… let's start then

Assumption

I am assuming that you have read this blog and configure aws CLI.

[What is AWS CLI? How to use AWS CLI?
Launch EC2 instance, create EBS and attach EBS volume to EC2 instance using aws CLI.medium.com](https://medium.com/@developer.shubham.rasal/what-is-aws-cli-how-to-use-aws-cli-6f1bdedabd2b "https://medium.com/@developer.shubham.rasal/what-is-aws-cli-how-to-use-aws-cli-6f1bdedabd2b")

Action Mode 🔥

What is S3? I mean what google says...
Amazon S3 or Amazon Simple Storage Service is a service offered by Amazon Web Services that provides object storage through a web service interface.

Create an S3 bucket

bucket name should be unique in your region so try to be a little creative and give the region you want to create a bucket.

Create and update permission of bucket and make it public

Create a policy

Create a new file and give the name s3_bucket_policy.txt and paste the below code. Don’t forget to change the bucket name in the “Resource” attribute

Update bucket policy

Now that we have created the policy we want to update the s3 bucket policy.

Check that the file exists in the current folder otherwise provide the full path.

Start static website hosting service of S3

Create one new file with the below code and save it with the name ‘website_configuration.txt’. Using this we want to enable and configure some setting of the S3 bucket

Update the error document and index document which you want to set..
In my case, I set index.html for both. but you may have your error page.

Enable Static website hosting

Create CloudFront distribution for S3 bucket 🤯

Here comes the interesting part.

What is CloudFront? This is what google says…

Create a new file containing the below code and name it cf_config.json

If you want more customization aws CLI to provide skeleton you can use that.

The above will create a new file cf_config.json file with the empty skeleton. You can add the details and you are ready…
I also used this approach.

I have used the S3 bucket name in DomainName.
Custom error page if forbidden and bad request. you can add more according to your need.
To know in detail what I did aws has very well described documentation.

$ aws cloudfront create-distribution help

Create CloudFront distribution

after successful, it will give you details about the CloudFront.

Note: If you create a file using the above command it has the “DistributionConfig” attribute and extra braces remove it and make it as the above code.
Error looks like

if you come here… you deserve a pat on the back 👏

Create an Angular project.

I assume that you have an angular project but if not let's create.

$ ng new

Create a build for deploy

$ ng build --prod

It will create build its output path. In my case dist folder in the root folder.

Upload your code and assets to S3

Time to upload code to s3 bucket. for that, we will use the s3 subcommand of AWS.
Update your folder name and bucket name in the below command.

the above command will copy all the files excluding ‘.svg’ images.
Now copy the .svg files to our S3 bucket.

You may wonder why two different commands to copy the files from the same folder .. why not use one command and copy all the files in one go... right?
If we copy all the files in one command then the .svg files do not load appears on the website. for that, we have to add a content type flag while copying SVG files.

Invalidate the CloudFront

As you know that what is CloudFront. It saves cache at edge locations. so if you are uploading files in again and again in the S3 like through CICD. you need to invalidate the CloudFront.
By Invalidating the files from edge caches. The next time a viewer requests the file, CloudFront returns to the origin to fetch the latest version of the file.

Get CloudFront URL

There are multiple ways to parse and get the output. I like to keep it simple.
for more details, you can check this article.

[Controlling command output from the AWS CLI
This topic describes the different ways to control the output from the AWS Command Line Interface (AWS CLI). The AWS…docs.aws.amazon.com](https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html "https://docs.aws.amazon.com/cli/latest/userguide/cli-usage-output.html")

Copy the CloudFront URL and paste it into the browser tab. let's hope for the best…..

Congratulations...😻 It is working superbly. Now you deserve a break... go and grab a cup of coffee ☕️

Next steps

Routing traffic to an Amazon CloudFront web distribution by using your domain name.
Implement CICD for an angular project for multiple environments.
Update CloudFront and S3 using AWS CLI

Wrapping Up 🥳

We have successfully deployed our angular app on aws S3 with Cloudfront.
You may think that we have invested lots of time in finding the attributes and values for our needs. we can do all this in 15 mins using the management console so why use aws CLI? It is a one-time investment and it will save a lot of time when you won't do all the steps for different projects or for different configurations like dev, stag,e, and prod. Few changes can save you many minutes and avoid the human errors that we tend to do.

What is AWS CLI ? How to use AWS CLI ?

Shubham Rasal — Sat, 17 Oct 2020 12:42:30 GMT

AWS CLI

AWS-9

Launch EC2 instance, create EBS and attach EBS volume to EC2 instance using aws CLI.

Introduction 🤓

AWS is a market leader and top innovator in the field of Cloud Computing. AWS has integrated building blocks that support any application architecture, regardless of scale, load, or complexity.
But AWS is more than beautiful and eye-catching AWS Web UI console. Let's discover the Amazon’s Command Line Interface — AWS CLI.

What is AWS CLI? ❤️

The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.

We can achieve more speed and customization through AWS CLI than the AWS management console. It improves the convenience and productivity of DevOps engineers and developers.

Installing AWS CLI ‍💻

There are two versions of AWS CLI.

Version 1.x
The previous version of the AWS CLI is available for backward compatibility.
Version 2.x It is the current version. It has more features than version 1

To install AWS CLI, there are multiple ways you can check that here. (Open). But I will show my favorite way to download and install AWS CLI.

#install awscli using pip tool
$ pip install awscli --upgrade --user

#check installation
$ aws --version

Note: If you don't have a pip tool then download get-pip.py from https://bootstrap.pypa.io/. and run this file using python.
$python get-pip.py
for more details (open)

Configure AWS CLI 🚀

1. Create an IAM user that has an EC2 creation policy.

I recommend you to create PowerAccessPolicy for Practice.

For more details, you can refer to this: How to create IAM user

2. You will get ACCESS KEY and SECRET KEY using that you can now configure to aws on CLI.

aws configure — profile “name”

How to use AWS CLI? 🧐

Using CLI you can achieve almost everything that you can do using AWS management console but with less efforts and clicks.

Lets try something using AWS CLI,

🔅 Create a key pair
🔅 Create a security group
🔅 Launch an instance using the above created key pair and security group.
🔅 Create an EBS volume of 1 GB.
🔅 The final step is to attach the above created EBS volume to the instance you created in the previous steps.

Mostly I use the help command instead of remembering commands.
AWS CLI documentation is informative and well described with examples.

$ aws help

Create Key Pair

To create keypair we have to use ec2 service of AWS. by using
$ aws ec2 help
you will see all the subcommands under ec2.

To create key pair and save it in proper format(.pem), use below command.

Note: Windows users use PowerShell for above commands. To activate PowerShell on normal cmd. Enter $ PowerShell command. OR Win + R and enter PowerShell to open Powershell.

Create a Security Group

You may want to create a security group in a specific VPC. For that, we need vpc id. Let's find VPC id using AWS CLI.

above command will return the list of VPC’s and tags associated with VPC.
copy the vpc id you want to create SG in and save it somewhere.
Now lets create security group.

Dont forget to edit the command according to your need. You can change name, description and vpc id.

Create security group rules

we want to add rules in the security group that we have created above.
If you copy-paste the security group id then skip the below command..
Lets find the security group name and id.

the above command will give you a list of security group names and ids.
copy the id that we want to add new rule and paste it in below command.

the above command will create a new rule for ssh.. you can always customize as per your need.. you know that right?
You can check more examples for multiple rules here..

Check that rule added successfully or not using describe subcommand.

Create a new instance

You may want to launch an instance in a specific availability zone. let's see with AZ are available in the region you specified while configuring.

We want to attach one more ebs volume to our instance and for that both should in same availability zone. Fix your availability zone for that.
AWS CLI has very well described documentation for ec2.. lets take help of that. It will give more examples and necessary information. You always need something different than I used here.. so go and use
$ aws ec2 help

To launch an instance we need ami id of the image that we want to launch..

$ aws ec2 describe-images

I have used the key pair and security group that we have created above. I choose the ap-south-1a availability zone and amazon Linux ami to launch this instance.
You can to much more customization to this command. Check this aws Documentation.(open)
copy instance id and save it somewhere.. we will see later where we need.

Create EBS volume

Our goal is to create a new volume and attach to an instance that we have just created. So we need to create it in the same Availability Zone.

Update the above command as per your need. (size in GiBs).
copy volume id and save it somewhere.

Attach EBS volume to instance.

update the instance id and volume id in the above command that we have copied and save.. remember?

That's all, you may find that lots of commands we used. but you can create one script combining all the commands. You can modify and run whenever it is necessary.
If you are already using AWS using WebUI. I recommend you to give to try to AWS CLI. You will notice the time you save by using CLI.

Wrapping Up 🥳

So far we have created a new key pair, security group, add ssh rule into it, provision new instance, create new ebs volume and attach to the instance.
but… you can do much more than this.
Tell me did you find this article helpful and will you use aws cli tool?
in the comment section below.
Thank you..

[Deploying Angular App to AWS S3 with CloudFront using AWS CLI
Deploy angular website on aws using aws CLImedium.com](https://medium.com/@developer.shubham.rasal/deploying-angular-app-to-aws-s3-with-cloudfront-using-aws-cli-ace33350a950 "https://medium.com/@developer.shubham.rasal/deploying-angular-app-to-aws-s3-with-cloudfront-using-aws-cli-ace33350a950")

Building Scalable Applications and Microservices on AWS.

Shubham Rasal — Mon, 21 Sep 2020 17:22:51 GMT

AWS-8

How is AWS helping to build microservices architecture to solve the new age problem?

Introduction 🤓

In today’s age, the world is moving to faster development and scalable approach.🌩️

What is microservices?

Microservices architectures are not a completely new approach to software engineering, but rather a combination of various successful and proven concepts such as:
• Agile software development
• Service-oriented architectures
• API-first design
• Continuous Integration/Continuous Delivery (CI/CD)

Microservices architectures make applications easier to scale and faster to develop, enabling innovation and accelerating time-to-market for new features.

Monolithic vs. Microservices Architecture

This is important to understand why we microservices architecture.

Monolithic architecture.🖥

With Monolithic architectures, all the processes are tightly bonded and it runs as a single service. Suppose you have an application and suddenly one of the processes of the application experiences spikes in demand and this makes your application slow response to requests. If you want to solve this issue you have to scale the entire architecture. Adding and improving the monolithic application’s feature becomes more complex as the code base grows. This process results in a limitation of experimentation and makes it difficult to implement new ideas. Monolithic architectures add risk for application availability because many dependent and tightly coupled processes increase the impact of single process failure.

Microservices Architecture 🖥🖥🖥

With a microservices architecture, an application is built as independent components that run each application process as a service. These services communicate via a well-defined interface using lightweight APIs. Services are built for business capabilities and each service performs a single function. Because they are independently run, each service can be updated, deployed, and scaled to meet the demand for specific functions of an application. Consider if you have one service and suddenly one service spike in demand at that time you can simply scale only that service.

There are much more interesting to know more characteristics and benefits of microservices. Recommended to this article.

https://aws.amazon.com/microservices/

Microservices Implementations

The Most Complete Platform for Microservices ❤️

AWS has integrated building blocks that support any application architecture, regardless of scale, load, or complexity

Containers on AWS

Amazon Elastic Container Service

The most secure, reliable, and scalable way to run containers. AWS is the #1 place for you to run containers and 80% of all containers in the cloud run on AWS. Customers such as Samsung, Expedia, KPMG, GoDaddy, and Snap choose to run their containers on AWS because of our security, reliability, and scalability.

ECS

Using the ECS service of AWS can run containerized applications or build microservices. Containers provide process isolation that makes it easy to break apart and run applications as independent components called microservices.

Serverless

AWS Lambda

AWS Lambda lets you run code without provisioning or managing servers. Just upload your code and Lambda manages everything that is required to run and scale your code with high availability.

Check how you can create a simple microservice using Lambda and API Gateway here.

Service Mesh

AWS App Mesh

AWS App Mesh makes it easy to monitor and control microservices running on AWS. App Mesh standardizes how your microservices communicate, giving you end-to-end visibility, and helping to ensure high-availability for your applications.

Container Orchestration

EKS

Amazon EKS is a managed service that makes it easy for you to run Kubernetes on AWS without needing to operate your own Kubernetes cluster.

There are many more services useful to implement a microservices architecture.

AWS is the most complete platform for microservices. AWS offers many services for computing, storage and database, Networking, messaging, logging, and monitoring, DevOps.

Now let’s get into some real and success stories to know how they implement microservices on AWS.

AWS Success Stories

Coursera

Coursera is an educational technology company with a mission to provide universal access to the world’s best curricula.

What challenges do they face?

Coursera had a large monolithic application for processing batch jobs that were difficult to run, deploy, and scale.
A new thread was created whenever a new job needed to be completed, and each job took up different amounts of memory and CPU, continually creating inefficiencies.
A lack of resource isolation allowed memory-limit errors to bring down the entire application.
The infrastructure engineering team attempted to move to a microservices architecture using Docker containers, but they ran into problems as they tried to use Apache Mesos to manage the cluster and containers — Mesos was complicated to set up and Coursera didn’t have the expertise or time required to manage a Mesos cluster.

How they used AWS?

They used Docker containers on Amazon EC2 Container Service(ECS) which enables Coursera to easily move to a microservices-based architecture.
Each job is created as a container and Amazon ECS schedules the container across the Amazon EC2 instance cluster.
Amazon ECS handles all the cluster management and container orchestration, and containers provide the necessary resource isolation.

What are the benefits?

Launched a prototype in less than two months
Reduced time to deploy software changes from hours to minutes
Reduced engineering time spent installing software and maintaining clusters

Read more about the Coursera case study(here).

Brainly

Using AWS, Brainly eliminates outages and reduces virtual server costs by 60 percent. Brainly is a peer-to-peer learning community and educational technology company. The Brainly platform runs on Amazon EC2 Reserved Instances and Spot Instances, with Amazon ElastiCache providing caching services, while Amazon Elastic Kubernetes Service orchestrates microservices containers.

Benefits of AWS

Eliminates outages
Reduces virtual server costs by 60 percent
Avoids the cost of building cloud expertise
Increases speed of orchestration services 2–3 times
Keeps IT team down to four people
Enables continuous innovation

Bridestory

Bridestory uses microservices and moved from monolithic architecture.

Same Architecture, Different Brand

The new microservices architecture has transformed operations. Hanafi reports, “Since the end of 2018, we have been hitting our metrics. Previously, we were averaging three weeks to launch one big feature, whereas now our developers are empowered to conduct small releases daily, with less than a 1 percent failure rate.” The company has also launched a new app called Parentstory using its containerized infrastructure. “Isolation is the key to our new multi-tenancy microservices model,” he explains. “We are constantly exploring with AWS architects how we can use the same architecture with the same source code, but with a totally different brand, different customers, and different database.” With this “recycled” approach, the team was able to launch the Parentstory app nearly three times faster than Bridestory.

Today, a new generation of companies is navigating a journey to AWS, and those companies have a very different set of challenges to overcome. They are not the same challenges faced by the first wave of born-in-the-cloud AWS users, but they are surmountable and are being continually addressed so that companies can reap the benefits of the cloud.
I hope you find this article informative and useful.
I would like to connect and learn your thoughts on the New Age of Cloud Computing and many more…
Connect with me on LinkedIn.

https://www.linkedin.com/in/shubham-rasal/

If you find this informative and helpful... Don’t forget to👏👏 .

What is Big Data? How big companies manage Big Data?

Shubham Rasal — Thu, 17 Sep 2020 14:39:28 GMT

1-Big Data

How big MNC’s like Google, Facebook, Instagram, etc stores, manages, and manipulate Thousands of Terabytes of data with High Speed and High Efficiency

What is Big Data?🤔

We are going to discuss something interesting today and that is “Big Data”.
Before moving next let's understand what is big data.
We are constantly generating data.. even nowadays our kitchen appliances are now connected to the internet and sharing and storing mountains of data.

The amount of information being collected around the world is soo much big to process. That’s why our topic comes in play i.e Big Data.

In simple words, big data has huge amount of raw data and it is too complex for traditional software to process it. we need to study that data for business to grow like Facebook and Walmart and many more.

Let's take some examples so we can get more idea of big data.🤯

People are generating 2.5 quintillion bytes of data each day.
Nearly 90% of all data has been created in the last two years
Walmart handles more than 1 million customer transactions every hour.
Facebook generates 500 Terabytes of data each day.
Google currently processes over 20 petabytes of data per day.

Daily smartphone and computer usage mean that the volume of data is expanding rapidly. The average user shares dozens of media links daily, and all of that has to be stored somewhere.

Boeing 737 — the plane used by many carriers on this route — the total amount of data generated would be a massive 240 terabytes of data.
You can read more about it. (click here)

Over 2.5 quintillion bytes of data are created every single day, and it’s only going to grow from there. By 2020, it’s estimated that 1.7MB of data will be created every second for every person on earth. (Read more)

https://www.socialmediatoday.com/

How to store this much data now? 🧐
Softwares are available to solve this issue like

Hadoop
HBase
Hive

We will discuss little about Hadoop not going any technical.
To solve the problem of Big data a new concept was introduced known as the distributed storage system, and the product of this concept is known as Hadoop.
What is Distributed Storage? A distributed storage system is an infrastructure that can split data across multiple physical servers, and often across more than one data center. It typically takes the form of a cluster of storage units, with a mechanism for data synchronization and coordination between cluster nodes.
let's take an example.. suppose we have one file of 50Gb and we want to store it somewhere. If we try to store it one hard disk it will take time to store it.. and if store this file into different 50 machines then so much less time to the previous one.

Big Data demands a cost-effective, innovative solution to store and analyze it. Hadoop is the answer to all Big Data requirements. So, let’s explore why Hadoop is so important.
“Hadoop Market is expected to reach $99.31B by 2022 at a CAGR of 42.1%”.

Why Hadoop? 🙂

Hadoop provides a cost-effective storage solution for business.
It facilitates businesses to easily access new data sources and tap into different types of data to produce value from that data.
It is a highly scalable storage platform
Hadoop is fault tolerance. When data is sent to an individual node, that data is also replicated to other nodes in the cluster, which means that in the event of failure, there is another copy available for use.
Hadoop is more than just a faster, cheaper database and analytics tool. It is designed as a scale-out architecture that can affordably store all of a company’s data for later use.

Hope you found this post to be informative.
Comment down your thoughts on Big data.
Share it with your friends and lets connects on LinkedIn to get more updates.
Please do not hesitate to keep 👏👏👏👏👏 it.

Thank you and stay motivated…

Deploy the WordPress application on Kubernetes and AWS RDS using terraform

Shubham Rasal — Tue, 01 Sep 2020 17:36:42 GMT

Deploy the WordPress application on Kubernetes and AWS RDS using terraform

Infrastructure as code using Terraform, which automatically deploys the WordPress application using AWS RDS service.

What do we want to do?

Deploy the WordPress application on Kubernetes and AWS using terraform including the following steps;

1. Write an Infrastructure as code using Terraform, which automatically deploy the WordPress application

2. On AWS, use RDS service for the relational database for WordPress application.

3. Deploy WordPress as a container on top of Minikube.

4. The WordPress application should be accessible from the public world if deployed on AWS or through workstation if deployed on Minikube.

My main intention is to show how we can use the RDS service of AWS in our WordPress application. Here I am using WordPress application on Kubernetes over Minikube.

Pre-requisite :

AWS account.
Download AWS CLI and Configure it.
Download terraform. Download
Download Minkube.

Let’s Start…

1. Configure the Providers

We want to use two services AWS and Kubernetes. for that we need to use AWS and Kubernetes providers in Terraform.

Providers

2. Start Minikube

To start Minikube open VirtualBox and start manually or give the command

$ minikube start

minikube start

3. Create RDS Service

We want to use RDS to store the WordPress database. so we will use MYSQL community edition.

first, we will create a security group that will allow traffic only for MySQL port.

Security Group

Now we will create an RDS instance for MYSQL.

RDS

Here we have taken 20GB Memory and t2.micro type for RDS instance. Added username and password attributes. You need to put your specifications. or you can also use a variable to take interactive input.

3. Launch WordPress on K8s

Here, for this demo, I am using locally installed minikube but you can use services like EKS or target services on AWS.

Create deployment for WordPress.

using deployment we will achieve high availability. I have selected WordPress 4.8-apache container. The main part here is we are setting some variables of WordPress like WORDPRESS_DB_HOST, WORDPRESS_DB_USER, WORDPRESS_DB_PASSWORD, WORDPRESS_DB_NAME. To set these properties we are getting values to form the RDS instance. but to access WordPress, we need to expose it.

Expose WordPress.

expose

Above kubernetes_service resource selects our WordPress pod and give service as Nodeport and exposing it.

3. Check output

output

The above code gives the URL and port number where WordPress is exposed. To get the first half of the URL which I had put statically here. you can check using below command

$ minikube ip

Select language and add admin details. after that, you will see the login tab like below.

Add Post

Publish and Check.

You will see the output like above.

Check the solution

Now delete the pod manually and it will be launch again…

Again check WordPress our previous blog exists or not.

Conclusion

We have successfully created Infrastructure as Code for deploying the WordPress application on Kubernetes and AWS RDS using terraform

I want to express my gratitude for everything that Mr. Vimal Daga sir have helped me achieve this knowledge and thank you for the cheering and mentoring us…..

If you like my efforts then don’t forget to give feedback in the comment box. Feel free to ask questions and doubts in the comments.

#LEARN — SHARE — GROW

AWS Networking using Terraform.

Shubham Rasal — Sun, 26 Jul 2020 11:14:11 GMT

Creating VPC, subnets, Internet Gateway, NAT Gateway, Route Table, Bastion host, Servers.

2 Creating VPC, subnets, Internet Gateway, NAT Gateway, Route Table, Bastion host, Servers.

What do we want to do?

Statement: We have to create a web portal for our company with all the security as much as possible. So, we use the WordPress software with a dedicated database server.

The database should not be accessible from the outside world for security purposes. We only need public WordPress for clients. So here are the steps for proper understanding!

Steps:

1. Write an Infrastructure as code using Terraform, which automatically creates a VPC.

2. In that VPC we have to create 2 subnets:

a) public subnet [ Accessible for Public World! ]

b) private subnet [ Restricted for Public World! ]

3. Create a public-facing internet gateway to connecting our VPC/Network to the internet world and attach this gateway to our VPC.

4. Create a routing table for Internet gateway so that instance can connect to outside world, update and associate it with the public subnet.

5. Create a NAT gateway to connecting our VPC/Network to the internet world and attach this gateway to our VPC in the public network

6. Update the routing table of the private subnet, so that to access the internet it uses the NAT gateway created in the public subnet

7. Launch an ec2 instance that has WordPress setup already having the security group allowing port 80 so that our client can connect to our WordPress site. Also, attach the key to the instance for further login into it.

8. Launch an ec2 instance that has MYSQL setup already with security group allowing port 3306 in a private subnet so that our WordPress VM can connect with the same. Also, attach the key with the same.

Note: WordPress instance has to be part of the public subnet so that our client can connect our site. MySQL instance has to be part of a private subnet so that the outside world can’t connect to it.

Don’t forget to add auto IP assign and auto DNS name assignment options to be enabled.

I have used Code snippets images for better visualization. You can find code this code file in this Github repository.

Pre-requisite :

AWS account.
Download AWS CLI and Configure it.
Download terraform. Download

Let’s Start…

1. Configure the Provider

2. Create VPC

Below code creates VPC with given cider block, the tenancy is default and we want DNS hostname URL for that we are enabling it. VPC will create in the region you have mentioned in the above provider resource.

2. Create two subnets in VPC

Here we are just creating two subnets and we named public and private. A VPC spans all of the Availability Zones in the Region. After creating a VPC, we want to add subnets in two different Availability Zone.

above code will create two subnets named ‘public_subnet’ and ‘private_subnet’ in ‘ap-south-1a’ and ‘ap-south-1b’

3. Create Internet Gateway

An internet gateway is a horizontally scaled, redundant, and highly available VPC component that allows communication between your VPC and the internet. Read More

we want to connect our VPC to the internet so that’s why we need to create Internet Gateway.

4. Create a Route Table

A route table contains a set of rules, called routes, that are used to determine where network traffic from your subnet or gateway is directed.

to create a route table we have used aws_route_table resource. We want to access the internet so add cider_block to 0.0.0.0/0 , (quad-zero route) and use the internet gateway that we create above. The destination for the route is, which represents all IPv4 addresses. The target is the internet gateway that's attached to your VPC.

To read more about the Route table you can refer to this. ROUTE_TABLE

5. Associate Route table with Public Subnet

Now we want to use the internet only for public subnet so associate route table with that subnet only. To do this we will use aws_route_table_association resource in terraform which takes subnet id and route table id.

Now we have created VPC and created two subnets in different Availability Zones in the region and given Internet connection by creating an Internet gateway for VPC. Now lets launch instances of WordPress and MySQL.

6. Create NAT Gateway

A Network Address Translation (NAT) gateway is a device that helps enabling EC2 instances in a private subnet to connect to the Internet and prevent the Internet from start off a connection with those instances.

1.Create EIP for NAT Gateway

2. NAT Gateway

Create NAT gateway using aws_nat_gateway resource of terraform. NAT gateway should be in public subnet and it requires EIP so that it can rapidly remap the address in case of failure.

7. Create Route Table for private subnet and NAT gateway

As per need, we want to get internet connection in a private subnet so that we create one more route table with nat gateway.

8. Associate Route Table to the private subnet.

above code creates association between route table which has NAT gateway and private subnet.

We want to use database instance which is in a private subnet that’s why we can not access it through ssh from our machine/internet. If we want to connect and manage instances in private subnet we have to create one more instance which will configure and manages instances in VPC. that instance is called as bastion host or Jump Server

10.Create Key Pair 🔑

We are using here tls_private_key for creating two resources and resource local_file is used to store this key locally. aws_key_pair is used to create key-pair in AWS and will attach this key in AWS.

9.Create a Security Group for Bastion Host or Jump Server

This security group is created for the bastion host and allow only ssh requests.

10.Create Bastion Host or Jump Server in Public subnet

Create an instance for bastion host in a public subnet, assign public IP and give Security Group that we have created allowing the only ssh.

11. Create a Security group which allow HTTP and SSH for Bastion Host

we have created one more security group which allows http inbound traffic from anywhere but ssh only from bastion host security group.

12. Create WordPress Instance in the public subnet

Create an instance for your web portal in public subnet allowing https and ssh security group. Here I have used amazon Linux and attach keypair that we have created. Also, assign public IP using that we can request this server.

13. Create a Security Group for MySQL and SSH for Bastion Host

Our Mysql server will be a private subnet. we want to open this server only for MySQL requests and ssh from the bastion host. Hence allow inbound/ingress for MySQL and ssh for bastion host only.

14. Create MYSQL Server in a private subnet

Create MySQL server in the private subnet.attach keypair and give security group that we have created for this instance.

15. How to Run this?😍🏆

Now sit back and have a coffee. It will take a little time to create all the infrastructure for you.

as Code, we are done here. now we will check our setup is working or not.

we will check do we have internet connection in instance in private subnet or not. for that, we will transfer the key to the bastion host and connect to MySQL instance.

For transferring key I am using the WinSCP tool.

Now I have transferred key to bastion host. we will check can we connect to instances in VPC through bastion host or jump server.

We can successfully ping to the outside world through MySQL instance in the private subnet.

Now, using bastion host you can configure webservers and database servers that we have created above. You can install WordPress on EC2 instance in the public subnet and install database service in the instance on private subnet which can go to the internet via NAT gateway.

You will find all the code in this GITHUB Repository.

Conclusion

We have successfully created Infrastructure as Code for VPC in AWS for classic VPC i.e VPC with Public and Private Subnets.

I want to express my gratitude for everything that Mr. Vimal Daga sir have helped me achieve this knowledge and thank you for the cheering and mentoring us…..

If you like my efforts then don’t forget to give feedback in the comment box. Feel free to ask questions and doubts in comments.

#LEARN — SHARE — GROW