What is Data Analysis?Data analysis is a process of inspecting, cleansing, transforming, and modeling data with the goal of discovering useful information, informing conclusions, and supporting decision-making. In today’s business world, data analysis plays a role in making decisions more scientific and helping businesses operate more effectively
Python is a great language for doing data analysis, primarily because of the fantastic ecosystem of data-centric Python packages. Pandas is one of those packages providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real-world data analysis in Python.
In this article, I have used Pandas to know more about doing data analysis.
Mainly pandas have two data structures, series,data frames, and Panel.

You can use array indexing or labels to access data in the series.
You can use array indexing or labels to access data in the seriesprint(a[1])print(a[‘test4’])

1

2

98.7

97.7

You can also apply mathematical operations on pandas series.
b = a * 2
c = a ** 1.5print(b)print(c)

1

2

3

4

5

6

7

8

9

10

11

test1200.0

test2197.4

test3196.8

test4195.4

dtype:float64

test11000.000000

test2980.563513

test3976.096258

test4965.699142

dtype:float64

You can even create a series of heterogeneous data.
s = pd.Series([‘test1’, 1.2, 3, ‘test2’], index=[‘test3’, ‘test4’, 2, ‘4.3’])

print(s)

1

2

3

4

5

test3test1

test41.2

23

4.3test2

dtype:object

pandas DataFrame

pandas DataFrame is a two-dimensional array with heterogeneous data.i.e., data is aligned in a tabular fashion in rows and columns.Structure
Let us assume that we are creating a data frame with the student’s data.

Name

Age

Gender

Rating

Steve

32

Male

3.45

Lia

28

Female

4.6

Vin

45

Male

3.9

Katie

38

Female

2

You can think of it as an SQL table or a spreadsheet data representation.
The table represents the data of a sales team of an organization with their overall performance rating. The data is represented in rows and columns. Each column represents an attribute and each row represents a person.
The data types of the four columns are as follows −

Column

Type

Name

String

Age

Integer

Gender

String

Rating

Float

Key Points
• Heterogeneous data
• Size Mutable
• Data Mutable

A pandas DataFrame can be created using the following constructor −pandas.DataFrame( data, index, columns, dtype, copy)

• data
data takes various forms like ndarray, series, map, lists, dict, constants and also another DataFrame.
• index
For the row labels, the Index to be used for the resulting frame is Optional Default np.arrange(n) if no index is passed.
• columns
For column labels, the optional default syntax is – np.arrange(n). This is only true if no index is passed.
• dtype
The data type of each column.
• copy
This command (or whatever it is) is used for copying of data if the default is False.

There are many methods to create DataFrames.
• Lists
• dict
• Series
• Numpy ndarrays
• Another DataFrame

Creating DataFrame from the dictionary of Series
The following method can be used to create DataFrames from a dictionary of pandas series.

You might have noticed that we got a DataFrame with NaN values in it. This is because we didn’t the data for that particular row and column.

Creating DataFrame from Text/CSV files
Pandas tool comes in handy when you want to load data from a CSV or a text file. It has built-in functions to do this for use.

df = pd.read_csv(‘happiness.csv’)

Yes, we created a DataFrame from a CSV file. This dataset contains the outcome of the European quality of life survey. This dataset is available here. Now we have stored the DataFrame in df, we want to see what’s inside. First, we will see the size of the DataFrame.

print(df.shape)

1

(105,4)

It has 105 Rows and 4 Columns. Instead of printing out all the data, we will see the first 10 rows.df.head(10)

1

2

3

4

5

6

7

8

9

10

11

CountryGenderMeanN=

0ATMale7.3471

1NaNFemale7.3570

2NaNBoth7.31041

3BEMale7.8468

4NaNFemale7.8542

5NaNBoth7.81010

6BGMale5.8416

7NaNFemale5.8555

8NaNBoth5.8971

9CYMale7.8433

There are many more methods to create a DataFrames. But now we will see the basic operation on DataFrames.

We are not interested in the unnamed column. So, let’s delete that first. Then we’ll see the statistics with one line of code.

1

2

3

4

5

6

7

8

9

dbhwoodbarkrootrootskbranch

count153.000000133.00000017.00000054.00000053.00000076.000000

mean26.3529411569.045113513.235294334.383333113.80226454.065789

std28.2736794071.380720632.467542654.641245247.22411865.606369

min3.0000003.0000007.0000000.3000000.0500004.000000

25%8.00000029.00000059.00000011.5000002.00000010.750000

50%15.000000162.000000328.00000041.00000011.00000035.000000

75%36.0000001000.000000667.000000235.00000045.00000077.750000

max145.00000025116.0000001808.0000003000.0000001030.000000371.000000

It’s simple as that. We can see all the statistics. Count, mean, standard deviation and other statistics. Now we are gonna find some other metrics which are not available in the describe() summary.

Mean :print(df.mean())

1

2

3

4

5

6

7

dbh26.352941

wood1569.045113

bark513.235294

root334.383333

rootsk113.802264

branch54.065789

dtype:float6

Min and Max print(df.min())

1

2

3

4

5

6

7

8

dbh3

wood3

bark7

root0.3

rootsk0.05

branch4

speciesAcacia mabellae

dtype:object

print(df.max())

1

2

3

4

5

6

7

8

dbh145

wood25116

bark1808

root3000

rootsk1030

branch371

speciesOther

dtype:object

Pairwise Correlation
df.corr()

1

2

3

4

5

6

7

dbhwoodbarkrootrootskbranch

dbh1.0000000.9051750.9654130.8993010.9349820.861660

wood0.9051751.0000000.9717000.9887520.9670820.821731

bark0.9654130.9717001.0000000.9610380.9713410.943383

root0.8993010.9887520.9610381.0000000.9369350.679760

rootsk0.9349820.9670820.9713410.9369351.0000000.621550

branch0.8616600.8217310.9433830.6797600.6215501.000000

Data Cleaning
We need to clean our data. Our data might contain missing values, NaN values, outliers, etc. We may need to remove or replace that data. Otherwise, our data might make any sense.
We can find null values using the following method.

print(df.isnull().any())

1

2

3

4

5

6

7

8

9

dbhFalse

woodTrue

barkTrue

rootTrue

rootskTrue

branchTrue

speciesFalse

fac26True

dtype:bool

We have to remove these null values. This can be done by the method shown below.

newdf = df.dropna()

print(newdf.shape)

1

2

3

4

5

dbhwoodbarkrootrootskbranchspecies fac26

12327550.0105.044.09.059.0B.myrtifoliaz

12426414.078.038.013.044.0B.myrtifoliaz

125942.08.05.01.37.0B.myrtifoliaz

1261285.013.017.02.216.0B.myrtifoliaz

print(newdf.shape)

1

(4,8)

Pandas .Panel()
A panel is a 3D container of data. The term Panel data is derived from econometrics and is partially responsible for the name pandas − pan(el)-da(ta)-s.
The names for the 3 axes are intended to give some semantic meaning to describing operations involving panel data. They are −
• items − axis 0, each item corresponds to a DataFrame contained inside.
• major_axis − axis 1, it is the index (rows) of each of the DataFrames.
• minor_axis − axis 2, it is the columns of each of the DataFrames.

A Panel can be created using the following constructor −
The parameters of the constructor are as follows −
• data – Data takes various forms like ndarray, series, map, lists, dict, constants and also another DataFrame
• items – axis=0
• major_axis – axis=1
• minor_axis – axis=2
• dtype – the Data type of each column
• copy – Copy data. Default, false

A Panel can be created using multiple ways like −
• From ndarrays
• From dict of DataFrames
• From 3D ndarray

1

2

3

4

5

# creating an empty panel

import pandas aspd

import numpy asnp

data=np.random.rand(2,4,5)

p=pd.Panel(data)

print(p)

1

2

3

4

5

output:

Dimensions:2(items)x4(major_axis)x5(minor_axis)

Items axis:0to1

Major_axis axis:0to3

Minor_axis axis:0to4

Note − Observe the dimensions of the empty panel and the above panel, all the objects are different.

From dict of DataFrame Objects

1

2

3

4

5

6

#creating an empty panel

import pandas aspd

import numpy asnp

data={'Item1':pd.DataFrame(np.random.randn(4,3)),

'Item2':pd.DataFrame(np.random.randn(4,2))}

p=pd.Panel(data)

print(p)

1

2

3

4

5

output:

Dimensions:2(items)x4(major_axis)x3(minor_axis)

Items axis:Item1 toItem2

Major_axis axis:0to3

Minor_axis axis:0to2

Selecting the Data from Panel
Select the data from the panel using −
• Items
• Major_axis
• Minor_axis

Using Items

1

2

3

4

5

6

# creating an empty panel

import pandas aspd

import numpy asnp

data={'Item1':pd.DataFrame(np.random.randn(4,3)),

'Item2':pd.DataFrame(np.random.randn(4,2))}

p=pd.Panel(data)

print p[‘Item1’]

1

2

3

4

5

6

output:

012

0-0.006795-1.156193-0.524367

10.0256101.5337410.331956

21.0676711.3096661.304710

30.6151961.348469-0.410289

We have two items, and we retrieved item1. The result is a DataFrame with 4 rows and 3 columns, which are the Major_axis and Minor_axis dimensions.

Using major_axis
Data can be accessed using the method panel.major_axis(index).

1

2

3

4

Item1Item2

00.027133-1.078773

10.115686-0.253315

2-0.473201NaN

Using minor_axis
Data can be accessed using the method panel.minor_axis(index).

To facilitate PCI compliant credit card collections Fusebill provides a AJAX Transparent Redirect endpoint which you can use to securely capture customer’s credit cards. If you are adding the first payment method on a customer, it will be set to the default payment method automatically.

This API action is authenticated with a separate Public API Key. If you do not have that key, please contact Fusebill Support. The Public Key can only be used to authenticate the Transparent Redirect action.

Google reCAPTCHA required.

Fusebill leverages reCAPTCHA technology to ensure payment method data captured is provided by a human and to protect against bots and scripting.

Object that allows specifying an amount to collect when creating the card.

Only works through Json{
"collectionAmount": 1.0
}

No

Object

+ Denotes a field required for Fusebill Payments API Risk Fields* Denotes fields required for AVS and may be required by your account’s Gateway. These fields are also required if using Fusebill Payments accounts as AVS is mandatory.

Slack is a great platform for team collaboration not just that it also has one of the best API interfaces to build Chatbots.

In this post, I will walk you through building a minimal Slack Bot with Django backend. The idea is to set up a Slack Bot that will notify event when greeted through a backend.

Before we start let us understand the Slack bots life cycle:

If you are new to Slack, It’s a messaging platform focused on team collaboration. Slack lets you create custom applications, including bots (sort of Messaging Platform as a Service). You will run the application back end to process business logic in your own server.

To start with, you need to be part of the Slack team and have admin privilege to create a new Slack App. If you are not part of a Slack team you may create one.

GIve the name of your company or team.

Enter Channel Name.

Click on See your channel in slack

We will create a Slack App for the Slack team then we will add a Bot User to the app.

We will create a Django based backend web application to post the messages into slack.

After setting up the Slack App and have the backend ready to notified events.

Create a Slack App

Start by creating a Slack app here, click Create App. Then proceed with app creation, give it a name and select the Slack team.

Then you will be taken to App configuration where you need do following to get our Bot up and running.

Create a Bot User

Install Slack App to your Team

Create a BOT User

On the left pane click on Bot User then choose a user name for the Bot and set “Always Show My Bot as Online” slider to on. Click on Add Bot User to create our shipment bot.

Install Slack App to Team

Now on the left pane click Install App and install the app to your Slack team.

Once installed you will get Bot User OAuth Access Token, note down this token we will need it later while configuring Django app. This token is the passphrase for our Bot to interact with the Slack Team.

Slack Client Credentials

Also, note down the App Credentials from Basic Information on the left pane. These credentials let us talk to Slack API, so every time we send a message to Slack we should send our Client ID(CLIENT_ID) & Client Secret(CLIENT_SECRET) to identify ourselves to Slack. Similarly, we can verify if an incoming message is sent by Slack checking if the Verification Token (VERIFICATION_TOKEN) in the message is the same as the one in App Credentials.

Now we should have four key values with us.

Client ID — SLACK_CLIENT_ID/li>

Client Secret — SLACK_CLIENT_SECRET

Verification Token — SLACK_VERIFICATION_TOKEN

Bot User Token — SLACK_BOT_USER_TOKEN

Environment Setup

Let us create a new virtual environment “venv” for our project with python version 3.6.x and activate the virtual environment.

1

virtualenv venv–python=python3.6

You need to activate the virtual environment before installation of other dependencies.

1

source venv/bin/activate

Now let’s install required packages

1

2

3

pip install django

pip install slacker

pip install slacker-log-handler

Create a Django Application

1

2

3

django-admin startproject shipment_portal

cd shipment

django-admin startapp shipment

Configure Django Settings

we need to add our own application shipment as a dependency. Add the line mentioned below in the file slack/settings.py

# slack/settings.py

1

2

3

4

5

6

7

8

9

INSTALLED_APPS=[

'django.contrib.admin',

'django.contrib.auth',

'django.contrib.contenttypes',

'django.contrib.sessions',

'django.contrib.messages',

'django.contrib.staticfiles',

'shipment',# <== add this line

]

Then add following configurations in slack_bot/settings.py with your authentication keys from Slack.

# slack/settings.py

1

2

3

4

5

6

7

# SLACK API Configurations

# ----------------------------------------------

# use your keys

SLACK_CLIENT_ID='20xxxxxxxxxx.20xxxxxxxxxx'

SLACK_CLIENT_SECRET='d29fe85a95c9xxxxxxxxxxxxxxxxxxxxx'

SLACK_VERIFICATION_TOKEN='xpxxxxxxxxxxxxxxxxxxxxxxxxx'

SLACK_BOT_USER_TOKEN='xoxb-xxxxxxxxxx-xxxxxxxxxxxxxxxxxxxxxxxxx'

Now start the Django development server

1

python manage.py runserver

Once the server is started it will print something similar to this

1

2

3

4

5

6

7

8

Performing system checks…

System check identified no issues(0silenced).

You have13unapplied migration(s).Your project may notwork properly until you apply the migrations forapp(s):admin,auth,contenttypes,sessions.

Run‘python manage.py migrate’toapply them.

July03,2017–17:30:32

Django version1.11.3,using settings‘shipment_portal.settings’

Starting development server at http://127.0.0.1:8000/

Quit the server with CONTROL-C.

Ignore the migration warnings and open the URL in your browser.

Create an API endpoint

Now that we have our app server up and running we need to create an endpoint for Slack to send event messages. We will create an API view with Django as follows:

Shipment/view.py

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

classShipmentCreate(LoginRequiredMixin,View):

def post(self,request,*args,**kwargs):

try:

user=request.user.id

===============

Your code goes here............

===============

response={

'status':200,

'type':'+OK',

'message':'Shipment created',

}

message='Shipment created by'+user

send_notification(message,channel='#general')

except Exception aserror:

response={

'status':500,

'type':'-ERR',

'message':'Internal Server Error',

}

returnJsonResponse(response,status=response.get('status'))

Configure Django Routes

If you are new to web applications, routing is the way to tell web server which functions to invoke when an URL is hit with a request. When the URL is hit with a request message the corresponding function will be invoked, passing the requester message as a parameter to the function.
Add following lines in shipment/urls.py to tie shipment API class to http://localhost:8000/shipment/

1

2

3

4

5

from.views import *

from django.conf.urls import url

urlpatterns=[

url(r'^shipment/$',Shipment.as_view(),name='shipment'),

]

slackapi.py

Functions written in slackapi.py are used to post notification/messages to slack.

If you have multiple IPs assigned on your Linux pc then there is a chance that you want to use different IPs for some applications than default one. Updating IP routes every time isn’t a good idea and you may mess up.