Agenda¶
- Introduction
- Datasets
- Train data ＆ Test data
- Data Preprocessing
- Machine Learning Algorithms
- Supervised learning
- Regression -- Linear Regression
- Regression -- Support Vector Regression
- Classification -- Logistic Regression
- Classification -- K-Nearest Neighbors
- Classification -- Support Vector Machine
- Unsupervised learning
- Clustering -- K-means
- Clustering -- Hierarchical clustering
- Supervised learning
Review¶
Reference : https://www.wikiwand.com/en/Data_analysis
Note¶
In [1]:
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
Introduction¶
- 資料探勘 and 機器學習
- 建立在 NumPy, SciPy ,Matplotlib 套件上
- 六大功能：
a. Preprocessing (資料預處理)
b. Dimensionality reduction (資料降維)
c. Regression (迴歸)
d. Classification (分類)
e. Clustering (分群)
f. Model selection (模型評估/選取)
Introduction¶
Note¶
- Supervised learning
- Regression
- Classification
- Unsupervised learning
- Clustering
Introduction¶
- Machine Learning Problem：
Reference : http://scikit-learn.org/stable/tutorial/machine_learning_map/
Introduction¶
Note¶
SciKits
Reference : https://scikits.appspot.com/scikits
Datasets¶
Note¶
- Scikit-Learn 針對不同資料集使用不同的 load 指令
- 格式：
load_資料集名稱()
In [2]:
from sklearn import datasets
- Dataset (boston)
In [3]:
boston = datasets.load_boston()
In [4]:
type(boston)
Out[4]:
The Bunch object in Scikit-Learn is simply a dictionary that exposes dictionary keys as properties so that you can access them with dot notation.
In [5]:
boston.keys()
Out[5]:
- data：解釋變數
- target：反應變數
- feature_names：變數名稱說明
- DESCR：資料描述
In [6]:
boston.DESCR
Out[6]:
In [7]:
boston.feature_names
Out[7]:
分别是房屋均價及周邊犯罪率、是否在河邊等相關信息
In [8]:
boston.data.shape
Out[8]:
- 確認 boston ( target ) 的維度
In [9]:
boston.target.shape
Out[9]:
In [10]:
from sklearn import datasets
- Dataset (iris)
In [11]:
iris = datasets.load_iris()
鳶尾花(iris)資料集是非常著名的生物資訊資料集之一，取自美國加州大學歐文分校的機械學習資料庫
In [12]:
iris.keys()
Out[12]:
- data：解釋變數
- target：反應變數
- target_names：反應變數的類別名稱
- feature_names：變數名稱說明
- DESCR：資料描述
iris.DESCR
In [13]:
iris.target_names
Out[13]:
In [14]:
iris.feature_names
Out[14]:
分别是花萼、花瓣等相關信息
In [15]:
iris.data.shape
Out[15]:
- 確認 iris ( target ) 的維度
In [16]:
iris.target.shape
Out[16]:
Question¶
想分析 Boston 這組資料。根據前幾堂課學到的方法做一些，你認為在進入分析(模型)前應該要做的事?¶
Review¶
Data science process flowchart¶
Reference : https://www.wikiwand.com/en/Data_analysis
Datasets -- Boston¶
Data Prepare
In [17]:
boston_df = pd.DataFrame( boston.data )
boston_df.columns = boston.feature_names
boston_df['PRICE'] = boston.target
確認資料維度
In [18]:
boston_df.shape
Out[18]:
Datastes -- Boston¶
可以利用前五筆資料，來看一下資料狀況
In [19]:
boston_df.head()
Out[19]:
Datastes -- Boston¶
確認資料中是否有遺失值
In [20]:
boston_df.isnull().sum()
Out[20]:
Datastes -- Boston¶
基本敘述統計
In [21]:
boston_df.describe()
Out[21]:
Datastes -- Boston¶
探索型資料視覺化
In [22]:
sns.pairplot(boston_df)
Out[22]:
Question¶
想分析 iris 這組資料。根據前幾堂課學到的方法做一些，先來探索一下這組資料吧！¶
Datasets -- Iris¶
Data Prepare
In [23]:
iris_df = pd.DataFrame( iris.data )
iris_df.columns = iris.feature_names
iris_df['species'] = iris.target_names[iris.target]
確認資料維度
In [24]:
iris_df.shape
Out[24]:
Datasets -- Iris¶
可以利用前五筆資料，來看一下資料狀況
In [25]:
iris_df.head()
Out[25]:
Datasets -- Iris¶
確認資料中是否有遺失值
In [26]:
iris_df.isnull().sum()
Out[26]:
Datasets -- Iris¶
基本敘述統計
In [27]:
iris_df.describe()
Out[27]:
In [28]:
iris_df["species"].value_counts()
Out[28]:
Datasets -- Iris¶
探索型資料視覺化
In [29]:
sns.pairplot(iris_df)
Out[29]:
Datasets -- Iris¶
探索型資料視覺化
In [30]:
sns.pairplot(iris_df ,hue='species')
Out[30]:
Datasets -- Iris¶
探索型資料視覺化
In [31]:
sns.pairplot(iris_df ,hue='species',diag_kind="kde")
Out[31]:
Datasets -- Iris¶
探索型資料視覺化
In [32]:
sns.boxplot(x="species", y="petal length (cm)", data=iris_df)
Out[32]:
Train data ＆ Test data¶
Train data ＆ Test data¶
Train data ＆ Test data¶
Train data ＆ Test data¶
Train data ＆ Test data -- Boston¶
Choose Data
In [33]:
B_X = boston_df.drop('PRICE',axis=1)
B_y = boston_df['PRICE']
Train data ＆ Test data -- Boston¶
Training and testing
In [34]:
from sklearn.model_selection import train_test_split
In [35]:
B_X_train, B_X_test, B_y_train, B_y_test = train_test_split(B_X , B_y , test_size = 0.2, random_state = 0)
Train data ＆ Test data -- Iris¶
Choose Data
In [36]:
I_X = iris_df.loc[:, ['petal length (cm)','petal width (cm)']]
I_y = iris.target
Train data ＆ Test data -- Iris¶
Training and testing
In [37]:
from sklearn.model_selection import train_test_split
In [38]:
I_X_train, I_X_test, I_y_train, I_y_test = train_test_split(I_X, I_y, test_size = 0.3, random_state = 0)
Machine Learning Algorithms -- Supervised learning¶
- Regression
- Linear Regression
- Support Vector Regression
- Classification
- Logistic Regression
- K-Nearest Neighbors
- Support Vector Machine
In [39]:
from sklearn.linear_model import LinearRegression
Use LinearRegression¶
In [40]:
lm = LinearRegression()
In [41]:
lm.fit( B_X_train.values, B_y_train.values )
Out[41]:
In [42]:
print(lm.intercept_)
Training Model ( coefficient )¶
In [43]:
print(lm.coef_)
Note¶
Training Model ( features and coefficients )¶
In [44]:
pd.DataFrame(list( zip(B_X_train.columns,lm.coef_) ),columns=['features','estimatedCoeffs'])
Out[44]:
In [45]:
from sklearn.feature_selection import f_regression
print(f_regression(B_X_train, B_y_train)[1])
Note¶
小數位數¶
In [46]:
print(f_regression(B_X_train, B_y_train)[1].round(decimals=5))
In [47]:
B_y_predict = lm.predict(B_X_test)
In [48]:
pd.DataFrame( list(zip(B_y_test.values,B_y_predict)), columns=['Measured','Predicted'] )
Out[48]:
In [49]:
plt.scatter(B_y_test.values,B_y_predict,s=2)
plt.plot([B_y_test.values.min(), B_y_test.values.max()], [B_y_test.values.min(), B_y_test.values.max()], 'k--', lw=2)
plt.ylabel('Predicted')
plt.xlabel('Measured')
Out[49]:
In [50]:
from sklearn.metrics import mean_squared_error
mse = mean_squared_error(B_y_test.values, B_y_predict)
print("MSE : ",mse)
In [51]:
R_2 = lm.score(B_X_train, B_y_train)
print("R-squared : ",R_2)
- Adjusted R-squared
In [52]:
adj_R_2 = R_2 - (1 - R_2) * (B_X_train.shape[1] / (B_X_train.shape[0] - B_X_train.shape[1] - 1))
print("Adjusted R-squared : ",adj_R_2)
In [53]:
from sklearn.svm import SVR
Use Support Vector Regression¶
In [54]:
svr = SVR(kernel='rbf')
rbf : 高斯徑向基底函數
In [55]:
svr.fit( B_X_train.values, B_y_train.values )
Out[55]:
In [56]:
B_y_predict_svr = svr.predict(B_X_test)
In [57]:
pd.DataFrame( list(zip(B_y_test.values,B_y_predict_svr)), columns=['Measured','Predicted'] )
Out[57]:
In [58]:
plt.scatter(B_y_test.values,B_y_predict_svr,s=2)
plt.plot([B_y_test.values.min(), B_y_test.values.max()], [B_y_test.values.min(), B_y_test.values.max()], 'k--', lw=2)
plt.ylabel('Predicted')
plt.xlabel('Measured')
Out[58]:
Question¶
WHY ??¶
資料標準化 ¶
In [59]:
from sklearn.preprocessing import StandardScaler
In [60]:
scaler = StandardScaler()
scaler.fit(B_X_train)
B_X_train_s = scaler.transform(B_X_train)
B_X_test_s = scaler.transform(B_X_test)
In [61]:
svr.fit( B_X_train_s, B_y_train.values )
Out[61]:
In [62]:
B_y_predict_svr_s = svr.predict(B_X_test_s)
In [63]:
plt.scatter(B_y_test.values,B_y_predict_svr_s,s=2)
plt.plot([B_y_test.values.min(), B_y_test.values.max()], [B_y_test.values.min(), B_y_test.values.max()], 'k--', lw=2)
plt.ylabel('Predicted')
plt.xlabel('Measured')
Out[63]:
Data Preprocessing¶
- StandardScaler $$ \acute{x} = \frac{x - \bar{x}}{\sigma} $$
- MinMaxScaler $$ \acute{x} = \frac{x - min(x)}{max(x) - min(x)} $$
- Normalizer $$ \acute{x} = \frac{x}{\lVert x \rVert} $$
Data Preprocessing¶
Note ¶
不要顯示科學計算符號，規定顯示小數五位¶
In [64]:
np.set_printoptions(precision = 5, suppress = True)
In [65]:
print("Mean : ",I_X.values.mean(axis=0))
print("Std : ",I_X.values.std(axis=0))
print("Max : ",I_X.values.max(axis=0))
print("Min : ",I_X.values.min(axis=0))
Data Preprocessing¶
In [66]:
from sklearn import preprocessing
In [67]:
standard = preprocessing.StandardScaler()
standard.fit(I_X)
I_X_s = standard.transform(I_X)
In [68]:
print("Mean : ",I_X_s.mean(axis=0))
print("Std : ",I_X_s.std(axis=0))
print("Max : ",I_X_s.max(axis=0))
print("Min : ",I_X_s.min(axis=0))
In [69]:
minmax = preprocessing.MinMaxScaler()
minmax.fit(I_X)
I_X_m = minmax.transform(I_X)
In [70]:
print("Mean : ",I_X_m.mean(axis=0))
print("Std : ",I_X_m.std(axis=0))
print("Max : ",I_X_m.max(axis=0))
print("Min : ",I_X_m.min(axis=0))
In [71]:
normal = preprocessing.Normalizer()
normal.fit(I_X)
I_X_n = normal.transform(I_X)
In [72]:
print("Mean : ",I_X_n.mean(axis=0))
print("Std : ",I_X_n.std(axis=0))
print("Max : ",I_X_n.max(axis=0))
print("Min : ",I_X_n.min(axis=0))
Machine Learning Algorithms -- Supervised learning¶
- Regression
- Linear Regression
- Support Vector Regression
- Classification
- Logistic Regression
- K-Nearest Neighbors
- Support Vector Machine
Reference : http://www.saedsayad.com/logistic_regression.htm
Note ¶
In [73]:
def SigmoidFunc(x):
return 1 / (1 + np.exp(-x))
Note ¶
In [74]:
Sigmoid_x = np.arange(-20, 20, 0.01)
Sigmoid_y = SigmoidFunc(Sigmoid_x)
plt.plot(Sigmoid_x, Sigmoid_y)
plt.axvline(0, ls='dotted', color='black', alpha=0.5)
plt.axhline(y=0, ls='dotted', color='black', alpha=0.5)
plt.axhline(y=0.5, ls = 'dotted', color='black', alpha=0.5)
plt.axhline(y=1, ls='dotted', color='black', alpha=0.5)
plt.yticks([0.0, 0.5, 1.0])
plt.ylim(-0.05, 1.05)
plt.title("Sigmoid Function")
plt.show()
In [75]:
from sklearn.linear_model import LogisticRegression
Use LogisticRegression¶
In [76]:
logit = LogisticRegression()
In [77]:
logit.fit( I_X_train, I_y_train )
Out[77]:
In [78]:
y_logit_predict = logit.predict( I_X_test )
Compare Predict Value and Measure Value
In [79]:
pd.DataFrame( list(zip( I_y_test, y_logit_predict)), columns=['Measured','Predicted'] )
Out[79]:
Compare Predict Value and Measure Value
In [80]:
pd.DataFrame( list(zip(iris.target_names[ I_y_test],
iris.target_names[ y_logit_predict])), columns=['Measured','Predicted'] )
Out[80]:
In [81]:
print(iris.target_names)
print(logit.predict_proba( I_X_test.iloc[0, :].values.reshape(1, 2)))
print(logit.predict_proba( I_X_test.iloc[1, :].values.reshape(1, 2)))
print(logit.predict_proba( I_X_test.iloc[2, :].values.reshape(1, 2)))
Note¶
In [82]:
from sklearn.metrics import accuracy_score
print('Accuracy:',accuracy_score( I_y_test, y_logit_predict))
Note¶
Decision boundary¶
In [83]:
from matplotlib.colors import ListedColormap
def DecisionBoundary_plot(X, y, method, h=.02):
markers = ('o', '*', '^')
colors = ('yellow', 'magenta', 'cyan')
colormap = ListedColormap(colors[:len(np.unique(y))])
x_min, x_max = X[:,0].min()-1, X[:,0].max()+1
y_min, y_max = X[:,1].min()-1, X[:,1].max()+1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
np.arange(y_min, y_max, h))
Z = method.predict(np.array([xx.ravel(), yy.ravel()]).T)
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, alpha=0.2, cmap=colormap)
plt.xlim(xx.min(), xx.max())
plt.ylim(yy.min(), yy.max())
for ix, lab in enumerate(np.unique(y)):
spec = iris.target_names[lab]
plt.scatter( x = X[y==lab,0], y = X[y==lab,1],
c=colormap(ix), marker=markers[ix], label=spec)
In [84]:
DecisionBoundary_plot(X= I_X.values, y= I_y, method=logit)
plt.xlabel('petal length (cm)')
plt.ylabel('petal width (cm)')
plt.legend(loc = 'upper left')
plt.show()
Question¶
利用經過 StandardScaler 轉換後的資料，建立Logistic Regression Model，並比較 Accuracy¶
Logistic Regression Model -- StandardScaler Data ¶
Data Prepare -- Training and testing
In [85]:
I_X_train_s, I_X_test_s, I_y_train_s, I_y_test_s = train_test_split( I_X_s, I_y, test_size = 0.3, random_state = 0)
Model
In [86]:
logit_s = LogisticRegression()
logit_s.fit( I_X_train_s, I_y_train_s )
Out[86]:
Logistic Regression Model -- StandardScaler Data ¶
Predict
In [87]:
y_logit_predict_s = logit_s.predict( I_X_test_s )
Accuracy
In [88]:
print('Accuracy(Standard):',accuracy_score( I_y_test_s, y_logit_predict_s))
經過標準化的資料， Accuracy 更好
Logistic Regression Model -- StandardScaler Data ¶
Plot ( Decision boundary )
In [89]:
DecisionBoundary_plot(X= I_X_s, y= I_y, method=logit_s)
plt.title('Iris (Standard)')
plt.xlabel('petal length (cm)')
plt.ylabel('petal width (cm)')
plt.legend(loc = 'upper left')
plt.show()
Note¶
Linear VS Nonlinear¶
Note¶
Classification of linearly nonseparable¶
- K-nearest neighbor (KNN)
- Support Vector Machine (SVM)
- Decision Tree
In [90]:
from sklearn.neighbors import KNeighborsClassifier
Use KNeighborsClassifier¶
In [91]:
knn = KNeighborsClassifier()
In [92]:
knn.fit( I_X_train_s, I_y_train_s)
Out[92]:
In [93]:
y_knn_predict = knn.predict( I_X_test_s )
Compare Predict Value and Measure Value
In [94]:
pd.DataFrame( list(zip(iris.target_names[ I_y_test_s],
iris.target_names[ y_knn_predict])), columns=['Measured','Predicted'] )
Out[94]:
In [95]:
from sklearn.metrics import accuracy_score
print('Accuracy:',accuracy_score( I_y_test_s, y_knn_predict))
In [96]:
DecisionBoundary_plot(X= I_X_s, y= I_y, method=knn)
plt.xlabel('petal length (cm)')
plt.ylabel('petal width (cm)')
plt.legend(loc = 'upper left')
plt.show()
Note¶
Support Vector Machine¶
- Radial basis function kernel $$ K( \, x,\acute{x} \, ) = exp(-\gamma \lVert x - \acute{x} \rVert^2 ) , where \, \gamma = \frac{1}{2 \sigma^2} $$
In [97]:
from sklearn.svm import SVC
Use SVC¶
In [98]:
svm = SVC(kernel = 'rbf', random_state = 0, gamma = 0.2)
In [99]:
svm.fit( I_X_train, I_y_train )
Out[99]:
In [100]:
y_svm_predict = svm.predict( I_X_test )
Compare Predict Value and Measure Value
In [101]:
pd.DataFrame( list(zip(iris.target_names[ I_y_test],
iris.target_names[ y_svm_predict])), columns=['Measured','Predicted'] )
Out[101]:
In [102]:
from sklearn.metrics import accuracy_score
print('Accuracy:',accuracy_score( I_y_test, y_svm_predict))
In [103]:
DecisionBoundary_plot(X= I_X.values , y= I_y, method=svm)
plt.xlabel('petal length (cm)')
plt.ylabel('petal width (cm)')
plt.legend(loc = 'upper left')
plt.show()
Question¶
利用經過 StandardScaler 轉換後的資料，建立SVM Model，並比較 Accuracy¶
Support Vector Machine ( SVM ) Model -- StandardScaler Data ¶
Data Prepare -- Training and testing
Model
In [104]:
svm_s = SVC(kernel = 'rbf', random_state = 0, gamma = 0.2)
svm_s.fit( I_X_train_s, I_y_train_s )
Out[104]:
Support Vector Machine ( SVM ) Model -- StandardScaler Data ¶
Predict
In [105]:
y_svm_predict_s = svm_s.predict( I_X_test_s )
Accuracy
In [106]:
print('Accuracy:',accuracy_score( I_y_test_s, y_svm_predict_s))
Support Vector Machine ( SVM ) Model -- StandardScaler Data ¶
Plot ( Decision boundary )
In [107]:
DecisionBoundary_plot(X= I_X_s, y=I_y, method=svm_s)
plt.xlabel('petal length (cm)')
plt.ylabel('petal width (cm)')
plt.legend(loc = 'upper left')
plt.show()
Note¶
Overfitting¶
Reference : http://mlwiki.org/index.php/Overfitting
Machine Learning Algorithms -- Unsupervised learning¶
- Unsupervised learning
- K-means
- Hierarchical clustering
Reference : : https://dotblogs.com.tw/dragon229/2013/02/04/89919
In [108]:
from sklearn.cluster import KMeans
Use KMeans¶
In [109]:
kmeans = KMeans(n_clusters = 3)
In [110]:
kmeans_fit = kmeans.fit( I_X )
In [111]:
kmeans_result = kmeans_fit.labels_
Compare Original and Cluster
In [112]:
pd.DataFrame( list(zip(iris.target_names[I_y],
iris.target_names[kmeans_result])), columns=['Original','Cluster'] )
Out[112]:
In [113]:
from sklearn.metrics import silhouette_score
print('Silhouette:',silhouette_score( I_X, kmeans_result))
分群演算法的績效可以使用 Silhouette 係數
Question¶
利用經過 MinMaxScaler 轉換後的資料，做 K-Means Cluster，並比較 Silhouette¶
K-MeansModel -- MinMaxScaler Data ¶
Model
In [114]:
kmeans_fit_m = kmeans.fit( I_X_m )
In [115]:
kmeans_result_m = kmeans_fit_m.labels_
In [116]:
print('Silhouette:',silhouette_score( I_X_m, kmeans_result_m ))
- Tries to combine or divide a dataset into clusters
- A tree-like hierarchical structure is created
- Can adopt two approaches :
- Agglomerative hierarchical clustering
- Divisive hierarchical clustering
Reference : https://quantdare.com/hierarchical-clustering/
In [117]:
from sklearn.cluster import AgglomerativeClustering
Use AgglomerativeClustering¶
In [118]:
hierarchical = AgglomerativeClustering(linkage = 'ward', affinity = 'euclidean', n_clusters = 3)
In [119]:
hierarchical_fit = hierarchical.fit( I_X )
In [120]:
hierarchical_result = hierarchical_fit.labels_
Compare Original and Cluster
In [121]:
pd.DataFrame( list(zip(iris.target_names[I_y],
iris.target_names[hierarchical_result])), columns=['Measured','Predicted'] )
Out[121]:
In [122]:
from sklearn.metrics import silhouette_score
print('Silhouette:',silhouette_score( I_X, hierarchical_result))