malware-detection-using-clustering.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"source":"<a href=\"https://www.kaggle.com/code/vidhikishorwaghela/malware-detection-using-clustering?scriptVersionId=143202960\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","source":"<a href=\"https://colab.research.google.com/gist/Vidhi1290/e3b51162bd38de6485c5e5b869112a89/malware-detection-clustering.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>","metadata":{"id":"view-in-github"}},{"cell_type":"markdown","source":"# CLUSTER ANALYSIS","metadata":{"id":"WKQTxiUk2qEr"}},{"cell_type":"code","source":"#from google.colab import drive\n#drive.mount('/content/drive')","metadata":{"id":"d-4OMvbg2C_F","outputId":"02787607-3524-4208-d5f6-1f36aaa2712a","execution":{"iopub.status.busy":"2023-09-16T11:46:31.09807Z","iopub.execute_input":"2023-09-16T11:46:31.098518Z","iopub.status.idle":"2023-09-16T11:46:31.104816Z","shell.execute_reply.started":"2023-09-16T11:46:31.098487Z","shell.execute_reply":"2023-09-16T11:46:31.103313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### WHAT WE GONNA DO IN THIS JUPYTER NOTEBOOK: MORE LIKE A CHECKLIST ✅","metadata":{"id":"HxGv2H75N6L6"}},{"cell_type":"markdown","source":"* Problem Statement\n* Understanding Data\n* Data Preparation\n    * Choose relevant features\n    * Missing value analysis\n    * Data standardization\n* K-Means Clustering\n* Clustering in practice\n    * Silhouette value of clusters\n    * Stability check of the clusters\n    * Categorize new samples into predefined clusters","metadata":{"id":"Cbzl2Jkj-UgW"}},{"cell_type":"markdown","source":"### LET'S UNDERSTAND THE PROBLEM STATEMENT:","metadata":{"id":"VwloE3ILOK-Y"}},{"cell_type":"markdown","source":"NOTE: I TOOK THIS PROBLEM AND DATASET FROM KAGGLE, I WOULD MENTION THE LINK AT THE END OF THIS NOTEBOOK, INCASE YOU WANNA REFER:\n\nLooking at current situation around us, it is obvious that the next world war is very much going to be the one fought out in cyber space.\nThe more successful the researchers become in cracking down malicious code, the more sophisticated mal-code appear in the wild, evading all kinds of detection mechanism established to counter it. With that in mind I decided to create a machine learning approach to detect malicious executable.","metadata":{"id":"Jt6HBKXWOOSX"}},{"cell_type":"markdown","source":"UNDERSTANDING THE DATA AND ATTRIBUTES DESCRIPTION:\n\nThe Dataset contains features extracted from malicious and non-malicious windows executable files.\n\n\nI have created this training file using hybrid features (binary hexadecimal + DLL calls) from windows executables.\n\n\nThe file consist of total 373 samples of which 301 are malicious files while rest 72 are non-malicious.\n\n\nThe dataset is imbalanced with malware samples more than regular samples.\nThere are 531 features represented from F1 all way to F531 and a label column stating if the file is malicious or non-malicious.\n\n\nAs representing binary hexadecimal feature names was difficult they have been represented as F1, F2,.. and so on, the same applies to DLL calls which are included in it. Some features out of 531 feature can be dropped as they hold very little importance, more to be explored as part of feature engineering.\n\n\nColumn label would hold true value of the executable file if it was a malware or not.\n","metadata":{"id":"JXJ6_pUZPanv"}},{"cell_type":"code","source":"#Import required packages\nimport pandas as pd\nimport numpy as np","metadata":{"id":"J-PcolS12yYb","execution":{"iopub.status.busy":"2023-09-16T11:46:31.107182Z","iopub.execute_input":"2023-09-16T11:46:31.107524Z","iopub.status.idle":"2023-09-16T11:46:31.118741Z","shell.execute_reply.started":"2023-09-16T11:46:31.107494Z","shell.execute_reply":"2023-09-16T11:46:31.11767Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# UNDERSTANDING THE DATA","metadata":{"id":"yC42JkR222bv"}},{"cell_type":"code","source":"malware = pd.read_csv(\"/kaggle/input/malware-executable-detection/uci_malware_detection.csv\")\nmalware.head()","metadata":{"id":"9ygeOOf020l2","outputId":"0dccd1e8-d050-4aa5-ad40-50ddbbb07f53","execution":{"iopub.status.busy":"2023-09-16T11:46:31.1201Z","iopub.execute_input":"2023-09-16T11:46:31.120963Z","iopub.status.idle":"2023-09-16T11:46:31.197588Z","shell.execute_reply.started":"2023-09-16T11:46:31.120928Z","shell.execute_reply":"2023-09-16T11:46:31.196376Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware.tail()","metadata":{"id":"bEgZoQfB291j","outputId":"df3b4477-c519-49d9-bfab-ac4c3beb7304","execution":{"iopub.status.busy":"2023-09-16T11:46:31.199288Z","iopub.execute_input":"2023-09-16T11:46:31.199683Z","iopub.status.idle":"2023-09-16T11:46:31.223454Z","shell.execute_reply.started":"2023-09-16T11:46:31.199649Z","shell.execute_reply":"2023-09-16T11:46:31.222131Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Checking the datatype of each attribute\nmalware.dtypes","metadata":{"id":"QUQbBqIQ3BX7","outputId":"4a6112fa-358f-494a-c843-84ffd6b109bc","execution":{"iopub.status.busy":"2023-09-16T11:46:31.227136Z","iopub.execute_input":"2023-09-16T11:46:31.227539Z","iopub.status.idle":"2023-09-16T11:46:31.238584Z","shell.execute_reply.started":"2023-09-16T11:46:31.227505Z","shell.execute_reply":"2023-09-16T11:46:31.237035Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Claculating the Statistics\nmalware.describe()","metadata":{"id":"iQns32-03KWF","outputId":"328a8a52-3441-4267-b4a7-d93fa3393e02","execution":{"iopub.status.busy":"2023-09-16T11:46:31.240417Z","iopub.execute_input":"2023-09-16T11:46:31.24082Z","iopub.status.idle":"2023-09-16T11:46:32.481914Z","shell.execute_reply.started":"2023-09-16T11:46:31.240789Z","shell.execute_reply":"2023-09-16T11:46:32.480614Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# PREPARING THE DATA","metadata":{"id":"cMKOjCDc3Rwu"}},{"cell_type":"markdown","source":"PERFORMING IMPUTATION METHOD TO DRP ROWS AND COLUMNS WHOSE THRESHOLD VALUE IS LESS THAN 70%\n\nThe simplest way to deal with missing data is to remove the rows or the entire column. There is not an optimal criterion for dropping however you may take 70% as an ideal percentage and try to eliminate the rows and columns who have missing data with greater than this threshold.","metadata":{"id":"dm1m9j-4gdbN"}},{"cell_type":"code","source":"threshold = 0.7\n#Dropping columns with missing value rate higher than threshold\nmalware = malware[malware.columns[malware.isnull().mean() < threshold]]\n\n#Dropping rows with missing value rate higher than threshold\nmalware = malware.loc[malware.isnull().mean(axis=1) < threshold]","metadata":{"id":"3nxzb36KUtpi","execution":{"iopub.status.busy":"2023-09-16T11:46:32.483727Z","iopub.execute_input":"2023-09-16T11:46:32.48406Z","iopub.status.idle":"2023-09-16T11:46:32.502263Z","shell.execute_reply.started":"2023-09-16T11:46:32.48403Z","shell.execute_reply":"2023-09-16T11:46:32.501106Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"##Choose required features","metadata":{"id":"QrW2vFKO-Uga"}},{"cell_type":"code","source":"malware['label'] = malware['Label']+ ')'\nmalware.head(3)","metadata":{"id":"AYOk0h8O-Uga","outputId":"1ee15d46-d475-4c84-e028-af258208ca1d","execution":{"iopub.status.busy":"2023-09-16T11:46:32.50423Z","iopub.execute_input":"2023-09-16T11:46:32.504558Z","iopub.status.idle":"2023-09-16T11:46:32.525927Z","shell.execute_reply.started":"2023-09-16T11:46:32.504516Z","shell.execute_reply":"2023-09-16T11:46:32.524972Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware.drop(['Label'], axis=1, inplace=True)\nmalware.head(3)","metadata":{"id":"FI0FSFDp-Uga","outputId":"4f126266-0c98-4975-fc83-572b52c0c747","execution":{"iopub.status.busy":"2023-09-16T11:46:32.527154Z","iopub.execute_input":"2023-09-16T11:46:32.52805Z","iopub.status.idle":"2023-09-16T11:46:32.553267Z","shell.execute_reply.started":"2023-09-16T11:46:32.528013Z","shell.execute_reply":"2023-09-16T11:46:32.552451Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware_label = malware['label']\n## Select all columns except \"label\"\nmalware.drop('label', axis=1, inplace=True)\nmalware.head(3)","metadata":{"id":"5KEWRU1N-Uga","outputId":"c8df562d-ba9c-4f86-f0cd-785f48bd2f8f","execution":{"iopub.status.busy":"2023-09-16T11:46:32.554339Z","iopub.execute_input":"2023-09-16T11:46:32.555355Z","iopub.status.idle":"2023-09-16T11:46:32.578036Z","shell.execute_reply.started":"2023-09-16T11:46:32.555323Z","shell.execute_reply":"2023-09-16T11:46:32.576742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware_label","metadata":{"id":"-XuBWS-Z-Ugb","outputId":"554bd484-ba62-4a28-da27-8aa48e92f03c","execution":{"iopub.status.busy":"2023-09-16T11:46:32.579622Z","iopub.execute_input":"2023-09-16T11:46:32.580205Z","iopub.status.idle":"2023-09-16T11:46:32.594814Z","shell.execute_reply.started":"2023-09-16T11:46:32.580174Z","shell.execute_reply":"2023-09-16T11:46:32.593989Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# MISSING VALUE ANALYSIS","metadata":{"id":"NfZKbMdZBFRx"}},{"cell_type":"code","source":"malware.isnull().sum(axis=0)","metadata":{"id":"v-4PzLAXAeXw","outputId":"ce42664d-082b-44ae-8694-a97eefa3d783","execution":{"iopub.status.busy":"2023-09-16T11:46:32.596091Z","iopub.execute_input":"2023-09-16T11:46:32.596624Z","iopub.status.idle":"2023-09-16T11:46:32.612712Z","shell.execute_reply.started":"2023-09-16T11:46:32.596595Z","shell.execute_reply":"2023-09-16T11:46:32.611312Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.impute import SimpleImputer\nmean_imputer = SimpleImputer(strategy='mean')\nmean_imputer.fit(malware)\nmean_imputer","metadata":{"id":"CQ82Jn9JG6Zv","outputId":"4b4f908b-6bd2-4bea-da97-9d390d679079","execution":{"iopub.status.busy":"2023-09-16T11:46:32.61961Z","iopub.execute_input":"2023-09-16T11:46:32.620259Z","iopub.status.idle":"2023-09-16T11:46:33.672995Z","shell.execute_reply.started":"2023-09-16T11:46:32.620207Z","shell.execute_reply":"2023-09-16T11:46:33.671944Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware = pd.DataFrame(mean_imputer.transform(malware),columns=malware.columns)","metadata":{"id":"S3HnIfKeHAN-","execution":{"iopub.status.busy":"2023-09-16T11:46:33.674569Z","iopub.execute_input":"2023-09-16T11:46:33.675138Z","iopub.status.idle":"2023-09-16T11:46:33.691397Z","shell.execute_reply.started":"2023-09-16T11:46:33.675104Z","shell.execute_reply":"2023-09-16T11:46:33.690439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware.isnull().sum(axis=0)","metadata":{"id":"nff7UzSFHHQl","outputId":"c8dbb882-e70b-4b8a-ccd9-980b5b10c0cd","execution":{"iopub.status.busy":"2023-09-16T11:46:33.693353Z","iopub.execute_input":"2023-09-16T11:46:33.693907Z","iopub.status.idle":"2023-09-16T11:46:33.711868Z","shell.execute_reply.started":"2023-09-16T11:46:33.693857Z","shell.execute_reply":"2023-09-16T11:46:33.710424Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# DATA STANDARDIZATION","metadata":{"id":"Yqk1fBFhHQc2"}},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nstandardizer = StandardScaler()\nstandardizer.fit(malware)\nmalware_std = pd.DataFrame(standardizer.transform(malware))\nmalware_std.head()","metadata":{"id":"sEQfyHVDHXvd","outputId":"9078315f-9184-4ac9-b52e-1815d148e6e2","execution":{"iopub.status.busy":"2023-09-16T11:46:33.713762Z","iopub.execute_input":"2023-09-16T11:46:33.7144Z","iopub.status.idle":"2023-09-16T11:46:33.767428Z","shell.execute_reply.started":"2023-09-16T11:46:33.714364Z","shell.execute_reply":"2023-09-16T11:46:33.765809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware_std.describe()","metadata":{"id":"JmYi0b5UIdDP","outputId":"5b758438-dc03-4c49-e9be-17b7206ed418","execution":{"iopub.status.busy":"2023-09-16T11:46:33.769336Z","iopub.execute_input":"2023-09-16T11:46:33.769809Z","iopub.status.idle":"2023-09-16T11:46:34.924179Z","shell.execute_reply.started":"2023-09-16T11:46:33.769767Z","shell.execute_reply":"2023-09-16T11:46:34.922935Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## K-MEANS CLUSTERING","metadata":{"id":"NI7fb0y7Iprr"}},{"cell_type":"markdown","source":"1. Kmeans is a distance based iterative technique, where the instances that are \"closer\" are \"grouped\" together forming a \"cluster\".\n2. This \"closeness\" is computed by distances,by default, Euclidean distances\n3. We need to specify prior, how many clusters we want to get.\n4. What is iterative in this case?\n\nA. We specify a number of clusters we need, so in the first iteration, centroids(centre) of the cluster are randomly picked in the data (this centroid need not be a data point but could be any other point as well). For eg: if we need 3 clusters, 3 centroids are randomly picked.\n\nB. Now with respect to each of these centroids, distance is computed for each of the points in the data and the data point is assigned to that cluster for which the point's distance is closest to its centroid. This is \"Assignment phase\".\n\nC .Once all points are assigned to the clusters, new centroids are computed from the points of each cluster (in 2d it is (x1+x2)/2, (y1+y2)/2)..remember this formula :)\n\nD. Once, this new centroids are computed, the assignment phase starts-- compute the distance between each of the data points with each of the new centroids and assign the point to the closest cluster. After assignment, the new cluster centroids are computed. This process continues until there is no change in cluster centroids from previous iteration\n","metadata":{"id":"uFCmioZiI7yy"}},{"cell_type":"code","source":"from sklearn.cluster import KMeans\nkmeans_object = KMeans(n_clusters=5, random_state=1240)\nkmeans_object.fit(malware_std)\nkmeans_clusters = kmeans_object.predict(malware_std)","metadata":{"id":"99wy8qVmJcBt","execution":{"iopub.status.busy":"2023-09-16T11:46:34.926048Z","iopub.execute_input":"2023-09-16T11:46:34.926491Z","iopub.status.idle":"2023-09-16T11:46:35.770049Z","shell.execute_reply.started":"2023-09-16T11:46:34.926448Z","shell.execute_reply":"2023-09-16T11:46:35.768386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kmeans_clusters","metadata":{"id":"mGT52-IKJ4Az","outputId":"93703a2f-8c93-4324-f487-ac6acb100457","execution":{"iopub.status.busy":"2023-09-16T11:46:35.772097Z","iopub.execute_input":"2023-09-16T11:46:35.77273Z","iopub.status.idle":"2023-09-16T11:46:35.787638Z","shell.execute_reply.started":"2023-09-16T11:46:35.77268Z","shell.execute_reply":"2023-09-16T11:46:35.785957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"np.unique(kmeans_clusters, return_counts=True)","metadata":{"id":"DoQPemDNJ5VX","outputId":"cb852533-a78d-499c-8c03-708d8d318bad","execution":{"iopub.status.busy":"2023-09-16T11:46:35.789517Z","iopub.execute_input":"2023-09-16T11:46:35.790641Z","iopub.status.idle":"2023-09-16T11:46:35.803397Z","shell.execute_reply.started":"2023-09-16T11:46:35.790595Z","shell.execute_reply":"2023-09-16T11:46:35.801843Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kmeans_results = pd.DataFrame({\"label\":malware_label,\"kmeans_cluster\":kmeans_clusters})\nkmeans_results.head(10)","metadata":{"id":"0DmAqISBKBiT","outputId":"877aac70-5b25-4cfa-b246-337745ef155c","execution":{"iopub.status.busy":"2023-09-16T11:46:35.805748Z","iopub.execute_input":"2023-09-16T11:46:35.806205Z","iopub.status.idle":"2023-09-16T11:46:35.824345Z","shell.execute_reply.started":"2023-09-16T11:46:35.806163Z","shell.execute_reply":"2023-09-16T11:46:35.823073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Inspecting cluster centroids to understand average statistics of each cluster","metadata":{"id":"_fDVXfh-KKlD"}},{"cell_type":"code","source":"## Using inverse_transform to retrive actual values from standardized data\ncluster_centroids = pd.DataFrame(standardizer.inverse_transform(kmeans_object.cluster_centers_),\n                                 columns=malware.columns)   \ncluster_centroids","metadata":{"id":"nV2Fm3VUKMoj","outputId":"792c4c5f-4af9-417a-c78a-554c7ed38b9c","execution":{"iopub.status.busy":"2023-09-16T11:46:35.826474Z","iopub.execute_input":"2023-09-16T11:46:35.82732Z","iopub.status.idle":"2023-09-16T11:46:35.862575Z","shell.execute_reply.started":"2023-09-16T11:46:35.827274Z","shell.execute_reply":"2023-09-16T11:46:35.861263Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cluster_centroids.to_csv(\"best_kmeans_cluster_centroids.csv\")","metadata":{"id":"ve8-_1_DKbL_","execution":{"iopub.status.busy":"2023-09-16T11:46:35.863978Z","iopub.execute_input":"2023-09-16T11:46:35.864283Z","iopub.status.idle":"2023-09-16T11:46:35.878325Z","shell.execute_reply.started":"2023-09-16T11:46:35.864255Z","shell.execute_reply":"2023-09-16T11:46:35.877439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"wss= {}\nfor k in range(1, 21):\n    kmeans_loop = KMeans(n_clusters=k,n_init=30,max_iter=300,random_state=1000).fit(malware_std)\n    clusters = kmeans_loop.labels_\n    wss[k] = kmeans_loop.inertia_ # Inertia: Sum of squared distances of samples to their closest cluster center\n","metadata":{"id":"toyqtltZKdMg","execution":{"iopub.status.busy":"2023-09-16T11:46:35.879468Z","iopub.execute_input":"2023-09-16T11:46:35.880192Z","iopub.status.idle":"2023-09-16T11:47:25.147633Z","shell.execute_reply.started":"2023-09-16T11:46:35.880156Z","shell.execute_reply":"2023-09-16T11:47:25.146629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n%matplotlib inline\nplt.figure()\n\nplt.plot(list(wss.keys()),list(wss.values()), 'bx--')\nplt.xlabel('Number of clusters')\nplt.ylabel('Total within sum of squares')\nplt.show()","metadata":{"id":"FigBqt-kLhQu","outputId":"b556a0d7-82f8-4fb8-c725-d232bf8c0bad","execution":{"iopub.status.busy":"2023-09-16T11:47:25.149258Z","iopub.execute_input":"2023-09-16T11:47:25.149954Z","iopub.status.idle":"2023-09-16T11:47:25.50301Z","shell.execute_reply.started":"2023-09-16T11:47:25.149912Z","shell.execute_reply":"2023-09-16T11:47:25.501918Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Silhouette Analysis\n\n## Higher the silhouette score better the clustering\n\nThe silhouette value is a measure of how similar an object is to its own cluster (cohesion) compared to other clusters (separation). The silhouette ranges from −1 to +1, where a high value indicates that the object is well matched to its own cluster and poorly matched to neighboring clusters. If most objects have a high value, then the clustering configuration is appropriate. If many points have a low or negative value, then the clustering configuration may have too many or too few clusters.","metadata":{"id":"-g1ifB5hhQZ_"}},{"cell_type":"code","source":"## Standard plotting code copied from sklearn documentation.\n## Just change \"X_matrix\" to the data of your choice\n\nfrom sklearn.metrics import silhouette_samples, silhouette_score\nimport matplotlib.cm as cm\n\n### Just change this to your dataframe\nX_matrix = malware_std.values\n\nrange_n_clusters = [3, 4, 5, 6, 7, 8, 9, 10, 11]\n\nfor n_clusters in range_n_clusters:\n    fig, (ax1) = plt.subplots()\n\n\n    # Initialize the clusterer with n_clusters value and a random generator\n    # seed of 10 for reproducibility.\n    clusterer = KMeans(n_clusters=n_clusters, random_state=10)\n    cluster_labels = clusterer.fit_predict(X_matrix)\n\n    # The silhouette_score gives the average value for all the samples.\n    # This gives a perspective into the density and separation of the formed\n    # clusters\n    silhouette_avg = silhouette_score(X_matrix, cluster_labels)\n\n    # Compute the silhouette scores for each sample\n    sample_silhouette_values = silhouette_samples(X_matrix, cluster_labels)\n\n    y_lower = 0\n    for i in range(n_clusters):\n        # Aggregate the silhouette scores for samples belonging to\n        # cluster i, and sort them\n        ith_cluster_silhouette_values = \\\n            sample_silhouette_values[cluster_labels == i]\n\n        ith_cluster_silhouette_values.sort()\n\n        size_cluster_i = ith_cluster_silhouette_values.shape[0]\n        y_upper = y_lower + size_cluster_i\n\n        color = cm.nipy_spectral(float(i) / n_clusters)\n        ax1.fill_betweenx(np.arange(y_lower, y_upper),\n                          0, ith_cluster_silhouette_values,\n                          facecolor=color, edgecolor=color, alpha=0.7)\n\n        # Label the silhouette plots with their cluster numbers at the middle\n        ax1.text(-0.05, y_lower + 0.5 * size_cluster_i, str(i))\n\n        # Compute the new y_lower for next plot\n        y_lower = y_upper + 10  # 10 for the 0 samples\n\n    ax1.set_title(\"The silhouette plot for the various clusters\")\n    ax1.set_xlabel(\"The silhouette coefficient values\")\n    ax1.set_ylabel(\"Cluster label\")\n\n    # The vertical line for average silhouette score of all the values\n    ax1.axvline(x=silhouette_avg, color=\"red\", linestyle=\"--\")\n\n    ax1.set_yticks([])  # Clear the yaxis labels / ticks\n    ax1.set_xticks([-0.2, 0, 0.2, 0.4, 0.6, 0.8, 1])\n\n    plt.suptitle((\"For %d clusters, silhouette avg coeff = %f \" % (n_clusters,silhouette_avg)),\n                 fontsize=14, fontweight='bold')\n    plt.show()","metadata":{"id":"9C0wRWqFLz5r","outputId":"050467c1-c7cb-42d3-f30a-3031f7899ac4","execution":{"iopub.status.busy":"2023-09-16T11:47:25.504777Z","iopub.execute_input":"2023-09-16T11:47:25.505459Z","iopub.status.idle":"2023-09-16T11:47:34.882673Z","shell.execute_reply.started":"2023-09-16T11:47:25.505415Z","shell.execute_reply":"2023-09-16T11:47:34.881571Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"FINAL NOTE:  Higher the silhouette score better the clustering.\n\nHence best K value for this dataset is 3","metadata":{"id":"Io1yTpDrhDbI"}},{"cell_type":"code","source":"best_kmeans = KMeans(n_clusters=8, random_state=1240)\nbest_kmeans.fit(malware_std)\nbest_kmeans_labels = best_kmeans.predict(malware_std)","metadata":{"id":"LVztULm0X-U5","execution":{"iopub.status.busy":"2023-09-16T11:47:34.884261Z","iopub.execute_input":"2023-09-16T11:47:34.885257Z","iopub.status.idle":"2023-09-16T11:47:35.628041Z","shell.execute_reply.started":"2023-09-16T11:47:34.885213Z","shell.execute_reply":"2023-09-16T11:47:35.626422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kmeans_results = pd.DataFrame({\"label\":malware_label,\"kmeans_cluster\":best_kmeans_labels})\nkmeans_results.head()","metadata":{"id":"eYKwlU0WhiyF","outputId":"1f5a7747-23e7-45f3-ee33-3cd36e136dcb","execution":{"iopub.status.busy":"2023-09-16T11:47:35.629757Z","iopub.execute_input":"2023-09-16T11:47:35.630586Z","iopub.status.idle":"2023-09-16T11:47:35.651692Z","shell.execute_reply.started":"2023-09-16T11:47:35.630492Z","shell.execute_reply":"2023-09-16T11:47:35.649957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware = pd.read_csv(\"/kaggle/input/malware-executable-detection/uci_malware_detection.csv\")\nmalware['label'] = malware['Label']+ ')'\nmalware.drop(['Label',], axis=1, inplace=True)\n\nfinal_cluster_data = pd.merge(malware, kmeans_results, on='label')\nfinal_cluster_data.head(10)","metadata":{"id":"qOU-A_4Ujsb3","outputId":"6d49d3e4-5f20-462d-d92e-d55f9d90ffd1","execution":{"iopub.status.busy":"2023-09-16T11:47:52.436064Z","iopub.execute_input":"2023-09-16T11:47:52.437233Z","iopub.status.idle":"2023-09-16T11:47:52.851057Z","shell.execute_reply.started":"2023-09-16T11:47:52.43719Z","shell.execute_reply":"2023-09-16T11:47:52.849882Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_cluster_data.shape","metadata":{"id":"9QoxGE1zMMFx","outputId":"148f0dca-8213-494a-8c05-d786ef935a42","execution":{"iopub.status.busy":"2023-09-16T11:47:53.754273Z","iopub.execute_input":"2023-09-16T11:47:53.754778Z","iopub.status.idle":"2023-09-16T11:47:53.763636Z","shell.execute_reply.started":"2023-09-16T11:47:53.754736Z","shell.execute_reply":"2023-09-16T11:47:53.76226Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"final_cluster_data.kmeans_cluster.value_counts()","metadata":{"id":"jIz-5pEmkdHs","outputId":"1aaad8d0-2d34-46db-d933-e1ad45299d8a","execution":{"iopub.status.busy":"2023-09-16T11:47:54.05426Z","iopub.execute_input":"2023-09-16T11:47:54.054685Z","iopub.status.idle":"2023-09-16T11:47:54.067004Z","shell.execute_reply.started":"2023-09-16T11:47:54.054651Z","shell.execute_reply":"2023-09-16T11:47:54.065959Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## CHECKING THE CLUSTER STABILITY\n\n","metadata":{"id":"2MqOQUnvMc11"}},{"cell_type":"code","source":"from sklearn.metrics import adjusted_rand_score\nimport numpy as np","metadata":{"id":"dbJNHQznMaP_","execution":{"iopub.status.busy":"2023-09-16T11:50:57.846648Z","iopub.execute_input":"2023-09-16T11:50:57.847128Z","iopub.status.idle":"2023-09-16T11:50:57.853449Z","shell.execute_reply.started":"2023-09-16T11:50:57.847089Z","shell.execute_reply":"2023-09-16T11:50:57.851921Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"indices=malware_std.sample(frac=0.9,random_state=123).index\nprint(indices)","metadata":{"id":"Dq-nsil-McV0","outputId":"53e41fd6-2027-4793-f3aa-e468b47e2f09","execution":{"iopub.status.busy":"2023-09-16T11:50:58.261934Z","iopub.execute_input":"2023-09-16T11:50:58.263003Z","iopub.status.idle":"2023-09-16T11:50:58.273895Z","shell.execute_reply.started":"2023-09-16T11:50:58.262944Z","shell.execute_reply":"2023-09-16T11:50:58.272309Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware_std_subset=malware_std.iloc[indices,:]","metadata":{"id":"MhcLUnUOMr5E","execution":{"iopub.status.busy":"2023-09-16T11:50:58.816694Z","iopub.execute_input":"2023-09-16T11:50:58.817141Z","iopub.status.idle":"2023-09-16T11:50:58.824546Z","shell.execute_reply.started":"2023-09-16T11:50:58.817107Z","shell.execute_reply":"2023-09-16T11:50:58.823343Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"malware_std_subset.shape","metadata":{"id":"NQdo-m_PMv1Z","outputId":"1698092f-8a6c-4acd-977b-c889f8291de1","execution":{"iopub.status.busy":"2023-09-16T11:50:59.780846Z","iopub.execute_input":"2023-09-16T11:50:59.781261Z","iopub.status.idle":"2023-09-16T11:50:59.788874Z","shell.execute_reply.started":"2023-09-16T11:50:59.781229Z","shell.execute_reply":"2023-09-16T11:50:59.787631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kmeans_object = KMeans(n_clusters=5,n_init=30,max_iter=300,random_state=1000)\nkmeans_object.fit(malware_std)\nclus1= kmeans_object.predict(malware_std)","metadata":{"id":"fUINi9Z4MzFO","execution":{"iopub.status.busy":"2023-09-16T11:51:00.519899Z","iopub.execute_input":"2023-09-16T11:51:00.520295Z","iopub.status.idle":"2023-09-16T11:51:01.646805Z","shell.execute_reply.started":"2023-09-16T11:51:00.520264Z","shell.execute_reply":"2023-09-16T11:51:01.645863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kmeans_object = KMeans(n_clusters=5,n_init=30,max_iter=300,random_state=1000)\nkmeans_object.fit(malware_std_subset)\nclus2= kmeans_object.predict(malware_std_subset)","metadata":{"id":"X4VaN1J2M4Op","execution":{"iopub.status.busy":"2023-09-16T11:51:02.088429Z","iopub.execute_input":"2023-09-16T11:51:02.088824Z","iopub.status.idle":"2023-09-16T11:51:03.871757Z","shell.execute_reply.started":"2023-09-16T11:51:02.088792Z","shell.execute_reply":"2023-09-16T11:51:03.870764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(len(clus1))\nprint(len(clus2))","metadata":{"id":"KUEHgTHgM9Cu","outputId":"28d98e70-6806-40b3-e0d7-14b75a21a743","execution":{"iopub.status.busy":"2023-09-16T11:51:07.855983Z","iopub.execute_input":"2023-09-16T11:51:07.856419Z","iopub.status.idle":"2023-09-16T11:51:07.861658Z","shell.execute_reply.started":"2023-09-16T11:51:07.856386Z","shell.execute_reply":"2023-09-16T11:51:07.86027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"clus1=clus1[indices]\nprint(len(clus1))","metadata":{"id":"MWLqGoWfM_Do","outputId":"2c21bcd4-7a93-464e-9e1a-f3adde5de359","execution":{"iopub.status.busy":"2023-09-16T11:51:08.105142Z","iopub.execute_input":"2023-09-16T11:51:08.105527Z","iopub.status.idle":"2023-09-16T11:51:08.110822Z","shell.execute_reply.started":"2023-09-16T11:51:08.105497Z","shell.execute_reply":"2023-09-16T11:51:08.109677Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"adjusted_rand_score(clus1,clus2)","metadata":{"id":"wMcnBSVqNDo2","outputId":"6c79cd9f-8a7a-41a7-c6cc-8810ad85e0c5","execution":{"iopub.status.busy":"2023-09-16T11:51:09.337842Z","iopub.execute_input":"2023-09-16T11:51:09.338327Z","iopub.status.idle":"2023-09-16T11:51:09.355594Z","shell.execute_reply.started":"2023-09-16T11:51:09.338273Z","shell.execute_reply":"2023-09-16T11:51:09.354154Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Categorize new samples into predefined clusters","metadata":{"id":"r-lmhSZ3-Ugf"}},{"cell_type":"code","source":"newdata=pd.read_csv(\"/kaggle/input/malware-executable-detection/uci_malware_detection.csv\") ## Assume this part is a newly added samples for demonstration ","metadata":{"id":"SEZ0clCo-Ugf","execution":{"iopub.status.busy":"2023-09-16T11:51:45.047183Z","iopub.execute_input":"2023-09-16T11:51:45.047688Z","iopub.status.idle":"2023-09-16T11:51:45.106815Z","shell.execute_reply.started":"2023-09-16T11:51:45.047646Z","shell.execute_reply":"2023-09-16T11:51:45.105741Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"newdata","metadata":{"id":"wRiI4Efm-Ugf","outputId":"4b720895-0971-47dc-f5e1-21cacaf11eb9","execution":{"iopub.status.busy":"2023-09-16T11:51:47.773216Z","iopub.execute_input":"2023-09-16T11:51:47.773602Z","iopub.status.idle":"2023-09-16T11:51:47.798654Z","shell.execute_reply.started":"2023-09-16T11:51:47.773572Z","shell.execute_reply":"2023-09-16T11:51:47.79752Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"newdata['label'] = newdata['Label']+ ')'\n\nnewdata.drop(['Label'], axis=1, inplace=True)\n\n\nnewdata_label = newdata['label']\n## Select all columns except \"label\"\nnewdata.drop('label', axis=1, inplace=True)\nnewdata","metadata":{"id":"Pvd1rQRn-Ugf","outputId":"143c8335-ee26-40c7-a73b-f14e966e1f0d","execution":{"iopub.status.busy":"2023-09-16T11:51:48.019188Z","iopub.execute_input":"2023-09-16T11:51:48.019595Z","iopub.status.idle":"2023-09-16T11:51:48.047111Z","shell.execute_reply.started":"2023-09-16T11:51:48.019565Z","shell.execute_reply":"2023-09-16T11:51:48.04628Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"newdata = pd.DataFrame(mean_imputer.transform(newdata),columns=newdata.columns)\nnewdata\n","metadata":{"id":"NVhZ2v3T-Ugf","outputId":"ebbe5d42-a9f6-4d55-e73c-dd3ded6dd4a1","execution":{"iopub.status.busy":"2023-09-16T11:51:48.28038Z","iopub.execute_input":"2023-09-16T11:51:48.280949Z","iopub.status.idle":"2023-09-16T11:51:48.343142Z","shell.execute_reply.started":"2023-09-16T11:51:48.280906Z","shell.execute_reply":"2023-09-16T11:51:48.342009Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"newdata_std = pd.DataFrame(standardizer.transform(newdata))\nnewdata_std","metadata":{"id":"h7ecKbHZ-Ugg","outputId":"177740e0-5715-48ea-b3fe-4646da34f64f","execution":{"iopub.status.busy":"2023-09-16T11:51:51.348792Z","iopub.execute_input":"2023-09-16T11:51:51.349727Z","iopub.status.idle":"2023-09-16T11:51:51.398496Z","shell.execute_reply.started":"2023-09-16T11:51:51.34969Z","shell.execute_reply":"2023-09-16T11:51:51.397449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_kmeans_labels = best_kmeans.predict(newdata_std)\nbest_kmeans_labels","metadata":{"id":"H_Z6GqJ2-Ugg","outputId":"ca75840b-7470-4e8e-f3a9-bc64bfdf5d3c","execution":{"iopub.status.busy":"2023-09-16T11:51:51.597677Z","iopub.execute_input":"2023-09-16T11:51:51.598067Z","iopub.status.idle":"2023-09-16T11:51:51.61947Z","shell.execute_reply.started":"2023-09-16T11:51:51.598035Z","shell.execute_reply":"2023-09-16T11:51:51.618493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kmeans_results = pd.DataFrame({\"label\":newdata_label,\"kmeans_cluster\":best_kmeans_labels})\nkmeans_results","metadata":{"id":"pxdvkgdh-Ugg","outputId":"77b20a7c-d826-4923-88c6-6f8e38329d04","execution":{"iopub.status.busy":"2023-09-16T11:51:55.420707Z","iopub.execute_input":"2023-09-16T11:51:55.421118Z","iopub.status.idle":"2023-09-16T11:51:55.435589Z","shell.execute_reply.started":"2023-09-16T11:51:55.421077Z","shell.execute_reply":"2023-09-16T11:51:55.434322Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Learning Outcomes","metadata":{"id":"AAaob3bJ-Ugg"}},{"cell_type":"markdown","source":"* Be able implement cluster analysis in Python\n    * Be able to pre-process data\n    * Be able to implement hierarchical clustering\n    * Be able to visualise dendrogram and variants of linkage methods\n    * Be able to implement the cut-off value to obtain clusters\n    * Be able to implement k-means clustering algorithm\n    * Be able to experiment with different values of ‘k’ and display the elbow plot of errors\n    * Be able to compute Silhouette value of clusters\n* Be able to perform cluster stability check\n    * Be able to implement the stability check of clusters in multiple ways such as: random sampling of data, changing the value of ‘k’, etc.\n* Be able to extract cluster results to interpret cluster profiles\n    * Be able to extract the data with annotated cluster id\n    * Be able to derive cluster properties by analysing data with cluster statistics\n    * Be able to predict the clusters for new samples","metadata":{"id":"zBeERXfM-Ugg"}}]}