CNN+LSTM_Dataset_Preparation.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CNN+LSTM_Dataset_Preparation.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "4wiBn8-9gwHX"
      ]
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4wiBn8-9gwHX"
      },
      "source": [
        "#Dataset Preparation for Sequence Modeling\n",
        "\n",
        "Data needs to be packed(= arranged) in sequences\n",
        "\n",
        "NOTE: Sequence data of shape [no_of_files, SEQ_LEN, C, H, W] for all bearing data files doesn't fit in memory of LSTM, so they need to be packed into sequences at runtime. (no_of_files = total no. of files present in single combined pickle file(bearing1_1.pkz, bearing1_2.pkz, bearing2_1.pkz, bearing2_2.pkz, bearing3_1.pkz, bearing3_2.pkz), SEQ_LEN = each sequence length, C = no. of channels or no. of filters, H = height of an image in pixels, W = width of an image in pixels)\n",
        "\n",
        "(runtime - The runtime environment is literally python.exe, It's the Python executable(executable file) that will interpret(= convert, explain) our Python code by transforming(= changing) it into CPU-readable bytecode.)\n",
        "\n",
        "References - (1) https://www.youtube.com/watch?v=CznICCPa63Q&t=35s  (2) https://www.allerin.com/blog/sequence-modeling-for-beginners\n",
        "\n",
        "Sequence - one input sample. sequence is an input sample which consists of multiple inputs and those inputs depend on each other. sequence is a series of inputs, and the next input in the series depends on all the previous inputs. (sequences - means input samples(input examples) and each input sample consists of multiple data points. There can be a variable(= changeable) number of these data points per each input sample or input example and the data points depend on each other )\n",
        "\n",
        "(sequence synonyms = order, series, consecutive, succession)\n",
        "\n",
        "Sequence Modeling - Sequence modeling, put simply, is the process of generating a sequence of values by analyzing a series of input values. These input values could be time series data. (time series data is the data varies(= alters, changes) over a period of time. Sequence Modeling is a task of predicting what comes next. Unlike FNN and CNN, in sequence modeling, the current output is dependent on the previous inputs and the length of the inputs is not fixed\n",
        "\n",
        "(Recurrent Neural Networks (RNN) is the best example for sequence modeling. RNN is a sequence model algorithm. Sequence models are the machine learning models that input or output sequences of data. Sequence data or data sequence includes time-series data, text streams, audio clips, video clips, etc. )\n",
        "\n",
        "(modeling synonyms = creating, designing, forming) \n",
        "\n",
        "Sequence Modeling problems:\n",
        "\n",
        "(1) Can't model long-term dependencies \n",
        "\n",
        "(2) Don't preserve(= maintain, support) order of data points\n",
        "\n",
        "(3) no parameter sharing\n",
        "\n",
        "To model sequences, need:\n",
        "\n",
        "(1) to deal with variable length sequences\n",
        "\n",
        "(2) to maintain sequence order\n",
        "\n",
        "(3) to keep track of(monitor) long-term dependencies\n",
        "\n",
        "(Long-term dependency - (long-term = longer period), long-term dependencies are those problems for which the desired output depends on inputs presented at times far in the past )\n",
        "\n",
        "(4) to share parameters across(= everywhere on) the sequence\n",
        "\n",
        "Note:\n",
        "\n",
        "Our dataset is time-series dataset. Sequence Modeling is quite useful for time-series prediction. By applying sequence modeling to the time-series input data, we can obtain more insights from that time-series input data and prediction becomes easy and accurate\n",
        "\n",
        "Dataset preparation also called as dataset preprocessing. Any Machine Learning and Deep Learning model requires preprocessed data for training. Therefore data preprocessing should be performed on the dataset which we used to build the prediction model. Data Preprocessing is required for cleaning and organizing(= formatting, arranging) the raw data to make it suitable for building and training machine learnig models and deep learning models. Raw data refers to data that has not yet been preprocessed or prepared and raw data is the data cannot be understood by machine learning model and deep learning model. Preprocessing or preparing the raw data also removes unwanted and faulty data(errors) from data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ElDV_3NEi9jM"
      },
      "source": [
        "#Mounting Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "o94_buEEjW1R",
        "outputId": "9ef8ab19-bf8d-4b35-dc11-97e33857ffd0"
      },
      "source": [
        "'''\n",
        "Mounting => Before your computer can use any kind of storage device (such as hard drive, Google drive), we or our operating system must make it\n",
        "accessible through the computer’s file system. This process is called mounting. we can only access files on mounted media.\n",
        "In Computers, to mount is to make a group of files in a file system structure accessible to a user or user group. In some usages, it means to make a\n",
        "device physically accessible. Mounting a file system (Google drive) attaches that Google drive to a directory (mount point) and makes it available to the\n",
        "system. In simple words, with mounting a Google drive, user and operating system can access to all the files present in the Google drive. A mounted disk \n",
        "(a mounted drive) is available to the operating system as a file system, for reading, writing, or both.\n",
        "'''\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Mounted at /content/drive\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5_hh1Ec-nDUs"
      },
      "source": [
        "#Importing the Required Libraries"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "P3OpeDW3lUnS"
      },
      "source": [
        "'''\n",
        "Python library = python library is the collection of modules(= python files) and this python library is the reusable(=able to use again) chunk(= part, section\n",
        ", block) of code we want to include in our python programs or projects to make the implementation easier and faster.\n",
        "\n",
        "import = send\n",
        "\n",
        "os module in python provides functions for interacting with the operating system. os module in Python provides functions for creating and removing a \n",
        "directory(folder), fetching its contents, os module used for changing and identifying the current directory, etc. Basically os module allows source code\n",
        "to communicate (interact) with operating system.\n",
        "\n",
        "numpy module allows us to work with numerical data. numpy provides an object called numpy array. numpy supports large multi-dimensional arrays & matrices. \n",
        "Basically numpy is a python library used for working with arrays. numpy used for arithmetic operations, statistical operations, bitwise operations, copying \n",
        "and viewing arrays, stacking, matrix operations, linear algebra, mathematical operations, searching, sorting, and counting.\n",
        "\n",
        "pywt library used to perform wavelet transform (both Continuous Wavelet Transform (CWT) and Discrete Wavelet Transform (DWT)). This pywt library is a \n",
        "package of various wavelets of CWT and DWT (pywt library contains different wavelets of CWT and DWT) to perform wavelet transform.\n",
        "(A wavelet is a wave-like oscillation with an amplitude(= the maximum displacement or distance moved by a point on a vibrating body or wave) that begins at zero, \n",
        "increases, and then decreases back to zero.)\n",
        "\n",
        "pandas library is used for data manipulation and data analysis. pandas module works with the tabular data (i.e. data in rows and columns). Pandas provide a\n",
        "2D table object called dataframe. pandas module offers data structures and operations for manipulating numerical tables and time series.\n",
        "\n",
        "Pickle in Python is primarily used in serializing and deserializing a Python object structure. In other words, it's the process of converting a python\n",
        "object into a byte stream to store it in a file/database. Basically pickle library is used to dump (store) all the files of a directory (folder) into \n",
        "single combined file (pickle(.pkz) file) for easy fetching and fast retrieval of data. pickle.dump() is used to create a pickle file, it is used to dump\n",
        "(store) data in a pickle file. pickle.load() is used to load(= start, activate) pickle file\n",
        "\n",
        "Matplotlib is the most popular plotting(=sketching, drawing, designing, outlining) library for Python. python library = Collection of modules \n",
        "(modules = python files). Pyplot is a Matplotlib module which provides a MATLAB-like interface. Each pyplot function makes some change to a figure. For\n",
        "example, creates a figure, creates a plotting area in a figure, plots some lines in a plotting area,  decorates the plot with labels, etc. The various plots\n",
        "we can utilize using Pyplot are Line Plot, Histogram, Scatter, 3D Plot, Image, Contour, Hexagonal binning plot and Polar. Can also import as  \n",
        "\"import matplotlib.pyplot as plt\"\n",
        "\n",
        "as keyword is used as alias (AKA, also called as)\n",
        "'''\n",
        "import os\n",
        "import numpy as np\n",
        "import pywt\n",
        "import pandas as pd\n",
        "import pickle as pkl\n",
        "from matplotlib import pyplot as plt"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0D_g6iw4yfGf"
      },
      "source": [
        "#Parameters or Required Variables"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bO4u5LhVayXC"
      },
      "source": [
        "'''\n",
        "DATA_POINTS_PER_FILE => Data points present in each file. In the dataset in every directory (folder), each file contains 2560 data points. Therefore we\n",
        "created a variable DATA_POINTS_PER_FILE and assigned that variable with 2560\n",
        "\n",
        "TIME_PER_REC => Time recorded for each vibration signal. we store each vibration recording time in TIME_PER_REC. In the dataset, every 10 seconds, 1\n",
        "vibration recording of 0.1 seconds is collected. That means for every 10 seconds, one vibration (both horizontal and vertical) is recorded for 0.1 seconds\n",
        "Therefore we created a variable TIME_PER_REC and assigned that variable with 0.1\n",
        "\n",
        "SAMPLING_FREQ => Sampling frequency or sampling rate defines the number of samples (data points) per second. vibration sample frequency is 25.6 KHz => for 1\n",
        "second we get 25600 data points. each 10 seconds one vibration recording of 0.1s is collected => therefore 25600 * 0.1 (i.e. 2560) data points in each file\n",
        "for example, Bearing1_1 set has total 2803 files, each file with 2560 data points => 2803*2560=7175680 total data points\n",
        "\n",
        "The sampling period (or) sampling interval is the time difference between two consecutive(= successive, pakkana pakkana, subsequent) samples. It is the \n",
        "inverse of the sampling frequency. For example: if the sampling frequency is 25600 Hz, the sampling period is 1/25600 = 0.000039 seconds (that means the \n",
        "samples are spaced approximately 39 microseconds apart (=away, aside, distant, far))\n",
        "\n",
        "since we assigned WIN_SIZE = 20, we take 20 data points at a time. WIN_SIZE is the fixed size window \n",
        "\n",
        "We are using morlet wavelet in our code. Therefore we created a variable called WAVELET_TYPE and assigned ‘morl’ to the variable. morl => indicates morlet \n",
        "wavelet. assigning morl to the variable, means assigning morlet wavelet and morlet wavelet operations and functionalities to that variable. Morlet Wavelet is\n",
        "the most commonly used Continuous Wavelet Transform (CWT) Wavelet to respresent a vibration signal in time and frequency domains\n",
        "\n",
        "created VAL_SPLIT variable to store the percentage of data split(= divide, separate) into validation data from training data.\n",
        "since we assigned, 0.1 to the VAL_SPLIT variable, we are separating 10% data from training data (total training data) and made it as validation data\n",
        "\n",
        "SEQ_LEN = 5 => SEQ_LEN - each sequence length, therefore each sequence contains 5 data points\n",
        "\n",
        "'''\n",
        "DATA_POINTS_PER_FILE = 2560\n",
        "TIME_PER_REC = 0.1\n",
        "SAMPLING_FREQ = 25600 # 25.6 KHz\n",
        "SAMPLING_PERIOD = 1.0/SAMPLING_FREQ\n",
        "\n",
        "WIN_SIZE = 20\n",
        "WAVELET_TYPE = 'morl'\n",
        "\n",
        "VAL_SPLIT = 0.1 \n",
        "\n",
        "SEQ_LEN = 5 # sequence length"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jQ3TDIInJNTP"
      },
      "source": [
        "#Helper Functions\n",
        "\n",
        "Helper Function = A helper function is a function that performs part of the computation (= operation, calculation, estimation, guess) of another function. Helper functions are used to make our programs (= codes) easier to read by giving names to computations. Helper functions also let you reuse computations, just as with functions in general. A helper function is a function we write because we need that particular functionality (= purpose, operation) of a function in multiple places in a code, and because it makes the code more readable. Instead of defining a particular functionality many times, insert(=put, embed) the functionality which we required many times in a helper function, so that we can use that particular functionality as many times we required without defining again\n",
        "\n",
        "(1) Loading(= activating, starting) pickle(.pkz) files (Bearing1_1, Bearing1_2, Bearing2_1, Bearing2_2, Bearing3_1, Bearing3_2)\n",
        "\n",
        "(2) Applying CWT(Continuous Wavelet Transform)\n",
        "\n",
        "(3) Data Normalization"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WPF5_p_MIGT7"
      },
      "source": [
        "'''\n",
        "using def keyword, defined(=created) a function named load_df and passed pkz_file as an argument to the function\n",
        "\n",
        "with keyword => automatically releases memory after allocation. Whenever we open the file with open() function, it allocates some resources and memory to the \n",
        "file. And we should use close() function to release or delete that memory from the file otherwise errors will come. Sometimes we forget to close() the file \n",
        "and we couldn’t find that we didn’t closed the file, so even the whole code is correct we might get errors and we may not be able to correct it. So it’s better \n",
        "to use “with” keyword along with open() function as “with” keyword automatically releases or deletes memory after process completion\n",
        "\n",
        "The open() function opens a file in text format by default. To open a file in binary format, add 'b' to the mode parameter. Hence the \"rb\" mode opens the file \n",
        "in binary format for reading, while the \"wb\" mode opens the file in binary format for writing. (Note: there are 2 basic mode parameters (r = read mode,\n",
        "w = write mode)). Unlike text files, binary files are not human-readable.\n",
        "\n",
        "as = The as keyword is used to create an alias (= aka, also known as, also called). In the code, we create an alias f when opening the pkz_file, and \n",
        "now we can refer to the pkz_file (or we can access the pkz_file) by using f instead of pkz_file.\n",
        "\n",
        "df => Dataframe, Basically The pickle(.pkz) file is created using Python pickle and the dump() method and is loaded (=started, activated) using Python pickle \n",
        "and the load() method. we imported(=send) pickle module as pkl in the code. Therefore pkl.dump() is used to create pickle(.pkz) file and pkl.load() is \n",
        "used to load(=start, activate) pickle file. Here, f is the pickle file.\n",
        "\n",
        "return keyword = The return keyword is used to exit (= come out from) a function and return a value. (return df - returns df and exits load_df function. here\n",
        "                 df is the dataframe that contains pickle file data, so return df returns a pickle file) (Dataframe looks like tabular data i.e. data present \n",
        "                 in rows and columns)\n",
        "'''\n",
        "def load_df(pkz_file):  #load data frame from pickle file\n",
        "    with open(pkz_file, 'rb') as f:\n",
        "        df=pkl.load(f)\n",
        "    return df"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jDRSmEMSLUhb"
      },
      "source": [
        "'''\n",
        "Range of values = values present between a lower limit (included) and an upper limit (excluded).\n",
        "\n",
        "with def keyword, created(= defined) a function called df_row_ind_to_data_range() function and passed an argument called ind to the function\n",
        "\n",
        "DATA_POINTS_PER_FILE = 2560, if ind = 0, then the function df_row_ind_to_data_range(ind) returns a range of values (from 0 to 2560). if ind = 1, then\n",
        "the function df_row_ind_to_data_range(ind) returns a range of values (from 2560 to 5120) and so on.\n",
        "'''\n",
        "def df_row_ind_to_data_range(ind):  #get range of values from data frame given file index\n",
        "    return (DATA_POINTS_PER_FILE*ind, DATA_POINTS_PER_FILE*(ind+1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "30XKypeotvrU"
      },
      "source": [
        "'''\n",
        "basically, here we are doing signal processing and data normalization. signal processing and feature extraction can be used interchangeably. signal processing \n",
        "is one of the sub-concepts of Data Preprocessing. (Signal Processing = Signal processing is the analysis, interpretation and manipulation of signals. Processing \n",
        "of signals includes storage, separation of data from noise(= loud and irritating sound) , feature extraction (2D image features extraction or Time-Frequency \n",
        "features extraction).)\n",
        "\n",
        "using def keyword, created(= defined) extract_feature_image() function and passed ind and feature_name as parameters(function variables or arguments) inside\n",
        "the function. horiz accel is assigned to the feature_name parameter\n",
        "\n",
        "Here, data_range is a variable and we assigned the function df_row_index_to_data_range(index) return value to the data_range variable. In other words, we call \n",
        "the function df_row_index_to_data_range(index) and assigned that function’s return value to the data_range variable.\n",
        "\n",
        "data = here, data is a variable and in this variable we stored the values of horizontal and vertical acceleration from 0 (included) to 2560 (excluded) \n",
        "(0 to 2559).\n",
        "\n",
        "data = here, data is a variable. np.array() = https://www.youtube.com/watch?v=NYPKbmE0H6E , np.mean() = https://www.youtube.com/watch?v=hSxslgMFQys ,\n",
        "       range() = https://www.w3schools.com/python/ref_func_range.asp , here, in data variable we stored the average of 1st 20 data points (0 to 20), then \n",
        "       average of 2nd 20 data points (20 to 40), then average of 3rd 20 data points (40 to 60) and so on average of nth 20 data points. since we assigned \n",
        "       WIN_SIZE = 20, we take 20 data points at a time. WIN_SIZE is the fixed size window \n",
        "\n",
        "CWT => https://www.youtube.com/watch?v=F_QvT_8kOfc , https://www.weisang.com/en/documentation/timefreqspectrumalgorithmscwt_en/ \n",
        "(CWT = The Continuous Wavelet Transform (CWT) is used to decompose(= break up, separate) a signal into wavelets. The CWT is used to construct a time-frequency \n",
        "representation of a signal that offers very good time and frequency localization which helps in understanding more insights about time domain and frequency\n",
        "domain. The CWT is an excellent tool for mapping the changing properties of non-stationary signals. The CWT is also an ideal tool for determining whether or not \n",
        "a signal is stationary in a global sense. )\n",
        "(A Wavelet is a wave-like oscillation with an amplitude(= length, size, magnitude, breadth) that begins at zero, increases, and then decreases back to zero. \n",
        "(Oscillation = movement back and forth(=forward) in a regular rhythm))\n",
        "(Different types of CWT Wavelets - https://pywavelets.readthedocs.io/en/latest/ref/cwt.html#:~:text=pywt.,data%20%3A%20array_like ..... there are various types\n",
        "of CWT Wavelets and we are using morlet waveltet among them)\n",
        "(1D data or 1D array = 1D array contains elements only in one dimension. In other words, the shape of the numpy array should contain only one value in the tuple. \n",
        "                       To create a one dimensional array (1D array) in numpy, you can use either of the array(), arange() or linspace() numpy functions.)\n",
        "(Here, coef = variable......  _ = variable (In python, it is conventional (= usual, traditional, normal) to use  underscore ( _ ) as variable name.)\n",
        "(pywt.cwt(data, np.linspace(1,128,128), WAVELET_TYPE) - The above is the syntax to perform continuous wavelet transform (CWT) on 1D data in python. )\n",
        "(np.linspace() = https://www.youtube.com/watch?v=NYPKbmE0H6E )\n",
        "\n",
        "here, coef = variable , np.log2() = https://www.geeksforgeeks.org/numpy-log2-python/ , ** operator = power operator in python (example => 2 ** 5 = 2 power 5 \n",
        "= 32 , np.log2() - This mathematical function helps user to calculate Base-2 logarithm of x where x belongs to all the input array elements. \n",
        "(log2 value = 0.3010)  )\n",
        "\n",
        "Data Normalization => https://towardsdatascience.com/data-normalization-in-machine-learning-395fdec69d02 \n",
        "Normalize or normalization = https://www.educative.io/edpresso/data-normalization-in-python\n",
        "Rescaling = resizing, adjusting (Data Normalization means rescaling all the data of a set in a particular range (say 0 to 1) )\n",
        "(coef - coef.min())/(coef.max() - coef.min()) - this is normalization syntax in python. min() = min() function returns the minimum value in the sequence.\n",
        "                                                max() = max() function returns the maximum value in the sequence\n",
        "\n",
        "return coef - return keyword is used to exit (= come out from) a function and return a value. return coef means returns coef and exits extract_feature_image()\n",
        "              funtion\n",
        "                                          \n",
        "'''\n",
        "#perform continuous wavelet transform (CWT) on 1D signals and return 2D feature image (Extracting 2D image features) (Converting 1D vibration signals into\n",
        "#2D CWT feature images(2D CWT feature images contains both time domain and frequency domain information) 2D CWT feature images are horiz accel feature images \n",
        "#and vert accel feature images )\n",
        "def extract_feature_image(ind, feature_name='horiz accel'):   \n",
        "    data_range = df_row_ind_to_data_range(ind)\n",
        "    data = df[feature_name].values[data_range[0]:data_range[1]]\n",
        "    # use window to process(= prepare, develop) 1D signal\n",
        "    data = np.array([np.mean(data[i:i+WIN_SIZE]) for i in range(0, DATA_POINTS_PER_FILE, WIN_SIZE)])\n",
        "    # perform CWT on 1D data(= 1D array)\n",
        "    coef, _ = pywt.cwt(data, np.linspace(1,128,128), WAVELET_TYPE)\n",
        "    # transform to power and apply logarithm?!\n",
        "    coef = np.log2(coef**2+0.001)\n",
        "    # normalize coef\n",
        "    coef = (coef - coef.min())/(coef.max() - coef.min()) \n",
        "    return coef"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MKKwBH79B1Rj"
      },
      "source": [
        "#Directory or Folder where pickle files(.pkz files) were stored in the Google Drive"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "i7EMtNNh_SLU"
      },
      "source": [
        "'''\n",
        "created main_dir variable, and assigned(= allocate, allot, set) the path where 6 bearing training dataset pickle files were saved in the pc or in the google \n",
        "drive.\n",
        "'''\n",
        "main_dir = '/content/drive/MyDrive/Colab Notebooks/'"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tkk7UDxty6WH"
      },
      "source": [
        "#Bearing1_1"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "KyFEFWSCEptH",
        "outputId": "5a682a62-37bb-46bf-f705-c438dcc6c0e6"
      },
      "source": [
        "'''\n",
        "In pkz_file variable, we stored the bearing1_1.pkz along with the bearing1_1.pkz path (i.e. main_dir). bearing1_1.pkz => is a pickle file.\n",
        "\n",
        "df = load_df(pkz_file) - calling load_df(pkz_file) function which we created(=defined) previously. load_df(pkz_file) loads dataframe from pickle file and \n",
        "                         returns a data frame (here, data frame is a pickle file(.pkz file), dataframe contains pickle file data. Dataframe looks like tabular \n",
        "                         data i.e. data present in rows and columns )\n",
        "                         (pickle file(.pkz file) - pickle file means collection of all files present in a single directory or folder. for example, Bearing1_1\n",
        "                         is a training or learning data directory which contains 2803 data files, bearing1_1.pkz is a training or learning data pickle file \n",
        "                         and bearing1_1.pkz is a single combined file which contains all 2803 data files of Bearing1_1)\n",
        "\n",
        "df.head() => returns the 1st 5 rows of the dataframe (bearing1_1.pkz) i.e. displays the 1st 5 rows of the dataframe (bearing1_1.pkz).\n",
        "\n",
        "'''\n",
        "pkz_file = main_dir + 'bearing1_1.pkz'\n",
        "df = load_df(pkz_file)\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>hour</th>\n",
              "      <th>minute</th>\n",
              "      <th>second</th>\n",
              "      <th>microsecond</th>\n",
              "      <th>horiz accel</th>\n",
              "      <th>vert accel</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>9</td>\n",
              "      <td>39</td>\n",
              "      <td>39</td>\n",
              "      <td>65664.0</td>\n",
              "      <td>0.552</td>\n",
              "      <td>-0.146</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>9</td>\n",
              "      <td>39</td>\n",
              "      <td>39</td>\n",
              "      <td>65703.0</td>\n",
              "      <td>0.501</td>\n",
              "      <td>-0.480</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>9</td>\n",
              "      <td>39</td>\n",
              "      <td>39</td>\n",
              "      <td>65742.0</td>\n",
              "      <td>0.138</td>\n",
              "      <td>0.435</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>39</td>\n",
              "      <td>39</td>\n",
              "      <td>65781.0</td>\n",
              "      <td>-0.423</td>\n",
              "      <td>0.240</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>9</td>\n",
              "      <td>39</td>\n",
              "      <td>39</td>\n",
              "      <td>65820.0</td>\n",
              "      <td>-0.802</td>\n",
              "      <td>0.020</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   hour  minute  second  microsecond  horiz accel  vert accel\n",
              "0     9      39      39      65664.0        0.552      -0.146\n",
              "1     9      39      39      65703.0        0.501      -0.480\n",
              "2     9      39      39      65742.0        0.138       0.435\n",
              "3     9      39      39      65781.0       -0.423       0.240\n",
              "4     9      39      39      65820.0       -0.802       0.020"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_tQFrM__Ous_",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "171fd459-d5d6-49ae-dfd4-2fb9dbb3190b"
      },
      "source": [
        "'''\n",
        "created no_of_rows variable to store the total no. of rows present in a data frame(pickle file) (i.e. to store the no.of rows present in each file of\n",
        "pickle file.)\n",
        "\n",
        "df.shape[0] => return total rows(total data points in bearing1_1) of a data frame (pickle file)\n",
        "\n",
        "created no_of_files variable to store the total no. of files present in single combined pickle file(bearing1_1.pkz file)\n",
        "\n",
        "printing(=displaying) no. of rows(total no. of data points) and no. of files(total no. of files) present in bearing1_1.pkz file.\n",
        "'''\n",
        "no_of_rows = df.shape[0]\n",
        "no_of_files = int(no_of_rows / DATA_POINTS_PER_FILE)\n",
        "print(no_of_rows, no_of_files)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "7175680 2803\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sUDKbO6G9RHH"
      },
      "source": [
        "#extracting 2D feature images for each data file in Bearing1_1 and converting into numpy array\n",
        "\n",
        "#storing the probability of failure"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "I_krMSWG0Gpz",
        "outputId": "6ce1ca69-8c26-4984-ede4-457f7e1aa67c"
      },
      "source": [
        "'''\n",
        "created dictionary named data with keys x and y. x key referred to an empty list. y key also referred to an empty list\n",
        "\n",
        "extracted(= derived, calculated, taken out)horiz accel features from file 0 to file 2802(2803 total files) and stored in coef_h\n",
        "extracted(= derived, calculated, taken out)vert accel features from file 0 to file 2802(2803 total files) and stored in coef_v\n",
        "\n",
        "created x_ variable to store the array(numpy array) of list of horiz accel feature values and vert accel feature values from file 0 to file 2802 (2803 total\n",
        "files)\n",
        "created y_ variable to store the probability(=chance, likelihood, possibility, anticipation, expectation) of failure. Probability as a number lies between\n",
        "(0 and 1). A probability of 0 means that the event will not happen. Here, event is the failure. Hence if y_ variable value is 0, then that means there is\n",
        "no machine failure, machine is working properly, if y_ variable is 1, then that means there is machine failure (complete machine breakdown). Since our data\n",
        "is run-to-failure data, first there will be no failure at all (i.e. 0) and eventually the machine will fail over time (i.e. 1). According to our code, \n",
        "0 => means no failure , 1 => means complete failure, >0.5 => means some failure. y_ variable stores the values like this => 0/2802, 1/2802,\n",
        "2/2802,--------2802/2802\n",
        "\n",
        "using append method added x_ variable values to the data dictionary with x key (i.e. data['x']). In data dictionary, x key represents an empty list\n",
        "therefore values of x_ variable added at the end of x empty list.\n",
        "using append method added y_ variable values to the end of data dictionary with y key (i.e. data['y']). In data dictionary, y key represents an empty list\n",
        "therefore values of y_ variable added at the end of y empty list.\n",
        "\n",
        "after appending, in data['x'] we stored the array of data['x'] values, in data['y'] we stored the array of data['y'] values. Therefore, data['x'] becomes\n",
        "an numpy array and data['y'] becomes an numpy array.\n",
        "array = collection of similar types of values\n",
        "\n",
        "with assert keyword, if the condition returns true, then nothing happens. (or) If the condition returns true, then that condition will be displayed \n",
        "or printed as it is.If the condition returns false, then an AssertionError is raised. The keyword assert functionality is somewhat similar to if condition.\n",
        "shape => https://www.w3schools.com/python/numpy/numpy_array_shape.asp (The shape of an array is the number of elements in each dimension.)\n",
        "\n",
        "Here, no_of_files = 2803 (bearing1_1 contains total 2803 files), 2 = represents no. of channels or no. of filters\n",
        "128, 128 = represents pixels (i.e. 128 x 128 pixels) Pixel => short form for picture element\n",
        "since the above condition is true, we printed no_of_files, shape of data['x'], and shape of data['y']\n",
        "data['x'].shape gives = no. of horiz accel & vert accel feature elements in array, and each feature image size (i,e. no. of pixels(128 x 128)) as an ouptut.\n",
        "data['y'].shape gives = no. of failure probabilities or no. of fault probability values as an output.\n",
        "shape in python displays output in tuple format.\n",
        "The shape of an array is the number of elements in each dimension.\n",
        "data['x'] array contains 4 dimensions, and data['y'] array contains 1 dimension.\n",
        "basically, computer stores an image in 0's and 1's.\n",
        "128 x 128 pixels => represents that each feature image is divided into 128 small parts(i.e. each image is represented as 128 rows and 128 columnns with\n",
        "0's and 1's)\n",
        "\n",
        "'''\n",
        "data = {'x': [], 'y': []}\n",
        "for i in range(0, no_of_files):\n",
        "    coef_h = extract_feature_image(i, feature_name='horiz accel')\n",
        "    coef_v = extract_feature_image(i, feature_name='vert accel')\n",
        "    x_ = np.array([coef_h, coef_v])\n",
        "    y_ = i/(no_of_files-1)\n",
        "    data['x'].append(x_)\n",
        "    data['y'].append(y_)\n",
        "data['x']=np.array(data['x'])\n",
        "data['y']=np.array(data['y'])\n",
        "\n",
        "assert data['x'].shape==(no_of_files, 2, 128, 128)\n",
        "print(no_of_files, data['x'].shape, data['y'].shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2803 (2803, 2, 128, 128) (2803,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3aM8UQ79LSBD"
      },
      "source": [
        "#Saving as pickle files (.pkz files)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X8QRKXNi-ov1"
      },
      "source": [
        "'''\n",
        "bearing1_1_all_data.pkz - means bearing1_1_train_data.pkz + bearing1_1_val_data.pkz\n",
        "\n",
        "The pickle(.pkz) file is created using Python pickle and the dump() method and is loaded (=started, activated) using Python pickle and the load() method.\n",
        "we imported(=send) pickle module as pkl in the code. Therefore pkl.dump() is used to create pickle(.pkz) file and pkl.load() is used to load(=start, activate) \n",
        "pickle file.\n",
        "pickle.dump() function => is used to store the object data(dump information) to the file. pickle.dump() function takes 3 arguments. The first argument is the\n",
        "object that you want to store. The second argument is the file object you get by opening the desired file in write-binary (wb) mode. third argument is \n",
        "optional.\n",
        "\n",
        "created out_file variable to store all the 2D CWT feature data extracted from bearing1_1.pkz file. file name => bearing1_1_all_data.pkz\n",
        "main_dir variable => path where 6 bearing training dataset pickle files were saved in the pc or in the google drive. now, bearing1_1_all_data.pkz also saved\n",
        "                     in the same path\n",
        "\n",
        "with keyword => automatically releases memory after allocation. Whenever we open the file with open() function, it allocates some resources and memory to the \n",
        "file. And we should use close() function to release or delete that memory from the file otherwise errors will come. Sometimes we forget to close() the file\n",
        "and we couldn’t find that we didn’t closed the file, so even the whole code is correct we might get errors and we may not be able to correct it. So it’s better\n",
        "to use “with” keyword along with open() function as “with” keyword automatically releases or deletes memory after process completion.\n",
        "\n",
        "The open() function opens a file in text format by default. To open a file in binary format, add 'b' to the mode parameter. Hence the \"wb\" mode opens the file \n",
        "in binary format for writing. Unlike text files, binary files are not human-readable.\n",
        "\n",
        "as = The as keyword is used to create an alias (= aka, also known as, also called). In the below code, we create an alias f when opening the pkz_file, and \n",
        "now we can refer to the pkz_file (or we can access the pkz_file) by using f instead of pkz_file.\n",
        "\n",
        "pkl.dump() creates file object named \"f\" and dumps(=stores) the data into \"f\" and saves \"f\" as bearing1_1_all_data.pkz in selected path (google drive)\n",
        "'''\n",
        "out_file = main_dir+'bearing1_1_all_data.pkz'\n",
        "with open(out_file, 'wb') as f:\n",
        "    pkl.dump(data, f)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KP4InmEZQ6UF"
      },
      "source": [
        "#Bearing1_2\n",
        "\n",
        "implementation is same as Bearing1_1\n",
        "\n",
        "(1) Extracting 2D feature images (both horiz accel feature images and vert accel feature images) from bearing1_2.pkz and converting into numpy array\n",
        "\n",
        "(2) storing the count(total number) of failure probability values or fault probability values in numpy array\n",
        "\n",
        "(3) Saving as pickle files (.pkz files)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "tg25lR0zJw5m",
        "outputId": "71d67905-c5d5-4bd2-f283-7bdb912ea1fa"
      },
      "source": [
        "pkz_file = main_dir + 'bearing1_2.pkz'\n",
        "df = load_df(pkz_file)\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>hour</th>\n",
              "      <th>minute</th>\n",
              "      <th>second</th>\n",
              "      <th>microsecond</th>\n",
              "      <th>horiz accel</th>\n",
              "      <th>vert accel</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>8</td>\n",
              "      <td>47</td>\n",
              "      <td>5</td>\n",
              "      <td>196910.0</td>\n",
              "      <td>0.050</td>\n",
              "      <td>-0.253</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8</td>\n",
              "      <td>47</td>\n",
              "      <td>5</td>\n",
              "      <td>196950.0</td>\n",
              "      <td>0.165</td>\n",
              "      <td>-0.140</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>47</td>\n",
              "      <td>5</td>\n",
              "      <td>196990.0</td>\n",
              "      <td>0.125</td>\n",
              "      <td>0.542</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8</td>\n",
              "      <td>47</td>\n",
              "      <td>5</td>\n",
              "      <td>197030.0</td>\n",
              "      <td>0.157</td>\n",
              "      <td>-0.261</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>8</td>\n",
              "      <td>47</td>\n",
              "      <td>5</td>\n",
              "      <td>197070.0</td>\n",
              "      <td>0.421</td>\n",
              "      <td>0.081</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   hour  minute  second  microsecond  horiz accel  vert accel\n",
              "0     8      47       5     196910.0        0.050      -0.253\n",
              "1     8      47       5     196950.0        0.165      -0.140\n",
              "2     8      47       5     196990.0        0.125       0.542\n",
              "3     8      47       5     197030.0        0.157      -0.261\n",
              "4     8      47       5     197070.0        0.421       0.081"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "L7QB5sRnZsLj",
        "outputId": "5d6f2014-aeb0-42e7-d915-02734570b4e6"
      },
      "source": [
        "no_of_rows = df.shape[0]\n",
        "no_of_files = int(no_of_rows / DATA_POINTS_PER_FILE)\n",
        "print(no_of_rows, no_of_files)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2229760 871\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Qo9Ap6q_Z7Fk",
        "outputId": "23c24e01-fd80-4cb6-b3b6-24f41e022b0d"
      },
      "source": [
        "data = {'x': [], 'y': []}\n",
        "for i in range(0, no_of_files):\n",
        "    coef_h = extract_feature_image(i, feature_name='horiz accel')\n",
        "    coef_v = extract_feature_image(i, feature_name='vert accel')\n",
        "    x_ = np.array([coef_h, coef_v])\n",
        "    y_ = i/(no_of_files-1)\n",
        "    data['x'].append(x_)\n",
        "    data['y'].append(y_)\n",
        "data['x']=np.array(data['x'])\n",
        "data['y']=np.array(data['y'])\n",
        "\n",
        "assert data['x'].shape==(no_of_files, 2, 128, 128)\n",
        "print(no_of_files, data['x'].shape, data['y'].shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "871 (871, 2, 128, 128) (871,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EpS-wgb4ab0f"
      },
      "source": [
        "out_file = main_dir+'bearing1_2_all_data.pkz'\n",
        "with open(out_file, 'wb') as f:\n",
        "    pkl.dump(data, f)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XBqdkKx8bIuj"
      },
      "source": [
        "#Bearing2_1\n",
        "\n",
        "implementation is same as Bearing1_1, Bearing1_2\n",
        "\n",
        "(1) Extracting 2D feature images (both horiz accel feature images and vert accel feature images) from bearing2_1.pkz and converting into numpy array\n",
        "\n",
        "(2) storing the count(total number) of failure probability values or fault probability values in numpy array\n",
        "\n",
        "(3) Saving as pickle files (.pkz files)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SNv33Ettao8J",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "c3fe3469-26dd-4d47-a9fa-422af7790fa2"
      },
      "source": [
        "pkz_file = main_dir + 'bearing2_1.pkz'\n",
        "df = load_df(pkz_file)\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>hour</th>\n",
              "      <th>minute</th>\n",
              "      <th>second</th>\n",
              "      <th>microsecond</th>\n",
              "      <th>horiz accel</th>\n",
              "      <th>vert accel</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>8</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "      <td>884410.0</td>\n",
              "      <td>-0.391</td>\n",
              "      <td>0.011</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "      <td>884450.0</td>\n",
              "      <td>0.292</td>\n",
              "      <td>0.133</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "      <td>884490.0</td>\n",
              "      <td>0.596</td>\n",
              "      <td>0.024</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "      <td>884530.0</td>\n",
              "      <td>0.230</td>\n",
              "      <td>0.272</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>8</td>\n",
              "      <td>14</td>\n",
              "      <td>15</td>\n",
              "      <td>884570.0</td>\n",
              "      <td>-0.225</td>\n",
              "      <td>0.272</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   hour  minute  second  microsecond  horiz accel  vert accel\n",
              "0     8      14      15     884410.0       -0.391       0.011\n",
              "1     8      14      15     884450.0        0.292       0.133\n",
              "2     8      14      15     884490.0        0.596       0.024\n",
              "3     8      14      15     884530.0        0.230       0.272\n",
              "4     8      14      15     884570.0       -0.225       0.272"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qofa7FLlZVfL",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "7689e927-2013-4e17-90e2-3c4e82062062"
      },
      "source": [
        "no_of_rows = df.shape[0]\n",
        "no_of_files = int(no_of_rows / DATA_POINTS_PER_FILE)\n",
        "print(no_of_rows, no_of_files)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2332160 911\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rJPWZ0GOacnQ",
        "outputId": "065dfce6-dbc4-45ef-90da-5f7614b44cc2"
      },
      "source": [
        "data = {'x': [], 'y': []}\n",
        "for i in range(0, no_of_files):\n",
        "    coef_h = extract_feature_image(i, feature_name='horiz accel')\n",
        "    coef_v = extract_feature_image(i, feature_name='vert accel')\n",
        "    x_ = np.array([coef_h, coef_v])\n",
        "    y_ = i/(no_of_files-1)\n",
        "    data['x'].append(x_)\n",
        "    data['y'].append(y_)\n",
        "data['x']=np.array(data['x'])\n",
        "data['y']=np.array(data['y'])\n",
        "\n",
        "assert data['x'].shape==(no_of_files, 2, 128, 128)\n",
        "print(no_of_files, data['x'].shape, data['y'].shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "911 (911, 2, 128, 128) (911,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5BcYWNHDajEc"
      },
      "source": [
        "out_file = main_dir+'bearing2_1_all_data.pkz'\n",
        "with open(out_file, 'wb') as f:\n",
        "    pkl.dump(data, f)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0nDQPVOdbGar"
      },
      "source": [
        "#Bearing2_2\n",
        "\n",
        "implementation is same as Bearing1_1, Bearing1_2, Bearing2_1\n",
        "\n",
        "(1) Extracting 2D feature images (both horiz accel feature images and vert accel feature images) from bearing2_2.pkz and converting into numpy array\n",
        "\n",
        "(2) storing the count(total number) of failure probability values or fault probability values in numpy array\n",
        "\n",
        "(3) Saving as pickle files (.pkz files)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "Lkm7iBgobGCk",
        "outputId": "97c4c993-973e-471b-acf5-b90a76d6cfb6"
      },
      "source": [
        "pkz_file = main_dir + 'bearing2_2.pkz'\n",
        "df = load_df(pkz_file)\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>hour</th>\n",
              "      <th>minute</th>\n",
              "      <th>second</th>\n",
              "      <th>microsecond</th>\n",
              "      <th>horiz accel</th>\n",
              "      <th>vert accel</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>7</td>\n",
              "      <td>40</td>\n",
              "      <td>33</td>\n",
              "      <td>540660.0</td>\n",
              "      <td>0.038</td>\n",
              "      <td>0.290</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>7</td>\n",
              "      <td>40</td>\n",
              "      <td>33</td>\n",
              "      <td>540700.0</td>\n",
              "      <td>0.125</td>\n",
              "      <td>-0.104</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>7</td>\n",
              "      <td>40</td>\n",
              "      <td>33</td>\n",
              "      <td>540740.0</td>\n",
              "      <td>0.035</td>\n",
              "      <td>-0.314</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>7</td>\n",
              "      <td>40</td>\n",
              "      <td>33</td>\n",
              "      <td>540780.0</td>\n",
              "      <td>-0.092</td>\n",
              "      <td>0.200</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>7</td>\n",
              "      <td>40</td>\n",
              "      <td>33</td>\n",
              "      <td>540820.0</td>\n",
              "      <td>0.033</td>\n",
              "      <td>0.211</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   hour  minute  second  microsecond  horiz accel  vert accel\n",
              "0     7      40      33     540660.0        0.038       0.290\n",
              "1     7      40      33     540700.0        0.125      -0.104\n",
              "2     7      40      33     540740.0        0.035      -0.314\n",
              "3     7      40      33     540780.0       -0.092       0.200\n",
              "4     7      40      33     540820.0        0.033       0.211"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "r6nwzbIEdA8z",
        "outputId": "54641586-a064-4cbe-e733-039de8dfadcb"
      },
      "source": [
        "no_of_rows = df.shape[0]\n",
        "no_of_files = int(no_of_rows / DATA_POINTS_PER_FILE)\n",
        "print(no_of_rows, no_of_files)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2040320 797\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9wsGILXVdKI_",
        "outputId": "c0dd7a01-f740-4f29-cd95-3a93fb4545dc"
      },
      "source": [
        "data = {'x': [], 'y': []}\n",
        "for i in range(0, no_of_files):\n",
        "    coef_h = extract_feature_image(i, feature_name='horiz accel')\n",
        "    coef_v = extract_feature_image(i, feature_name='vert accel')\n",
        "    x_ = np.array([coef_h, coef_v])\n",
        "    y_ = i/(no_of_files-1)\n",
        "    data['x'].append(x_)\n",
        "    data['y'].append(y_)\n",
        "data['x']=np.array(data['x'])\n",
        "data['y']=np.array(data['y'])\n",
        "\n",
        "assert data['x'].shape==(no_of_files, 2, 128, 128)\n",
        "print(no_of_files, data['x'].shape, data['y'].shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "797 (797, 2, 128, 128) (797,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HSzsehnUdqf5"
      },
      "source": [
        "out_file = main_dir+'bearing2_2_all_data.pkz'\n",
        "with open(out_file, 'wb') as f:\n",
        "    pkl.dump(data, f)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FisP4bked7zj"
      },
      "source": [
        "#Bearing3_1\n",
        "\n",
        "implementation is same as Bearing1_1, Bearing1_2, Bearing2_1, Bearing2_2\n",
        "\n",
        "(1) Extracting 2D feature images (both horiz accel feature images and vert accel feature images) from bearing3_1.pkz and converting into numpy array\n",
        "\n",
        "(2) storing the count(total number) of failure probability values or fault probability values in numpy array\n",
        "\n",
        "(3) Saving as pickle files (.pkz files)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "CyZUxLSJd1K8",
        "outputId": "5c6fe493-0719-4013-e9ba-847aca6f1e6c"
      },
      "source": [
        "pkz_file = main_dir + 'bearing3_1.pkz'\n",
        "df = load_df(pkz_file)\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>hour</th>\n",
              "      <th>minute</th>\n",
              "      <th>second</th>\n",
              "      <th>microsecond</th>\n",
              "      <th>horiz accel</th>\n",
              "      <th>vert accel</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>9</td>\n",
              "      <td>10</td>\n",
              "      <td>39</td>\n",
              "      <td>118790.0</td>\n",
              "      <td>0.338</td>\n",
              "      <td>-0.263</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>9</td>\n",
              "      <td>10</td>\n",
              "      <td>39</td>\n",
              "      <td>118830.0</td>\n",
              "      <td>0.278</td>\n",
              "      <td>0.285</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>9</td>\n",
              "      <td>10</td>\n",
              "      <td>39</td>\n",
              "      <td>118870.0</td>\n",
              "      <td>0.143</td>\n",
              "      <td>0.590</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>9</td>\n",
              "      <td>10</td>\n",
              "      <td>39</td>\n",
              "      <td>118910.0</td>\n",
              "      <td>0.090</td>\n",
              "      <td>-0.193</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>9</td>\n",
              "      <td>10</td>\n",
              "      <td>39</td>\n",
              "      <td>118940.0</td>\n",
              "      <td>0.035</td>\n",
              "      <td>-0.109</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   hour  minute  second  microsecond  horiz accel  vert accel\n",
              "0     9      10      39     118790.0        0.338      -0.263\n",
              "1     9      10      39     118830.0        0.278       0.285\n",
              "2     9      10      39     118870.0        0.143       0.590\n",
              "3     9      10      39     118910.0        0.090      -0.193\n",
              "4     9      10      39     118940.0        0.035      -0.109"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DlVrscrQfcUQ",
        "outputId": "f2079e50-3983-40fe-d076-717956d86cc2"
      },
      "source": [
        "no_of_rows = df.shape[0]\n",
        "no_of_files = int(no_of_rows / DATA_POINTS_PER_FILE)\n",
        "print(no_of_rows, no_of_files)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1318400 515\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "D1vxFppiftsG",
        "outputId": "25e07fb3-4bb0-4baa-faf4-f369bdfab549"
      },
      "source": [
        "data = {'x': [], 'y': []}\n",
        "for i in range(0, no_of_files):\n",
        "    coef_h = extract_feature_image(i, feature_name='horiz accel')\n",
        "    coef_v = extract_feature_image(i, feature_name='vert accel')\n",
        "    x_ = np.array([coef_h, coef_v])\n",
        "    y_ = i/(no_of_files-1)\n",
        "    data['x'].append(x_)\n",
        "    data['y'].append(y_)\n",
        "data['x']=np.array(data['x'])\n",
        "data['y']=np.array(data['y'])\n",
        "\n",
        "assert data['x'].shape==(no_of_files, 2, 128, 128)\n",
        "print(no_of_files, data['x'].shape, data['y'].shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "515 (515, 2, 128, 128) (515,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KTXWi_T-gEyT"
      },
      "source": [
        "out_file = main_dir+'bearing3_1_all_data.pkz'\n",
        "with open(out_file, 'wb') as f:\n",
        "    pkl.dump(data, f)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "MkQH2H1-gW39"
      },
      "source": [
        "#Bearing3_2\n",
        "\n",
        "implementation is same as Bearing1_1, Bearing1_2, Bearing2_1, Bearing2_2, Bearing3_1\n",
        "\n",
        "(1) Extracting 2D feature images (both horiz accel feature images and vert accel feature images) from bearing3_2.pkz and converting into numpy array\n",
        "\n",
        "(2) storing the count(total number) of failure probability values or fault probability values in numpy array\n",
        "\n",
        "(3) Saving as pickle files (.pkz files)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "id": "FEIPB_hggVdg",
        "outputId": "79a0ac7e-79be-4e20-8b8b-b1e36584fb97"
      },
      "source": [
        "pkz_file = main_dir + 'bearing3_2.pkz'\n",
        "df = load_df(pkz_file)\n",
        "df.head()"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>hour</th>\n",
              "      <th>minute</th>\n",
              "      <th>second</th>\n",
              "      <th>microsecond</th>\n",
              "      <th>horiz accel</th>\n",
              "      <th>vert accel</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>8</td>\n",
              "      <td>34</td>\n",
              "      <td>41</td>\n",
              "      <td>978160.0</td>\n",
              "      <td>-0.291</td>\n",
              "      <td>0.181</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>8</td>\n",
              "      <td>34</td>\n",
              "      <td>41</td>\n",
              "      <td>978200.0</td>\n",
              "      <td>0.146</td>\n",
              "      <td>0.185</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>8</td>\n",
              "      <td>34</td>\n",
              "      <td>41</td>\n",
              "      <td>978240.0</td>\n",
              "      <td>0.404</td>\n",
              "      <td>-0.159</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>8</td>\n",
              "      <td>34</td>\n",
              "      <td>41</td>\n",
              "      <td>978280.0</td>\n",
              "      <td>0.191</td>\n",
              "      <td>-0.179</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>8</td>\n",
              "      <td>34</td>\n",
              "      <td>41</td>\n",
              "      <td>978320.0</td>\n",
              "      <td>-0.180</td>\n",
              "      <td>0.072</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   hour  minute  second  microsecond  horiz accel  vert accel\n",
              "0     8      34      41     978160.0       -0.291       0.181\n",
              "1     8      34      41     978200.0        0.146       0.185\n",
              "2     8      34      41     978240.0        0.404      -0.159\n",
              "3     8      34      41     978280.0        0.191      -0.179\n",
              "4     8      34      41     978320.0       -0.180       0.072"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 31
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EBfsIE1RhzIF",
        "outputId": "af46e5d5-6597-40a9-f16e-32cad3c2e95e"
      },
      "source": [
        "no_of_rows = df.shape[0]\n",
        "no_of_files = int(no_of_rows / DATA_POINTS_PER_FILE)\n",
        "print(no_of_rows, no_of_files)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "4190720 1637\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "wYoooLK8iA7A",
        "outputId": "5e67b704-2eb7-488d-a2a3-8f2edd398476"
      },
      "source": [
        "data = {'x': [], 'y': []}\n",
        "for i in range(0, no_of_files):\n",
        "    coef_h = extract_feature_image(i, feature_name='horiz accel')\n",
        "    coef_v = extract_feature_image(i, feature_name='vert accel')\n",
        "    x_ = np.array([coef_h, coef_v])\n",
        "    y_ = i/(no_of_files-1)\n",
        "    data['x'].append(x_)\n",
        "    data['y'].append(y_)\n",
        "data['x']=np.array(data['x'])\n",
        "data['y']=np.array(data['y'])\n",
        "\n",
        "assert data['x'].shape==(no_of_files, 2, 128, 128)\n",
        "print(no_of_files, data['x'].shape, data['y'].shape)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "1637 (1637, 2, 128, 128) (1637,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "a5k4HmSOijrk"
      },
      "source": [
        "out_file = main_dir+'bearing3_2_all_data.pkz'\n",
        "with open(out_file, 'wb') as f:\n",
        "    pkl.dump(data, f)"
      ],
      "execution_count": null,
      "outputs": []
    }
  ]
}