Fusión_de_Datos_de_Panóptico_de_Twitch_0_1_0.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "#Fusión de Bases de Datos del Panóptico de Twitch v0.1.0"
      ],
      "metadata": {
        "id": "-dttYO5V9jBk"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "La siguiente aplicación tiene como objetivo el recopilar las bases de datos generadas durante la primera fase experimental y construir una única base de datos interpretable utilizando cualquier aplicación de análisis estadístico.\n",
        "\n",
        "Si bien el diseño de esta aplicación impide su uso sobre bases de datos no estructurados bajo el formato propuesto en el Panóptico de Twitch, es posible realizar modificaciones a la estructura del texto para poder aplicar el modelo a cualquier formato de base de datos basada en CSV.\n",
        "\n",
        "Panóptico de Twitch v0.1.0: https://drive.google.com/file/d/1XiJ9mDe6EcDp5lYHilbDIABkYlHJPD-A/view?usp=sharing"
      ],
      "metadata": {
        "id": "Oj5wedXJ9sQq"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "T-6u1bjc9W2U"
      },
      "outputs": [],
      "source": [
        "import socket\n",
        "import time\n",
        "import re\n",
        "import glob\n",
        "import pandas as pd\n",
        "import os.path\n",
        "from os import path\n",
        "\n",
        "from google.colab import drive\n",
        "drive.mount('/content/drive')\n",
        "\n",
        "output_dir = \"/content/drive/MyDrive/Panoptico_Twitch/\"\n",
        "df_concat = pd.DataFrame()\n",
        "exc = 0\n",
        "all_filename_count = 0\n",
        "extension = 'csv'\n",
        "output_dir_var = \"\"\n",
        "gender = 0"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Selección de Streamers**\n",
        "\n",
        "Selecciona el nombre de los streamers a fusionar en la base de datos. Genera la lista \"streamers\" que se utilizará en el futuro para distintas funciones (selección de carpeta, renombrar, generizar, etc).\n",
        "\n"
      ],
      "metadata": {
        "id": "b4GsGtiT9hcd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "streamers = \"auronplay\", \"iamcristinini\", \"ibai\", \"illojuan\", \"rivers_gg\", \"staryuuki\""
      ],
      "metadata": {
        "id": "O5o3h6Dd_Gjf"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Primera fusión: Bases de Datos de cada Streamer**\n",
        "\n",
        "Este algoritmo se compone de una estructura de tres bucles:\n",
        "  1. Selección de streamer\n",
        "  2. Selección de día\n",
        "  3. Carga en la memoria y concatenación de los datos en .CSV\n",
        "\n",
        "En definitiva, la aplicación generará un archivo \"Fusión_[Nombre].csv\" por cada streamer explicitado en la lista \"streamers\"."
      ],
      "metadata": {
        "id": "wDjh4lefAsV_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for streamrs in streamers:\n",
        "  os.chdir(output_dir + streamrs)\n",
        "\n",
        "  list_days = os.listdir()\n",
        "  print(list_days)\n",
        "\n",
        "  for j in list_days:\n",
        "    j = str(j)\n",
        "    print(output_dir, streamrs,\"/\",j,sep='')\n",
        "    output_dir_var = output_dir + streamrs + \"/\" + j\n",
        "\n",
        "    os.chdir(output_dir_var)\n",
        "\n",
        "    all_filenames = [z for z in glob.glob('*.{}'.format(extension))]\n",
        "    \n",
        "    for i in range(0,len(all_filenames)):\n",
        "      if os.path.isfile(all_filenames[i]) and os.path.getsize(all_filenames[i]) > 1:\n",
        "        df_concat = pd.concat([pd.read_csv(all_filenames[i], low_memory=False, on_bad_lines='skip'), df_concat], ignore_index=True)\n",
        "        exc = exc + 1\n",
        "        all_filename_count = all_filename_count + len(all_filenames)\n",
        "\n",
        "    print(df_concat)\n",
        "    print(exc, \"/\", len(all_filenames))\n",
        "    all_filename_count = 0\n",
        "    exc = 0\n",
        "\n",
        "  output_dir = \"/content/drive/MyDrive/Panoptico_Twitch/\"\n",
        "  os.chdir(output_dir)\n",
        "  df_concat.to_csv(\"Fusion_\" + streamrs + \".csv\", index=False, encoding='utf-8-sig')\n",
        "  df_concat = pd.DataFrame()"
      ],
      "metadata": {
        "id": "fXMC70W8AqnB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Generización y Nombre**\n",
        "\n",
        "Debido a que en la tabulación de los datos en el Panóptico de Twitch no se establecen datos referidos al streamer, ésta función persigue establecer datos relevantes indicados en la investigación en la tabulación: para permitir la comparación entre streamers, en primer lugar, adjuntamos el nombre del canal dodne se escribió cada mensaje; y el género de streamer, codificado en éste algoritmo\n",
        "\n"
      ],
      "metadata": {
        "id": "gxSqw8aOMy1C"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "output_dir = \"/content/drive/MyDrive/Panoptico_Twitch/\"\n",
        "os.chdir(output_dir)\n",
        "\n",
        "for streamrs in streamers:\n",
        "\n",
        "  if streamrs in (\"illojuan\", \"ibai\", \"auronplay\"):\n",
        "    gender = \"male\" \n",
        "  if streamrs in (\"iamcristinini\", \"rivers_gg\", \"staryuuki\"):\n",
        "    gender = \"female\"\n",
        "\n",
        "  file = \"Fusion_\" + streamrs + \".csv\"\n",
        "\n",
        "  df = pd.DataFrame()\n",
        "\n",
        "  df = pd.read_csv(file, low_memory=False)\n",
        "\n",
        "  df[\"streamer\"] = streamrs\n",
        "  df[\"gender\"] = gender\n",
        "\n",
        "  print(df)\n",
        "\n",
        "  df.to_csv(\"Fusion_\" + streamrs + \"_final.csv\", index=False, encoding='utf-8-sig')\n"
      ],
      "metadata": {
        "id": "YdNru8GhMzQz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Segunda fusión: Generación de la Base de Datos final**\n",
        "\n",
        "Esta fusión se compone, sencillamente, de la carga de las bases de datos previamente generadas y la creación de un nuevo archivo \"Base_de_datos.csv\"."
      ],
      "metadata": {
        "id": "67miTo5QRFIA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "output_dir = \"/content/drive/MyDrive/Panoptico_Twitch/\"\n",
        "os.chdir(output_dir)\n",
        "df = pd.DataFrame()\n",
        "\n",
        "for streamrs in streamers:\n",
        "  file = \"Fusion_\" + streamrs + \"_final.csv\"\n",
        "\n",
        "  df = pd.concat([pd.read_csv(file, low_memory=False, on_bad_lines='skip'), df], ignore_index=True)\n",
        "  print(df)\n",
        "\n",
        "df.to_csv(\"Base_de_datos.csv\", index=False, encoding='utf-8-sig')\n"
      ],
      "metadata": {
        "id": "ZVFo_fmpRFjO"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "**Limpieza de la Base de Datos**\n",
        "\n"
      ],
      "metadata": {
        "id": "2nQ50ARCTydO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "with open('Base_de_datos.csv', 'r') as in_file, open('Base_de_datos_limpia.csv', 'w') as out_file:\n",
        "    seen = set() # set for fast O(1) amortized lookup\n",
        "    for line in in_file:\n",
        "        if line in seen: continue # skip duplicate\n",
        "\n",
        "        seen.add(line)\n",
        "        out_file.write(line)"
      ],
      "metadata": {
        "id": "lQBBVrJpTy2w"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Función de limpieza de Base de Datos."
      ],
      "metadata": {
        "id": "HdC7SAmpUm6-"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "all_filenames = [z for z in glob.glob('*.{}'.format(extension))]\n",
        "\n",
        "for i in all_filenames:\n",
        "  if i != \"Base_de_datos_limpia.csv\":\n",
        "    print(\"Eliminando \", i, \"...\")\n",
        "    os.remove(i)"
      ],
      "metadata": {
        "id": "pSVv55_DUmDY"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}