duplicate_remover.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dbd91f37",
   "metadata": {},
   "source": [
    "Adam Peetz, Regis Jesuit University 2023\n",
    "\n",
    "# Removing Duplicate Images \n",
    "\n",
    " \n",
    " \n",
    "\n",
    "Reddit users often repost the same content to a subreddit to gain karma. Karma is Reddit's running tally of how many likes a user's content has received. This results in duplicate images occurring in content scraped from the website. <br>  \n",
    "\n",
    "Leaving duplicate images in the dataset would cause a model to increase the importance of features displayed in those images. It may also result in data leakage if those images happened to be split between the training and test sets. To prevent this from happening, duplicate images will be removed using the difPy library. <br> \n",
    "\n",
    " \n",
    " \n",
    "\n",
    "DifPy is a library created by Elise Landman (2022). DifPy compares the tensors in an image and removes images that have similar tensors from the dataset. Running DifPy on the images sourced from Reddit effectively removes duplicate tensors from the dataset.   \n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "36b0e2a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DifPy preparing files: [838/838] [100%]\n",
      "DifPy comparing images: [838/838] [100%]\n",
      "Found 0 images with one or more duplicate/similar images in 126.0877 seconds.\n"
     ]
    }
   ],
   "source": [
    "# duplicate image remover\n",
    "\n",
    "# import library\n",
    "from difPy import dif\n",
    "\n",
    "# define image path for duplicate search\n",
    "image_path = \"C:\\\\Users\\\\adamg\\\\Documents\\\\MSDS_692\\\\Week_2\\\\scraping_reddit\\\\datasets\\\\0\\\\\"\n",
    "\n",
    "# execute image duplicate search using difpy\n",
    "search = dif(image_path, delete=True)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "74a2b5f7",
   "metadata": {},
   "source": [
    "# References\n",
    "\n",
    "Landman, Elise. (2022). Duplicate Image Finder. retreived 01/20/2023 from https://github.com/elisemercury/Duplicate-Image-Finder/wiki/difPy-Usage-Documentation"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}