Recommendations_with_IBM-zh.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# IBM 推荐系统\n",
    "\n",
    "在此 notebook 中，你将运用所学的推荐技能处理 IBM Watson Studio 平台的真实数据。 \n",
    "\n",
    "\n",
    "你可以通过此 workspace 提交 notebook，或者在本地机器上操作并在下个页面提交 notebook。无论是哪种方式，都请确保代码符合项目[审阅标准](https://review.udacity.com/#!/rubrics/2632/view)。**请定期保存代码。**\n",
    "\n",
    "跟着目录操作，你将能够创建多个不同的推荐方法，这些方法可以用于不同的情形。 \n",
    "\n",
    "\n",
    "## 目录\n",
    "\n",
    "I. [探索性数据分析](#Exploratory-Data-Analysis)<br>\n",
    "II.[基于排名的推荐方法](#Rank)<br>\n",
    "III.[基于用户-用户的协同过滤](#User-User)<br>\n",
    "IV.[基于内容的推荐方法（选修内容）](#Content-Recs)<br>\n",
    "V. [矩阵分解](#Matrix-Fact)<br>\n",
    "VI.[其他内容和总结](#conclusions)\n",
    "\n",
    "你可以在 notebook 的结尾处找到提交 notebook 的指南。首先导入必要的库并读取数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import project_tests as t\n",
    "import pickle\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "df = pd.read_csv('data/user-item-interactions.csv')\n",
    "df_content = pd.read_csv('data/articles_community.csv')\n",
    "del df['Unnamed: 0']\n",
    "del df_content['Unnamed: 0']\n",
    "\n",
    "# Show df to get an idea of the data\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show df_content to get an idea of the data\n",
    "df_content.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a class=\"anchor\" id=\"Exploratory-Data-Analysis\">第一部分：探索性数据分析</a>\n",
    "\n",
    "通过以下字典和单元格了解数据的描述性统计信息。\n",
    "\n",
    "`1.` 用户与数据集中的多少篇文章互动了？分布如何？以图表的形式描述每个用户与某篇文章互动的次数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill in the median and maximum number of user_article interactios below\n",
    "\n",
    "median_val = # 50% of individuals interact with ____ number of articles or fewer.\n",
    "max_views_by_user = # The maximum number of user-article interactions by any 1 user is ______."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` 发现并移除 **df_content** dataframe 中的重复文章。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Find and explore duplicate articles"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove any rows that have the same article_id - only keep the first"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` 在以下单元格中查找：\n",
    "\n",
    "**a.**用户与之互动的唯一文章数量。  \n",
    "**b.**数据集中的唯一文章数量（无论用户是否与之互动了）。<br>\n",
    "**c.**数据集中的唯一用户数量。（不包括空值）<br>\n",
    "**d.**数据集中的用户-文章互动次数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_articles = # The number of unique articles that have at least one interaction\n",
    "total_articles = # The number of unique articles on the IBM platform\n",
    "unique_users = # The number of unique users\n",
    "user_article_interactions = # The number of user-article interactions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` 在以下单元格中查找查看次数最多的文章的 **article_id**，以及被查看频率。在与公司领导讨论后，`email_mapper` 函数被视为将用户映射到 ID 的合理方式。有少数几个空值，并且所有这些空值都可能属于一个用户（我们以这种方式使用以下函数存储了这些值）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "most_viewed_article_id = # The most viewed article in the dataset as a string with one value following the decimal \n",
    "max_views = # The most viewed article in the dataset was viewed how many times?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## No need to change the code here - this will be helpful for later parts of the notebook\n",
    "# Run this cell to map the user email to a user_id column and remove the email column\n",
    "\n",
    "def email_mapper():\n",
    "    coded_dict = dict()\n",
    "    cter = 1\n",
    "    email_encoded = []\n",
    "    \n",
    "    for val in df['email']:\n",
    "        if val not in coded_dict:\n",
    "            coded_dict[val] = cter\n",
    "            cter+=1\n",
    "        \n",
    "        email_encoded.append(coded_dict[val])\n",
    "    return email_encoded\n",
    "\n",
    "email_encoded = email_mapper()\n",
    "del df['email']\n",
    "df['user_id'] = email_encoded\n",
    "\n",
    "# show header\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## If you stored all your results in the variable names above, \n",
    "## you shouldn't need to change anything in this cell\n",
    "\n",
    "sol_1_dict = {\n",
    "    '`50% of individuals have _____ or fewer interactions.`': median_val,\n",
    "    '`The total number of user-article interactions in the dataset is ______.`': user_article_interactions,\n",
    "    '`The maximum number of user-article interactions by any 1 user is ______.`': max_views_by_user,\n",
    "    '`The most viewed article in the dataset was viewed _____ times.`': max_views,\n",
    "    '`The article_id of the most viewed article is ______.`': most_viewed_article_id,\n",
    "    '`The number of unique articles that have at least 1 rating ______.`': unique_articles,\n",
    "    '`The number of unique users in the dataset is ______`': unique_users,\n",
    "    '`The number of unique articles on the IBM platform`': total_articles\n",
    "}\n",
    "\n",
    "# Test your dictionary against the solution\n",
    "t.sol_1_test(sol_1_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a class=\"anchor\" id=\"Rank\">第二部分：基于排名的推荐方法</a>\n",
    "\n",
    "与之前的课程不同，我们没有关于用户是否喜欢某篇文章的评分。我们只知道用户与文章互动了。在这些情形下，文章的热门程度只能通过用户与文章的互动频率来判断。\n",
    "\n",
    "`1.` 填写以下函数，使其返回前 **n** 篇文章，按照互动次数从高到低排序。使用以下测试测试你的函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_top_articles(n, df=df):\n",
    "    '''\n",
    "    INPUT:\n",
    "    n - (int) the number of top articles to return\n",
    "    df - (pandas dataframe) df as defined at the top of the notebook \n",
    "    \n",
    "    OUTPUT:\n",
    "    top_articles - (list) A list of the top 'n' article titles \n",
    "    \n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    return top_articles # Return the top article titles from df (not df_content)\n",
    "\n",
    "def get_top_article_ids(n, df=df):\n",
    "    '''\n",
    "    INPUT:\n",
    "    n - (int) the number of top articles to return\n",
    "    df - (pandas dataframe) df as defined at the top of the notebook \n",
    "    \n",
    "    OUTPUT:\n",
    "    top_articles - (list) A list of the top 'n' article titles \n",
    "    \n",
    "    '''\n",
    "    # Your code here\n",
    " \n",
    "    return top_articles # Return the top article ids"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(get_top_articles(10))\n",
    "print(get_top_article_ids(10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test your function by returning the top 5, 10, and 20 articles\n",
    "top_5 = get_top_articles(5)\n",
    "top_10 = get_top_articles(10)\n",
    "top_20 = get_top_articles(20)\n",
    "\n",
    "# Test each of your three lists from above\n",
    "t.sol_2_test(get_top_articles)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a class=\"anchor\" id=\"User-User\">第三部分：基于用户-用户的协同过滤</a>\n",
    "\n",
    "\n",
    "`1.` 使用以下函数调整 **df** dataframe 的形状，使行表示用户，并使列表示文章。  \n",
    "\n",
    "* 每个**用户**只能在每**行**中出现一次。\n",
    "\n",
    "\n",
    "* 每篇**文章**只能在每**列**中出现一次。  \n",
    "\n",
    "\n",
    "* **如果用户与某篇文章互动了，则在该文章所在的列与用户行形成的单元格中填充 1**。无论用户与文章互动了多少次，都填充 1。  \n",
    "\n",
    "\n",
    "* **如果用户与文章没有互动，则在该文章所在的列与用户行形成的单元格中填充 0**。 \n",
    "\n",
    "使用以下测试检验矩阵的基本结构是否与解答中的结构一致。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create the user-article matrix with 1's and 0's\n",
    "\n",
    "def create_user_item_matrix(df):\n",
    "    '''\n",
    "    INPUT:\n",
    "    df - pandas dataframe with article_id, title, user_id columns\n",
    "    \n",
    "    OUTPUT:\n",
    "    user_item - user item matrix \n",
    "    \n",
    "    Description:\n",
    "    Return a matrix with user ids as rows and article ids on the columns with 1 values where a user interacted with \n",
    "    an article and a 0 otherwise\n",
    "    '''\n",
    "    # Fill in the function here\n",
    "    \n",
    "    return user_item # return the user_item matrix \n",
    "\n",
    "user_item = create_user_item_matrix(df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Tests: You should just need to run this cell.  Don't change the code.\n",
    "assert user_item.shape[0] == 5149, \"Oops!  The number of users in the user-article matrix doesn't look right.\"\n",
    "assert user_item.shape[1] == 714, \"Oops!  The number of articles in the user-article matrix doesn't look right.\"\n",
    "assert user_item.sum(axis=1)[1] == 36, \"Oops!  The number of articles seen by user 1 doesn't look right.\"\n",
    "print(\"You have passed our quick tests!  Please proceed!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` 完成以下函数，该函数应该接受 user_id，并提供与该用户最相似的有序用户列表（从最相似到最不相似）。返回的列表不应包含提供的 user_id，因为我们知道每个用户都与其本身相似。因为每个用户的结果是二元的，所以建议用两个用户的点积表示相似性。 \n",
    "\n",
    "使用测试测试你的函数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_similar_users(user_id, user_item=user_item):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int) a user_id\n",
    "    user_item - (pandas dataframe) matrix of users by articles: \n",
    "                1's when a user has interacted with an article, 0 otherwise\n",
    "    \n",
    "    OUTPUT:\n",
    "    similar_users - (list) an ordered list where the closest users (largest dot product users)\n",
    "                    are listed first\n",
    "    \n",
    "    Description:\n",
    "    Computes the similarity of every pair of users based on the dot product\n",
    "    Returns an ordered\n",
    "    \n",
    "    '''\n",
    "    # compute similarity of each user to the provided user\n",
    "\n",
    "    # sort by similarity\n",
    "\n",
    "    # create list of just the ids\n",
    "   \n",
    "    # remove the own user's id\n",
    "       \n",
    "    return most_similar_users # return a list of the users in order from most to least similar\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do a spot check of your function\n",
    "print(\"The 10 most similar users to user 1 are: {}\".format(find_similar_users(1)[:10]))\n",
    "print(\"The 5 most similar users to user 3933 are: {}\".format(find_similar_users(3933)[:5]))\n",
    "print(\"The 3 most similar users to user 46 are: {}\".format(find_similar_users(46)[:3]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`3.` 创建了为每个用户提供最相似用户的函数后，你需要使用这些用户查找可以推荐的文章。完成以下函数，以返回向每个用户推荐的文章。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_article_names(article_ids, df=df):\n",
    "    '''\n",
    "    INPUT:\n",
    "    article_ids - (list) a list of article ids\n",
    "    df - (pandas dataframe) df as defined at the top of the notebook\n",
    "    \n",
    "    OUTPUT:\n",
    "    article_names - (list) a list of article names associated with the list of article ids \n",
    "                    (this is identified by the title column)\n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    return article_names # Return the article names associated with list of article ids\n",
    "\n",
    "\n",
    "def get_user_articles(user_id, user_item=user_item):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int) a user id\n",
    "    user_item - (pandas dataframe) matrix of users by articles: \n",
    "                1's when a user has interacted with an article, 0 otherwise\n",
    "    \n",
    "    OUTPUT:\n",
    "    article_ids - (list) a list of the article ids seen by the user\n",
    "    article_names - (list) a list of article names associated with the list of article ids \n",
    "                    (this is identified by the doc_full_name column in df_content)\n",
    "    \n",
    "    Description:\n",
    "    Provides a list of the article_ids and article titles that have been seen by a user\n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    return article_ids, article_names # return the ids and names\n",
    "\n",
    "\n",
    "def user_user_recs(user_id, m=10):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int) a user id\n",
    "    m - (int) the number of recommendations you want for the user\n",
    "    \n",
    "    OUTPUT:\n",
    "    recs - (list) a list of recommendations for the user\n",
    "    \n",
    "    Description:\n",
    "    Loops through the users based on closeness to the input user_id\n",
    "    For each user - finds articles the user hasn't seen before and provides them as recs\n",
    "    Does this until m recommendations are found\n",
    "    \n",
    "    Notes:\n",
    "    Users who are the same closeness are chosen arbitrarily as the 'next' user\n",
    "    \n",
    "    For the user where the number of recommended articles starts below m \n",
    "    and ends exceeding m, the last items are chosen arbitrarily\n",
    "    \n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    return recs # return your recommendations for this user_id    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check Results\n",
    "get_article_names(user_user_recs(1, 10)) # Return 10 recommendations for user 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test your functions here - No need to change this code - just run this cell\n",
    "assert set(get_article_names(['1024.0', '1176.0', '1305.0', '1314.0', '1422.0', '1427.0'])) == set(['using deep learning to reconstruct high-resolution audio', 'build a python app on the streaming analytics service', 'gosales transactions for naive bayes model', 'healthcare python streaming application demo', 'use r dataframes & ibm watson natural language understanding', 'use xgboost, scikit-learn & ibm watson machine learning apis']), \"Oops! Your the get_article_names function doesn't work quite how we expect.\"\n",
    "assert set(get_article_names(['1320.0', '232.0', '844.0'])) == set(['housing (2015): united states demographic measures','self-service data preparation with ibm data refinery','use the cloudant-spark connector in python notebook']), \"Oops! Your the get_article_names function doesn't work quite how we expect.\"\n",
    "assert set(get_user_articles(20)[0]) == set(['1320.0', '232.0', '844.0'])\n",
    "assert set(get_user_articles(20)[1]) == set(['housing (2015): united states demographic measures', 'self-service data preparation with ibm data refinery','use the cloudant-spark connector in python notebook'])\n",
    "assert set(get_user_articles(2)[0]) == set(['1024.0', '1176.0', '1305.0', '1314.0', '1422.0', '1427.0'])\n",
    "assert set(get_user_articles(2)[1]) == set(['using deep learning to reconstruct high-resolution audio', 'build a python app on the streaming analytics service', 'gosales transactions for naive bayes model', 'healthcare python streaming application demo', 'use r dataframes & ibm watson natural language understanding', 'use xgboost, scikit-learn & ibm watson machine learning apis'])\n",
    "print(\"If this is all you see, you passed all of our tests!  Nice job!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` 现在我们将提高上述 **user_user_recs** 函数的一致性。  \n",
    "\n",
    "* 当所有用户与给定用户的邻近程度都一样时，我们并非随意选择用户，而是先选择总互动次数最多的用户，然后选择互动次数第二多的用户。\n",
    "\n",
    "\n",
    "* 当推荐的文章数量以低于 m 的数字开始并以高于 m的数字结束时，我们并非随意选择文章，而是先选择总互动次数最多的文章，然后选择总互动次数第二多的文章。你可以利用之前编写的 **top_articles** 函数获得这种排名。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_top_sorted_users(user_id, df=df, user_item=user_item):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int)\n",
    "    df - (pandas dataframe) df as defined at the top of the notebook \n",
    "    user_item - (pandas dataframe) matrix of users by articles: \n",
    "            1's when a user has interacted with an article, 0 otherwise\n",
    "    \n",
    "            \n",
    "    OUTPUT:\n",
    "    neighbors_df - (pandas dataframe) a dataframe with:\n",
    "                    neighbor_id - is a neighbor user_id\n",
    "                    similarity - measure of the similarity of each user to the provided user_id\n",
    "                    num_interactions - the number of articles viewed by the user - if a u\n",
    "                    \n",
    "    Other Details - sort the neighbors_df by the similarity and then by number of interactions where \n",
    "                    highest of each is higher in the dataframe\n",
    "     \n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    return neighbors_df # Return the dataframe specified in the doc_string\n",
    "\n",
    "\n",
    "def user_user_recs_part2(user_id, m=10):\n",
    "    '''\n",
    "    INPUT:\n",
    "    user_id - (int) a user id\n",
    "    m - (int) the number of recommendations you want for the user\n",
    "    \n",
    "    OUTPUT:\n",
    "    recs - (list) a list of recommendations for the user by article id\n",
    "    rec_names - (list) a list of recommendations for the user by article title\n",
    "    \n",
    "    Description:\n",
    "    Loops through the users based on closeness to the input user_id\n",
    "    For each user - finds articles the user hasn't seen before and provides them as recs\n",
    "    Does this until m recommendations are found\n",
    "    \n",
    "    Notes:\n",
    "    * Choose the users that have the most total article interactions \n",
    "    before choosing those with fewer article interactions.\n",
    "\n",
    "    * Choose articles with the articles with the most total interactions \n",
    "    before choosing those with fewer total interactions. \n",
    "   \n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    return recs, rec_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Quick spot check - don't change this code - just use it to test your functions\n",
    "rec_ids, rec_names = user_user_recs_part2(20, 10)\n",
    "print(\"The top 10 recommendations for user 20 are the following article ids:\")\n",
    "print(rec_ids)\n",
    "print()\n",
    "print(\"The top 10 recommendations for user 20 are the following article names:\")\n",
    "print(rec_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`5.` 请利用上述函数正确填写以下字典。然后对照解答检验该字典。按照以下注释提供必要的代码。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "### Tests with a dictionary of results\n",
    "\n",
    "user1_most_sim = # Find the user that is most similar to user 1 \n",
    "user131_10th_sim = # Find the 10th most similar user to user 131"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Dictionary Test Here\n",
    "sol_5_dict = {\n",
    "    'The user that is most similar to user 1.': user1_most_sim, \n",
    "    'The user that is the 10th most similar to user 131': user131_10th_sim,\n",
    "}\n",
    "\n",
    "t.sol_5_test(sol_5_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`6.` 如果是新用户，你可以使用上述哪个函数做出推荐？请解释。你能想到更好的推荐方法吗？在以下单元格中解释向新用户做出推荐的更好方法。\n",
    "\n",
    "**请在此处填写答案。**\n",
    "\n",
    "`7.` 利用现有函数向以下新用户提供前 10 篇推荐文章。你可以对照我们的解答测试你的函数，确保在如何做出推荐方面与我们的想法一致。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_user = '0.0'\n",
    "\n",
    "# What would your recommendations be for this new user '0.0'?  As a new user, they have no observed articles.\n",
    "# Provide a list of the top 10 article ids you would give to \n",
    "new_user_recs = # Your recommendations here\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert set(new_user_recs) == set(['1314.0','1429.0','1293.0','1427.0','1162.0','1364.0','1304.0','1170.0','1431.0','1330.0']), \"Oops!  It makes sense that in this case we would want to recommend the most popular articles, because we don't know anything about these users.\"\n",
    "\n",
    "print(\"That's right!  Nice job!\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a class=\"anchor\" id=\"Content-Recs\">第四部分：基于内容的推荐方法（选修内容）</a>\n",
    "\n",
    "另一种推荐方法是对与某个术语相关的文章进行从高到低排名。内容可以是 **doc_body**、**doc_description** 或**doc_full_name**。创建基于内容的推荐系统并非只有一种方式，尤其是考虑到每列都包含与内容相关的信息。  \n",
    "\n",
    "`1.` 使用以下函数主体创建一个基于内容的推荐系统。由于这个推荐系统的正确答案不止一个，所以没有提供测试函数。如果你想尝试一种需要更多输入值的方法，可以更改函数输入。当前的输入值考虑到了你可能会使用基于内容的推荐方法。此外，你可能会使用满足“内容标准”的最热门推荐方法，总之，你在做出这些推荐时可以灵活选择方法。\n",
    "\n",
    "### 这部分并非通过项目必须完成的任务。但是，你可以借机展示你的技能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_content_recs():\n",
    "    '''\n",
    "    INPUT:\n",
    "    \n",
    "    OUTPUT:\n",
    "    \n",
    "    '''"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` 你已经创建了基于内容的推荐系统，接着在以下单元格中简要说明下这一基于内容的推荐系统是如何运行的。你觉得你的函数有哪些值得改进的地方吗？这一基于内容的推荐系统有什么新奇的地方吗？\n",
    "\n",
    "### 这部分并非通过项目必须完成的任务。但是，你可以借机展示你的技能。\n",
    "\n",
    "**在此处解释下这一基于内容的推荐系统。**\n",
    "\n",
    "`3.` 根据注释使用这一基于内容的推荐系统对以下情形做出推荐。我们没有提供测试，因为在创建此基于内容的推荐系统时，可以有多个正确答案。\n",
    "\n",
    "### 这部分并非通过项目必须完成的任务。但是，你可以借机展示你的技能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# make recommendations for a brand new user\n",
    "\n",
    "\n",
    "# make a recommendations for a user who only has interacted with article id '1427.0'\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <a class=\"anchor\" id=\"Matrix-Fact\">第五部分：矩阵分解</a>\n",
    "\n",
    "在此部分，你将利用矩阵分解向 IBM Watson Studio 平台上的用户推荐文章。\n",
    "\n",
    "`1.` 你在上述**第三部分**的**第一个问题**中已经创建了 **user_item** 矩阵。接下来的第一个问题需要你运行单元格，为**第五部分**的其他步骤做好准备。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Load the matrix here\n",
    "user_item_matrix = pd.read_pickle('user_item_matrix.p')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# quick look at the matrix\n",
    "user_item_matrix.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`2.` 在此部分，你可以对用户-项目矩阵运用[numpy](https://docs.scipy.org/doc/numpy-1.14.0/reference/generated/numpy.linalg.svd.html) 的奇异值分解方法。在单元格中执行 SVD，并解释为何与课程中的步骤不一样。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform SVD on the User-Item Matrix Here\n",
    "\n",
    "u, s, vt = # use the built in to get the three matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**请在此处填写答案。**\n",
    "\n",
    "`3.` 如何确定潜在特征的数量？这个问题比较难。运行以下单元格后你会发现，随着潜在特征数量的增加，用户-项目矩阵中 1 和 0 值的预测错误率会降低。运行以下单元格，了解当潜在特征的数量增加时，准确率会如何提高。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_latent_feats = np.arange(10,700+10,20)\n",
    "sum_errs = []\n",
    "\n",
    "for k in num_latent_feats:\n",
    "    # restructure with k latent features\n",
    "    s_new, u_new, vt_new = np.diag(s[:k]), u[:, :k], vt[:k, :]\n",
    "    \n",
    "    # take dot product\n",
    "    user_item_est = np.around(np.dot(np.dot(u_new, s_new), vt_new))\n",
    "    \n",
    "    # compute error for each prediction to actual value\n",
    "    diffs = np.subtract(user_item_matrix, user_item_est)\n",
    "    \n",
    "    # total errors and keep track of them\n",
    "    err = np.sum(np.sum(np.abs(diffs)))\n",
    "    sum_errs.append(err)\n",
    "    \n",
    "    \n",
    "plt.plot(num_latent_feats, 1 - np.array(sum_errs)/df.shape[0]);\n",
    "plt.xlabel('Number of Latent Features');\n",
    "plt.ylabel('Accuracy');\n",
    "plt.title('Accuracy vs. Number of Latent Features');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`4.` 通过上述单元格，我们无法判断要使用多少个潜在特征，因为能够更好地预测矩阵的 1 和 0 值，并不表明我们就能做出很好的推荐。我们可以将数据集划分为训练集和测试集，如以下单元格所示。  \n",
    "\n",
    "根据第三个问题的代码判断，不同的潜在特征数量对训练集和测试集的准确率有何影响。使用以下划分方法： \n",
    "\n",
    "* 我们可以对测试集中的多少个用户做出预测？  \n",
    "* 由于冷启动问题，我们无法对多少个用户做出预测？\n",
    "* 我们可以对测试集中的多少篇文章做出预测？  \n",
    "* 由于冷启动问题，我们无法对多少篇文章做出预测？"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df.head(40000)\n",
    "df_test = df.tail(5993)\n",
    "\n",
    "def create_test_and_train_user_item(df_train, df_test):\n",
    "    '''\n",
    "    INPUT:\n",
    "    df_train - training dataframe\n",
    "    df_test - test dataframe\n",
    "    \n",
    "    OUTPUT:\n",
    "    user_item_train - a user-item matrix of the training dataframe \n",
    "                      (unique users for each row and unique articles for each column)\n",
    "    user_item_test - a user-item matrix of the testing dataframe \n",
    "                    (unique users for each row and unique articles for each column)\n",
    "    test_idx - all of the test user ids\n",
    "    test_arts - all of the test article ids\n",
    "    \n",
    "    '''\n",
    "    # Your code here\n",
    "    \n",
    "    return user_item_train, user_item_test, test_idx, test_arts\n",
    "\n",
    "user_item_train, user_item_test, test_idx, test_arts = create_test_and_train_user_item(df_train, df_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace the values in the dictionary below\n",
    "a = 662 \n",
    "b = 574 \n",
    "c = 20 \n",
    "d = 0 \n",
    "\n",
    "\n",
    "sol_4_dict = {\n",
    "    'How many users can we make predictions for in the test set?': # letter here, \n",
    "    'How many users in the test set are we not able to make predictions for because of the cold start problem?': # letter here, \n",
    "    'How many movies can we make predictions for in the test set?': # letter here,\n",
    "    'How many movies in the test set are we not able to make predictions for because of the cold start problem?': # letter here\n",
    "}\n",
    "\n",
    "t.sol_4_test(sol_4_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`5.` 现在对上述 **user_item_train** 数据集进行奇异值分解，并得出 U、S 和 V 转置矩阵。然后判断在使用不同的潜在特征数量时，可以使用此矩阵分解方法对 **user_item_test** 数据集中的多少行做出预测，并根据测试数据的准确率确定应该保留多少个潜在特征。这个问题需要运用在第 `2` - `4`.个问题中完成的步骤。\n",
    "\n",
    "通过以下单元格了解 SVD 在测试数据上做出推荐预测的效果如何。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit SVD on the user_item_train matrix\n",
    "u_train, s_train, vt_train = # fit svd similar to above then use the cells below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Use these cells to see how well you can use the training \n",
    "# decomposition to predict on test data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`6.` 在以下单元格中解释上个问题的结果。根据你得出的结果，讨论下你会如何判断上述推荐系统是否比用户目前查找文章的方式更好。 \n",
    "\n",
    "**请在此处填写答案。**\n",
    "\n",
    "<a id='conclusions'></a>\n",
    "### 其他内容\n",
    "你现在可以保存为每个用户推荐的文章，开发一个类来作出新的预测并更新结果，以及创建一个部署结果的应用。这些任务并不是此项目必须完成的任务。但是，在学完课程知识后，你肯定能够继续完成这些任务并改进你的项目。\n",
    "\n",
    "\n",
    "## 总结\n",
    "\n",
    "> 恭喜！你已经完成 IBM 推荐系统项目。 \n",
    "\n",
    "> **小贴士**：当你对项目满意后，请检查报告并看看是否满足所有[审阅标准](https://review.udacity.com/#!/rubrics/2632/view)。请删除所有的**小贴士**（例如上方小贴士），使演示尽可能流畅。\n",
    "\n",
    "\n",
    "## 提交指南\n",
    "\n",
    "> 在提交项目之前，你需要在 workspace 的此部分创建 notebook 的 .html 或 .pdf 版本。运行以下单元格即可创建这两种版本。如果操作正确，系统会返回代码 0，并且你可以在 workspace 目录（点击左上角的橙色 Jupyter 图标）中看到生成的 .html 文件。\n",
    "\n",
    "> 或者，你可以通过**文件** > **下载为**子菜单将此报告下载为 .html 文件，然后手动将报告上传到 workspace 目录中：点击左上角的橙色 Jupyter 图标，然后点击“上传”按钮。\n",
    "\n",
    "> 完成这些步骤后，你可以点击右下角的“提交项目”按钮，提交项目。这样便会创建和提交一个 zip 文件，其中包含此 .ipynb 文件和你创建的 .html 或 .pdf 文件。恭喜！ \n",
    "\n",
    "\n",
    "```python\n",
    "from subprocess import call\n",
    "call(['python', '-m', 'nbconvert', 'Recommendations_with_IBM.ipynb'])\n",
    "```"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}