{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finishing the web crawler, last effort!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The code so far"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fixInternalLink(address,url):\n",
    "    if url[0:4]==\"http\":\n",
    "        return url\n",
    "    else:\n",
    "        if url[0:0]==\"/\" and address[-1]==\"/\":\n",
    "            url=address[:-2]+url\n",
    "        elif  url[0:0]!=\"/\" and address[-1]!=\"/\":\n",
    "            url=address+\"/\"+url\n",
    "        else:\n",
    "            url=address+url\n",
    "        return url\n",
    "\n",
    "            \n",
    "def getLink(p,n): # this function gets the first link in p starting from position n, adjusted to deal with single quotations\n",
    "    startPosition1=p.find('href=\"',n)\n",
    "    startPosition2=p.find(\"href=\\\\'\",n)\n",
    "    if startPosition1>-1 and (startPosition2==-1 or startPosition1<startPosition2):\n",
    "        startPosition=startPosition1+6\n",
    "        endPosition=p.find('\"',startPosition)-1\n",
    "        url=p[startPosition:endPosition+1]\n",
    "        return endPosition,url\n",
    "    elif startPosition2>-1 and (startPosition1==-1 or startPosition2<startPosition1):\n",
    "        startPosition=startPosition2+7\n",
    "        endPosition=p.find(\"\\\\'\",startPosition)-1\n",
    "        url=p[startPosition:endPosition+1]\n",
    "        return endPosition,url\n",
    "    else:\n",
    "        return -1,\"\"\n",
    "\n",
    "\n",
    "import requests\n",
    "from hashlib import sha256 # do not forget this, or you won't have sha256 available\n",
    "limitVisitedPages=200\n",
    "depthLimit=5\n",
    "Crawled=[]\n",
    "Skipped=[]\n",
    "skippedEqual=[]\n",
    "visitedHashes=[]\n",
    "toCrawl=[\"http://www.paolocoletti.it/test\"]\n",
    "toCrawlWithDepth=[[\"http://www.paolocoletti.it/test\",1]] # new structure with URL and depth\n",
    "BadExtensions=[\".png\",\".gif\",\".jpeg\",\".jpg\",\".php\",\".json\",\".ico\",\"#\"]\n",
    "\n",
    "while len(toCrawl)>0 and len(Crawled)<limitVisitedPages: \n",
    "    pageAddress,depth = toCrawlWithDepth.pop(0) # pay attention that now I pop out the url and the depth\n",
    "    toCrawl.pop(0) \n",
    "    page = requests.get(pageAddress) \n",
    "    sha=sha256(page.content).hexdigest()\n",
    "    if sha in visitedHashes:\n",
    "        skippedEqual.append(pageAddress)\n",
    "    else:\n",
    "        visitedHashes.append(sha)\n",
    "        print(\"I am crawling page: \" + pageAddress) # I put this to have an idea of what the program is doing\n",
    "        Crawled.append(pageAddress)\n",
    "\n",
    "        n=0\n",
    "        while n>=0:\n",
    "            n,url=getLink(str(page.content),n+1)\n",
    "            if n>-1 and url!=\"\":\n",
    "                url=fixInternalLink(pageAddress,url)\n",
    "                if url[-1:] not in BadExtensions and url[-4:] not in BadExtensions and url[-5:] not in BadExtensions: # otherwise we completely forget about it\n",
    "                    if url not in toCrawl and url not in Crawled:\n",
    "                        if depth<depthLimit:\n",
    "                            toCrawlWithDepth.append([url,depth+1]) # if the depth of the current page is below the limit, I add it to the list\n",
    "                            toCrawl.append(url) \n",
    "                        elif depth==depthLimit:\n",
    "                            if url not in Skipped:\n",
    "                                Skipped.append(url) # otherwise if the depth of the current page is AT THE LIMIT, then the link is beyond the limit and I add it to the list of the Skipped\n",
    "                        else:\n",
    "                            print(\"Error: what am I doing here????\") # it should never happen that the depth of this page is above the limit!\n",
    "                    \n",
    "print(len(Crawled),len(toCrawl),len(Skipped))\n",
    "print(Crawled)\n",
    "print(toCrawl)\n",
    "print(Skipped)\n",
    "print(skippedEqual)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Indexing the content"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our last effort!\n",
    "\n",
    "Until now we have built a perfect crawler which, however, is good only at retrieving links. We would like also to associate search terms with those links."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First of all, we need to remove all the tags from the web page, i.e. all the things such as <whatever....>, because these are not words which appear on the web page but only technical information. Thus we build together a function which does it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a clean text we can use the function splitString, built some time ago, to get all the words of the page as a list. I paste here the old function for your convenience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def splitString(s):\n",
    "    Ortho=[\" \",\";\",\".\",\"!\",\"?\",\":\",\",\",\"-\"]\n",
    "    L=[]\n",
    "    if len(s)==0: # you never know....\n",
    "        return L\n",
    "    current=0\n",
    "    start=0\n",
    "    # first of all I skip all the starting ortho symbols\n",
    "    while current < len(s) and s[current] in Ortho: \n",
    "        current=current+1\n",
    "    start=current\n",
    "    # now we are sure that start and current are at the beginning of the first real word (or that the string is over)\n",
    "    while current < len(s):\n",
    "        if s[current] in Ortho:\n",
    "            # I found an orthograhic symbol. \n",
    "            # first I append the word from start up tu current-1\n",
    "            L.append(s[start:current])\n",
    "            # But there might be several symbols, so I go on until I find a non-orthographic symbol or until the string is over\n",
    "            while current < len(s) and s[current] in Ortho:\n",
    "                current=current+1\n",
    "            # now current is on the first non-ortho symbol\n",
    "            start=current # and thus start is set equal to current, as we will have to re-start the function.\n",
    "        else:\n",
    "            current=current+1\n",
    "    # I am now outside the loop. I might be outside because the last character is an orthographic symbol and in this case fine!\n",
    "    # but I might be outside because the string ended with a letter and thus I am still \"inside\" a word! And I have to add it\n",
    "    if s[-1] not in Ortho:\n",
    "        L.append(s[start:])\n",
    "    return L"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the two function in combination, we can get easily the list of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now the final effort! Modify the previous program adding a dict **words** which associates each word with a list of the links in which that word is found. E.g. if word \"Test\" is found in three different webpage, words[\"Test\"] will contain as value a list with those three links.\n",
    "\n",
    "Go on to exe file..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}