{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finishing the web crawler, last effort!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The code so far"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "I am crawling page: http://www.paolocoletti.it/test\n",
      "I am crawling page: http://www.paolocoletti.it/test/test2.html\n",
      "I am crawling page: http://www.paolocoletti.it/test/test3.html\n",
      "I am crawling page: http://www.paolocoletti.it/test/test4.html\n",
      "4 0 0\n",
      "['http://www.paolocoletti.it/test', 'http://www.paolocoletti.it/test/test2.html', 'http://www.paolocoletti.it/test/test3.html', 'http://www.paolocoletti.it/test/test4.html']\n",
      "[]\n",
      "[]\n",
      "['http://www.paolocoletti.it/test/test.html']\n"
     ]
    }
   ],
   "source": [
    "def fixInternalLink(address,url):\n",
    "    if url[0:4]==\"http\":\n",
    "        return url\n",
    "    else:\n",
    "        if url[0:0]==\"/\" and address[-1]==\"/\":\n",
    "            url=address[:-2]+url\n",
    "        elif  url[0:0]!=\"/\" and address[-1]!=\"/\":\n",
    "            url=address+\"/\"+url\n",
    "        else:\n",
    "            url=address+url\n",
    "        return url\n",
    "\n",
    "            \n",
    "def getLink(p,n): # this function gets the first link in p starting from position n, adjusted to deal with single quotations\n",
    "    startPosition1=p.find('href=\"',n)\n",
    "    startPosition2=p.find(\"href=\\\\'\",n)\n",
    "    if startPosition1>-1 and (startPosition2==-1 or startPosition1<startPosition2):\n",
    "        startPosition=startPosition1+6\n",
    "        endPosition=p.find('\"',startPosition)-1\n",
    "        url=p[startPosition:endPosition+1]\n",
    "        return endPosition,url\n",
    "    elif startPosition2>-1 and (startPosition1==-1 or startPosition2<startPosition1):\n",
    "        startPosition=startPosition2+7\n",
    "        endPosition=p.find(\"\\\\'\",startPosition)-1\n",
    "        url=p[startPosition:endPosition+1]\n",
    "        return endPosition,url\n",
    "    else:\n",
    "        return -1,\"\"\n",
    "\n",
    "\n",
    "import requests\n",
    "from hashlib import sha256 # do not forget this, or you won't have sha256 available\n",
    "limitVisitedPages=200\n",
    "depthLimit=5\n",
    "Crawled=[]\n",
    "Skipped=[]\n",
    "skippedEqual=[]\n",
    "visitedHashes=[]\n",
    "toCrawl=[\"http://www.paolocoletti.it/test\"]\n",
    "toCrawlWithDepth=[[\"http://www.paolocoletti.it/test\",1]] # new structure with URL and depth\n",
    "BadExtensions=[\".png\",\".gif\",\".jpeg\",\".jpg\",\".php\",\".json\",\".ico\",\"#\"]\n",
    "\n",
    "while len(toCrawl)>0 and len(Crawled)<limitVisitedPages: \n",
    "    pageAddress,depth = toCrawlWithDepth.pop(0) # pay attention that now I pop out the url and the depth\n",
    "    toCrawl.pop(0) \n",
    "    page = requests.get(pageAddress) \n",
    "    sha=sha256(page.content).hexdigest()\n",
    "    if sha in visitedHashes:\n",
    "        skippedEqual.append(pageAddress)\n",
    "    else:\n",
    "        visitedHashes.append(sha)\n",
    "        print(\"I am crawling page: \" + pageAddress) # I put this to have an idea of what the program is doing\n",
    "        Crawled.append(pageAddress)\n",
    "\n",
    "        n=0\n",
    "        while n>=0:\n",
    "            n,url=getLink(str(page.content),n+1)\n",
    "            if n>-1 and url!=\"\":\n",
    "                url=fixInternalLink(pageAddress,url)\n",
    "                if url[-1:] not in BadExtensions and url[-4:] not in BadExtensions and url[-5:] not in BadExtensions: # otherwise we completely forget about it\n",
    "                    if url not in toCrawl and url not in Crawled:\n",
    "                        if depth<depthLimit:\n",
    "                            toCrawlWithDepth.append([url,depth+1]) # if the depth of the current page is below the limit, I add it to the list\n",
    "                            toCrawl.append(url) \n",
    "                        elif depth==depthLimit:\n",
    "                            if url not in Skipped:\n",
    "                                Skipped.append(url) # otherwise if the depth of the current page is AT THE LIMIT, then the link is beyond the limit and I add it to the list of the Skipped\n",
    "                        else:\n",
    "                            print(\"Error: what am I doing here????\") # it should never happen that the depth of this page is above the limit!\n",
    "                    \n",
    "print(len(Crawled),len(toCrawl),len(Skipped))\n",
    "print(Crawled)\n",
    "print(toCrawl)\n",
    "print(Skipped)\n",
    "print(skippedEqual)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "    Test 4 web page    Test 4  Dice, dire, duty, dividend, desire, destroy  This is just test web page numer four. Here  a BACK link to test original page .  Here  another link to test 3 .   \n"
     ]
    }
   ],
   "source": [
    "def removeTags(s):\n",
    "    # returns a string equal to s but without all the tags\n",
    "    i=0\n",
    "    r=\"\"\n",
    "    while i<len(s):\n",
    "        if s[i]!=\"<\":\n",
    "            r=r+s[i]\n",
    "        else:\n",
    "            while i<len(s) and s[i]!=\">\":\n",
    "                i=i+1\n",
    "            # at this point i is on the \">\" symbol\n",
    "            # just to avoid having a word attached to another when there is a tag in the middle, such as Paolo<...>Coletti -> PaoloColetti, we introduce a space here\n",
    "            r=r+\" \"\n",
    "        i=i+1\n",
    "    return r\n",
    "\n",
    "print(removeTags('<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\" \"http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd\"><html><head><title>Test 4 web page</title></head><body><h1>Test 4</h1><p>Dice, dire, duty, dividend, desire, destroy</p><p>This is just test web page numer four. Here <a href=\"http://www.paolocoletti.it/test/test.html\">a BACK link to test original page</a>.</p><p>Here <a href=\"http://www.paolocoletti.it/test/test3.html\">another link to test 3</a>.</p></body></html>'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def splitString(s):\n",
    "    Ortho=[\" \",\";\",\".\",\"!\",\"?\",\":\",\",\",\"-\"]\n",
    "    L=[]\n",
    "    if len(s)==0: # you never know....\n",
    "        return L\n",
    "    current=0\n",
    "    start=0\n",
    "    # first of all I skip all the starting ortho symbols\n",
    "    while current < len(s) and s[current] in Ortho: \n",
    "        current=current+1\n",
    "    start=current\n",
    "    # now we are sure that start and current are at the beginning of the first real word (or that the string is over)\n",
    "    while current < len(s):\n",
    "        if s[current] in Ortho:\n",
    "            # I found an orthograhic symbol. \n",
    "            # first I append the word from start up tu current-1\n",
    "            L.append(s[start:current])\n",
    "            # But there might be several symbols, so I go on until I find a non-orthographic symbol or until the string is over\n",
    "            while current < len(s) and s[current] in Ortho:\n",
    "                current=current+1\n",
    "            # now current is on the first non-ortho symbol\n",
    "            start=current # and thus start is set equal to current, as we will have to re-start the function.\n",
    "        else:\n",
    "            current=current+1\n",
    "    # I am now outside the loop. I might be outside because the last character is an orthographic symbol and in this case fine!\n",
    "    # but I might be outside because the string ended with a letter and thus I am still \"inside\" a word! And I have to add it\n",
    "    if s[-1] not in Ortho:\n",
    "        L.append(s[start:])\n",
    "    return L"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the two function in combination, we can get easily the list of words."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['Test', '4', 'web', 'page', 'Test', '4', 'Dice', 'dire', 'duty', 'dividend', 'desire', 'destroy', 'This', 'is', 'just', 'test', 'web', 'page', 'numer', 'four', 'Here', 'a', 'BACK', 'link', 'to', 'test', 'original', 'page', 'Here', 'another', 'link', 'to', 'test', '3']\n"
     ]
    }
   ],
   "source": [
    "print(splitString(removeTags('<!DOCTYPE html PUBLIC \"-//W3C//DTD HTML 4.01 Transitional//EN\" \"http://www.w3.org/TR/1999/REC-html401-19991224/loose.dtd\"><html><head><title>Test 4 web page</title></head><body><h1>Test 4</h1><p>Dice, dire, duty, dividend, desire, destroy</p><p>This is just test web page numer four. Here <a href=\"http://www.paolocoletti.it/test/test.html\">a BACK link to test original page</a>.</p><p>Here <a href=\"http://www.paolocoletti.it/test/test3.html\">another link to test 3</a>.</p></body></html>')))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now the final effort! Modify the previous program adding a dict **words** which associates each word with a list of the links in which that word is found. E.g. if word \"Test\" is found in three different webpage, words[\"Test\"] will contain as value a list with those three links.\n",
    "\n",
    "<p><font color=PURPLE>Strategy: after having thought at the problem, if you can't come up with a working algorithm click below to reveal it. I said <b>after having thought</b>\n",
    "<details>You need to create an empty dict **words**. After _Crawled.append(pageAddress)_ you get the list of words contained in that page. You go through the list and check the corresponding values in the dict **words**. If that key exists, you append the pageAddress to the list of links that is contained in words[<that key>], otherwise you create a new element with the word as key and a list with only element pageAddress as value.\n",
    "<br>A final touch: a word can appear several times inside the same page. We do not like, however, to put the same link several times in the list. So, before inserting the link in an already existing list, check that the link be not already inside.\n",
    "    </details>\n",
    "    \n",
    "<p><font color=BLACK>Finally, test your program asking for the value of words[\"another\"] and words[\"SINGLE\"]."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}