webcheck branch master updated. 1.10.4-72-g8de60b5
[
Date Prev][
Date Next]
[
Thread Prev][
Thread Next]
webcheck branch master updated. 1.10.4-72-g8de60b5
- From: Commits of the webcheck project <webcheck-commits [at] lists.arthurdejong.org>
- To: webcheck-commits [at] lists.arthurdejong.org
- Reply-to: webcheck-users [at] lists.arthurdejong.org
- Subject: webcheck branch master updated. 1.10.4-72-g8de60b5
- Date: Sun, 22 Sep 2013 22:07:53 +0200 (CEST)
This is an automated email from the git hooks/post-receive script. It was
generated because a ref change was pushed to the repository containing
the project "webcheck".
The branch, master has been updated
via 8de60b500dc50357d11637f39f8ce01fd09ec0ca (commit)
via 6b650e7eb3861a690484de0282071fbc8677b388 (commit)
via fee3ccb29c0a83e0bb7deac249278d27e14d3055 (commit)
via c5ea12def1487fb8d34018e5d6545d8e449fdd96 (commit)
via bdca8863fc3b565978f8fd560c1726496aab0379 (commit)
via 58dcfbf3ff7152016bf3906bbf5805e097a1ee0e (commit)
from 44fc843ea803118aeffc4914f17414eaee040e0b (commit)
Those revisions listed above that are new to this repository have
not appeared on any other notification email; so we list those
revisions in full, below.
- Log -----------------------------------------------------------------
http://arthurdejong.org/git/webcheck/commit/?id=8de60b500dc50357d11637f39f8ce01fd09ec0ca
commit 8de60b500dc50357d11637f39f8ce01fd09ec0ca
Merge: 44fc843 6b650e7
Author: Arthur de Jong <arthur@arthurdejong.org>
Date: Sun Sep 22 22:06:29 2013 +0200
Use Jinja templates to render report
The switch to Jinja removes the need for custom escaping and Python code
to write HTML output and instead uses easy to read templates.
As a result of the switch, this drops more than 450 lines of Python code
while adding a little over 400 lines of HTML template code.
http://arthurdejong.org/git/webcheck/commit/?id=6b650e7eb3861a690484de0282071fbc8677b388
commit 6b650e7eb3861a690484de0282071fbc8677b388
Author: Arthur de Jong <arthur@arthurdejong.org>
Date: Sun Sep 22 15:38:07 2013 +0200
Remove unused code
Most of this is removed because of the switch to the Jinja template
engine.
diff --git a/webcheck/parsers/__init__.py b/webcheck/parsers/__init__.py
index f0f5f97..9b7224a 100644
--- a/webcheck/parsers/__init__.py
+++ b/webcheck/parsers/__init__.py
@@ -53,11 +53,3 @@ def get_parsermodule(mimetype):
if mimetype in _parsermodules:
return _parsermodules[mimetype]
return None
-
-
-def get_mimetypes():
- """Return a list of supported mime types that can be parsed
- by the installed parsers."""
- if _parsermodules == {}:
- _init_modules()
- return _parsermodules.keys()
diff --git a/webcheck/parsers/html/__init__.py
b/webcheck/parsers/html/__init__.py
index ac7b70d..3a372df 100644
--- a/webcheck/parsers/html/__init__.py
+++ b/webcheck/parsers/html/__init__.py
@@ -41,28 +41,6 @@ mimetypes = ('text/html', 'application/xhtml+xml',
'text/x-server-parsed-html')
_entitypattern = re.compile('&(#[0-9]{1,6}|[a-zA-Z]{2,10});')
-def htmlescape(txt):
- """HTML escape the given string and return an ASCII clean string with
- known entities and character entities for the other values."""
- # check for empty string
- if not txt:
- return u''
- # convert to unicode object
- if not isinstance(txt, unicode):
- txt = unicode(txt)
- # the output string
- out = ''
- # loop over the characters of the string
- for c in txt:
- if ord(c) in htmlentitydefs.codepoint2name:
- out += '&%s;' % htmlentitydefs.codepoint2name[ord(c)]
- elif ord(c) > 126:
- out += '&#%d;' % ord(c)
- else:
- out += c.encode('utf-8')
- return out
-
-
def _unescape_entity(match):
"""Helper function for htmlunescape().
This funcion unescapes a html entity, it is passed to the sub()
diff --git a/webcheck/plugins/__init__.py b/webcheck/plugins/__init__.py
index 4d032a4..e69de29 100644
--- a/webcheck/plugins/__init__.py
+++ b/webcheck/plugins/__init__.py
@@ -1,229 +0,0 @@
-
-# __init__.py - plugin function module
-#
-# Copyright (C) 1998, 1999 Albert Hopkins (marduk)
-# Copyright (C) 2002 Mike W. Meyer
-# Copyright (C) 2005, 2006, 2007, 2009, 2011, 2013 Arthur de Jong
-#
-# This program is free software; you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation; either version 2 of the License, or
-# (at your option) any later version.
-#
-# This program is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with this program; if not, write to the Free Software
-# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
-#
-# The files produced as output from the software do not automatically fall
-# under the copyright of the software, unless explicitly stated otherwise.
-
-"""This package groups all the plugins.
-
-When generating the report each plugin is called in turn with
-the generate() function. Each plugin should export the following
-fields:
-
- generate(crawler)
- Based on the site generate all the output files as needed.
- __title__
- A short description of the plugin that is used when linking
- to the output from the plugin.
- __author__
- The author(s) of the plugin.
- __outputfile__
- The file the plugin generates (for linking to).
- docstring
- The docstring is used as description of the plugin in the
- report.
-
-Pluings can use the functions exported by this module."""
-
-import time
-
-from sqlalchemy.orm import joinedload
-
-import webcheck
-from webcheck import config
-from webcheck.db import Link
-from webcheck.parsers.html import htmlescape
-from webcheck.output import open_file
-
-
-def _floatformat(f):
- """Return a float as a string while trying to keep it within three
- characters."""
- txt = '%.1f' % f
- # remove period from too long strings
- if len(txt) > 3:
- txt = txt[:txt.find('.')]
- return txt
-
-
-def get_size(i):
- """Return the size in bytes as a readble string."""
- K = 1024
- M = K * 1024
- G = M * 1024
- if i > 1024 * 1024 * 999:
- return _floatformat(float(i) / float(G)) + 'G'
- elif i > 1024 * 999:
- return _floatformat(float(i) / float(M)) + 'M'
- elif i >= 1024:
- return _floatformat(float(i) / float(K)) + 'K'
- else:
- return '%d' % i
-
-
-def _get_info(link):
- """Return a string with a summary of the information in the link."""
- info = u'url: %s\n' % link.url
- if link.status:
- info += u'%s\n' % link.status
- if link.title:
- info += u'title: %s\n' % link.title.strip()
- if link.author:
- info += u'author: %s\n' % link.author.strip()
- if link.is_internal:
- info += u'internal link'
- else:
- info += u'external link'
- if link.yanked:
- info += u', not checked (%s)\n' % link.yanked
- else:
- info += u'\n'
- if link.redirectdepth:
- if link.children.count() > 0:
- info += u'redirect: %s\n' % link.children.first().url
- else:
- info += u'redirect (not followed)\n'
- count = link.count_parents
- if count == 1:
- info += u'linked from 1 page\n'
- elif count > 1:
- info += u'linked from %d pages\n' % count
- if link.mtime:
- info += u'last modified: %s\n' % time.ctime(link.mtime)
- if link.size:
- info += u'size: %s\n' % get_size(link.size)
- if link.mimetype:
- info += u'mime-type: %s\n' % link.mimetype
- if link.encoding:
- info += u'encoding: %s\n' % link.encoding
- for problem in link.linkproblems:
- info += u'problem: %s\n' % problem.message
- # trim trailing newline
- return info.strip()
-
-
-def make_link(link, title=None):
- """Return an <a>nchor to a url with title. If url is in the Linklist and
- is external, insert "class=external" in the <a> tag."""
- return '<a href="%(url)s" %(target)sclass="%(cssclass)s"
title="%(info)s">%(title)s</a>' % \
- dict(url=htmlescape(link.url),
- target='target="_blank" ' if
config.REPORT_LINKS_IN_NEW_WINDOW else '',
- cssclass='internal' if link.is_internal else 'external',
- info=htmlescape(_get_info(link)).replace('\n', ' '),
- title=htmlescape(title or link.title or link.url))
-
-
-def print_parents(fp, link, indent=' '):
- """Write a list of parents to the output file descriptor.
- The output is indeted with the specified indent."""
- # if there are no parents print nothing
- count = link.count_parents
- if not count:
- return
- parents = link.parents.order_by(Link.title,
Link.url).options(joinedload(Link.linkproblems))[:config.PARENT_LISTLEN]
- fp.write(
- indent + '<div class="parents">\n' +
- indent + ' referenced from:\n' +
- indent + ' <ul>\n')
- more = link.count_parents
- for parent in parents:
- fp.write(
- indent + ' <li>%(parent)s</li>\n'
- % {'parent': make_link(parent)})
- more -= 1
- if more:
- fp.write(
- indent + ' <li>%(more)d more...</li>\n'
- % {'more': more})
- fp.write(
- indent + ' </ul>\n' +
- indent + '</div>\n')
-
-
-def _print_navbar(fp, selected, crawler):
- """Return an html fragement representing the navigation bar for a page."""
- fp.write(' <ul class="navbar">\n')
- for plugin in crawler.plugins:
- # skip if no outputfile
- if not hasattr(plugin, '__outputfile__'):
- continue
- # generate a link to the plugin page
- selected = ''
- if plugin == selected:
- selected = ' class="selected"'
- fp.write(
- ' <li><a href="%(pluginfile)s"%(selected)s
title="%(description)s">%(title)s</a></li>\n'
- % {'pluginfile': plugin.__outputfile__,
- 'selected': selected,
- 'title': htmlescape(plugin.__title__),
- 'description': htmlescape(plugin.__doc__)})
- fp.write(' </ul>\n')
-
-
-def open_html(plugin, crawler):
- """Print an html fragment for the start of an html page."""
- # open the file
- fp = open_file(plugin.__outputfile__)
- # get the first base url
- base = crawler.bases[0]
- # write basic html head
- fp.write(
- '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>\n'
- '<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd";>\n'
- '<html xmlns="http://www.w3.org/1999/xhtml";>\n'
- ' <head>\n'
- ' <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"
/>\n'
- ' <title>Webcheck report for %(sitetitle)s (%(plugintitle)s)</title>\n'
- ' <link rel="stylesheet" type="text/css" href="webcheck.css" />\n'
- ' <link rel="icon" href="favicon.ico" type="image/ico" />\n'
- ' <link rel="shortcut icon" href="favicon.ico" />\n'
- ' <script type="text/javascript" src="fancytooltips.js"></script>\n'
- ' <meta name="Generator" content="webcheck %(version)s" />\n'
- ' </head>\n'
- ' <body>\n'
- ' <h1 class="basename">Webcheck report for <a
href="%(siteurl)s">%(sitetitle)s</a></h1>\n'
- % {'sitetitle': htmlescape(base.title or base.url),
- 'plugintitle': htmlescape(plugin.__title__),
- 'siteurl': base.url,
- 'version': webcheck.__version__})
- # write navigation bar
- _print_navbar(fp, plugin, crawler)
- # write plugin heading
- fp.write(' <h2>%s</h2>\n' % htmlescape(plugin.__title__))
- # write plugin contents
- fp.write(' <div class="content">\n')
- return fp
-
-
-def close_html(fp):
- """Print an html fragment as footer of an html page."""
- fp.write(' </div>\n')
- # write bottom of page
- fp.write(
- ' <p class="footer">\n'
- ' Generated %(time)s by <a href="%(homepage)s">webcheck
%(version)s</a>\n'
- ' </p>\n'
- ' </body>\n'
- '</html>\n'
- % {'time': htmlescape(time.ctime(time.time())),
- 'homepage': webcheck.__homepage__,
- 'version': htmlescape(webcheck.__version__)})
- fp.close()
http://arthurdejong.org/git/webcheck/commit/?id=fee3ccb29c0a83e0bb7deac249278d27e14d3055
commit fee3ccb29c0a83e0bb7deac249278d27e14d3055
Author: Arthur de Jong <arthur@arthurdejong.org>
Date: Sun Sep 22 17:02:56 2013 +0200
Switch plugins to use template
The sitemap module has been somewhat rewritten to use generators to
provide the structure of the website. The problems module has also been
simplified a bit.
diff --git a/webcheck/plugins/about.py b/webcheck/plugins/about.py
index 014f1b7..47e6028 100644
--- a/webcheck/plugins/about.py
+++ b/webcheck/plugins/about.py
@@ -28,87 +28,13 @@ __title__ = 'about webcheck'
__author__ = 'Arthur de Jong'
__outputfile__ = 'about.html'
-import time
-
-import webcheck
from webcheck.db import Session, Link
-import webcheck.plugins
+from webcheck.output import render
def generate(crawler):
"""Output a list of modules, it's authors and the webcheck version."""
- fp = webcheck.plugins.open_html(webcheck.plugins.about, crawler)
session = Session()
- # TODO: xxx links were fetched, xxx pages were examined and a total of xxx
notes and problems were found
- # TODO: include some runtime information (e.g. supported schemes, user
configuration, etc)
- # output some general information about the report
- fp.write(
- ' <p>\n'
- ' This is a website report generated by <tt>webcheck</tt>\n'
- ' %(version)s. <tt>webcheck</tt> is a website checking tool for\n'
- ' webmasters. It crawls a given website and does a number of tests\n'
- ' to see if links and pages are valid.\n'
- ' More information about <tt>webcheck</tt> can be found at the\n'
- ' <tt>webcheck</tt> homepage which is located at\n'
- ' <a href="%(homepage)s">%(homepage)s</a>.\n'
- ' </p>\n'
- ' <p>\n'
- ' This report was generated on %(time)s, a total of %(numurls)d\n'
- ' links were found.\n'
- ' </p>\n\n'
- % {'version': webcheck.plugins.htmlescape(webcheck.__version__),
- 'time': webcheck.plugins.htmlescape(time.ctime(time.time())),
- 'numurls': session.query(Link).count(),
- 'homepage': webcheck.__homepage__})
- # output copyright information
- fp.write(
- ' <h3>Copyright</h3>\n'
- ' <p>\n'
- ' <tt>webcheck</tt> was originally named <tt>linbot</tt> which was\n'
- ' developed by Albert Hopkins (marduk).\n'
- ' Versions up till 1.0 were maintained by Mike W. Meyer who changed\n'
- ' the name to <tt>webcheck</tt>.\n'
- ' After that Arthur de Jong did a complete rewrite.\n'
- ' <tt>webcheck</tt> is <i>free software</i>; you can redistribute
it\n'
- ' and/or modify it under the terms of the\n'
- ' <a href="http://www.gnu.org/copyleft/gpl.html";>GNU General Public
License</a>\n'
- ' (version 2 or later).\n'
- ' There is no warranty; not even for merchantability or fitness for
a\n'
- ' particular purpose. See the source for further details.\n'
- ' </p>\n'
- ' <p>\n'
- ' Copyright © 1998-2013 Albert Hopkins (marduk),\n'
- ' Mike W. Meyer and Arthur de Jong\n'
- ' </p>\n'
- ' <p>\n'
- ' The files in this generated report do not automatically fall
under\n'
- ' the copyright of the software, unless explicitly stated
otherwise.\n'
- ' </p>\n'
- ' <p>\n'
- ' <tt>webcheck</tt> includes the\n'
- ' <a
href="http://victr.lm85.com/projects/fancytooltips/";>FancyTooltips</a>\n'
- ' javascript library to display readable tooltips. FancyTooltips is\n'
- ' distributed under the MIT license and has the following copyright\n'
- ' notices (see <tt>fancytooltips.js</tt> for details):\n'
- ' </p>\n'
- ' <p>\n'
- ' Copyright © 2003-2005 Stuart Langridge, Paul McLanahan,\n'
- ' Peter Janes, Brad Choate, Dunstan Orchard, Ethan Marcotte,\n'
- ' Mark Wubben and Victor Kulinski\n'
- ' </p>\n\n')
- # output plugin information
- fp.write(
- ' <h3>Plugins</h3>\n'
- ' <ul>\n')
- for plugin in crawler.plugins:
- fp.write(
- ' <li>\n'
- ' <strong>%s</strong><br />\n'
- % webcheck.plugins.htmlescape(plugin.__title__))
- if hasattr(plugin, '__doc__'):
- fp.write(' %s<br />\n' %
webcheck.plugins.htmlescape(plugin.__doc__))
- fp.write(' </li>\n')
- fp.write(
- ' </ul>\n')
- webcheck.plugins.close_html(fp)
+ render(__outputfile__, crawler=crawler, title=__title__,
+ numlinks=session.query(Link).count())
session.close()
diff --git a/webcheck/plugins/badlinks.py b/webcheck/plugins/badlinks.py
index 716f1f2..47411d3 100644
--- a/webcheck/plugins/badlinks.py
+++ b/webcheck/plugins/badlinks.py
@@ -31,7 +31,7 @@ __outputfile__ = 'badlinks.html'
from sqlalchemy.orm import joinedload
from webcheck.db import Session, Link
-import webcheck.plugins
+from webcheck.output import render
def postprocess(crawler):
@@ -52,41 +52,8 @@ def postprocess(crawler):
def generate(crawler):
"""Present the list of bad links."""
session = Session()
- # find all links with link problems
- links =
session.query(Link).filter(Link.linkproblems.any()).order_by(Link.url).options(joinedload(Link.linkproblems))
- # present results
- fp = webcheck.plugins.open_html(webcheck.plugins.badlinks, crawler)
- if not links:
- fp.write(
- ' <p class="description">\n'
- ' There were no problems retrieving links from the website.\n'
- ' </p>\n')
- webcheck.plugins.close_html(fp)
- return
- fp.write(
- ' <p class="description">\n'
- ' These links could not be retrieved during the crawling of the
website.\n'
- ' </p>\n'
- ' <ol>\n')
- for link in links:
- # list the link
- fp.write(
- ' <li>\n'
- ' %(badurl)s\n'
- ' <ul class="problems">\n'
- % {'badurl': webcheck.plugins.make_link(link, link.url)})
- # list the problems
- for problem in link.linkproblems:
- fp.write(
- ' <li>%(problem)s</li>\n'
- % {'problem': webcheck.plugins.htmlescape(problem)})
- fp.write(
- ' </ul>\n')
- # present a list of parents
- webcheck.plugins.print_parents(fp, link, ' ')
- fp.write(
- ' </li>\n')
- fp.write(
- ' </ol>\n')
- webcheck.plugins.close_html(fp)
+ links = session.query(Link).filter(Link.linkproblems.any())
+ links = links.order_by(Link.url).options(joinedload(Link.linkproblems))
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links)
session.close()
diff --git a/webcheck/plugins/external.py b/webcheck/plugins/external.py
index 9a7681f..4e86460 100644
--- a/webcheck/plugins/external.py
+++ b/webcheck/plugins/external.py
@@ -31,39 +31,14 @@ __outputfile__ = 'external.html'
from sqlalchemy.orm import joinedload
from webcheck.db import Session, Link
-import webcheck.plugins
+from webcheck.output import render
def generate(crawler):
"""Generate the list of external links."""
session = Session()
- # get all external links
links = session.query(Link).filter(Link.is_internal !=
True).order_by(Link.url)
- # present results
- fp = webcheck.plugins.open_html(webcheck.plugins.external, crawler)
- if not links:
- fp.write(
- ' <p class="description">'
- ' No external links were found on the website.'
- ' </p>\n')
- webcheck.plugins.close_html(fp)
- return
- fp.write(
- ' <p class="description">'
- ' This is the list of all external urls encountered during the'
- ' examination of the website.'
- ' </p>\n'
- ' <ol>\n')
- for link in links.options(joinedload(Link.linkproblems)):
- fp.write(
- ' <li>\n'
- ' %(link)s\n'
- % {'link': webcheck.plugins.make_link(link)})
- # present a list of parents
- webcheck.plugins.print_parents(fp, link, ' ')
- fp.write(
- ' </li>\n')
- fp.write(
- ' </ol>\n')
- webcheck.plugins.close_html(fp)
+ links = links.options(joinedload(Link.linkproblems))
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links)
session.close()
diff --git a/webcheck/plugins/images.py b/webcheck/plugins/images.py
index 05c9369..e18d544 100644
--- a/webcheck/plugins/images.py
+++ b/webcheck/plugins/images.py
@@ -29,7 +29,7 @@ __author__ = 'Arthur de Jong'
__outputfile__ = 'images.html'
from webcheck.db import Session, Link
-import webcheck.plugins
+from webcheck.output import render
def generate(crawler):
@@ -40,24 +40,6 @@ def generate(crawler):
links = links.filter((Link.is_page != True) | (Link.is_page == None))
links = links.filter(Link.mimetype.startswith('image/'))
links = links.order_by(Link.url)
- # present results
- fp = webcheck.plugins.open_html(webcheck.plugins.images, crawler)
- if not links:
- fp.write(
- ' <p class="description">\n'
- ' No images were linked on the website.\n'
- ' </p>\n'
- ' <ol>\n')
- webcheck.plugins.close_html(fp)
- return
- fp.write(
- ' <p class="description">\n'
- ' This is the list of all images found linked on the website.\n'
- ' </p>\n'
- ' <ol>\n')
- for link in links:
- fp.write(' <li>%s</li>\n' % webcheck.plugins.make_link(link,
link.url))
- fp.write(
- ' </ol>\n')
- webcheck.plugins.close_html(fp)
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links)
session.close()
diff --git a/webcheck/plugins/new.py b/webcheck/plugins/new.py
index 96392e4..8f3c678 100644
--- a/webcheck/plugins/new.py
+++ b/webcheck/plugins/new.py
@@ -32,7 +32,7 @@ import time
from webcheck import config
from webcheck.db import Session, Link
-import webcheck.plugins
+from webcheck.output import render
SECS_PER_DAY = 60 * 60 * 24
@@ -41,38 +41,9 @@ SECS_PER_DAY = 60 * 60 * 24
def generate(crawler):
"""Output the list of recently modified pages."""
session = Session()
- # the time for which links are considered new
newtime = time.time() - SECS_PER_DAY * config.REPORT_WHATSNEW_URL_AGE
- # get all internal pages that are new
links = session.query(Link).filter_by(is_page=True, is_internal=True)
links = links.filter(Link.mtime > newtime).order_by(Link.mtime.desc())
- # present results
- fp = webcheck.plugins.open_html(webcheck.plugins.new, crawler)
- if not links.count():
- fp.write(
- ' <p class="description">\n'
- ' No pages were found that were modified within the last %(new)d
days.\n'
- ' </p>\n'
- % {'new': config.REPORT_WHATSNEW_URL_AGE})
- webcheck.plugins.close_html(fp)
- return
- fp.write(
- ' <p class="description">\n'
- ' These pages have been recently modified (within %(new)d days).\n'
- ' </p>\n'
- ' <ul>\n'
- % {'new': config.REPORT_WHATSNEW_URL_AGE})
- for link in links:
- age = (time.time() - link.mtime) / SECS_PER_DAY
- fp.write(
- ' <li>\n'
- ' %(link)s\n'
- ' <ul class="problems">\n'
- ' <li>age: %(age)d days</li>\n'
- ' </ul>\n'
- ' </li>\n'
- % {'link': webcheck.plugins.make_link(link),
- 'age': age})
- fp.write(' </ul>\n')
- webcheck.plugins.close_html(fp)
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links, now=time.time(), SECS_PER_DAY=SECS_PER_DAY)
session.close()
diff --git a/webcheck/plugins/notchkd.py b/webcheck/plugins/notchkd.py
index eecf025..d5b61b5 100644
--- a/webcheck/plugins/notchkd.py
+++ b/webcheck/plugins/notchkd.py
@@ -31,39 +31,14 @@ __outputfile__ = 'notchkd.html'
from sqlalchemy.orm import joinedload
from webcheck.db import Session, Link
-import webcheck.plugins
+from webcheck.output import render
def generate(crawler):
"""Output the list of not checked pages."""
session = Session()
- # get all yanked urls
links = session.query(Link).filter(Link.yanked != None).order_by(Link.url)
- # present results
- fp = webcheck.plugins.open_html(webcheck.plugins.notchkd, crawler)
- if not links.count():
- fp.write(
- ' <p class="description">\n'
- ' All links have been checked.\n'
- ' </p>\n')
- webcheck.plugins.close_html(fp)
- return
- fp.write(
- ' <p class="description">\n'
- ' This is the list of all urls that were encountered but not
checked\n'
- ' at all during the examination of the website.\n'
- ' </p>\n'
- ' <ol>\n')
- for link in links.options(joinedload(Link.linkproblems)):
- fp.write(
- ' <li>\n'
- ' %(link)s\n'
- % {'link': webcheck.plugins.make_link(link, link.url)})
- # present a list of parents
- webcheck.plugins.print_parents(fp, link, ' ')
- fp.write(
- ' </li>\n')
- fp.write(
- ' </ol>\n')
- webcheck.plugins.close_html(fp)
+ links = links.options(joinedload(Link.linkproblems))
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links)
session.close()
diff --git a/webcheck/plugins/notitles.py b/webcheck/plugins/notitles.py
index 605619a..f40cc13 100644
--- a/webcheck/plugins/notitles.py
+++ b/webcheck/plugins/notitles.py
@@ -31,7 +31,7 @@ __outputfile__ = 'notitles.html'
from sqlalchemy.sql.functions import char_length
from webcheck.db import Session, Link
-import webcheck.plugins
+from webcheck.output import render
def postprocess(crawler):
@@ -50,30 +50,9 @@ def postprocess(crawler):
def generate(crawler):
"""Output the list of pages without a title."""
session = Session()
- # get all internal pages without a title
links = session.query(Link).filter_by(is_page=True, is_internal=True)
links = links.filter((char_length(Link.title) == 0) |
(Link.title == None)).order_by(Link.url)
- # present results
- fp = webcheck.plugins.open_html(webcheck.plugins.notitles, crawler)
- if not links.count():
- fp.write(
- ' <p class="description">\n'
- ' All pages had a title specified.\n'
- ' </p>\n')
- webcheck.plugins.close_html(fp)
- return
- fp.write(
- ' <p class="description">\n'
- ' This is the list of all (internal) pages without a proper title\n'
- ' specified.\n'
- ' </p>\n'
- ' <ol>\n')
- for link in links:
- fp.write(
- ' <li>%(link)s</li>\n'
- % {'link': webcheck.plugins.make_link(link, link.url)})
- fp.write(
- ' </ol>\n')
- webcheck.plugins.close_html(fp)
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links)
session.close()
diff --git a/webcheck/plugins/old.py b/webcheck/plugins/old.py
index e061248..ada9682 100644
--- a/webcheck/plugins/old.py
+++ b/webcheck/plugins/old.py
@@ -30,9 +30,9 @@ __outputfile__ = 'old.html'
import time
-from webcheck.db import Session, Link
from webcheck import config
-import webcheck.plugins
+from webcheck.db import Session, Link
+from webcheck.output import render
SECS_PER_DAY = 60 * 60 * 24
@@ -41,40 +41,9 @@ SECS_PER_DAY = 60 * 60 * 24
def generate(crawler):
"""Output the list of outdated pages to the specified file descriptor."""
session = Session()
- # the time for which links are considered old
oldtime = time.time() - SECS_PER_DAY * config.REPORT_WHATSOLD_URL_AGE
- # get all internal pages that are old
links = session.query(Link).filter_by(is_page=True, is_internal=True)
links = links.filter(Link.mtime < oldtime).order_by(Link.mtime)
- # present results
- fp = webcheck.plugins.open_html(webcheck.plugins.old, crawler)
- if not links.count():
- fp.write(
- ' <p class="description">\n'
- ' No pages were found that were older than %(old)d days old.\n'
- ' </p>\n'
- % {'old': config.REPORT_WHATSOLD_URL_AGE})
- webcheck.plugins.close_html(fp)
- return
- fp.write(
- ' <p class="description">\n'
- ' These pages have been modified a long time ago (older than
%(old)d\n'
- ' days) and may be outdated.\n'
- ' </p>\n'
- ' <ul>\n'
- % {'old': config.REPORT_WHATSOLD_URL_AGE})
- for link in links:
- age = (time.time() - link.mtime) / SECS_PER_DAY
- fp.write(
- ' <li>\n'
- ' %(link)s\n'
- ' <ul class="problems">\n'
- ' <li>age: %(age)d days</li>\n'
- ' </ul>\n'
- ' </li>\n'
- % {'link': webcheck.plugins.make_link(link),
- 'age': age})
- fp.write(
- ' </ul>\n')
- webcheck.plugins.close_html(fp)
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links, now=time.time(), SECS_PER_DAY=SECS_PER_DAY)
session.close()
diff --git a/webcheck/plugins/problems.py b/webcheck/plugins/problems.py
index 19f71d2..ba1cb1a 100644
--- a/webcheck/plugins/problems.py
+++ b/webcheck/plugins/problems.py
@@ -28,21 +28,19 @@ __title__ = 'problems by author'
__author__ = 'Arthur de Jong'
__outputfile__ = 'problems.html'
+import collections
+import re
+
from webcheck.db import Session, Link
-import webcheck.plugins
+from webcheck.output import render
-def _mk_id(name):
- """Convert the name to a string that may be used inside an
- ID attribute."""
- # convert to lowercase first
+def mk_id(name):
+ """Convert the name to a string that may be used inside an ID
+ attribute."""
name = name.lower()
- import re
- # strip any leading non alpha characters
name = re.sub('^[^a-z]*', '', name)
- # remove any non-allowed characters
name = re.sub('[^a-z0-9_:.]+', '-', name)
- # we're done
return name
@@ -50,79 +48,17 @@ def generate(crawler):
"""Output the overview of problems per author."""
session = Session()
# make a list of problems per author
- problem_db = {}
+ problem_db = collections.defaultdict(list)
# get internal links with page problems
links = session.query(Link).filter_by(is_internal=True)
links = links.filter(Link.pageproblems.any()).order_by(Link.url)
for link in links:
- # make a normal name for the author
- if link.author:
- author = link.author.strip()
- else:
- author = unicode('Unknown')
- # store the problem
- if author in problem_db:
- problem_db[author].append(link)
- else:
- problem_db[author] = [link]
- fp = webcheck.plugins.open_html(webcheck.plugins.problems, crawler)
- if not problem_db:
- fp.write(
- ' <p class="description">\n'
- ' No problems were found on this site, hurray.\n'
- ' </p>\n')
- webcheck.plugins.close_html(fp)
- return
- # print description
- fp.write(
- ' <p class="description">\n'
- ' This is an overview of all the problems on the site, grouped by\n'
- ' author.\n'
- ' </p>\n')
- # get a list of authors
+ author = link.author.strip() if link.author else u'Unknown'
+ problem_db[author].append(link)
+ # get a sorted list of authors
authors = problem_db.keys()
authors.sort()
- # generate short list of authors
- if len(authors) > 1:
- fp.write(' <ul class="authorlist">\n')
- for author in authors:
- fp.write(
- ' <li><a href="#author_%(authorref)s">Author:
%(author)s</a></li>\n'
- % {'authorref': webcheck.plugins.htmlescape(_mk_id(author)),
- 'author': webcheck.plugins.htmlescape(author)})
- fp.write(' </ul>\n')
- # generate problem report
- fp.write(' <ul>\n')
- for author in authors:
- fp.write(
- ' <li id="author_%(authorref)s">\n'
- ' Author: %(author)s\n'
- ' <ul>\n'
- % {'authorref': webcheck.plugins.htmlescape(_mk_id(author)),
- 'author': webcheck.plugins.htmlescape(author)})
- # sort pages by url
- problem_db[author].sort(lambda a, b: cmp(a.url, b.url))
- # list problems for this author
- for link in problem_db[author]:
- # present the links
- fp.write(
- ' <li>\n'
- ' %(link)s\n'
- ' <ul class="problems">\n'
- % {'link': webcheck.plugins.make_link(link)})
- # list the problems
- for problem in link.pageproblems:
- fp.write(
- ' <li>%(problem)s</li>\n'
- % {'problem': webcheck.plugins.htmlescape(problem)})
- # end the list item
- fp.write(
- ' </ul>\n'
- ' </li>\n')
- fp.write(
- ' </ul>\n'
- ' </li>\n')
- fp.write(
- ' </ul>\n')
- webcheck.plugins.close_html(fp)
+ authors = [(x, problem_db[x]) for x in authors]
+ render(__outputfile__, crawler=crawler, title=__title__,
+ authors=authors, mk_id=mk_id)
session.close()
diff --git a/webcheck/plugins/sitemap.py b/webcheck/plugins/sitemap.py
index da94460..55bff59 100644
--- a/webcheck/plugins/sitemap.py
+++ b/webcheck/plugins/sitemap.py
@@ -30,66 +30,47 @@ __outputfile__ = 'index.html'
from webcheck import config
from webcheck.db import Link
-import webcheck.plugins
+from webcheck.output import render
-def add_pagechildren(link, children, explored):
+def get_children(link, explored):
"""Determine the page children of this link, combining the children of
embedded items and following redirects."""
# get all internal children
qry = link.children.filter(Link.is_internal == True)
if link.depth:
qry = qry.filter((Link.depth > link.depth) | (Link.depth == None))
- # follow redirects
- children.update(y
- for y in (x.follow_link() for x in qry)
- if y and y.is_page and y.is_internal and y.id not in
explored)
- explored.update(x.id for x in children)
+ # follow redirects and return all direct children
+ for child in (x.follow_link() for x in qry):
+ if child and child.is_page and child.is_internal and child.id not in
explored:
+ explored.add(child.id)
+ yield child
# add embedded element's pagechildren (think frames)
for embed in link.embedded.filter(Link.is_internal ==
True).filter(Link.is_page == True):
- # TODO: put this in a query
if embed.id not in explored and \
(embed.depth == None or embed.depth > link.depth):
- add_pagechildren(embed, children, explored)
+ for child in get_children(embed, explored):
+ yield child
-def _explore(fp, link, explored, depth=0, indent=' '):
+def explore(links, explored=None, depth=0):
"""Recursively do a breadth first traversal of the graph of links on the
- site. Prints the html results to the file descriptor."""
- # output this link
- fp.write(indent + '<li>\n')
- fp.write(indent + ' ' + webcheck.plugins.make_link(link) + '\n')
- # only check children if we are not too deep yet
- if depth <= config.REPORT_SITEMAP_LEVEL:
- # figure out the links to follow and ensure that they are only
- # explored from here
- children = set()
- add_pagechildren(link, children, explored)
- # remove None which could be there as a result of follow_link()
- children.discard(None)
- if children:
- children = list(children)
- # present children as a list
- fp.write(indent + ' <ul>\n')
+ site."""
+ if explored is None:
+ explored = set(x.id for x in links)
+ for link in links:
+ children = []
+ if depth <= config.REPORT_SITEMAP_LEVEL:
+ children = list(get_children(link, explored))
children.sort(lambda a, b: cmp(a.url, b.url))
- for child in children:
- _explore(fp, child, explored, depth + 1, indent + ' ')
- fp.write(indent + ' </ul>\n')
- fp.write(indent + '</li>\n')
+ if children:
+ yield link, explore(children, explored, depth + 1)
+ else:
+ yield link, None
def generate(crawler):
"""Output the sitemap."""
- fp = webcheck.plugins.open_html(webcheck.plugins.sitemap, crawler)
- # output the site structure using breadth first traversal
- fp.write(
- ' <p class="description">\n'
- ' This an overview of the crawled site.\n'
- ' </p>\n'
- ' <ul>\n')
- explored = set(x.id for x in crawler.bases)
- for l in crawler.bases:
- _explore(fp, l, explored)
- fp.write(
- ' </ul>\n')
- webcheck.plugins.close_html(fp)
+ links = explore(crawler.bases)
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links)
diff --git a/webcheck/plugins/size.py b/webcheck/plugins/size.py
index 2d5b570..7f6703c 100644
--- a/webcheck/plugins/size.py
+++ b/webcheck/plugins/size.py
@@ -28,12 +28,12 @@ __title__ = "what's big"
__author__ = 'Arthur de Jong'
__outputfile__ = 'size.html'
-from webcheck.db import Session, Link
from webcheck import config
-import webcheck.plugins
+from webcheck.db import Session, Link
+from webcheck.output import render
-def _getsize(link, seen=None):
+def get_size(link, seen=None):
"""Return the size of the link and all its embedded links, counting each
link only once."""
# make a new list
@@ -48,7 +48,7 @@ def _getsize(link, seen=None):
# add sizes of embedded objects
for embed in link.embedded:
if embed not in seen:
- size += _getsize(embed, seen)
+ size += get_size(embed, seen)
link.total_size = size
return link.total_size
@@ -56,41 +56,10 @@ def _getsize(link, seen=None):
def generate(crawler):
"""Output the list of large pages."""
session = Session()
- # get all internal pages and get big links
links = session.query(Link).filter_by(is_page=True, is_internal=True)
links = [x for x in links
- if _getsize(x) >= config.REPORT_SLOW_URL_SIZE * 1024]
- # sort links by size (biggest first)
+ if get_size(x) >= config.REPORT_SLOW_URL_SIZE * 1024]
links.sort(lambda a, b: cmp(b.total_size, a.total_size))
- # present results
- fp = webcheck.plugins.open_html(webcheck.plugins.size, crawler)
- if not links:
- fp.write(
- ' <p class="description">\n'
- ' No pages over %(size)dK were found.\n'
- ' </p>\n'
- % {'size': config.REPORT_SLOW_URL_SIZE})
- webcheck.plugins.close_html(fp)
- return
- fp.write(
- ' <p class="description">\n'
- ' These pages are probably too big (over %(size)dK) which could be\n'
- ' slow to download.\n'
- ' </p>\n'
- ' <ul>\n'
- % {'size': config.REPORT_SLOW_URL_SIZE})
- for link in links:
- size = webcheck.plugins.get_size(link.total_size)
- fp.write(
- ' <li>\n'
- ' %(link)s\n'
- ' <ul class="problem">\n'
- ' <li>size: %(size)s</li>\n'
- ' </ul>\n'
- ' </li>\n'
- % {'link': webcheck.plugins.make_link(link),
- 'size': size})
- fp.write(
- ' </ul>\n')
- webcheck.plugins.close_html(fp)
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links)
session.close()
diff --git a/webcheck/plugins/urllist.py b/webcheck/plugins/urllist.py
index d3ae8cf..2999f31 100644
--- a/webcheck/plugins/urllist.py
+++ b/webcheck/plugins/urllist.py
@@ -27,24 +27,13 @@ __author__ = 'Arthur de Jong'
__outputfile__ = 'urllist.html'
from webcheck.db import Session, Link
-import webcheck.plugins
+from webcheck.output import render
def generate(crawler):
"""Output a sorted list of URLs."""
session = Session()
- fp = webcheck.plugins.open_html(webcheck.plugins.urllist, crawler)
- fp.write(
- ' <p class="description">\n'
- ' This is the list of all urls encountered during the examination
of\n'
- ' the website. It lists internal as well as external and\n'
- ' non-examined urls.\n'
- ' </p>\n'
- ' <ol>\n')
links = session.query(Link).order_by(Link.url)
- for link in links:
- fp.write(' <li>' + webcheck.plugins.make_link(link, link.url) +
'</li>\n')
- fp.write(
- ' </ol>\n')
- webcheck.plugins.close_html(fp)
+ render(__outputfile__, crawler=crawler, title=__title__,
+ links=links)
session.close()
diff --git a/webcheck/templates/about.html b/webcheck/templates/about.html
new file mode 100644
index 0000000..cee0830
--- /dev/null
+++ b/webcheck/templates/about.html
@@ -0,0 +1,87 @@
+{#
+ # about.html - template for webcheck about plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% block content %}
+ <p>
+ This is a website report generated by <tt>webcheck</tt> {{
webcheck.__version__ }}.
+ <tt>webcheck</tt> is a website checking tool for webmasters. It crawls a
+ given website and does a number of tests to see if links and pages are
+ valid.
+ More information about <tt>webcheck</tt> can be found at the
+ <tt>webcheck</tt> homepage which is located at
+ <a href="{{ webcheck.__homepage__ }}">{{ webcheck.__homepage__ }}</a>.
+ </p>
+ <p>
+ This report was generated on {{ time }}, a total of {{ numlinks }}
+ links were found.
+ </p>
+
+ <h3>Copyright</h3>
+ <p>
+ <tt>webcheck</tt> was originally named <tt>linbot</tt> which was
+ developed by Albert Hopkins (marduk).
+ Versions up till 1.0 were maintained by Mike W. Meyer who changed the
+ name to <tt>webcheck</tt>.
+ After that Arthur de Jong did a complete rewrite.
+ <tt>webcheck</tt> is <i>free software</i>; you can redistribute it
+ and/or modify it under the terms of the
+ <a href="http://www.gnu.org/copyleft/gpl.html";>GNU General Public
License</a>
+ (version 2 or later).
+ There is no warranty; not even for merchantability or fitness for a
+ particular purpose. See the source for further details.
+ </p>
+ <p>
+ Copyright © 1998-2013 Albert Hopkins (marduk),
+ Mike W. Meyer and Arthur de Jong
+ </p>
+ <p>
+ The files in this generated report do not automatically fall under
+ the copyright of the software, unless explicitly stated otherwise.
+ </p>
+ <p>
+ <tt>webcheck</tt> includes the
+ <a href="http://victr.lm85.com/projects/fancytooltips/";>FancyTooltips</a>
+ javascript library to display readable tooltips. FancyTooltips is
+ distributed under the MIT license and has the following copyright
+ notices (see <tt>fancytooltips.js</tt> for details):
+ </p>
+ <p>
+ Copyright © 2003-2005 Stuart Langridge, Paul McLanahan,
+ Peter Janes, Brad Choate, Dunstan Orchard, Ethan Marcotte,
+ Mark Wubben and Victor Kulinski
+ </p>
+
+ <h3>Plugins</h3>
+ <ul>
+ {% for plugin in crawler.plugins %}
+ <li>
+ <strong>{{ plugin.__title__ }}</strong><br />
+ {% if plugin.__doc__ %}
+ {{ plugin.__doc__ }}<br />
+ {% endif %}
+ </li>
+ {% endfor %}
+ </ul>
+{% endblock %}
diff --git a/webcheck/templates/badlinks.html b/webcheck/templates/badlinks.html
new file mode 100644
index 0000000..919b717
--- /dev/null
+++ b/webcheck/templates/badlinks.html
@@ -0,0 +1,51 @@
+{#
+ # badlinks.html - template for webcheck bad links plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% from 'macros.html' import make_link, link_parents with context %}
+
+{% block content %}
+ {% if not links.count() %}
+ <p class="description">
+ There were no problems retrieving links from the website.
+ </p>
+ {% else %}
+ <p class="description">
+ These links could not be retrieved during the crawling of the website.
+ </p>
+ <ol>
+ {% for link in links %}
+ <li>
+ {{ make_link(link, link.url) }}
+ <ul class="problems">
+ {% for problem in link.linkproblems %}
+ <li>{{ problem }}</li>
+ {% endfor %}
+ </ul>
+ {{ link_parents(link) }}
+ </li>
+ {% endfor %}
+ </ol>
+ {% endif %}
+{% endblock %}
diff --git a/webcheck/templates/external.html b/webcheck/templates/external.html
new file mode 100644
index 0000000..8276017
--- /dev/null
+++ b/webcheck/templates/external.html
@@ -0,0 +1,47 @@
+{#
+ # external.html - template for webcheck external links plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% from 'macros.html' import make_link, link_parents with context %}
+
+{% block content %}
+ {% if not links.count() %}
+ <p class="description">
+ No external links were found on the website.
+ </p>
+ {% else %}
+ <p class="description">
+ This is the list of all external urls encountered during the
+ examination of the website.
+ </p>
+ <ol>
+ {% for link in links %}
+ <li>
+ {{ make_link(link) }}
+ {{ link_parents(link) }}
+ </li>
+ {% endfor %}
+ </ol>
+ {% endif %}
+{% endblock %}
diff --git a/webcheck/templates/images.html b/webcheck/templates/images.html
new file mode 100644
index 0000000..0b195a4
--- /dev/null
+++ b/webcheck/templates/images.html
@@ -0,0 +1,46 @@
+{#
+ # images.html - template for webcheck images plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% from 'macros.html' import make_link, link_parents with context %}
+
+{% block content %}
+ {% if not links.count() %}
+ <p class="description">
+ No images were linked on the website.
+ </p>
+ {% else %}
+ <p class="description">
+ This is the list of all images found linked on the website.
+ </p>
+ <ol>
+ {% for link in links %}
+ <li>
+ {{ make_link(link, link.url) }}
+ {{ link_parents(link) }}
+ </li>
+ {% endfor %}
+ </ol>
+ {% endif %}
+{% endblock %}
diff --git a/webcheck/templates/index.html b/webcheck/templates/index.html
new file mode 100644
index 0000000..9d54512
--- /dev/null
+++ b/webcheck/templates/index.html
@@ -0,0 +1,41 @@
+{#
+ # index.html - template for webcheck sitemap plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% block content %}
+ <p class="description">
+ This an overview of the crawled site.
+ </p>
+ <ul>
+ {% for link, children in links recursive %}
+ <li>
+ {{ make_link(link) }}
+ {% if children %}
+ <ul>
+{{ loop(children) }} </ul>
+ {% endif %}
+ </li>
+ {% endfor %}
+ </ul>
+{% endblock %}
diff --git a/webcheck/templates/new.html b/webcheck/templates/new.html
new file mode 100644
index 0000000..fcbe594
--- /dev/null
+++ b/webcheck/templates/new.html
@@ -0,0 +1,50 @@
+{#
+ # new.html - template for webcheck recently modified pages plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% from 'macros.html' import make_link with context %}
+
+{% block content %}
+ {% if not links.count() %}
+ <p class="description">
+ No pages were found that were modified within the last
+ {{ config.REPORT_WHATSNEW_URL_AGE }} days.
+ </p>
+ {% else %}
+ <p class="description">
+ These pages have been recently modified (within
+ {{ config.REPORT_WHATSNEW_URL_AGE }} days).
+ </p>
+ <ul>
+ {% for link in links %}
+ <li>
+ {{ make_link(link) }}
+ <ul class="problems">
+ <li>age: {{ (now - link.mtime) / SECS_PER_DAY }} days</li>
+ </ul>
+ </li>
+ {% endfor %}
+ </ul>
+ {% endif %}
+{% endblock %}
diff --git a/webcheck/templates/notchkd.html b/webcheck/templates/notchkd.html
new file mode 100644
index 0000000..a633aa2
--- /dev/null
+++ b/webcheck/templates/notchkd.html
@@ -0,0 +1,47 @@
+{#
+ # notchkd.html - template for webcheck unfollowed links plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% from 'macros.html' import make_link, link_parents with context %}
+
+{% block content %}
+ {% if not links.count() %}
+ <p class="description">
+ All links have been checked.
+ </p>
+ {% else %}
+ <p class="description">
+ This is the list of all urls that were encountered but not checked
+ at all during the examination of the website.
+ </p>
+ <ol>
+ {% for link in links %}
+ <li>
+ {{ make_link(link) }}
+ {{ link_parents(link) }}
+ </li>
+ {% endfor %}
+ </ol>
+ {% endif %}
+{% endblock %}
diff --git a/webcheck/templates/notitles.html b/webcheck/templates/notitles.html
new file mode 100644
index 0000000..46e301a
--- /dev/null
+++ b/webcheck/templates/notitles.html
@@ -0,0 +1,46 @@
+{#
+ # notitles.html - template for webcheck pages without titles plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% from 'macros.html' import make_link with context %}
+
+{% block content %}
+ {% if not links.count() %}
+ <p class="description">
+ All pages had a title specified.
+ </p>
+ {% else %}
+ <p class="description">
+ This is the list of all (internal) pages without a proper title
+ specified.
+ </p>
+ <ol>
+ {% for link in links %}
+ <li>
+ {{ make_link(link, link.url) }}
+ </li>
+ {% endfor %}
+ </ol>
+ {% endif %}
+{% endblock %}
diff --git a/webcheck/templates/old.html b/webcheck/templates/old.html
new file mode 100644
index 0000000..8915ee5
--- /dev/null
+++ b/webcheck/templates/old.html
@@ -0,0 +1,50 @@
+{#
+ # old.html - template for webcheck old pages plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% from 'macros.html' import make_link with context %}
+
+{% block content %}
+ {% if not links.count() %}
+ <p class="description">
+ No pages were found that were older than
+ {{ config.REPORT_WHATSOLD_URL_AGE }} days.
+ </p>
+ {% else %}
+ <p class="description">
+ These pages have been modified a long time ago (older than
+ {{ config.REPORT_WHATSOLD_URL_AGE }} days) and could be outdated.
+ </p>
+ <ul>
+ {% for link in links %}
+ <li>
+ {{ make_link(link) }}
+ <ul class="problems">
+ <li>age: {{ (now - link.mtime) / SECS_PER_DAY }} days</li>
+ </ul>
+ </li>
+ {% endfor %}
+ </ul>
+ {% endif %}
+{% endblock %}
diff --git a/webcheck/templates/problems.html b/webcheck/templates/problems.html
new file mode 100644
index 0000000..d184356
--- /dev/null
+++ b/webcheck/templates/problems.html
@@ -0,0 +1,39 @@
+{% extends 'base.html' %}
+
+{% block content %}
+ {% if not authors %}
+ <p class="description">
+ No problems were found on this site, hurray.
+ </p>
+ {% else %}
+ <p class="description">
+ This is an overview of all the problems on the site, grouped by author.
+ </p>
+ {# index of authors #}
+ {% if authors|length > 1 %}
+ <ul class="authorlist">
+ {% for author, links in authors %}
+ <li><a href="#author_{{ mk_id(author) }}">Author: {{ author
}}</a></li>
+ {% endfor %}
+ </ul>
+ {% endif %}
+ <ul>
+ {% for author, links in authors %}
+ <li id="author_{{ mk_id(author) }}">
+ Author: {{ author }}
+ <ul>
+ {% for link in links %}
+ <li>
+ {{ make_link(link) }}
+ <ul class="problems">
+ {% for problem in link.pageproblems %}
+ <li>{{ problem }}</li>
+ {% endfor %}
+ </ul>
+ </li>
+ {% endfor %}
+ </ul>
+ {% endfor %}
+ </ul>
+ {% endif %}
+{% endblock %}
diff --git a/webcheck/templates/size.html b/webcheck/templates/size.html
new file mode 100644
index 0000000..14ac810
--- /dev/null
+++ b/webcheck/templates/size.html
@@ -0,0 +1,49 @@
+{#
+ # size.html - template for webcheck big pages plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% from 'macros.html' import make_link with context %}
+
+{% block content %}
+ {% if not links %}
+ <p class="description">
+ No pages over {{ config.REPORT_SLOW_URL_SIZE }}K were found.
+ </p>
+ {% else %}
+ <p class="description">
+ These pages are probably too big (over
+ {{ config.REPORT_SLOW_URL_SIZE }}K) which could be slow to download.
+ </p>
+ <ul>
+ {% for link in links %}
+ <li>
+ {{ make_link(link) }}
+ <ul class="problems">
+ <li>size: {{ link.total_size|filesizeformat(binary=True) }}</li>
+ </ul>
+ </li>
+ {% endfor %}
+ </ul>
+ {% endif %}
+{% endblock %}
diff --git a/webcheck/templates/urllist.html b/webcheck/templates/urllist.html
new file mode 100644
index 0000000..0c7f1cf
--- /dev/null
+++ b/webcheck/templates/urllist.html
@@ -0,0 +1,41 @@
+{#
+ # urllist.html - template for webcheck url list plugin
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{% extends 'base.html' %}
+
+{% from 'macros.html' import make_link with context %}
+
+{% block content %}
+ <p class="description">
+ This is the list of all urls encountered during the examination of
+ the website. It lists internal as well as external and
+ non-examined urls.
+ </p>
+ <ol>
+ {% for link in links %}
+ <li>
+ {{ make_link(link, link.url) }}
+ </li>
+ {% endfor %}
+ </ol>
+{% endblock %}
http://arthurdejong.org/git/webcheck/commit/?id=c5ea12def1487fb8d34018e5d6545d8e449fdd96
commit c5ea12def1487fb8d34018e5d6545d8e449fdd96
Author: Arthur de Jong <arthur@arthurdejong.org>
Date: Sun Sep 22 17:14:07 2013 +0200
Introduce template macros for rendering links
diff --git a/webcheck/output.py b/webcheck/output.py
index b0441d0..356cf30 100644
--- a/webcheck/output.py
+++ b/webcheck/output.py
@@ -135,6 +135,8 @@ def render(output_file, **kwargs):
crawler = kwargs.get('crawler', None)
if crawler:
kwargs.setdefault('sitename', crawler.bases[0].title or
crawler.bases[0].url)
+ kwargs.setdefault('Link', Link)
+ kwargs.setdefault('config', config)
template = env.get_template(output_file)
fp = open_file(output_file)
fp.write(template.render(**kwargs))
diff --git a/webcheck/templates/macros.html b/webcheck/templates/macros.html
new file mode 100644
index 0000000..f1f550d
--- /dev/null
+++ b/webcheck/templates/macros.html
@@ -0,0 +1,84 @@
+{#
+ # macros.html - macros that are used in other templates
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+
+{# output a space separated list of CSS classes for the provided link #}
+{% macro link_css_classes(link) -%}
+ {% if link.is_internal %}internal{% else %}external{% endif %}
+{%- endmacro %}
+
+{# output an overview of the link information #}
+{% macro link_info(link, separator=' ') -%}
+ url: {{ link.url }}{{ separator }}
+ {%- if link.status %}{{ link.status }}{{ separator }}{% endif -%}
+ {%- if link.title %}title: {{ link.title|trim }}{{ separator }}{% endif -%}
+ {%- if link.author %}author: {{ link.author|trim }}{{ separator }}{% endif
-%}
+ {%- if link.is_internal %}internal link{% else %}external link{% endif -%}
+ {%- if link.yanked %}, not checked ({{ link.yanked }}){% endif -%}{{
separator }}
+ {%- if link.redirectdepth -%}
+ {%- if link.children.count() > 0 -%}
+ redirect: {{ link.children.first().url }}
+ {%- else -%}
+ redirect (not followed)
+ {%- endif -%}{{ separator }}
+ {%- endif -%}
+ {%- set count = link.count_parents -%}
+ {%- if count == 1 -%}
+ linked from 1 page{{ separator }}
+ {%- elif count > 1 -%}
+ linked from {{ count }} pages{{ separator }}
+ {%- endif -%}
+ {%- if link.mtime %}last modified: {{ time.ctime(link.mtime) }}{{ separator
}}{% endif -%}
+ {%- if link.size %}size: {{ link.size|filesizeformat(binary=True) }}{{
separator }}{% endif -%}
+ {%- if link.mimetype %}mime-type: {{ link.mimetype }}{{ separator }}{% endif
-%}
+ {%- if link.encoding %}encoding: {{ link.encoding }}{{ separator }}{% endif
-%}
+ {%- for problem in link.linkproblems -%}
+ problem: {{ problem.message }}{{ separator }}
+ {%- endfor -%}
+{%- endmacro %}
+
+{# render a link embedded in an <a> with a title and information #}
+{% macro make_link(link, title=None) -%}
+ <a href="{{ link.url }}" class="{{ link_css_classes(link) }}"
+ {%- if config.REPORT_LINKS_IN_NEW_WINDOW %} target="_blank"{%- endif %}
+ title="{{ link_info(link) }}">{{ title or link.title or link.url }}</a>
+{%- endmacro %}
+
+{# return a <div> containing a list of parent links #}
+{% macro link_parents(link) %}
+ {% set count = link.count_parents %}
+ {% if count %}
+ {% set parents = link.parents.order_by(Link.title,
Link.url)[:config.PARENT_LISTLEN] %}
+ <div class="parents">
+ referenced from:
+ <ul>
+ {% for parent in parents %}
+ <li>{{ make_link(parent) }}</li>
+ {% endfor %}
+ {% set more = count - parents|length %}
+ {% if more %}
+ <li>{{ more }} more...</li>
+ {% endif %}
+ </ul>
+ </div>
+ {% endif %}
+{% endmacro %}
http://arthurdejong.org/git/webcheck/commit/?id=bdca8863fc3b565978f8fd560c1726496aab0379
commit bdca8863fc3b565978f8fd560c1726496aab0379
Author: Arthur de Jong <arthur@arthurdejong.org>
Date: Sun Sep 22 14:51:45 2013 +0200
Introduce a base template
This sets up the basic layout for the report. The plugins are expected
to supply a crawler instance.
diff --git a/webcheck/output.py b/webcheck/output.py
index d49ed21..b0441d0 100644
--- a/webcheck/output.py
+++ b/webcheck/output.py
@@ -129,6 +129,12 @@ env = jinja2.Environment(
def render(output_file, **kwargs):
"""Render the output file with the specified context variables."""
+ kwargs.setdefault('webcheck', webcheck)
+ kwargs.setdefault('output_file', output_file)
+ kwargs.setdefault('time', time.ctime(time.time()))
+ crawler = kwargs.get('crawler', None)
+ if crawler:
+ kwargs.setdefault('sitename', crawler.bases[0].title or
crawler.bases[0].url)
template = env.get_template(output_file)
fp = open_file(output_file)
fp.write(template.render(**kwargs))
diff --git a/webcheck/templates/base.html b/webcheck/templates/base.html
new file mode 100644
index 0000000..0cdb2d1
--- /dev/null
+++ b/webcheck/templates/base.html
@@ -0,0 +1,58 @@
+{#
+ # base.html - base template for webcheck reports
+ #
+ # Copyright (C) 2013 Arthur de Jong
+ #
+ # This program is free software; you can redistribute it and/or modify
+ # it under the terms of the GNU General Public License as published by
+ # the Free Software Foundation; either version 2 of the License, or
+ # (at your option) any later version.
+ #
+ # This program is distributed in the hope that it will be useful,
+ # but WITHOUT ANY WARRANTY; without even the implied warranty of
+ # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ # GNU General Public License for more details.
+ #
+ # You should have received a copy of the GNU General Public License
+ # along with this program; if not, write to the Free Software
+ # Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+ #
+ # The files produced as output from the software do not automatically fall
+ # under the copyright of the software, unless explicitly stated otherwise.
+ #}
+<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.1//EN"
"http://www.w3.org/TR/xhtml11/DTD/xhtml11.dtd";>
+
+<html xmlns="http://www.w3.org/1999/xhtml";>
+ <head>
+ <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
+ <title>Webcheck report for {{ sitename }} ({{ title }})</title>
+ <link rel="stylesheet" type="text/css" href="webcheck.css" />
+ <link rel="icon" href="favicon.ico" type="image/ico" />
+ <link rel="shortcut icon" href="favicon.ico" />
+ <script type="text/javascript" src="fancytooltips.js"></script>
+ <meta name="Generator" content="webcheck {{ webcheck.__version__ }}" />
+ </head>
+ <body>
+ <h1 class="basename">Webcheck report for {{ sitename }}</a></h1>
+
+ <ul class="navbar">
+ {% for plugin in crawler.plugins %}
+ {% if plugin.__outputfile__ %}
+ <li><a href="{{ plugin.__outputfile__ }}"{% if plugin.__outputfile__
== output_file %} class="selected"{% endif %} title="{{ plugin.__doc__ }}">{{
plugin.__title__ }}</a></li>
+ {% endif %}
+ {% endfor %}
+ </ul>
+
+ <h2>{{ title }}</h2>
+
+ <div class="content">
+ {% block content %}
+ {% endblock %}
+ </div>
+
+ <p class="footer">
+ Generated {{ time }} by <a href="{{ webcheck.__homepage__ }}">webcheck
{{ webcheck.__version__ }}</a>
+ </p>
+ </body>
+</html>
http://arthurdejong.org/git/webcheck/commit/?id=58dcfbf3ff7152016bf3906bbf5805e097a1ee0e
commit 58dcfbf3ff7152016bf3906bbf5805e097a1ee0e
Author: Arthur de Jong <arthur@arthurdejong.org>
Date: Sun Sep 22 14:33:10 2013 +0200
Provide function for template-based report rendering
This uses the Jinja template engine to produce the report HTML files.
This also renames the util module to output to better describe its
purpose.
diff --git a/webcheck/crawler.py b/webcheck/crawler.py
index 0c1a2da..abd1828 100644
--- a/webcheck/crawler.py
+++ b/webcheck/crawler.py
@@ -41,7 +41,7 @@ import urlparse
from webcheck import config
from webcheck.db import Session, Base, Link, truncate_db
-from webcheck.util import install_file
+from webcheck.output import install_file
import webcheck.parsers
from sqlalchemy import create_engine
diff --git a/webcheck/util.py b/webcheck/output.py
similarity index 87%
rename from webcheck/util.py
rename to webcheck/output.py
index 911e3a2..d49ed21 100644
--- a/webcheck/util.py
+++ b/webcheck/output.py
@@ -1,5 +1,5 @@
-# util.py - utility functions for webcheck
+# output.py - utility functions for webcheck
#
# Copyright (C) 1998, 1999 Albert Hopkins (marduk)
# Copyright (C) 2002 Mike W. Meyer
@@ -22,15 +22,22 @@
# The files produced as output from the software do not automatically fall
# under the copyright of the software, unless explicitly stated otherwise.
+"""Utility functions for generating the report."""
+
import codecs
import logging
import os
import shutil
import sys
+import time
import urllib
import urlparse
+import jinja2
+
from webcheck import config
+from webcheck.db import Link
+import webcheck
logger = logging.getLogger(__name__)
@@ -111,3 +118,18 @@ def install_file(source, is_text=False):
# close files
tfp.close()
sfp.close()
+
+
+env = jinja2.Environment(
+ loader=jinja2.PackageLoader('webcheck'),
+ extensions=['jinja2.ext.autoescape'],
+ autoescape=True,
+ trim_blocks=True, lstrip_blocks=True, keep_trailing_newline=True)
+
+
+def render(output_file, **kwargs):
+ """Render the output file with the specified context variables."""
+ template = env.get_template(output_file)
+ fp = open_file(output_file)
+ fp.write(template.render(**kwargs))
+ fp.close()
diff --git a/webcheck/plugins/__init__.py b/webcheck/plugins/__init__.py
index 4c534fd..4d032a4 100644
--- a/webcheck/plugins/__init__.py
+++ b/webcheck/plugins/__init__.py
@@ -51,7 +51,7 @@ import webcheck
from webcheck import config
from webcheck.db import Link
from webcheck.parsers.html import htmlescape
-from webcheck.util import open_file
+from webcheck.output import open_file
def _floatformat(f):
-----------------------------------------------------------------------
Summary of changes:
webcheck/crawler.py | 2 +-
webcheck/{util.py => output.py} | 32 +++++-
webcheck/parsers/__init__.py | 8 --
webcheck/parsers/html/__init__.py | 22 ----
webcheck/plugins/__init__.py | 229 -------------------------------------
webcheck/plugins/about.py | 80 +------------
webcheck/plugins/badlinks.py | 43 +------
webcheck/plugins/external.py | 33 +-----
webcheck/plugins/images.py | 24 +---
webcheck/plugins/new.py | 35 +-----
webcheck/plugins/notchkd.py | 33 +-----
webcheck/plugins/notitles.py | 27 +----
webcheck/plugins/old.py | 39 +------
webcheck/plugins/problems.py | 92 +++------------
webcheck/plugins/sitemap.py | 67 ++++-------
webcheck/plugins/size.py | 45 ++------
webcheck/plugins/urllist.py | 17 +--
webcheck/templates/about.html | 87 ++++++++++++++
webcheck/templates/badlinks.html | 51 +++++++++
webcheck/templates/base.html | 58 ++++++++++
webcheck/templates/external.html | 47 ++++++++
webcheck/templates/images.html | 46 ++++++++
webcheck/templates/index.html | 41 +++++++
webcheck/templates/macros.html | 84 ++++++++++++++
webcheck/templates/new.html | 50 ++++++++
webcheck/templates/notchkd.html | 47 ++++++++
webcheck/templates/notitles.html | 46 ++++++++
webcheck/templates/old.html | 50 ++++++++
webcheck/templates/problems.html | 39 +++++++
webcheck/templates/size.html | 49 ++++++++
webcheck/templates/urllist.html | 41 +++++++
31 files changed, 845 insertions(+), 719 deletions(-)
rename webcheck/{util.py => output.py} (81%)
create mode 100644 webcheck/templates/about.html
create mode 100644 webcheck/templates/badlinks.html
create mode 100644 webcheck/templates/base.html
create mode 100644 webcheck/templates/external.html
create mode 100644 webcheck/templates/images.html
create mode 100644 webcheck/templates/index.html
create mode 100644 webcheck/templates/macros.html
create mode 100644 webcheck/templates/new.html
create mode 100644 webcheck/templates/notchkd.html
create mode 100644 webcheck/templates/notitles.html
create mode 100644 webcheck/templates/old.html
create mode 100644 webcheck/templates/problems.html
create mode 100644 webcheck/templates/size.html
create mode 100644 webcheck/templates/urllist.html
hooks/post-receive
--
webcheck
--
To unsubscribe send an email to
webcheck-commits-unsubscribe@lists.arthurdejong.org or see
http://lists.arthurdejong.org/webcheck-commits/
- webcheck branch master updated. 1.10.4-72-g8de60b5,
Commits of the webcheck project
