PageRenderTime 50ms CodeModel.GetById 20ms RepoModel.GetById 1ms app.codeStats 0ms

/bin/compile.py

https://github.com/imrehg/website
Python | 927 lines | 774 code | 17 blank | 136 comment | 23 complexity | d317d2c22935f6c33e85d0231c86aea4 MD5 | raw file
  1. #!/usr/bin/env python
  2. """
  3. What we have right now looks like this:
  4. * All of our pages and blog posts are .html files, rather than
  5. Markdown or reST. This is primarily because we inherited most of
  6. those pages from an earlier website, and it was easier to leave them
  7. in that format than convert.
  8. * We use Jinja2 as a templating engine. A small number of pages are
  9. one-of-a-kind (e.g., the ./index.html home page for the site).
  10. * Three kinds of pages are highly repeated, and each currently
  11. requires special handling in our Python compilation script.
  12. * bootcamps/yyyy-mm-site.html (e.g., bootcamps/2012-07-paris.html):
  13. there are about 25 of these right now, and we expect to add
  14. several per month going forward (one for each two-day workshop we
  15. run). A lot of the content (e.g., setup instructions for
  16. students) is generic, and is contained in other .html files that
  17. are included by reference. However, in order to construct the
  18. bootcamps/index.html page, we need to read in all the
  19. bootcamp-specific pages, sort by date, and divide them into two
  20. groups (those coming up and those past). We figure out what files
  21. to load using a file glob, since we can get the start date needed
  22. for sorting from the metadata embedded in the bootcamp files.
  23. * 4_0/lecture/topic.html (e.g., 4_0/python/func.html): there are
  24. about 140 of these pages. Each is a short tutorial on a
  25. particular subject (e.g., Python functions), and they are grouped
  26. into directories (e.g., one for Python, one for regular
  27. expressions, and so on). The special processing here is that
  28. 4_0/index.html needs a list of all the lecture titles, and the
  29. lectures all need lists of the topics they contain. Right now, we
  30. do this via explicit includes: there's a bunch of <meta...> tags
  31. at the top of 4_0/index.html referring to the
  32. 4_0/lecture/index.html pages, and each of those pages uses
  33. <meta...> tags to refer to tutorial files, e.g.:
  34. <meta name="subfile" content="func.html" />
  35. What makes things more complicated is that each lecture's
  36. index.html page (e.g., 4_0/python/index.html) has a couple of
  37. paragraphs at the top to introduce key points, enclosed in:
  38. {% block introduction %}
  39. ...stuff...
  40. {% endblock introduction %}
  41. We extract these blocks from those files and insert them into
  42. 4_0/index.html when compiling to produce the little 'ads' you see
  43. in http://dev.software-carpentry.org/4_0/index.html under each
  44. section title.
  45. * blog/yyyy/mm/name-name-name.html (e.g.,
  46. blog/2012/11/web-4-science.html). These are blog posts, with a
  47. bit of metadata at the top (in <meta...> tags) with the author,
  48. the date, and a serial number:
  49. {% extends "_blog.html" %}
  50. {% block file_metadata %}
  51. <meta name="post_id" content="5239" />
  52. <meta name="author_id" content="pipitone.j" />
  53. <meta name="title" content="Pelican Guts: on content management for Software Carpentry" />
  54. <meta name="post_date" content="2012-11-01" />
  55. <meta name="category" content="tooling" />
  56. {% endblock file_metadata %}
  57. {% block content %}
  58. <p>...actual blog post content...</p>
  59. {% endblock content %}
  60. Again, we find posts using a file glob, extract the post_id
  61. fields, sort, and use that information to order them, fill in the
  62. index, etc. (We have to use a serial post ID rather than the date
  63. because we often have several posts on the same date. That's also
  64. what we inherited when we extracted all of this from WordPress.)
  65. """
  66. import sys
  67. import os
  68. import glob
  69. import re
  70. import getopt
  71. import json
  72. import jinja2
  73. import time
  74. import datetime
  75. try: # Python 3
  76. from urllib.parse import urlparse, urljoin
  77. except ImportError: # Python 2
  78. from urlparse import urlparse, urljoin
  79. from PyRSS2Gen import RSS2, RSSItem, Guid
  80. #----------------------------------------
  81. USAGE = """compile.py [options] initial_file_path: rebuild Software Carpentry web site
  82. -c calendar_file_name optional
  83. -d today's date YYY-MM-DD
  84. -h show this help and exit
  85. -m blog_metadata_json_file_path
  86. -o output_directory_path
  87. -p jinja2_template_search_path may be used multiple times
  88. -r blog_rss_file_path
  89. -s site_url
  90. -v make verbose
  91. -x shorten blog excerpts
  92. """
  93. CONTACT_EMAIL = 'info@software-carpentry.org'
  94. TWITTER_NAME = '@swcarpentry'
  95. TWITTER_URL = 'https://twitter.com/swcarpentry'
  96. METADATA_TEMPLATE = r'<meta\s+name="%s"\s+content="([^"]*)"\s*/>'
  97. MONTHS = {
  98. '01' : 'Jan', '02' : 'Feb', '03' : 'Mar', '04' : 'Apr',
  99. '05' : 'May', '06' : 'Jun', '07' : 'Jul', '08' : 'Aug',
  100. '09' : 'Sep', '10' : 'Oct', '11' : 'Nov', '12' : 'Dec'
  101. }
  102. BLOG_DESCRIPTION = 'Helping scientists make better software since 1998'
  103. BLOG_HISTORY_LENGTH = 10
  104. BLOG_TITLE = 'Software Carpentry'
  105. BLOG_EXCERPT_LENGTH = 200
  106. BLOG_CONTENT_PATTERN = re.compile(r'{% block content %}(.+){% endblock content %}', re.DOTALL)
  107. BLOG_TAG_REPLACEMENT_PATTERN = re.compile(r'<[^>]+>')
  108. #----------------------------------------
  109. class Application(object):
  110. """
  111. Manage the application:
  112. * Parse command-line arguments.
  113. * Manage a dictionary of standard Jinja2 variables used in all pages.
  114. * Create the Jinja2 template expansion environment.
  115. * Manage blog metadata (mapping aliases to author names, topic names, etc.).
  116. """
  117. def __init__(self, args):
  118. """
  119. Initialize settings, parse command line, create rendering environment.
  120. """
  121. self.env = None
  122. self.metadata = None
  123. self.output_dir = None
  124. self.blog_filename = None
  125. self.icalendar_filename = None
  126. self.search_path = []
  127. self.site = None
  128. self.today = None
  129. self.verbosity = 0
  130. self.shorten_blog_excerpts = False
  131. self.filenames = self._parse(args)
  132. self._build_env()
  133. self._load_metadata()
  134. def standard(self, filename):
  135. """
  136. Return dictionary of standard page elements for a file.
  137. (Filename needed so we can calculate root path.)
  138. """
  139. depth = len([x for x in os.path.dirname(filename).split('/') if x])
  140. if depth == 0:
  141. root_path = '.'
  142. else:
  143. root_path = '/'.join([os.pardir] * depth)
  144. return {'contact_email' : CONTACT_EMAIL,
  145. 'filename' : filename,
  146. 'root_path' : root_path,
  147. 'site' : self.site,
  148. 'timestamp' : timestamp(),
  149. 'today' : self.today,
  150. 'twitter_name' : TWITTER_NAME,
  151. 'twitter_url' : TWITTER_URL}
  152. def _parse(self, args):
  153. """
  154. Parse command-line options.
  155. """
  156. options, filenames = getopt.getopt(args, 'c:d:hm:o:p:r:s:vx')
  157. for opt, arg in options:
  158. if opt == '-c':
  159. assert self.icalendar_filename is None, \
  160. 'iCalendar filename specified multiple times'
  161. self.icalendar_filename = arg
  162. elif opt == '-d':
  163. self.today = arg
  164. elif opt == '-h':
  165. usage(0)
  166. elif opt == '-m':
  167. assert self.metadata is None, \
  168. 'Blog metadata specified multiple times'
  169. self.metadata = arg
  170. elif opt == '-o':
  171. assert self.output_dir is None, \
  172. 'Destination directory specified multiple times'
  173. self.output_dir = arg
  174. elif opt == '-p':
  175. assert os.path.isdir(arg), \
  176. 'Search path directory "%s" not found' % arg
  177. self.search_path.append(arg)
  178. elif opt == '-r':
  179. assert self.blog_filename is None, \
  180. 'RSS filename specified multiple times'
  181. self.blog_filename = arg
  182. elif opt == '-s':
  183. self.site = arg
  184. elif opt == '-v':
  185. self.verbosity += 1
  186. elif opt == '-x':
  187. self.shorten_blog_excerpts = True
  188. else:
  189. assert False, \
  190. 'Unknown option %s' % opt
  191. assert self.today is not None, \
  192. 'No date set (use -d)'
  193. assert self.output_dir is not None, \
  194. 'No destination directory specified (use -o)'
  195. assert self.search_path, \
  196. 'No search path directories specified (use -p)'
  197. assert self.site is not None, \
  198. 'No site specified (use -s)'
  199. return filenames
  200. def _build_env(self):
  201. """
  202. Create template expansion environment.
  203. """
  204. loader = jinja2.FileSystemLoader(self.search_path)
  205. self.env = jinja2.Environment(loader=loader,
  206. autoescape=True)
  207. def _load_metadata(self):
  208. """
  209. Load blog metadata translation information (if specified).
  210. """
  211. if self.metadata is None:
  212. self.metadata = {}
  213. else:
  214. with open(self.metadata, 'r') as reader:
  215. self.metadata = json.load(reader)
  216. #----------------------------------------
  217. class GenericPage(object):
  218. """
  219. Store information gleaned from a Jinja2 template page.
  220. * KEYS defines the names of <meta...> tag elements looked for.
  221. Subclasses must inherit this variable or define one of their own
  222. with the same name.
  223. * UPLINK is how to get 'up' in the hierarchy (e.g., up to the
  224. index page for a lesson).
  225. """
  226. KEYS = '*subfile *subglob title'.split()
  227. UPLINK = ''
  228. def __init__(self, app, factory, filename, original, parent):
  229. """
  230. Initialize page representation from file. This is the
  231. template method pattern: derived classes may override any or
  232. all of file loading, metadata extraction, finalizing this
  233. object, loading subfiles, and finalizing the child objects
  234. representing those children.
  235. """
  236. self.app = app
  237. self.filename = filename
  238. self.original = original
  239. self.parent = parent
  240. self.children = []
  241. self.uplink = self.UPLINK
  242. self._factory = factory
  243. self._directory = os.path.dirname(filename)
  244. self._sort_key = None
  245. self._load_file()
  246. self._get_metadata()
  247. self._finalize_self()
  248. self._load_subfiles()
  249. self._finalize_children()
  250. def link(self):
  251. """
  252. Return the URL for linking to this page from a sibling.
  253. May be overriden in derived classes.
  254. """
  255. return os.path.basename(self.filename)
  256. def render(self):
  257. """
  258. Render this page and its children.
  259. """
  260. self._render()
  261. for child in self.children:
  262. child.render()
  263. def _load_file(self):
  264. """
  265. Load file data, which is then stored as a single block of
  266. characters and as a list of lines (since we need both in
  267. different cases).
  268. """
  269. with open(self.filename, 'r') as reader:
  270. self._data = reader.read()
  271. self._block = self._data
  272. self._lines = self._block.split('\n')
  273. def _get_patterns(self):
  274. """
  275. Return triples of (field name, default value, regexp) for
  276. matching metadata fields. Requires this class to have a
  277. 'KEYS' member. If a key name starts with a '*', the key can
  278. have multiple values.
  279. """
  280. result = []
  281. for k in self.KEYS:
  282. multi, default = k.startswith('*'), None
  283. if multi:
  284. k, default = k[1:], []
  285. r = re.compile(METADATA_TEMPLATE % k)
  286. result.append([k, default, r])
  287. return result
  288. def _get_metadata(self):
  289. """
  290. Extract metadata embedded in <meta...> tags using the patterns
  291. constructed by _get_patterns. These values are stored as
  292. member variables in this object.
  293. """
  294. for (field, default, pat) in self._get_patterns():
  295. self._set_metadata(field, default)
  296. for line in self._lines:
  297. match = pat.search(line)
  298. if match:
  299. self._set_metadata(field, match.group(1))
  300. def _set_metadata(self, field, value):
  301. """
  302. Store metadata extracted from a <meta...> tag as a member
  303. variable of this object.
  304. """
  305. # Member variable doesn't exist at all yet, so assign value.
  306. # If the key can have multiple values, the initial value must
  307. # be a list.
  308. if field not in self.__dict__:
  309. assert value in (None, []), \
  310. 'Must initialize to None (single) or [] (multi)'
  311. self.__dict__[field] = value
  312. # Member variable already exists and is a list, so the
  313. # variable can be multi-valued, so append.
  314. elif type(self.__dict__[field]) is list:
  315. self.__dict__[field].append(value)
  316. # Member variable already exists, so check that its value is
  317. # None (i.e., that it hasn't already been initialized), and
  318. # then set its value.
  319. else:
  320. assert self.__dict__[field] is None, \
  321. 'Single-valued field %s being reset' % field
  322. self.__dict__[field] = value
  323. def _load_subfiles(self):
  324. """
  325. Load sub-files recursively.
  326. """
  327. # Get children that are named explicitly.
  328. self.children = [self._factory(os.path.join(self._directory, sf), sf, self)
  329. for sf in self.subfile]
  330. # Get children by globbing.
  331. sort = False
  332. if 'subglob' in self.__dict__:
  333. sort = True
  334. for sg in self.subglob:
  335. whole_glob = os.path.join(self._directory, sg)
  336. matches = [self._factory(m, None, self)
  337. for m in glob.glob(whole_glob)]
  338. self.children += matches
  339. # If anything was globbed, sort everything (including children
  340. # that were loaded explicitly).
  341. if sort:
  342. self.children.sort(key=lambda x: (x._sort_key, id(x)))
  343. def _finalize_children(self):
  344. """
  345. Create prev/next links between children.
  346. """
  347. for (i, child) in enumerate(self.children):
  348. child.prev = None if (i == 0) \
  349. else self.children[i-1].link()
  350. child.next = None if (i == len(self.children)-1) \
  351. else self.children[i+1].link()
  352. def _render(self):
  353. """
  354. Render and save this page.
  355. """
  356. if self.app.verbosity > 0:
  357. sys.stderr.write(self.filename)
  358. sys.stderr.write('\n')
  359. # Render.
  360. template = self.app.env.get_template(self.filename)
  361. result = template.render(page=self,
  362. **self.app.standard(self.filename))
  363. # Make sure the output directory exists.
  364. dest = os.path.join(self.app.output_dir, self.filename)
  365. directory = os.path.dirname(dest)
  366. if not os.path.isdir(directory):
  367. os.makedirs(directory)
  368. # Save the rendered text.
  369. with open(dest, 'w') as writer:
  370. writer.write(result)
  371. def _finalize_self(self):
  372. """
  373. Template method: finalize elements in this object.
  374. """
  375. pass
  376. #----------------------------------------
  377. class BootCampPage(GenericPage):
  378. """
  379. Represent information about a boot camp.
  380. The 'Instances' class variable keeps track of all created instances,
  381. so that they can be used to render the iCalendar feed.
  382. """
  383. KEYS = GenericPage.KEYS + \
  384. ['venue', 'latlng', 'date', 'startdate', 'enddate',
  385. 'eventbrite_key']
  386. UPLINK = 'index.html'
  387. Instances = []
  388. def __init__(self, *args):
  389. GenericPage.__init__(self, *args)
  390. BootCampPage.Instances.append(self)
  391. def _finalize_self(self):
  392. """
  393. Finish creating this object:
  394. * Create normalized date for display.
  395. * Create slug.
  396. * Create sort key (start date and venue).
  397. """
  398. self._merge_dates()
  399. self.slug = os.path.splitext(os.path.basename(self.filename))[0]
  400. self._sort_key = (self.startdate, self.venue)
  401. def _merge_dates(self):
  402. """
  403. Merge start and end dates into human-readable form.
  404. """
  405. start, end = self.startdate, self.enddate
  406. start_year, start_month, start_day = start.split('-')
  407. start_month_name = MONTHS[start_month]
  408. # One-day workshop.
  409. if end is None:
  410. self.date = '%s %s, %s' % \
  411. (start_month_name, start_day, start_year)
  412. return
  413. end_year, end_month, end_day = end.split('-')
  414. end_month_name = MONTHS[end_month]
  415. # Spans two years.
  416. if start_year < end_year:
  417. self.date = '%s %s, %s - %s %s, %s' % \
  418. (start_month_name, start_day, start_year,
  419. end_month_name, end_day, end_year)
  420. # Spans two months.
  421. elif start_month < end_month:
  422. self.date = '%s %s - %s %s, %s' % \
  423. (start_month_name, start_day,
  424. end_month_name, end_day, end_year)
  425. # All in one month.
  426. elif start_day < end_day:
  427. self.date = '%s %s-%s, %s' % \
  428. (start_month_name, start_day,
  429. end_day, end_year)
  430. # End date is before start date?
  431. else:
  432. assert False, \
  433. 'Bad date range %s -- %s' % (start, end)
  434. def index_link(self):
  435. """
  436. Link from index page to this post.
  437. FIXME: must be a better way than special-casing this.
  438. """
  439. return 'bootcamps/{0}'.format(self.link())
  440. #----------------------------------------
  441. class LessonPage(GenericPage):
  442. """
  443. Represent information about a lesson made up of topics.
  444. """
  445. UPLINK = '../index.html'
  446. KEYPOINTS = re.compile(r'<ul\s+class="keypoints"\s*>(.+?)</ul>', re.DOTALL)
  447. def link(self):
  448. """
  449. Return the URL for linking to this page from a sibling by
  450. hacking around with the file path.
  451. """
  452. lowest_dir = self._directory.split('/')[-1]
  453. return os.path.join(os.pardir, lowest_dir, 'index.html')
  454. def _finalize_self(self):
  455. """
  456. Finish creating this lessage page:
  457. * The slug for a lesson is the last directory in path (since the
  458. main lesson file is always 'index.html').
  459. * Extract the keypoints from this lesson for inclusion in the main
  460. index page.
  461. """
  462. self.slug = os.path.dirname(self.filename).split('/')[-1]
  463. m = self.KEYPOINTS.search(self._data)
  464. assert m, \
  465. 'No keypoints found in %s' % self.filename
  466. self.keypoints = m.group(1)
  467. #----------------------------------------
  468. class TopicPage(GenericPage):
  469. """
  470. Represent information about a single topic within a lesson.
  471. """
  472. UPLINK = 'index.html'
  473. def _finalize_self(self):
  474. """
  475. Finish creating this topic's page object:
  476. * The slug for a topic is the base of the filename.
  477. """
  478. self.slug = os.path.splitext(os.path.basename(self.filename))[0]
  479. #----------------------------------------
  480. class BlogIndexPage(GenericPage):
  481. """
  482. Singleton to store information about all blog posts.
  483. """
  484. def __init__(self, *args):
  485. GenericPage.__init__(self, *args)
  486. self.blog_history_length = BLOG_HISTORY_LENGTH
  487. def _finalize_children(self):
  488. """
  489. Extract information about years and months for creating the
  490. table of blog posts as well as linking children.
  491. """
  492. # Link children as usual.
  493. GenericPage._finalize_children(self)
  494. # Extract information needed for index table.
  495. self.years = sorted(set(c.year for c in self.children))
  496. self.months = sorted(MONTHS.keys())
  497. self._posts = {}
  498. for child in self.children:
  499. year, month = child.year, child.month
  500. if (year, month) not in self._posts:
  501. self._posts[(year, month)] = []
  502. self._posts[(year, month)].append(child)
  503. def posts(self, year, month):
  504. """
  505. Get all blog posts for a specific period.
  506. """
  507. return self._posts.get((year, month), [])
  508. def month_name(self, month):
  509. """
  510. Convert 2-digit month number into month name (helper function
  511. for template expansion).
  512. """
  513. return MONTHS[month]
  514. #----------------------------------------
  515. class BlogPostPage(GenericPage):
  516. """
  517. Represent information about a single blog post.
  518. The 'Instances' class variable keeps track of all created instances,
  519. so that they can be used to render the blog feed.
  520. """
  521. KEYS = GenericPage.KEYS + \
  522. ['post_id', 'post_date', 'author_id', '*category']
  523. UPLINK = '../../index.html'
  524. Instances = []
  525. def __init__(self, *args):
  526. GenericPage.__init__(self, *args)
  527. BlogPostPage.Instances.append(self)
  528. def link(self):
  529. """
  530. Return the URL for linking to this page from a sibling. This
  531. is completely tied to the year/month/post directory structure
  532. right now.
  533. """
  534. return '/'.join(['..', '..', self.year, self.month, self.name])
  535. def index_link(self):
  536. """
  537. Link from index page to this post.
  538. FIXME: must be a better way than special-casing this.
  539. """
  540. return '/'.join([self.year, self.month, self.name])
  541. def excerpt(self, excerpt_filename):
  542. """
  543. Return an excerpt of the page for display in the RSS reader by:
  544. * finding the content block
  545. * expanding it
  546. and optionally (if asked to shorten):
  547. * replacing HTML tags
  548. * taking the first few hundred characters
  549. * going back from the end of that to the first space
  550. * hoping the result isn't too horribly mangled.
  551. The right way to do this would be to have an explicit excerpt
  552. div or span in the blog posts, but that's not what we inherited
  553. from WordPress.
  554. """
  555. # No text in blog post.
  556. if not self.content:
  557. return ''
  558. # Translate the content block.
  559. template = jinja2.Template(self.content)
  560. result = template.render(page=self,
  561. **self.app.standard(excerpt_filename))
  562. if not self.app.shorten_blog_excerpts:
  563. return result
  564. # Have content and want to shorten it, so shorten it.
  565. result = BLOG_TAG_REPLACEMENT_PATTERN.sub('', result)
  566. result = result[:BLOG_EXCERPT_LENGTH]
  567. if ' ' in result:
  568. result = result[:result.rindex(' ')]
  569. if result:
  570. result += ' [...]'
  571. return result
  572. def _finalize_self(self):
  573. """
  574. Record this page's creation so that the index can be constructed.
  575. Translate metadata.
  576. """
  577. def _tx(key, value):
  578. return self.app.metadata[key][value]
  579. self._sort_key = int(self.post_id)
  580. self.year, self.month, self.name = self.filename.split('/')[-3:]
  581. self.content = None
  582. m = BLOG_CONTENT_PATTERN.search(self._block)
  583. if m:
  584. self.content = m.group(1)
  585. for key in self.app.metadata:
  586. if key not in self.__dict__:
  587. pass
  588. elif type(self.__dict__[key]) is str:
  589. self.__dict__[key] = _tx(key, self.__dict__[key])
  590. elif type(self.__dict__[key]) is list:
  591. self.__dict__[key] = [_tx(key, v) for v in self.__dict__[key]]
  592. else:
  593. assert False, 'Bad metadata translation setup'
  594. #----------------------------------------
  595. class PageFactory(object):
  596. """
  597. Construct the right kind of page object for a page by finding a
  598. pageclass comment in the page or its parent(s) and looking up the
  599. corresponding class in this script. Page class mappings are
  600. cached to avoid repeatedly reading the same handful of generic
  601. (base) Jinja2 templates.
  602. """
  603. PAGE_CLASS_PAT = re.compile(r'<!--\s+pageclass:\s+\b(.+)\b\s+-->')
  604. EXTENDS_PAT = re.compile(r'{%\s*extends\s+"([^"]+)"\s*%}')
  605. def __init__(self, app):
  606. self.app = app
  607. self.cache = {}
  608. def __call__(self, filename, original, parent):
  609. """
  610. Make a page object based on metadata embedded in the page itself.
  611. """
  612. with open(filename, 'r') as reader:
  613. cls = self._find_page_class(filename)
  614. assert cls in globals(), \
  615. 'Unknown page class %s' % cls
  616. return globals()[cls](self.app, self, filename, original, parent)
  617. def _find_page_class(self, original_filename):
  618. """
  619. Search recursively for page class, returning the page's class.
  620. """
  621. # Page class has already been determined and cached.
  622. filename = self._find_file(original_filename)
  623. if filename in self.cache:
  624. return self.cache[filename]
  625. with open(filename, 'r') as reader:
  626. # Explicit page class declaration in this page.
  627. data = reader.read()
  628. m = self.PAGE_CLASS_PAT.search(data)
  629. if m:
  630. cls = m.group(1)
  631. self.cache[filename] = cls
  632. return cls
  633. # This page extends something else.
  634. m = self.EXTENDS_PAT.search(data)
  635. if m:
  636. base_filename = m.group(1)
  637. cls = self._find_page_class(base_filename)
  638. self.cache[filename] = cls
  639. return cls
  640. # No page class.
  641. assert False, \
  642. 'Unable to find page class for %s' % filename
  643. def _find_file(self, filename):
  644. """
  645. Search for a file in various directories by name. This
  646. duplicates the machinery inside Jinja2 for finding template
  647. pages that are being extended, but there's no easy way to get
  648. Jinja2 to do the finding for us.
  649. """
  650. for d in self.app.search_path:
  651. f = os.path.join(d, filename)
  652. if os.path.isfile(f):
  653. return f
  654. assert False, \
  655. 'File %s not found in search path %s' % (filename, self.app.search_path)
  656. #----------------------------------------
  657. class ContentEncodedRSSItem(RSSItem):
  658. def __init__(self, **kwargs):
  659. self.content = kwargs.get('content', None)
  660. if 'content' in kwargs:
  661. del kwargs['content']
  662. RSSItem.__init__(self, **kwargs)
  663. def publish_extensions(self, handler):
  664. if self.content:
  665. handler._out.write('<%(e)s><![CDATA[%(c)s]]></%(e)s>' %
  666. { 'e':'content:encoded', 'c':self.content})
  667. #----------------------------------------
  668. class ContentEncodedRSS2(RSS2):
  669. def __init__(self, **kwargs):
  670. RSS2.__init__(self, **kwargs)
  671. self.rss_attrs['xmlns:content']='http://purl.org/rss/1.0/modules/content/'
  672. #----------------------------------------
  673. class ICalendarWriter(object):
  674. """
  675. iCalendar generator for boot camps.
  676. The format is defined in RFC 5545: http://tools.ietf.org/html/rfc5545
  677. """
  678. def __call__(self, filename, site, bootcamps):
  679. lines = [
  680. 'BEGIN:VCALENDAR',
  681. 'VERSION:2.0',
  682. 'PRODID:-//Software Carpentry/Boot Camps//NONSGML v1.0//EN',
  683. ]
  684. for bootcamp in bootcamps:
  685. lines.extend(self.bootcamp(site, bootcamp))
  686. lines.extend(['END:VCALENDAR', ''])
  687. content = '\r\n'.join(lines)
  688. # From RFC 5545, section 3.1.4 (Character Set):
  689. # The default charset for an iCalendar stream is UTF-8.
  690. with open(filename, 'wb') as writer:
  691. writer.write(content.encode('utf-8'))
  692. def bootcamp(self, site, bootcamp):
  693. uid = '{0}@{1}'.format(bootcamp.link().replace('.html', ''),
  694. urlparse(site).netloc or 'software-carpentry.org')
  695. url = urljoin(site, bootcamp.index_link())
  696. if bootcamp.enddate:
  697. end_fields = [int(x) for x in bootcamp.enddate.split('-')]
  698. else: # one day boot camp?
  699. end_fields = [int(x) for x in bootcamp.startdate.split('-')]
  700. end = datetime.date(*end_fields)
  701. dtend = end + datetime.timedelta(1) # non-inclusive end date
  702. lines = [
  703. 'BEGIN:VEVENT',
  704. 'UID:{0}'.format(uid),
  705. 'DTSTAMP:{0}'.format(timestamp()),
  706. 'DTSTART;VALUE=DATE:{0}'.format(bootcamp.startdate.replace('-', '')),
  707. 'DTEND;VALUE=DATE:{0}'.format(dtend.strftime('%Y%m%d')),
  708. 'SUMMARY:{0}'.format(self.escape(bootcamp.venue)),
  709. 'DESCRIPTION;ALTREP="{0}":{0}'.format(url),
  710. 'URL:{0}'.format(url),
  711. 'LOCATION:{0}'.format(self.escape(bootcamp.venue)),
  712. ]
  713. if bootcamp.latlng:
  714. lines.append('GEO:{0}'.format(bootcamp.latlng.replace(',', ';')))
  715. lines.append('END:VEVENT')
  716. return lines
  717. def escape(self, value):
  718. """
  719. Escape text following RFC 5545.
  720. """
  721. for char in ['\\', ';', ',']:
  722. value = value.replace(char, '\\' + char)
  723. value.replace('\n', '\\n')
  724. return value
  725. #----------------------------------------
  726. def create_rss(filename, site, posts):
  727. """
  728. Generate RSS2 feed.xml file for blog.
  729. """
  730. items = []
  731. slice = posts[-BLOG_HISTORY_LENGTH:]
  732. slice.reverse()
  733. for post in slice:
  734. template_vars = post.app.standard(post.filename)
  735. template_vars['root_path'] = site
  736. path = os.path.join(site, 'blog', post.index_link())
  737. rendered_content = jinja2.Template(post.content).render(**template_vars)
  738. items.append(ContentEncodedRSSItem(title=post.title,
  739. author=post.author_id,
  740. link=path,
  741. description=post.excerpt(filename),
  742. content=rendered_content,
  743. pubDate=post.post_date))
  744. rss = ContentEncodedRSS2(title=BLOG_TITLE,
  745. link=site,
  746. description=BLOG_DESCRIPTION,
  747. lastBuildDate=datetime.datetime.utcnow(),
  748. items=items)
  749. with open(filename, 'w') as writer:
  750. rss.write_xml(writer)
  751. #----------------------------------------
  752. def timestamp():
  753. """
  754. Return the current UTC time formatted in ISO 8601
  755. """
  756. return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())
  757. #----------------------------------------
  758. def usage(exit_status):
  759. """
  760. Show usage and exit.
  761. """
  762. print >> sys.stderr, USAGE
  763. sys.exit(exit_status)
  764. #----------------------------------------
  765. def main(args):
  766. """
  767. Main driver:
  768. * construct an application manager
  769. * construct a page factory
  770. * create and render page objects for each page (recursively)
  771. * generate the blog's feed.xml file if asked to do so
  772. """
  773. app = Application(args)
  774. factory = PageFactory(app)
  775. for filename in app.filenames:
  776. page = factory(filename, filename, None)
  777. page.render()
  778. if app.blog_filename:
  779. create_rss(app.blog_filename, app.site, BlogPostPage.Instances)
  780. if app.icalendar_filename:
  781. icw = ICalendarWriter()
  782. icw(app.icalendar_filename, app.site, BootCampPage.Instances)
  783. #----------------------------------------
  784. if __name__ == '__main__':
  785. main(sys.argv[1:])