Penflip-header-logo@2x

Penflip updates, bug fixes, and ramblings

Folders!

I can't even tell you how many requests I've had for nested folder support. Too many. The general idea is this: add folders to a project to split it into parts. Each part has several chapters, creating a content hierarchy.

Well, you can finally do it! Adding a new folder is as simple as clicking the New Folder icon from the Penflip editor sidebar. Drag chapters into a folder and rearrange them inside the folder. You can also rearrange folders. Nesting files and toggling folders open and closed results in a much cleaner editing interface. So that's pretty cool.

new folder

But the real magic happens outside the editor. On your project page, folders are represented as sections. You're no longer limited to an unwieldy flat file list. With multiple sections and nested content, you can quickly and easily create some pretty nice looking documentation. In fact, we're already doing it with Penflip Help:

project sections

But wait... that's not all! Folders are turned into 'parts' when downloading as a PDF. You can now have a multi-part book by splitting your content into folders. The table of contents will show parts and the chapters nested beneath them. Download the Penflip Help project as a PDF to see what this looks like.

So, folders give you: a cleaner editing interface, more organized project pages, and multi-part books. Go make some folders!

Folders documentation

Book cover images

A previous update gave you some control over the title page of your book, but a book doesn't really feel like a book without a proper cover image.

Good news: Penflip now has support for cover images! Simply upload an image titled either cover.png or cover.jpg to your project's images folder, and it will be set as the cover image when downloading as a PDF or ePub. Ensure that the image matches the dimensions of your book: 8.5x11" for Business books, Technical books, and documents; 6x9" for Fiction books. A high resolution (300 PPI) is recommended for best quality.

See Help: Cover Images for more information.

New copyright options

Creative Commons! Yep, that's right.

From your project settings page, you can choose between a traditional copyright (e.g. ©2014 John Doe) or a Creative Commons public copyright license. Penflip uses the latest Creative Commons Version 4.0 licenses. Several options let you customize the license to fit your needs.

These licenses are primarily for inclusion in compiled PDFs, but you can also choose to display the selected copyright on your project page.

cc licenses

YAML front matter support

In an effort to make content cross-compatible for multiple purposes (e.g. a Jekyll blog and a PDF ebook), Penflip now has support for basic YAML front matter. Basic is an understatement, really, since the only attribute that is supported right now is title. Exactly like Jekyll, the front matter must be the first thing in the file and must take the form of valid YAML set between triple-dashed lines.

Your title front matter should look like this:

---
title: Learning CSS
---

This will be displayed as an H1 element (markdown #) in previews, downloads, etc.

You can import your Jekyll blog into Penflip and all your post titles will be converted to chapter titles. Oh yeah!

Project navigation links

Small but good: you can jump between consectutive chapters of your book with links at the bottom of each chapter. See the Help project, for example:

links

Project labels

Labels are a great way to sort and organize your projects. From your 'projects' page, hover over a project for the + label option. Once you've labeled your projects, click a label to show only projects with the selected label. You can toggle labels on and off with the toolbar, and manage labels from your account settings.

  • Projects can have multiple labels
  • You can only manage labels for projects you own (not projects you've joined)
  • Labels are not displayed on public projects in the Discover section

project labels

Improved download & editor performance

Both download conversions and the Penflip editor have always been a bit cumbersome, so I spent some time improving the performance of both. If you're interested in a little bit of what's going on under the hood, keep reading, otherwise here's the short of it: everything is much, much better now. Faster and more robust. Enjoy! :)

Downloads

Projects are converted using Pandoc plus quite a bit of preprocessing with ruby, including some heavy lifting like downloading remote images. Depending on a number of factors, namely project size and images, downloads can take a while to compile. This is particularly a problem for PDFs, which are converted from markdown to LaTeX to PDF, taking three passes to fully compile.

Here are the main improvements to downloads:

  • Background jobs: Originally, conversions were handled synchronously in a single web request. This is very bad, and I was fully aware of the problem when I wrote it, but for the sake of time I released it anyway. Penflip Unicorn workers are killed after 30s, so any downloads taking longer than about 30s to compile would error out. I've seen some projects take up to two minutes to compile a PDF.

    This was fixed by adding download jobs to a Sidekiq queue and processing them in the background. The browser polls for completion, and when the compiled project is detected, it's downloaded.

  • Caching downloads: Whenever a project is compiled, the result is stored on a Penflip server. If the project content or settings change, the project will be recompiled on the next download, otherwise, the stored version will be returned (which takes < 1 second vs 30+ seconds).

  • Caching remote images: Remote images are downloaded and lightly processed with ImageMagick during conversions. Previously, they'd be downloaded on every project conversion (which is redundant and time consuming). Now they're stored on Penflip servers. Even if the text of a project changes and downloads are recompiled, the remote images aren't downloaded again - they're simply pulled from cache. This greatly improves recompile time with remote images.

Web Editor

The Penflip editor is a fork of Prose, a beautiful markdown editor built to work with GitHub using the GitHub API. The GitHub API wasn't exactly designed for that kind of utility, however, so the Prose creators cleverly hacked around the API shortcomings by doing things like stringing several requests together for basic functions.

When I forked Prose, I kept those hacks intact and layered my own on top, adding up to a huge mess. Larger projects could take several seconds to fully load the editor (or fail outright), which is totally unacceptable. I got fed up and decided to to fix it.

  • Fewer API requests: I originally mimicked the GitHub API to make Prose work with Penflip (and directed the requests to the Penflip API instead of the GitHub API), leaving the Penflip API with the same shortcomings of the GitHub API. The Prose creators couldn't change the GitHub API, but fortunately I have full control over the Penflip API, so I optimized it for the web editor. Not sure how that will play out in the long term, but right now the API is only used by the Penflip editor, so it's fine.

    Anywhere with several requests for a simple action (e.g. 3 API requests to load the content of a file), I consolidated into a single request. The less requests, the better.

  • Caching fragments: Some operations within the API requests are pretty heavy, like building the contents list of a project. While I'm not caching entire API responses, I'm caching these time consuming operations - especially ones that are used in several API requests. This can trim the response time of a single request significantly.

Between reducing the number of API requests and making those individual requests more efficient, the Penflip editor is now much faster. I don't have any hard benchmarking data, but one ~150ms request is undoubtedly faster than three ~200ms requests to acheive the same result.

UPDATE: Easier sharing

Penflip is a collaborative writing platform, so it makes sense that it should be as easy to share as possible. Here are a few recent improvements on that front:

  • You can now invite collaborators while writing, which makes for a much smoother workflow

  • Click any Share button to invite multiple people at a time via email; no more copy + pasting links (but you can still do that if you want to)

  • If you need feedback on a specific chapter, you can optionally direct your invites to that chapter

sharing

Project search

Guess what. You can finally find all those awesome projects you've been looking for! The search is "instant", so just type away and see your results. You can search by project title or owner name.

Search is also available within your own projects, which is especially helpful when you have quite a few projects under your belt. Also instant, also awesome.

project search

Project shortcuts

Gone are the days of clicking into your project, then clicking again for settings or sharing[1]!

Now you can quickly get to edit, discussions, sharing, and settings for any of your projects directly from your project listings page. Time saver!

project-shortcuts.png

[1] ... you still can if you want to, I guess!

MathJax integration

MathJax is the de facto standard for displaying mathematical equations online. With Penflip, you can write LaTeX equations wrapped in MathJax tags, and they'll be rendered beautifully in both the browser as well as in your downloads.

Inline example

mathjax inline

Display example:

mathjax display

Helpful resources: MathJax documentation and samples.

Project filtering

When looking at projects you contribute to, you can choose to display projects started by you, projects started by others, or all projects you're a member of. This is especially helpful if you have many projects to sift through.

projects

Smarter lists

The toolbar buttons for creating markdown lists are more intuitive now. Selecting text and clicking a list button will turn your selection into a list, instead of creating a new empty list (as it did previously). It works if single lines or multiple lines are selected, and for both bullet lists and numeric lists. You can also toggle between these two types.

smarter lists

Clicking a list button on any empty line will create a new list, as it did before.

Quick fix: improper MathJax rendering

MathJax was trying to render equations where it should have displayed plain text instead. This was affecting merge requests and revision history views. It was the result of an improper configuration setting, which caused anything between backticks ( ` ) to be interpreted as math equations.

The configuration has been updated, and it should be fixed now.

- Loren

Bug fixes: PDF downloads, MathJax and more

Just a quick update. The following bugs have been fixed:

  • issue with new documents created in the past few days that resulted in broken PDF downloads (books were not affected)
  • issue with Chinese / Japanese characters resulting in blank chapter titles (foreign character support for chapter titles
  • double clicking save for 'New Chapter' created duplicate chapters
  • MathJax rendering issues (certain equations were not being displayed in preview)

.. and some other small issues.

Write on!

Tiny update: headers in the margin

Markdown hash marks are now displayed in the margin, which improves two things:

1) aligns header text with the rest of the content

2) allows for easy scanning and identifying headers within a large document

Penflip.com markdown headers


Spaces between hash and text are optional. Up to five hash marks are displayed in the margin.

UPDATE: Advanced formatting and publishing options

book covers

Many Penflip writers have been asking for a way to fine-tune the format and styling of downloads - particularly PDFs. There were a few options, but they were limited. Now there are over two dozen settings, including colors, page size, margin size, code block styling, and more.

You can access the new set of options from your project page, by clicking Settings > Publishing Options. There's also a shortcut from the download popup.

The whole shebang

A complete list of the current options:

Book options:
Table of contents, page breaks, page numbers

Cover style:
Cover page, cover color, title color, author color

Text options:
Font style, paragraph indent, font size, line spacing, link color

Page formatting:
Page width, page height, top margin, bottom margin, left margin, right margin

Code block styling:
Line numbers, border, border color, background color, keyword color, string color, comment color

Up next for formatting and publishing

In no particular order:

  • Copyright options
  • Download as kindle .mobi format
  • Upload cover image
  • Preset styles
  • More font options

Happy publishing!
- Loren

Suggest changes or updates to this post

Nerding out: a funky image regex

An ongoing issue with Penflip is compiling projects for download (e.g. to PDF). The basic idea is simple and most of the time it works flawlessly, but with arbitrary user input there will be curveballs. And I see every single curveball imagineable.

Even with fantastic open source libraries like Pandoc, I still have quite a bit of code for pre-processing, for example: downloading images that are referenced in projects. Before converting to PDF, markdown files are converted to LaTeX. Image urls are parsed from the LaTeX for downloading. This was my regex:

/(https|http):\/\/.*?(jpeg|jpg|png|gif|bmp)/i

Look for an image url and stop at the first image extension. Simple enough. Turns out an image extension can be included twice in a url. In hindsight, it's obvious that this is possible from a technical standpoint, but I didn't consider it to be a likely scenario. So of course it happened. Here's a url that broke the regex:

http://upload.wikimedia.org/wikipedia/commons/thumb/d/d6/Pachinko_balls.jpg/800px-Pachinko_balls.jpg

Note the two .jpg extensions. Ugh. My regex returned:

http://upload.wikimedia.org/wikipedia/commons/thumb/d/d6/Pachinko_balls.jpg

Here's a better regex, using positive lookbehinds and lookaheads to capture the full url:

/(?<=\\includegraphics{)(https|http):\/\/\S*?\.(jpeg|jpg|png|gif|bmp)(?=})/i

Problem solved!

Hello, world!

I'm Loren, founder of Penflip.com.

New features, improvements and bug fixes are pushed to Penflip almost every single day. It's useful to have a place to keep track of those things, so this will be that place. Occasionally I send newsletter emails, but this will be more frequently updated. I also plan to share some things I learn along the way while building Penflip.com.

If you have any feedback to make Penflip better, feel free to suggest it, tweet, or email me.

- Loren