Penflip updates, bug fixes, and ramblings

Improved download & editor performance

Both download conversions and the Penflip editor have always been a bit cumbersome, so I spent some time improving the performance of both. If you're interested in a little bit of what's going on under the hood, keep reading, otherwise here's the short of it: everything is much, much better now. Faster and more robust. Enjoy! :)


Projects are converted using Pandoc plus quite a bit of preprocessing with ruby, including some heavy lifting like downloading remote images. Depending on a number of factors, namely project size and images, downloads can take a while to compile. This is particularly a problem for PDFs, which are converted from markdown to LaTeX to PDF, taking three passes to fully compile.

Here are the main improvements to downloads:

  • Background jobs: Originally, conversions were handled synchronously in a single web request. This is very bad, and I was fully aware of the problem when I wrote it, but for the sake of time I released it anyway. Penflip Unicorn workers are killed after 30s, so any downloads taking longer than about 30s to compile would error out. I've seen some projects take up to two minutes to compile a PDF.

    This was fixed by adding download jobs to a Sidekiq queue and processing them in the background. The browser polls for completion, and when the compiled project is detected, it's downloaded.

  • Caching downloads: Whenever a project is compiled, the result is stored on a Penflip server. If the project content or settings change, the project will be recompiled on the next download, otherwise, the stored version will be returned (which takes < 1 second vs 30+ seconds).

  • Caching remote images: Remote images are downloaded and lightly processed with ImageMagick during conversions. Previously, they'd be downloaded on every project conversion (which is redundant and time consuming). Now they're stored on Penflip servers. Even if the text of a project changes and downloads are recompiled, the remote images aren't downloaded again - they're simply pulled from cache. This greatly improves recompile time with remote images.

Web Editor

The Penflip editor is a fork of Prose, a beautiful markdown editor built to work with GitHub using the GitHub API. The GitHub API wasn't exactly designed for that kind of utility, however, so the Prose creators cleverly hacked around the API shortcomings by doing things like stringing several requests together for basic functions.

When I forked Prose, I kept those hacks intact and layered my own on top, adding up to a huge mess. Larger projects could take several seconds to fully load the editor (or fail outright), which is totally unacceptable. I got fed up and decided to to fix it.

  • Fewer API requests: I originally mimicked the GitHub API to make Prose work with Penflip (and directed the requests to the Penflip API instead of the GitHub API), leaving the Penflip API with the same shortcomings of the GitHub API. The Prose creators couldn't change the GitHub API, but fortunately I have full control over the Penflip API, so I optimized it for the web editor. Not sure how that will play out in the long term, but right now the API is only used by the Penflip editor, so it's fine.

    Anywhere with several requests for a simple action (e.g. 3 API requests to load the content of a file), I consolidated into a single request. The less requests, the better.

  • Caching fragments: Some operations within the API requests are pretty heavy, like building the contents list of a project. While I'm not caching entire API responses, I'm caching these time consuming operations - especially ones that are used in several API requests. This can trim the response time of a single request significantly.

Between reducing the number of API requests and making those individual requests more efficient, the Penflip editor is now much faster. I don't have any hard benchmarking data, but one ~150ms request is undoubtedly faster than three ~200ms requests to acheive the same result.