For a long time, I've been using Calibre to keep my e-books and other electronic documents organized (epub, pdf, etc). I've used the desktop clients for both Windows and OSX, and while it's a little old-fashioned looking, it has some extremely powerful features. Calibre lets you edit the metadata stored with a file, and will pull in information from multiple sources for you. You get full-text search, filtering by tags, authors and more, and it displays a nice cover image for you while you browse. The files (your "library") still need to be stored somewhere, and I've been using Google Drive so I can sync between machines.
Well, since I've been on this kick to take more control of my data, doing things like dumping Facebook and moving all my notes and documents off of Google and into my own secure DokuWiki instance, I thought I'd take a stab at installing Calibre server. My goal was to move my books onto the same instance where I installed DokuWiki, then configure Calibre to (securely, via TLS and password protection) serve up my books in the browser. I had high hopes.
Installation
Before I get to the bad and ugly, here's how I installed it. Hopefully I won't forget anything. If you try this out and anything seems fishy, leave a comment below.
Spin up an Ubuntu server
You'll need somewhere to run all this. If you already have an Ubuntu server, you're good to go. Personally, I spun up an Ubuntu 17 instance over at DigitalOcean. It's easy to create machines in minutes, back them up all you like (automatically or manually), and destroy them when you're done. Once you create an account, just click "Create" and then "Droplets" at the top, and you'll see Ubuntu as an available distribution. If you need more detailed steps, I wrote about this before.
Note: Pay close attention to the SSH options. You'll want to be able to SSH into your new machine. Read this if you're unfamiliar with SSH: How To Use SSH Keys with DigitalOcean Droplets
After it's spun up, you'll get an email and you should be able to login with root. The first thing you should is create a sudo user - an account that can run things as root when needed, but that isn't actually root.
Install Python
Before you can install Calibre, make sure Python is installed. I had Python3 installed, but not Python2, and the Calibre script didn't seem to like that. You can check by just calling "python". If it's installed you'll end up at a prompt where you can run Python commands (type exit() to close it). Otherwise you'll get a message like this:
=> python
The program 'python' can be found in the following packages:
* python-minimal
* python3
Try: sudo apt install <selected package>
=> sudo apt-get update && sudo apt-get -y upgrade
=> sudo apt-get install python
eading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
libpython-stdlib libpython2.7-minimal libpython2.7-stdlib python-minimal python2.7 python2.7-minimal
Suggested packages:
python-doc python-tk python2.7-doc binutils binfmt-support
The following NEW packages will be installed:
libpython-stdlib libpython2.7-minimal libpython2.7-stdlib python python-minimal python2.7 python2.7-minimal
0 upgraded, 7 newly installed, 0 to remove and 0 not upgraded.
Need to get 3943 kB of archives.
After this operation, 16.7 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
...
...
... a bunch of output as it installs your files.....
Install Calibre
Go to the Calibre site and copy the first command you see. Run that via the command line after installing Python. For me, it looked like this:
sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.py | sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"
That produced a lot of output, starting with it downloading a Python script to run, which then downloads the Calibre setup file and runs some other magic. The end of the output will report something about "desktop integration failed" but since we're just trying to serve it up in the browser, don't worry about it. If you get any other strange looking errors, check out the Calibre link again - there's a lot of notes about other prerequisites, although I seemed to have them all other than Python2.
grant@pretend-wiki:~$ sudo -v && wget -nv -O- https://download.calibre-ebook.com/linux-installer.py | sudo python -c "import sys; main=lambda:sys.stderr.write('Download failed\n'); exec(sys.stdin.read()); main()"
2018-04-20 05:08:03 URL:https://download.calibre-ebook.com/linux-installer.py [27755/27755] -> "-" [1]
Installing to /opt/calibre
Downloading tarball signature securely...
Will download and install calibre-3.21.0-x86_64.txz
Downloading calibre-3.21.0-x86_64.txz
100% [==================================================================================================================================================================]
Downloaded 62593388 bytes
Checking downloaded file integrity...
Extracting files to /opt/calibre ...
Extracting application files...
Creating symlinks...
Symlinking /opt/calibre/ebook-convert to /usr/bin/ebook-convert
Symlinking /opt/calibre/lrf2lrs to /usr/bin/lrf2lrs
Symlinking /opt/calibre/fetch-ebook-metadata to /usr/bin/fetch-ebook-metadata
...
...
... whole lot more output.....
...
...
____________________ WARNING ____________________
Setting up desktop integration failed with error:
__________________________________________________
Traceback (most recent call last):
File "site-packages/calibre/linux.py", line 859, in setup_desktop_integration
File "subprocess.py", line 541, in check_call
CalledProcessError: Command 'xdg-desktop-menu install --noupdate ./calibre-gui.desktop' returned non-zero exit status 3
Creating un-installer: /usr/bin/calibre-uninstall
There were 1 warnings
* Setting up desktop integration failed with error:
Run "calibre" to start calibre
Install Dependencies (maybe?)
There's a tutorial on DigitalOcean called "How To Create a Calibre Ebook Server on Ubuntu 14.04" that served as a decent starting point, although it's a few years old and Calibre has been updated since then. Under Step 2, Gareth says to run two commands to install xvfb and ImageMagick.
Installing them did not affect my installation at all, but I'm not sure they're necessary anymore. Based on other stuff I read after installing them, I'm almost certain that you can, at the very least, forego installing ImageMagick.
sudo apt-get install xvfb
sudo apt-get install imagemagick
Uploading Your Library
If you don't have a Calibre library yet, you're sorta on your own. I don't think it's hard though! But it's not the path I took so I can't speak to it. The same tutorial referenced above has a section on creating a new library.
If you already have a library, like me, then you'll want to upload it. I just discovered the rsync command, and it's amazing. I didn't want to kill my home Internet connection for everyone else in the family, so I limited uploads to 70 KB/s, and passed it several options to show output, recover from network interruptions, etc. Over the course of a day or so, my entire library slowly got uploaded. You can read more about rsync here.
rsync --bwlimit=70 -e SSH -azP <your-local-library-directory> <your-username>@<your-ubuntu-ipaddress>:<remote-directory-to-copy-library-to>
Installing Self-Signed Cert (for secure communication)
If you're planning on assigning a domain name to your server, you might consider securing your site with a free certificate from Let's Encrypt. That's beyond the scope of this though.
If you're just using the droplet's IP address like I did, or you're already familiar with self-signed certificates, then follow these DigitalOcean instructions: How To Create a Self-Signed SSL Certificate for Apache in Ubuntu 16.04
When you're done, you should end up with a couple files (a public cert and a private key) - just mark the locations of each.
Add a Calibre User (for secure login)
If you don't want just anyone being able to view your library and download your books, you'll need to setup a user and password to login to your Calibre library.
calibre-server --userdb /srv/calibre/users.sqlite --manage-users
It's just a simple menu system. Follow the prompts to add your new user.
Run Calibre Server
Now that you have a new Calibre user and a self-signed certificate, you're ready fire up Calibre and access it securely. In order to run the Calibre service as soon as your server starts up, create a "service" file:
sudo nano /etc/systemd/system/calibre-server.service
And then paste all this into it (with necessary adjustments of course).
- The "user" is whatever user you created above.
- The "group" should probably be that user's group as well.. maybe? I set it to "ssl-cert" because that's the group my ssl key file was part of, and if the groups didn't match, then it couldn't load my key file on the next line. If I knew what the heck I was doing in the *nix world, I'm sure I could've come up with a better solution.
- The magic happens in the
ExecStartparameter, and there's a lot going on there. You can see all available options.- The "enable-auth" parameter forces you to login using the user you created.
- The "certfile" and "keyfile" parameters point to wherever you saved your self-signed certificate above.
- The last path is to wherever you created/uploaded your Calibre library.
[Unit]
Description=calibre content server
After=network.target
[Service]
Type=simple
User=<your-sudo-username>
Group=ssl-cert
ExecStart=/opt/calibre/calibre-server --userdb /srv/calibre/users.sqlite --enable-auth --ssl-certfile "/path/to/your/certificate" --ssl-keyfile "/path/to/your/key" "/path/to/your/calibre/library"
[Install]
WantedBy=multi-user.target
Start it up with this:
systemctl daemon-reload
sudo systemctl start calibre-server
If you want to verify that it's okay, run this:
=> sudo systemctl status calibre-server
● calibre-server.service - calibre content server
Loaded: loaded (/etc/systemd/system/calibre-server.service; disabled; vendor preset: enabled)
Active: active (running) since Sun 2018-04-22 02:29:07 UTC; 2s ago
Main PID: 1375 (calibre-server)
Tasks: 14 (limit: 4915)
Memory: 103.7M
CPU: 1.181s
CGroup: /system.slice/calibre-server.service
└─1375 /opt/calibre/bin/calibre-server /your/calibre/library/path
And if you have to make a change, then stop it first, and then reload and start it again:
sudo systemctl stop calibre-server
Your server should now be accessible via <your-ip-address>:8080. Sign in and check out your library... or add some books to it. It should all be secured over SSL as well.
The Bad.. and the Ugly
Mysterious Timeouts
My Google-fu totally failed me on this one. I even scoured pages 2 and 3 of the results, and could find nothing to explain why large downloads fail with:
I assumed it was my subpar Internet connection, so I tried again at work. I uploaded a few large files at one time, something that should've happened fairly quickly, but it hung for awhile and then displayed the same error. 😠
And why 3 minutes and 20 seconds? I scoured the source code for "200" but could find nothing relevant. I searched the Calibre issue tracker and got nada. I even checked out the systemd.service documentation to see if there was something else I could add to the "service" file - zilch.
The Calibre server doesn't use Nginx or Apache, but I'm not sure what it does use, so I'm at an impasse. I've uploaded some very large documents to the DokuWiki instance I setup, and it's had no problems even when it takes awhile... but it uses Apache so...
Similarly, sometimes I'd try to open a large document, and after waiting 5 or 6 minutes for it to attempt to "render" it, it just time out. This is on a fast Internet connection, so I can only assume the droplet I setup didn't have enough resources? It has 1GB of memory - how much more could it need?
The error mentions that something is "hung", but the DokuWiki instance that's running on the same box wasn't suffering at all, so whatever was hung wasn't affecting the entire machine. Sigh.
Poor Rendering
I really wanted to be able to find a book (full-text search works on the server too) and then render it right there in the browser. All my books, secure and in my control, available and viewable from anywhere in the world.
Then I opened up a book and saw this. Here's the cover page and first three pages of a BSA merit badge book.
It's supposed to look like this, which it does in a normal PDF viewer:
I thought perhaps it was somehow Firefox's fault, so I tried again on Google. Identical results. I tried again, with a different book on a mobile phone. And, well, it started off okay....
Waiting a few minutes for rendering..
Gotta download the entire thing now, making me wonder what in the world all that "rendering" is doing...
Oh yeah, looking good. :p
Most formatting lost. Code blocks look terrible.
Thinking somehow it might still be something I'm doing, I did some searching, and after about 30 seconds found this and this. I'm not alone in my disappointment.
Uninstalling
This should be easy-peasy. When you install Calibre, one of the last things it outputs is the path to a Python script that you should be able to run to uninstall it.
/usr/bin/calibre-uninstall
I don't have it. It said it installed it, but it didn't actually install it. I did a find / -name calibre-uninstall to search the entire file system, and it's nowhere. Awesome.
So instead, I stopped the server, removed the entire installation directory and the service file, and then rebooted to make sure any running processes were out of memory.
sudo systemctl stop calibre-server
sudo rm -rf /opt/calibre
sudo rm /etc/systemd/system/calibre-server.service
sudo reboot now
It must do a fair amount of browser caching, because even after doing all this, the browser would still show the main Calibre page. But when I opened a "private" session, which has no cache, the site failed to load as I'd expect it too.
Back to DokuWiki...
Note that your library is still on your server. You could also do a sudo rm -rf /path/to/your/library or you could just leave it there if you have other plans. For me, I plan on finding a way to migrate it all into DokuWiki and accessing it from there. With the help of the filelist plugin, originally written by Gina Häußge, you can end up with a nice looking list of files that load quickly. In the few weeks I've been using it, DokuWiki seems to use very little resources.
Here's a page in my wiki, with a list of user manuals. Easy to find, quick to render in the browser, can download it if I want too.
