Software – Brushing Up Science

Jupyter Notebooks are gone from my scientific workflow

TL;DR: I’ve just learned that the text editor Sublime Text can display images within Markdown files. Gone therefore is my need to use Jupyter Notebooks.

I was never a true convert to Jupyter Notebooks. I used them for several years, and saw their appeal, but they just didn’t quite feel right to me.

Most complaints against Notebooks are technical ones: they’re awkward to version control, they’re hard to debug, and they promote poor programming practices. But these issues are tangential to my complaints against Notebooks, which are are less concrete:

I’m always scrolling. It’s inefficient.
I don’t want to do work in a browser. Maybe it’s a weak reason, but I like keeping my scientific and programming tools separate from the browser.
Editing and navigating Notebooks feels clumsy. Maybe it’s a lack of practice, but I’d rather leverage the time I’ve invested in learning and setting up my text editor than spend time learning a bunch of new shortcuts specific to Notebooks.

Non-scientific software that helps me get science done

This is a shout out to all the software that helps my science happen despite not necessarily being developed for scientific purposes.

Fair warning, the list skews toward Linux programs since that’s what I use in my day-to-day work.

Tmux

I spend a lot of time at the command line. Or rather, command lines (note the plural). I often have four open at once. And I want to see all four at once, and jump back and forth between them all. Separate terminal windows or tabs don’t cut it. But Tmux does.

Here’s a pared-down example of how I might typically use Tmux: two panes, with one for editing text and the other exploring exploring directories.

Not gonna lie, Tmux is awkward to start with. The default keyboard shortcuts aren’t intuitive, simple things like copy/paste functionality don’t necessary work as you’d expect them to, and many online resources are outdated because older versions of Tmux used configuration commands that are no longer compatible.

But Tmux is well worth the learning curve.

A better way to code up scientific figures

I typically write 100–200 lines of code each time I develop a scientific figure that is destined for publication. This is a dangerous length because it’s easy to create a functioning mess. With shorter code fragments, it’s feasible to start over from scratch, and with thousands of lines of code, it makes sense to invest time upfront to organise and plan. But in between these extremes lurks the appeal to write a script that feels coherent at the time, but just creates problems for future you.

Let’s say you want to create a moderately complicated figure like this:

A script for this figure could be envisaged as a series of sequential steps:

Read data in from a csv file
Remove any flagged data
Create four subplots
Plot the first line of data against time
Label the y axis
Set the y axis limit
Repeat steps 4–6 for the second and third lines of data
Add the coloured contours and grey contour lines
Label the time axis
Add various annotations

Organise scripts and figures easily with Jupyter Notebooks

Keeping track of scripts used to generate figures is difficult. Before realising that Jupyter Notebooks could solve most of my problems, I would have directories with dozens of scripts with filenames of varying levels of ambiguity. Names that probably meant something to me at the time, but are hardly descriptive months or years later. Names like ISW_plume_plots.m, new_ISW_model_plots.m, and plot_model_behaviour.m. A certain PhD comic springs to mind.

Regardless of whether its Python, R, Julia, Matlab, or pretty much any other type of code, Jupyter Notebooks solve the problem. For example, I use a single notebook to archive the code for all figures in a paper and, more importantly, I can associate each set of code with the figure it generates. Rather than trying to remember what file I want, I need only remember which figure I want. (I say archive because I much prefer to do the bulk of my exploratory analysis in an editor. Alternatively, JupyterLab may work better for you.)

Continue reading “Organise scripts and figures easily with Jupyter Notebooks”

An improved Matlab command line (plus Jupyter Notebooks for Matlab)

The command line is a large part of any Matlab user’s workflow. This vital tool, however, isn’t as user friendly as it should be: it’s cumbersome to recall multi-line commands from the history, there’s no support for Vim key bindings, and there’s no syntax highlighting if using the nodesktop option (on a remote computer, say). Fortunately, there’s an alternative that avoids these problems: IMatlab. This kernel also lets you run Matlab code in Jupyter Notebooks (originally known as IPython Notebooks).

Continue reading “An improved Matlab command line (plus Jupyter Notebooks for Matlab)”

Invest in a good text editor

Scientists should invest time in a good text editor: pay the upfront cost of learning to use and customising a single editor for all of your text needs. This may be obvious to programmers, but less so to scientists who may have yet to recognise the benefits of a good editor.

Much scientific analysis and documentation can be achieved with plain text files (e.g., .py, .m, .f, .r, .tex, or .md). The default method to work with multiple file types is to use multiple IDEs (Integrated Development Environments): Matlab for m-files, Spyder or IPython notebooks for python scripts, TexStudio or TeXnicCenter for latex files, RStudio for R, or one of the countless editors for Markdown currently available.

Using a single editor has many benefits over using a range of editors within each IDE:

Continue reading “Invest in a good text editor”

Image conversion with a right-click

Image file manipulation is typically associated with programs like Inkscape, Photoshop, or even MS Paint. A graphical user interface, however, isn’t a pre-requisite for manipulating images. Often it is overkill. Simple tasks such as resizing or converting file type can be achieved with little more than a right click (and the power of ImageMagick).

ImageMagick is a command-line, Swiss-army-knife of image manipulation. It can achieve all the usual manipulations: sharpen, rotate, resize, convert to grayscale, convert to a sketch. The list goes on… Instead of harnessing all its creative power, here I show how ImageMagick commands can be added to the right-click menu (in either Windows or Linux) to achieve simple changes without ever opening the file. Continue reading “Image conversion with a right-click”

A better web reading experience

Web pages are typically filled with numerous links, menus, and widgets that are intended to make it easy to find what you want. However, these become distracting when actually reading content online. Further, many websites appear to ignore guidelines for readability. Fortunately, there are ways to improve your own experience when reading.

Continue reading “A better web reading experience”

LyX: a middle-ground to LaTeX and Word

LyX is a document processor that provides the power and professional-looking typesetting of LaTeX with the familiarity of an easy-to-use graphical interface à la MS Word. Effectively, it provides the best of both worlds. For someone without knowledge of LaTeX, LyX is less imposing and has a smaller learning curve. But even seasoned LaTeX users who have no desire to leave their favourite text editor can take advantage of some of LyX’s features.

Continue reading “LyX: a middle-ground to LaTeX and Word”

Using Inkscape for scientific figures, posters, and presentations

If I could offer only one piece of advice to any scientist on how to improve the figures and presentations they produce, it would be ‘Learn how to use Inkscape’. If you ended up here by searching for “Inkscape scientists” or some variant of that, then what follows is just confirmation that you’re looking for the right thing.

Inkscape is a cross-platform, open-source vector graphics editor. Let’s break down all those adjectives to see how they help:

Continue reading “Using Inkscape for scientific figures, posters, and presentations”