Function plotting and the inverse cdf

I prevously wrote about using the inverse-cdf transform not just for its classic application of efficient random sampling, but as an optimal histogramming-binning generator mapping exactly to expected population quantiles.

(If you think of the random process that populates the histogram as itself being done via uniform random sampling of the unit cdf interval, mapped back into the coordinate value using inverse-cdf, it's even clearer why a uniform binning of the cdf interval would be optimal.)

In this post I'd like to show that we can usefully bend the inverse cdf at least a little further, into applications where there isn't even traditionally a positive-definite pdf. My motivation is efficient function plotting.

In the YODA new plotting system, I'd like us to be able to overlay analytic -- or at least Pythony -- functional forms on top of data- or MC-populated histograms. But just creating a linspace() of lots of points in $x$, and hoping that'll be enough to smoothly render the most rapidly varying bits of a plot -- the usual approach -- is unsatisfying.

If that plot has a mixture of curvy bits and straight sections, we could distribute a given number of points more efficiently, as straight parts need only one point at either end to draw a (Postscript or PDF) line between, and the points budget would be better spent in the curvy parts where straight lines are nowhere a good approximation. If we're stuck with using straight-line segments -- which we more or less are with matplotlib -- then distributing their endpoints more efficiently means we can get a good visual effect everywhere on the plot with fewer points, and hence a smaller file-size.

Let's think a little: as intimated above, the key issue is not the gradient $df/dx$ of the function $f(x)$, as I can draw a steep-gradient straight line just as effectively with two points as I can a flat one, but the second derivative $d^2f/dx^2$, aka the "curvature". So let's make a guess that what our brains want to see proportionally mapped out is point density in proportion to the curvature. So we can use the inverse-cdf method again, right: the cdf is $$F(x) = \int \frac{d^2 f}{dx^2} dx = \frac{df}{dx} ,$$ right?

Ah, but curvature can be both positive and negative: that messes things up. We really need $$F(x) = \int \left| \frac{d^2 f}{dx^2} \right| dx ,$$ but that's a more awkward object: we'd need to decompose into the set of positive-curvature and negative curvature intervals, and add up the integrals from each, e.g. $$F(x) = \sum_i \pm \int_{\omega_i} \frac{df}{dx} \Big|_i ,$$ with appropriately identified $\pm$ signs.

We could maybe do this for arbitrary functions with a symbolic algebra library like sympy but it's awkward. And doesn't solve the elephant-in-the-room issue that most monotonic functions, and hence most cdfs, don't have an analytic inverse. (This problem is also the major limit of the inverse-cdf method in general, of course.)

When in doubt, sample to victory! Let's just throw lots of evenly spaced points at our function, numerically compute an array of second derivatives and take their absolute values, then compute a numerical cdf by taking the cumulative sum of the array:

xs = np.linspace(XMIN, XMAX, 1000)
ys = f(xs)
ypps = np.diff(ys, 2)
ayps = np.cumsum(abs(ypps))
ayps /= ayps[-1]

where the final line normalises to make sure the cdf really adds up to 1 as intended. Then we can throw a more modest number of points linearly into the cdf interval, and numerically invert via straight-line interpolations:

def G(q):
    return np.interp(q, ayps, xs[1:-1])
qs = np.linspace(0, 1, N)
xs_opt = G(qs)
ys_opt = f(xs_opt)

Let's see the result on a pretty nasty function, $$f(x) = (2.5 - x) + e^{-2x} \cos(20 x^{0.7}) ,$$ using just 50 points:

/images/curvplot.png

Not bad, huh? The red line is this optimal sampling approach, the green (with visibly bad-approximation straight lines in the wibbly left-hand part) is the uniform straight-line approximation, and the true function is hidden underneath in blue. The grey lines in the background show the "optimal" sampling points, uniform in the $y$-axis of the cdf, overlaid in orange.

Now, I've cheated a little here, because to my eyes that assumption that the (magnitude of the) second derivative is proportional to the optimal sampling density isn't quite correct. Unsurprisingly, our visual cortexes are a bit better than that; in particular we seem to be fairly good at compensating for scale differences, so we see deviations from low-amplitude bits of smooth curve not so differently from on high-amplitude versions of the same shape. In the full code, reproduced below, I added a micture fraction $f$, so you can fade smoothly between purely "optimal polling" and "uniform polling" strategies. There are probably smarter ways.

I hope this was interesting. It feels to me like there's more potential to play and improve here, but hopefully we'll deploy this feature into YODA's plotting before too long!

curvplot.py (Source)

#! /usr/bin/env python

import numpy as np
import matplotlib.pyplot as plt

N = 50
XMIN, XMAX = 0.0, 2.5
F = 0.6

def f(x):
    "Function to plot"
    #return np.exp(-x) * np.cos(10*x)
    return (2.5 - x) + np.exp(-2*x) * np.cos(20*x**0.7)

## Naive version
xs_lin = np.linspace(XMIN, XMAX, N)
ys_lin = f(xs_lin)

## Densely sample to build an cdf of |f''|
xs = np.linspace(XMIN, XMAX, 1000)
ys = f(xs)
ypps = np.diff(ys, 2)
#ypps /= (ys[1:-1] + np.mean(ys))
ayps = np.cumsum(abs(ypps))
ayps /= ayps[-1]

## Mix with linear in F:1-F ratio
byps = np.cumsum([1. for x in xs[1:-1]])
byps /= byps[-1]
ayps = F*ayps + (1-F)*byps

## Invert cdf
# TODO: make more efficient!
def G(q):
    return np.interp(q, ayps, xs[1:-1])
qs = np.linspace(0, 1, N)
xs_opt = G(qs)
ys_opt = f(xs_opt)

## Plot
for x in xs_opt:
    plt.axvline(x, color="lightgray", linewidth=0.8)
plt.plot(xs, ys, label="$f(x)$")
plt.plot(xs[1:-1], ayps, "--", label="CDF, $F(x)$")
#plt.plot(xs[1:-1], ayps, "*")
plt.plot(xs_lin, ys_lin, label="Linear")
plt.plot(xs_opt, ys_opt, label="Optimal")
plt.legend()
for f in [".png", ".pdf"]:
    plt.savefig("curvplot2"+f, dpi=150)
#plt.show()

Optimal binning and the inverse cdf

One of my favourite tricks in numerical methods is [inverse transform sampling](https://en.wikipedia.org/wiki/Inverse_transform_sampling), which is a supremely elegant way to efficiently sample from a probability distribution.

If you know a distribution's cumulative density function (cdf) -- or can approximate it; see the follow-up post -- and the distribution is positive everywhere, as probability densities should be, then uniform sampling from the unit interval and applying the inverse function of the cdf to the samples is exactly the transform required. It's elegant and intuitive -- once you've seen it -- because of course large fractions of the [0..1] cdf interval are taken by regions of high density in which the cdf grows quickly.

The problem is of course that most interesting functions -- even the Gaussian, dammit -- don't have analytic cdfs. But nevertheless, use of the inverse transform is the analytic endpoint for lots of other strategies such as importance sampling, where an analytic distribution close to the desired one is a key ingredient.

What I see remarked upon much less is the equivalence of sampling and binning of histograms. In fact, a culture of making histograms assuming uniform widths for all bins is so engrained that you'll find [eight different strategies](https://numpy.org/doc/stable/reference/generated/numpy.histogram_bin_edges.html) "to calculate the optimal bin width and consequently the number of bins". This remarkably ignores that there is a well-defined ideal strategy for achieving equal relative statistical errors across a histogram, and that is to bin with variable widths in proportion to the reciprocal of the expected density function!

When following this recipe, by construction the product of density and width then gives equal bin populations and hence equal statistical stability. You can then choose the number of bins by dividing the sample size by the desired statistically stable population of each bin. Extending to fix a minimum bin width to respect non-statistical limits on binnable resolution is a fairly simple task.

Let's see this in action for a classic function with a huge dynamic range of densities: the [Lorentzian or Breit-Wigner distribution](https://en.wikipedia.org/wiki/Cauchy_distribution), describing physical resonance effects, with pdf $$f(x) = \frac{1}{ \pi \gamma \left[1 + \left(\frac{x-x_0}{\gamma}\right)^2 \right] }$$ and cdf $$F(x) = \frac{1}{\pi} \arctan\left(\frac{x-x_0}{\gamma}\right) + \frac12 .$$

With uniform binning, a BW distribution is doomed either to be so coarsely grained that all detail will be missing from the statistically robust peak, or (with finer binning) to have wild instabilities in the low-population tails. Like this:

/images/mee.png

But we can use variable binning, with uniformly spaced samples in \(F(x)\) corresponding to distribution quantiles, mapped back into the expected distribution as bin-edge positions using the inverse cdf: $$x = \gamma \tan((F - \frac12) \pi) + x_0 . $$

Given that these bin edges correspond to distribution quantiles, I guess we could call this approach "expected quantile" binning, or similar, if it needs a name. Here's a bit of Python/numpy code implementing this binning strategy for the Breit-Wigner of the Z-boson mass peak around 91.2 GeV:

M, Gamma = 91.2, 5.5
gamma, x0 = M*Gamma, M**2
qmin, qmax = 0.05, 0.95 #< quantile range to map
qs = np.linspace(qmin, qmax, NBINS)
xs = gamma * np.tan( (qs-0.5) * np.pi ) + x0
binedges = np.sqrt(xs[xs > 0]) #< eliminate any negative E^2s

And giving the following edge distribution (this version actually engineered to place the qmin, qmax quantiles at 70 and 120 GeV):

/images/bwedges.png

And finally the dynamically binned distribution:

/images/mee-dyn.png

Nice, huh? The full code listing follows. I'll follow this up with a post on how to use a variation of the same idea to optimally sample a function for visual smoothness, based on sampling density proportional to curvature...

bwedges.py (Source)

#! /usr/bin/env python3

import argparse
ap = argparse.ArgumentParser()
ap.add_argument("DATFILE", help="unbinned data file to read in")
ap.add_argument("OUTNAME", nargs="?", default="mee",
                help="hist name to write out as .dat and .pdf")
ap.add_argument("--dyn", dest="DYNBIN", action="store_true",
                help="hist name to write out as .dat and .pdf")
args = ap.parse_args()

import numpy as np
vals = np.loadtxt(args.DATFILE)


## Binning
NBINS = 50
RANGE = [70,120]
binedges = np.linspace(*RANGE, NBINS)
if args.DYNBIN:
    ## Dynamic binning, by inversion of the Breit-Wigner CDF:
    ##  https://en.wikipedia.org/wiki/Cauchy_distribution
    ## PDF = 1 / [pi gamma (1 + (x-x0)^2/gamma^2)]
    ##   with x = E2,  x0 = M2, gamma = M Gamma
    ## CDF = arctan( (x - x0) / gamma) / pi + 1/2
    ##   -> x_samp = gamma tan((rand - 0.5) pi) + x0
    M, Gamma = 91.2, 5.5
    gamma, x0 = M*Gamma, M**2
    qmin, qmax = 0.05, 0.95 #< quantile range to map
    qs = np.linspace(qmin, qmax, NBINS)
    xs = gamma * np.tan( (qs-0.5) * np.pi ) + x0
    binedges = np.sqrt(xs[xs > 0]) #< eliminate any negative E^2s


## Plot and save
import matplotlib.pyplot as plt
fig = plt.figure(figsize=(10,7))
fig.patch.set_alpha(0.0)
counts, edges, _ = plt.hist(vals, bins=binedges,
                            density=True, histtype="step")
plt.xlim(RANGE)
plt.xlabel("$e^+ e^-$ pair invariant mass, $m_{ee}$ [GeV]")
plt.ylabel("$\mathrm{d}N / \mathrm{d}m_{ee}$ [count/GeV]")
plt.yscale("log")
for ext in [".pdf", ".png"]:
    plt.savefig(args.OUTNAME+ext, dpi=100) # transparent=True
np.savetxt(args.OUTNAME+"-hist.dat",
           np.stack((edges[:-1], edges[1:], counts)).T)

Showing you care

I recently did an interview with the David Hume Institute -- through a friend -- about how (or if) personal financial issues affect my work. I won't forensically document how that aspect went, except to say it was a fun chat and they are doing good things. But the thing that's stayed with me is how I involuntarily laughed at the suggestion that my employer would provide any sort of benefits in addition to pay.

The thought had never even occurred to me! I've used government-organised schemes through work, like Cycle Plus and Childcare Vouchers, but the employer is a pretty passive partner in those. It turns out that a bunch of companies do actually offer staff discounts -- as a loss-leader, of course, and not entirely enticing: I have yet to make use of my small discounts at Kilt Warehouse or Beauty Boutique -- and again this is zero-effort from the uni. Virtually nothing involving the uni actively working to provide better quality of life for its employees. Should it be?

This brought into relief the contrast I see via engineers I play with in a band: their employers shower them with subsidised meals, massages, music tuition, rewards for patents, you name it. It'd be enough to make jealous, except they also sound like brutal workplaces in other ways, and despite the overwork and underappreciation, academia's relative freedom and autonomy is something to be cherished. But the very idea of a work culture that tries to make employees feel valued is alien if you've always worked in universities.

This doesn't surprise me: the UK university sector is in a terrible bind. Outside Scotland, the switch to funding through student fees at just over £9000/year led to a pullback of previously index-linked government funding, and there has not been political courage to significantly increase the charges in line with inflation over the last 12 years. In Scotland, the situation is even worse as the government payment per student has not even kept pace with the rest-of-UK figure. The sector is chronically underfunded, full of people working several job's worth of tasks because they believe in the mission, at a time when increased expectations from students (on the teaching side) and for project-management diligence (on the research side) mean we need substantially more staff across the board. I could also add the intrinsic inefficiencies and perverse incentives of a system that's evolved to recruit staff primarily for their research expertise, then drowns them in teaching and departmental admin, while their research colleagues get used to them "not being useful anymore". And that genius governmental ideas like systematic 80% FTE funding of research projects (i.e. systematically making research financially unattractive) make it impossible to balance the books without rinsing international students for fee money.

There is no capacity in the university system to make meaningful change, and were I a VC I'd probably do the same: try to keep the locomotive on the tracks for as long as possible despite it being chronically overburdened, lobby government in the background, and hope not to be in charge when the crash comes. But that fatalism seems even to have propagated to cost-free indications of caring about the frontline staff. In 20+ years I have never once seen a VC or dean of college visit a Physics department or take staff Q&As. How hard would that be, once a year? I also happened to chat with a previous vice-chancellor last year during strikes, who enthused "sensible folk, physicists" at the news there weren't many Physics staff refusing to mark exams; he, of course, was a perfectly nice chap but never taught or researched in a university in his life. How would he know what it's like? Well, he might have asked...

In the big picture, senior management giving staff their time and attention would say a lot more than trinkets. Or the occasional lick of paint on our decaying buildings -- I should say that the culture in departments is generally good, with the infuriating obliqueness of central management a source of solidarity! But in UK academia, for now, the old mood music remains the same: keep heads down, be grateful, and for god's sake keep the paying students coming...

More heat than light

Another week, another row about particle-physics methodology involving the field's latest engagingly controversialist internal critic -- older readers may feel a pang of deja vu from the "Not Even Wrong" years. But this time, the maelstrom has somehow escaped Twitter and been platformed in Guardian Science.

I feel a pang of guilt about criticising this article. After all, as scientists we are meant to question ourselves constantly -- the Royal Society, with a decent claim to being the leading grouping of natural philosophers as scientific method established itself in the mid 1600s, after all adopted a Latinised "Take nobody's word for it" as a motto. And within the field, I'd be lying if I claimed never to have felt frustration at perceived timidity and herd instinct. There's also a good practical reason not to comment, since that's probably what is hoped for, all publicity being good publicity when you have wares to promote.

But this really is a terrible piece, and on the whole I think better to engage than let such things slide and enter public consciousness unopposed. It starts with quirkily hypothesised portmanteau animals and the cunning plan of an invented group of zoologists to travel the world in search for them -- then asserts that this is what particle physicists, or at least beyond-Standard-Model (BSM) theoretical physicists do with their days. Experimentalists don't get let off easy: we are apparently slack-jawed rubes, so uneducated or uncritical about physics that we hang on every theorist's word. I get the feeling Sabine has not tried selling any theories to a CERN experimentalist audience recently.

This is deeply disingenuous stuff. First off, it's a gross mischaracterisation of the model-building process. Even as a non-expert, I know that the majority of models are proposed not just willy-nilly, but to solve a perceived problem -- or ideally, more than one. Where most of us differ from Sabine's value system is in what we consider an above-threshold modelling problem. She has asserted many times that the Standard Model can accommodate everything that has been observed, which is not true: neutrino masses require a mechanism not established in the SM, cosmological matter-antimatter asymmetry requires a mechanism of CP violation far stronger than achievable in the SM quark sector, and so-on. These seem fairly unambiguous areas where new mechanics are needed, and I've not even mentioned her preferred touchstone of dark-matter particle vs. MOND.

But most of us also take seriously, though perhaps not as seriously, vaguer questions of model stability (the hierarchy problems) and of why our model contains the components it does in the form it does. If we should take nobody's word for it, we should also be sceptical of fringe calls to just give up and accept the world as it seems to be. It is an entirely reasonable scientific endeavour to try and understand why things are the way they are. To deny that this is rational requires either a particularly naive take on philosophy of science, or bad faith. Just because the likes of the anthropic principle (things are the way they are because we're here to see it) have some intellectual merit doesn't mean that fundamental scientists must Eeyorishly resign ourselves to not even trying.

Most "organising" theories that might solve big conundrums of this sort -- ranging from more technical data-model discrepancies to the borderline-philosophical -- have consequences that could potentially be measured, and so we should search for them and cut away the models that fail to appear. And, to give us some credit, some such organising principles have borne fruit before, in the forms of the W, Z, and Higgs bosons, and various exotic hadrons. This is a long way from hypothesising acontextual flying cave-worms: it's more like -- to extend an analogy in a field I know as little of as Sabine does -- observing several separate evolutionary responses to selection pressures, hypothesising that they could interact interestingly, and proposing to look for them in places with the appropriate conditions. Maybe that's the sort of thing zoologists should be funding, maybe it's not, but it's not a category error to consider it.

This brings me to the final, and I think most offensive, aspect of the article, which is the argument that we either pursue these hypothetical hints of organising principles through clueless herd instinct or through rampant careerism. And the reason this annoys me so much is that there is undoubtedly a kernel of truth here. I think everyone in the field has at some point encountered a physicist who can't explain why they're interested in what they're doing, but it's what the group or their PI is interested in, or because they just like the process, or because it's an area publishing lots of papers and they'd like to ride that bandwagon (cf. the absolutely correct criticism of LHC 3sigma-anomaly chasing). Pin the blame for that on our intrumentalised version of research-performance measurement, a superheated academic job market (guess what, folks want a job in a stimulating area they spent their intellectually formative years mastering), and the raging bin-fire that is the rentierist academic publication business. By overextending this reasonable criticism to the sort of gasp-inspiring cartoon that gets one a Guardian splash, the whole argument jumps the shark and we learn nothing.

But, by-and-by, most of us know about this problem. Most research-active academics are trying to find areas where they can do something impactful, not just be a cog in the machinery... and actually, proposing or searching for unmotivated exotic new particles is not a rational bet. I've seen properly cynical, unmotivated models, and no-one outside the proposer's group works on them or pays the blindest bit of attention. Blunderbuss criticism in a very public forum also risks destabilising institutional support for the whole field. Funding agencies generally recognise particle physics as mostly worthwhile and balance their involvement across its facets, but this could become harder to do if populist tales of careerist physicists cynically living it up on taxpayer funds find purchase in the wrong ears.

So, not everything said is wrong. But it is dressed up in such a pantomime-dame version of the critique that it can't be taken seriously. And that's a shame: there are conversations here which could perhaps usefully be made more open and explicit. There are horrifying degrees of rentierism and perverse incentive in academic careers, publishing, and conferences -- let's talk about them, too. But straw-man arguments about modelling whimsy and bad faith distract from these real problems and more nuanced questions of scientific value; as quintessentially rational people, we need to reject them and platform the valuable discussions instead.

Leaving Labour

Just sent:

Dear <CLP Secretary>,

It's with sadness that I write to tell you I have decided to leave the Labour Party.

Despite having initially welcomed the arrival of a nominally more socialist national party leader, the Corbyn team have backtracked, vacillated, and failed to deliver any coherent progressive economic message. In their hands, Labour has become an incoherent policy vacuum with constantly bungled media management. I don't believe that leftist or revolutionary policies make a party unelectable -- look at the two big electoral shocks of the last year -- but an incoherent party that occasionally strikes radical poses (before immediately backtracking) is going nowhere.

This has been going on for many months, but I have remained a member in the hope that the party would rediscover its purpose -- currently to hold the government to account, at a time that that is needed more than ever before in my lifetime. I continue to respect the commitment and competence many of the Party's MPs and MSPs. But Labour's failure over the last 6 months to provide an alternative to the Government's partisan & scorched-earth attitude to Brexit, crowned by last week's (yes, again bungled) declaration that Labour MPs will be either encouraged or whipped to vote for the Government's A50 bill, without demanding answers to the detail, has been the final straw for me.

I appreciate that many of these issues are not primarily the remit of the Scottish Labour Party, especially with its current paltry Westminster representation, and that by cancelling my membership I am unfairly tarring you with the same brush. But this is the only way I can make my displeasure known to the Party as a whole, and with the rhetoric coming from Theresa May and other representatives of "Little England" I cannot even say that my commitment to the Union -- the main distinction between Labour and SNP -- is as unwavering as it used to be.

Yours, Andy Buckley

Science and the English language

Having just completed the tortuous process of publishing an ATLAS data analysis, in particular 6 months of back-and-forward text-iteration, I find myself thinking of the excellent guidance on writing in English provided by George Orwell in his essay Politics and the English Language.

This is largely concerned with how banal, obfuscated and characterless English language can act as a cloak for vapid thought, and a smokescreen for vile political acts. (We've certainly seen plenty of that in the UK, USA, and parts of Europe in the last few years.) It is crucial short reading for anyone who aspires to good communication of facts and ideas, and I point all my students at it -- especially those who seem to believe that to be convincingly sciencey, a report has to be a Jackson Pollock composed of obfuscation and undefined technical jargon.

The pithy take-home message from the essay is a list of 6 excellent rules on language, emphasising clarity above all. The final rule is the one to rule them all: "Break any of these rules sooner than say anything outright barbarous." This invaluable instruction is expressed within a few short pages, to which I can't help but compare the ATLAS experiment's 51-page style guide, which says nothing remotely as interesting or profound.

But, as I was recently and painfully reminded, this tedious document is held up as the touchstone of English language by several individuals within the experiment. Frankly, the sort of people who will volunteer themselves to comment on hyphenation, the existence or not of possessive 's'es on words ending in s, or detailed footnote-marker placement are precisely the sort of busybodies who should be kept a good pole's length from any sort of editorial review. Rather than act as critical maintainers of ATLAS paper readability, they have made themselves the priests and guardians of an arcane grammar style guide, far more concerned with the trees than the wood. The joke is that the target journals (themselves hardly paragons of quality English) will anyway revert half these decisions, whether by design or accident.

As scientists, and particularly as scientists in a discipline with quite mathematical foundations, pedantry can come naturally. But we should -- must -- primarily concern ourselves with clear and correct description of our methods and results. After that comes readability, a nebulous concept involving not just clarity but also character. Formal grammatical and punctuational correctness honestly do not get a look-in: have a look through any of the best books on your shelf, either fiction or non-fiction, and you'll find "errors". Except they are not errors, they are character, voice, and awareness of what is important and what is trivial noise.

English language, like any other, is not subject to rules of algebraic correctness. It lives, breathes, and evolves. And its worth is tied into its variety and willingness to be interesting -- a style guide that thinks it appropriate to ban the use of the present perfect tense everywhere is a problem rather than a solution, by flattening the resulting language into a bland and repetitive drone.

Uniformly applied present tense, a limited and repetitive set of overly-adjectived nouns and ... this is the textual equivalent of speaking with someone on strong personality-suppressing drugs. Why would we want to inflict that on our readers in the name of "correctness"?

You should be concerned when a bunch of physicists or mathematicians appoint themselves as guardians of language. Unlike those subjects, language -- in general, and in its written representation -- does not have well-defined rights and wrongs. There are some robust don'ts for technical writing rather than creative, but far fewer of them than many seem to think. What matters is the readability of whole phrases or sections, not an algebraic absolutism on the level of grammatical atoms.

The experience genuinely made me question my interest in doing physics with ATLAS. That was several months ago, and fortunately it passed, but look at it this way: we already waive a lot of individuality by working in a big collaboration -- it involves following a lot of internal rules, of not having your name clearly identified with work which you led, or not having the right to be primary choice of conference speaker on your own work. So we have to make do with small things, and in corollary the small things matter... such as the small freedom to "voice" your own paper. It's hard enough to have busybodies with poorly calibrated comment filters make dubious change requests on work that they played next to no role in. And harder still to be asked to revert those comments in the next round of review, and so-on. But then to find that the collaboration places so little trust in its members' collective ability to produce a high-quality scientific paper, that it explicitly employs someone to make a bunch of acontextual (and frequently wrong) grammar complaints... blimey.

This is not to say that the feedback on language has been unhelpful. There have been places where phrasing or clarity has been improved. But it's a question of threshold: not a single hyphenation, nitpick on precise choice of sub-tense, or anal retentive replacement of "systematic" with "systematic uncertainty", or change from "observable" to "observable's value" has influenced scientific readability. And in some cases being technically correct really misses the point about communication: it has to be approachable and engaging, otherwise already dry scientific papers easily become about as captivating as the phone book.

So sure, give me feedback on language: but please keep to the stuff that makes a difference. Without this nonsense we'd have published nearly 6 months earlier, with a more readable document, and with my nerves significantly less shredded! But on the plus side, it's certainly revitalised my empathy for the authors of papers where I'm the reviewer. Proportionality and knowing when to stop... not natural physicist traits, but we need to learn -- especially when there's so bloody many of us.

WTF is the 'problem' with in-work benefits?

As 4 months of EU referendum nonsense kicks off, I heard Prince of Darkness and modern Machiavelli George Osborne on the radio saying that "migrants must contribute to the exchequer" for 4 yrs to get in-work benefits.

I don't understand why there isn't more media challenge of this line. Or rather, I do but it makes me sad to acknowledge how pathetic our media is, even the part of it that's meant to be unbiased. The desire for a simplistic dichotomy on the news means that the "debate" has swiftly been reduced to whether or not Cameron's EU deal has "done enough" to protect Britain from the rapacious ways of those sneaky Europeans. To my great annoyance, the third, liberal, compassionate, and largely fact-supported view that actually we don't need protecting, that the EU while flawed has largely benefitted us, and that the "deal" is unfair and unnecessary... is nowhere to be seen or heard. And while we're immersed the political fantasy land of debating between two obnoxious counterfactual viewpoints, there's not much point in holding anyone to account is there? That would spoil all the fun.

But let's examine that "contribute to the exchequer" thing. After all, it's being used to justify 4 years of pay bias against a group of UK workers primarily identified by their country of origin. It's not "benefits" as commonly demonised by sections of the right-wing press so much as the conditions under which UK employment contracts are formed. Isn't it unfairness, racism and xenophobia to want to treat one group of workers differently in UK law due to -- frankly -- their ethnicity?

Now restricting access to unemployment benefit -- if done right -- seems kind-of reasonable. I can get on board with the idea that we welcome immigrant workers, but what country wants to invite non-workers to come purely on the basis of our unemployment terms? Ok, it turns out there aren't many such people and the policy would probably have little practical effect, but I can see logic in it and it's not obviously unfair -- again, if done right.

Child benefit was one of the other contentious points in the deal-making, and here I don't have the knee-jerk lefty liberal gut reaction that I'm perhaps meant to. I honestly get a hinky sort of feel about child benefit for non-resident children, which I guess many do. But the analysis isn't straightforward: for example, because we're not in the Eurozone, that benefit money can't really "leave the country" -- it goes into banks with a UK presence via forex, and in principle gets reinvested.

But restricting in-work benefits just feels like straight-out discrimination to me. A year ago a hot news topic was how in-work benefits accounted for a large fraction of welfare spending because of substandard basic wages; if this is the case, then in-work benefits are a government subsidy to employers. Removing that subsidy based on a worker's origin isn't "reasonable" as claimed, nor does it have anything to do with a largely Daily-Mail-fantasy "something-for-nothing culture", but simply ensures a non-level playing field. Which is probably the whole point, playing to the favoured tune of DM & Express readers, but to sell it as "fair" is disingenuous, and IMHO it was reasonable for EU states, particularly those who supply much migrant labour, to oppose it. (It is unclear to me from reports exactly what has been agreed for now, other than a vague "7 year term".)

And the "contribute to the exchequer" line is pure nonsense. A quick Google found this briefing document which claims that 90% of UK income tax, from 2/3 of employees is collected via PAYE. And since 2013 PAYE reports and payments need to be made every month. A newly-arrived migrant worker with a job will have "contributed to the exchequer" within a month, and of course we have been over the statistics that EU migrants actually make more of a net contribution to govt funds than natives do. So where did the magic "4-year delay" figure come from, other than an out-of-a-hat election-sweetener for bigots and xenophobes?

Debt, deficit, and economic incompetence

I was nearly about to post some of my own thoughts on this Fiscal Charter guff, then realised Richard Murphy has again said it for me. Annoying man, what with being reasonable, and right, and all that.

His brief point on the debt being only payable through growth is important -- the debt is £1.5 trillion. Trillion. In a super-optimistic scenario where the govt manages to run a surplus of £15 bn every year, it would take more than a century to pay that off. It's not going to happen. Even substantial reduction would require decades of consistent budget surplus. And "optimistic" here is debateable -- "surplus" sounds good, but I wonder if people would be so keen on "responsibility" if Labour started pushing the actually quite right-wing/centrist message that a government surplus means the state taking more from them in tax than it pays back into the economy?

Saying that we need to fix the roof while the sun shines (a motherhood & apple pie truism) is disingenuous if taken, as clearly implied, to mean now -- we just announced a second quarter of negative inflation. Amazingly, the Tory policy of repeatedly saying that we are in strong recovery is being parrotted by the media just like the "Labour caused the crash" message, so plenty of people do actually believe that the UK is right now living the economic high life. And simultaneously believing that we need to cut public services at the same time as we are "booming" -- I sense a psychological parallel with self-flagellating pilgrims, loudly declaring penitence for our sinful overspending. That and the mistaken belief that it's some Other scroungers who are going to take the hit.

Anyway, effectively the national debt is so large that its repayment is orthogonal to the annual deficit: direct repayments (as if we are sending a cheque to... who? every month) are not a means by which it will ever be significantly reduced. I think an important factor in understanding the debt is that it is not like the UK took out a bank loan and has a single fixed rate of repayment in perpetuity. Instead, we acquire public debt by issuing gilt bonds to buyers (typically banks). We choose how many gilts to issue, their expiry dates, and the repayment terms. If the economy is growing, the UK is a very safe investment, and we can sell plenty of gilts despite offering paltry interest rates -- thus the size of the debt (i.e. number of active gilts) could be maintained, but the interest bill (currently £43 bn / yr) can be reduced. It's even better than that -- only about 25% of gilts are index-linked to inflation (cf. the Government Debt Management Office) so not only does economic growth allow issuing of gilts at more favourable rates, it allows the debt to be "inflated away". Provided that wages grow with that mild (few-percent) inflation, all is well. But this strategy requires an economic committment to growth, which is not what's being offered.

We can't aim to really reduce the debt through cuts. The state supports business. The state _does_ business. Cutting the services that support the people who do the work and generate economic activity is a great way to stagnate that activity. The logical end-point of cuts is for the government to do nothing and the whole economy to grind to a halt -- hurrah, no deficit, but boo, no _anything_. Just about every independent economist under the sun agrees that publicly-stimulated growth rather than cuts is the best way to address economic stagnation -- that's completely standard economic policy that everyone somehow forgot via a collective brain fart in the last 5 years.

I now have a horse of my own in this race, since anticipation of severe research council cuts after the comprehensive spending review (perhaps coupled with an expensive re-organisation of research councils cf. PPARC/CCLRC -> STFC, just for irony value) has led to STFC cutting particle physics rolling grant funding by 10%. Many excellent postdocs have already lost their funding, and the UK brain drain has begun -- what took many years to build up can be squandered in an instant. This would be bad enough if it were necessary -- that it's because of an economically illiterate policy being pursued for the personal power ambitions of the Conservative Party in general, and George Osborne in particular, is inexcusable.

Oh look, I did write something after all. Catharsis.

POSTSCRIPT: Oh look, another Murphy piece making the same points, but better.

What I learned this summer

British politics has been providing some fascinating viewing recently. In particular I've been spending more time than ever reading political and economics blogs, and even the odd textbook, since the General Election in May.

In particular, Simon Wren-Lewis' [Mainly Macro](https://mainlymacro.blogspot.com/) blog on macroeconomics and particularly the "deficit fetishism" of the previous government (as a stick with which to beat Labour for being in power during a global crash) has been a profound influence in my finally growing up and learning a bit about how economics works.

The thing that's most captivated me is how as a country we voted fairly convincingly for a party which told serial untruths about our collective economic state of affairs, and got away with selling the opposite of established, evidence-based macroeconomics as if it was inarguable prudence.

And since the election, the Conservatives have turned out -- as if it were even remotely surprsing that this should be so -- to have a rapacious appetite for cutting things that they either promised to protect, or weasel-worded around during the campaign. And guess what, the promised extra childcare allowance was immedaitely pushed back to the end of this parliament (if ever), and the double-talk of a "7-day NHS" amongst "efficiency savings" doesn't really stack up. Well I never.

We certainly get what we deserve, but did we really earn this situation? Humans may not be the robotic rational actors beloved of undergraduate economics [On which note, I've found that it also loves triangles -- curves and integration are too hard, you see. Oh what I'd give for an economics textbook that serves the mathematically conversant.] but it would be doing the electorate a disservice to completely discount their decision-making abilities. But that requires that the information we're presented with be of good quality, and the media manifestly failed in that respect.

We should, if sceptically armoured, expect the majority of the print media to be compromised, but that the state-owned BBC failed to alert its audience to the disconnect between government economic narrative and the opinion of the vast bulk of macroeconomists beggars belief. Even the news that the IMF and the UK government's own Office for Budget Responsibility credited austerity with damaging Britain's economic recovery after the 2008 crash made nary a dent on the media and hence public consciousness. Simon Wren-Lewis has written very convincingly of how the election was won by "mediamacro" -- the consistent presentation of austerity as economic consensus when the exact opposite was true, until it became the accepted norm. Ed Miliband was berated for not mentioning the deficit at a time when other countries were not at all concerned by their much larger ones.

The deficit was a convenient political red herring to justify cuts, and it's one that we swallowed compliantly. The moment when Miliband's argument that Labour didn't overspend before 2008 was shouted down angrily by a television audience member -- "yes you did!" -- was where it was most obvious that it had all gone wrong. Rather than stand his ground and use statesmanlike gifts of passion, rhetoric, and er, fact to turn the moment, Ed capitulated and most of Labour has been wallowing in contrite austerity-lite speak ever since. Some of them may even believe it. There is a case that Labour mildly overspent -- through the boom years Keynes would have advocated running a small budget surplus but Brown kept postponing identification of the point in his infamous "economic cycle" when that should have happened. Actually checking the numbers shows that Labour's overspending was not extreme and certainly didn't cause the crash, but the bulk of the deficit came from the country's automatic stabilisers responding to the crash -- and due to the perverse economic policy pursued through the Coalition years we are still not back to where we were at the end of those allegedly profligate Labour times.

I think a significant factor in how the country got hoodwinked is hard-coded into the British psyche. It's rooted in the national feeling that it -- whatever it is -- is probably our fault. Sorry. You could see when Mr Yes-You-Did got so flustered, that he Really Believed that Labour had overspent. A quick Google for a UK deficit time series graph would have sorted that out, but there was no need to check because he Knew. We had earned it, you see: the narrative that it was our fault and now we're paying the price is almost irresistable to Brits. Which is peculiar because it really seems to me like the sort of self-flagellating we-are-but-worms logic that Catholic theology does so well, and Britain is if anything one of the world's more secular states -- but maybe that's because even we can't handle the combination of our depressive national psyche and a religion that thrives on generating guilt and contrition.

It's also very British to not belive the alternative story. That as a nation with our own sovereign currency we are able, at some level, to print money and buy our way out of our own debt. That seems like cheating. Not cricket. A definite feeling of sleight of hand going on.

But that feeling is somewhat more understandable. It's like believing the world to be flat -- based on local observations, a parochial observer of nature could reasonably conclude that the world is, if not perfectly flat, at least quite a decent approximation to it. You have to go beyond common everyday experience to get the global picture that shows the local one to be so much horse manure. Money is similar: pretty much everyone thinks they know what it is based on personal experience, but that's just the local picture. In personal experience money is conserved: your employer gives you some, so you now have it and they don't; you spend it, and now you don't have it and some shopkeeper does. Zero sum. Conserved. And let's not think too hard about why people will give you useful things like food, TVs and cars in exchange for what are actually just bits of slightly fancy paper...

The global view in this case is that money has evolved since the 1600s when trans-European trade and the emergence of financial centres started to erode the inconvenient old system of actually transporting quantities of rare minerals around the planet in order to exchange them for commodities. Barring a sudden discovery of vast gold, silver, or precious stone reserves in a mountain or cave somewhere (and an intelligent miner would keep schtum about that and certainly not dump it all on the market at once), money back then was pretty much the intuitive zero-sum quantity. But the arrival of fiat currency -- money which only has value because an institution, namely a government, says so -- changed the nature of money on a large scale. Fiat currency is a con trick at some level: it only works as long as people believe in it. If all the debts that had to be created to put money into circulation were to suddenly and spontaneously be cancelled, the whole system would whimper out of existence. But that doesn't happen. It feels to me like there's a parallel with entropy going on here: sure, that solution is possible, and indeed all the particle quanta in the universe could just resolve themselves back to the vacuum state -- but there is essentially only one way for that to happen, while there are umpteen gazillion ways for the money machine to keep on tumbling, tossing, turning, and generating emergent behaviours for the forseeable future.

A neat thing about this crazy con world of financial collective excitations is that if people will sell things, including their own time and effort, in exchange for that special paper, then a monetarily sovereign government can make more of that paper to get people to behave in a way that is collectively useful to the economy at large. For example, we are still effectively in a recession -- not technically, according to the politicised definition cooked up by Nixon, but certainly a heck of a sustained slump with zero interest rates, zero non-housing-bubble inflation, and still-high unemployment. Since the rates can't go lower than zero without people just taking their money out of the banking system altogether, attempting to solve the stagnancy problem by interest rate changes isn't an option. Notably, there's evidence that businesses and the rich are not even doing the beneficial investment job that capitalism is meant to do [Note, it's long-term investment that returns social value. This fact seems to have bypassed the starry-eyed fans of financial markets when things went the way of high-frequency trading and short-selling.], and are just hanging on to their cash reserves at present. This is precisely the situation in which it's the role of the state to step in and provide financial stimulus to engage the unused capacity in the labour market. And it can do so via the con trick -- but a useful one -- of printing more of that special paper.

In practice this mechanism -- quantitative easing -- is conducted via a lot of fiddly technical apparatus, such as issue of government bonds with particular rules, which may or may not be immediately bought by the central bank in the partial fiction that it is separate from the government and has simply spontaneously decided to do so, but the big picture is: It's Money, But Not As You Know It. So naive narratives about fiscal prudence need some serious updating, lest we cause ourselves serious harm in the name of applying common-sense rules to the new situation where much of our "debt" is owed to ourselves, and much of the rest is actually private savings. Can we be that sensible? And will the mainstream news media manage to keep up?

(Terrifying thought that the govt has not actually been bothered about causing real pain and impeding the economy to the tune of 1 trillion pounds, in the name of a political conviction that flies in the face of daily evidence -- even the evidence from their own watchdog)

It's probably our fault

It's been a "fun" (aka horrifying) few months for liberal watchers of British politics since the May election gave us a Tory majority for the first time in a generation. I've found myself reading an awful lot of economics articles, books, etc. in that time, hopefully not to the detriment of my day job, because from a physicist's perspective it's intriguing to see how any progress can be made in a discipline which studies a chaotic system without repeatability or control over variables, and with undeclared political bias ever present in interpretations.

Still, there does seem to be academic consensus that the Coalition's signature austerity policy, now set to go into overdrive in this untrammeled Tory administration, damaged growth rather than ensured it. False but superficially compelling comparisons to household budgeting and the situation in Greece, via a compliant media, ensured that austerity was the dominant narrative through the electioneering, and Labour's "austerity lite" sales pitch failed to convince. (Several good summary articles on this from very respectable economists: Simon Wren-Lewis, Paul Krugman, and Robert Skidelsky (again). The blogs of Wren-Lewis, Richard Murphy, Frances Coppola, and Steve Keen have also provided me with much food for thought this summer.)

The pop psychologist in me wonders if the attachment to the austerity narrative is particularly British. We are of course (clichedly) notorious for not just gallons of tea, never saying what we mean, and social awkwardness, but also for a national obsession with self-deprecation and defeatism. Which is not to say we're not proud to be who we are, at least deep down, but that quite a bit of that national pride is about not thinking too highly of ourselves... quite the paradox.

Believing -- even when confronted by good numerical evidence to the contrary -- that a) Labour overspent badly and caused the 2008 crash, and b) austerity was a necessary and successful response to that by a financially responsible government, seems to tick several boxes of Britishness.

First, it was Our Fault. We all know deep down that it's always our fault; it's almost relieving to have it laid out clearly by Serious People in charge. I don't know why this would be such a British trait -- surely guilt should primarily be the burden of institutionally Catholic countries, and we've been suppressing that for centuries. Maybe we're just jealous.

It wasn't the Hard Working People who did it, though. We all like to think that we're the HWPs being talked about. But unscrupulous cheats and foreigners taking advantage of our much-vaunted hospitality. Ooh, don't you just hate it when people take advantage, and being British we were too polite to tell them to stop: hurrah for the plain-speaking Tories ready to grasp the nettle!

And second, obviously you can't spend your way out of a hole. After all we know that our nice-but-dim Labour government had overspent so badly (George Osborne told us so, and he looks trustworthy), and we have collectively absorbed centuries of Protestant teaching about the moral failures of spendthrifts who can't balance their household budgets.

Let's ignore that -- as surely anyone who slightly thinks about it must realise -- experience of household budgeting has little to do with running a national economy. Most particularly an economy with its own fiat currency... but then it's definitely too good to be true that we could not just spend our way out of trouble but also print our own new money to do so! "Too good to be true" re. QE is a great rallying call for British fatalism; perfectly engineered to be wisely muttered over the beer-wet bartops that Nigel Farage used to so often be pictured guffawing over (at least until UKIP and others were shafted by our electoral system and he ceased to be the must-have feature of the daily news cycle for another year or so... there's little Faragian guffawing afoot now, I suspect).

I don't have the experience of other countries' politics (despite meaning to follow the Swiss and French versions when in situ) to know if this suspicion is really true: are the miserabilist British actually more convinced of our own culpability, and more sceptical of hopeful glimmers than other nations? No idea, but we certainly wear it well, even when digging ourselves into a self-flagellating austerity hole for the second electoral cycle in a row.

Like many, though, I've found the rise of Jeremy Corbyn to offer just the sort of hope that my national instincts want to squish. It's second-order hope, because it's still around 5 years before the party he now leads has a pop at being in charge and enacting the policies that he's been espowsing for the last few months. And he has yet to convince his parliamentary party, significantly right of the party membership, to follow him. And because the hostile media are surely going to land shambling-unpatriotic-back-to-the-80s-Michael-Foot-2.0 attacks any minute now, and while bollocks they could still be highly destructive. I'm also really not a fan of his foreign policy history -- exactly the reactionary liberalism that champions illiberal causes so well documented by Nick Cohen.

But Corbyn's economic policy at least is more in line with academic consensus than the current regime, who must be worried that simply by mentioning the unmentionable from a high profile position as leader of the opposition, the Overton Window of economic narrative will be dragged leftward. Economics matters because it's central to all the other business of state, and without the excuse of "necessary" austerity there is little motivation for the ideological service cuts that the Cons have presided over and are gleefully extending.

There will be a plan to discredit Corbyn -- and admittedly he has plenty of potential chinks in his armour. But for a grizzled old Trot he showed plenty of media management nous in his leadership campaign, and ran rings around the gang of allegedly "professional" politician drones that were ranged against him. It's all to play for -- if only the collective British psyche will tolerate discussion of hope for a little while longer.