Just sent:

Dear <CLP Secretary>,

It's with sadness that I write to tell you I have decided to leave the Labour Party.

Despite having initially welcomed the arrival of a nominally more socialist national party leader, the Corbyn team have backtracked, vacillated, and failed to deliver any coherent progressive economic message. In their hands, Labour has become an incoherent policy vacuum with constantly bungled media management. I don't believe that leftist or revolutionary policies make a party unelectable -- look at the two big electoral shocks of the last year -- but an incoherent party that occasionally strikes radical poses (before immediately backtracking) is going nowhere.

This has been going on for many months, but I have remained a member in the hope that the party would rediscover its purpose -- currently to hold the government to account, at a time that that is needed more than ever before in my lifetime. I continue to respect the commitment and competence many of the Party's MPs and MSPs. But Labour's failure over the last 6 months to provide an alternative to the Government's partisan & scorched-earth attitude to Brexit, crowned by last week's (yes, again bungled) declaration that Labour MPs will be either encouraged or whipped to vote for the Government's A50 bill, without demanding answers to the detail, has been the final straw for me.

I appreciate that many of these issues are not primarily the remit of the Scottish Labour Party, especially with its current paltry Westminster representation, and that by cancelling my membership I am unfairly tarring you with the same brush. But this is the only way I can make my displeasure known to the Party as a whole, and with the rhetoric coming from Theresa May and other representatives of "Little England" I cannot even say that my commitment to the Union -- the main distinction between Labour and SNP -- is as unwavering as it used to be.

Yours, Andy Buckley

Having just completed the tortuous process of publishing an ATLAS data analysis, in particular 6 months of back-and-forward text-iteration, I find myself thinking of the excellent guidance on writing in English provided by George Orwell in his essay Politics and the English Language.

This is largely concerned with how banal, obfuscated and characterless English language can act as a cloak for vapid thought, and a smokescreen for vile political acts. (We've certainly seen plenty of that in the UK, USA, and parts of Europe in the last few years.) It is crucial short reading for anyone who aspires to good communication of facts and ideas, and I point all my students at it -- especially those who seem to believe that to be convincingly sciencey, a report has to be a Jackson Pollock composed of obfuscation and undefined technical jargon.

The pithy take-home message from the essay is a list of 6 excellent rules on language, emphasising clarity above all. The final rule is the one to rule them all: "Break any of these rules sooner than say anything outright barbarous." This invaluable instruction is expressed within a few short pages, to which I can't help but compare the ATLAS experiment's 51-page style guide, which says nothing remotely as interesting or profound.

But, as I was recently and painfully reminded, this tedious document is held up as the touchstone of English language by several individuals within the experiment. Frankly, the sort of people who will volunteer themselves to comment on hyphenation, the existence or not of possessive 's'es on words ending in s, or detailed footnote-marker placement are precisely the sort of busybodies who should be kept a good pole's length from any sort of editorial review. Rather than act as critical maintainers of ATLAS paper readability, they have made themselves the priests and guardians of an arcane grammar style guide, far more concerned with the trees than the wood. The joke is that the target journals (themselves hardly paragons of quality English) will anyway revert half these decisions, whether by design or accident.

As scientists, and particularly as scientists in a discipline with quite mathematical foundations, pedantry can come naturally. But we should -- must -- primarily concern ourselves with clear and correct description of our methods and results. After that comes readability, a nebulous concept involving not just clarity but also character. Formal grammatical and punctuational correctness honestly do not get a look-in: have a look through any of the best books on your shelf, either fiction or non-fiction, and you'll find "errors". Except they are not errors, they are character, voice, and awareness of what is important and what is trivial noise.

English language, like any other, is not subject to rules of algebraic correctness. It lives, breathes, and evolves. And its worth is tied into its variety and willingness to be interesting -- a style guide that thinks it appropriate to ban the use of the present perfect tense everywhere is a problem rather than a solution, by flattening the resulting language into a bland and repetitive drone.

Uniformly applied present tense, a limited and repetitive set of overly-adjectived nouns and ... this is the textual equivalent of speaking with someone on strong personality-suppressing drugs. Why would we want to inflict that on our readers in the name of "correctness"?

You should be concerned when a bunch of physicists or mathematicians appoint themselves as guardians of language. Unlike those subjects, language -- in general, and in its written representation -- does not have well-defined rights and wrongs. There are some robust don'ts for technical writing rather than creative, but far fewer of them than many seem to think. What matters is the readability of whole phrases or sections, not an algebraic absolutism on the level of grammatical atoms.

The experience genuinely made me question my interest in doing physics with ATLAS. That was several months ago, and fortunately it passed, but look at it this way: we already waive a lot of individuality by working in a big collaboration -- it involves following a lot of internal rules, of not having your name clearly identified with work which you led, or not having the right to be primary choice of conference speaker on your own work. So we have to make do with small things, and in corollary the small things matter... such as the small freedom to "voice" your own paper. It's hard enough to have busybodies with poorly calibrated comment filters make dubious change requests on work that they played next to no role in. And harder still to be asked to revert those comments in the next round of review, and so-on. But then to find that the collaboration places so little trust in its members' collective ability to produce a high-quality scientific paper, that it explicitly employs someone to make a bunch of acontextual (and frequently wrong) grammar complaints... blimey.

This is not to say that the feedback on language has been unhelpful. There have been places where phrasing or clarity has been improved. But it's a question of threshold: not a single hyphenation, nitpick on precise choice of sub-tense, or anal retentive replacement of "systematic" with "systematic uncertainty", or change from "observable" to "observable's value" has influenced scientific readability. And in some cases being technically correct really misses the point about communication: it has to be approachable and engaging, otherwise already dry scientific papers easily become about as captivating as the phone book.

So sure, give me feedback on language: but please keep to the stuff that makes a difference. Without this nonsense we'd have published nearly 6 months earlier, with a more readable document, and with my nerves significantly less shredded! But on the plus side, it's certainly revitalised my empathy for the authors of papers where I'm the reviewer. Proportionality and knowing when to stop... not natural physicist traits, but we need to learn -- especially when there's so bloody many of us.

As 4 months of EU referendum nonsense kicks off, I heard Prince of Darkness and modern Machiavelli George Osborne on the radio saying that "migrants must contribute to the exchequer" for 4 yrs to get in-work benefits.

I don't understand why there isn't more media challenge of this line. Or rather, I do but it makes me sad to acknowledge how pathetic our media is, even the part of it that's meant to be unbiased. The desire for a simplistic dichotomy on the news means that the "debate" has swiftly been reduced to whether or not Cameron's EU deal has "done enough" to protect Britain from the rapacious ways of those sneaky Europeans. To my great annoyance, the third, liberal, compassionate, and largely fact-supported view that actually we don't need protecting, that the EU while flawed has largely benefitted us, and that the "deal" is unfair and unnecessary... is nowhere to be seen or heard. And while we're immersed the political fantasy land of debating between two obnoxious counterfactual viewpoints, there's not much point in holding anyone to account is there? That would spoil all the fun.

But let's examine that "contribute to the exchequer" thing. After all, it's being used to justify 4 years of pay bias against a group of UK workers primarily identified by their country of origin. It's not "benefits" as commonly demonised by sections of the right-wing press so much as the conditions under which UK employment contracts are formed. Isn't it unfairness, racism and xenophobia to want to treat one group of workers differently in UK law due to -- frankly -- their ethnicity?

Now restricting access to unemployment benefit -- if done right -- seems kind-of reasonable. I can get on board with the idea that we welcome immigrant workers, but what country wants to invite non-workers to come purely on the basis of our unemployment terms? Ok, it turns out there aren't many such people and the policy would probably have little practical effect, but I can see logic in it and it's not obviously unfair -- again, if done right.

Child benefit was one of the other contentious points in the deal-making, and here I don't have the knee-jerk lefty liberal gut reaction that I'm perhaps meant to. I honestly get a hinky sort of feel about child benefit for non-resident children, which I guess many do. But the analysis isn't straightforward: for example, because we're not in the Eurozone, that benefit money can't really "leave the country" -- it goes into banks with a UK presence via forex, and in principle gets reinvested.

But restricting in-work benefits just feels like straight-out discrimination to me. A year ago a hot news topic was how in-work benefits accounted for a large fraction of welfare spending because of substandard basic wages; if this is the case, then in-work benefits are a government subsidy to employers. Removing that subsidy based on a worker's origin isn't "reasonable" as claimed, nor does it have anything to do with a largely Daily-Mail-fantasy "something-for-nothing culture", but simply ensures a non-level playing field. Which is probably the whole point, playing to the favoured tune of DM & Express readers, but to sell it as "fair" is disingenuous, and IMHO it was reasonable for EU states, particularly those who supply much migrant labour, to oppose it. (It is unclear to me from reports exactly what has been agreed for now, other than a vague "7 year term".)

And the "contribute to the exchequer" line is pure nonsense. A quick Google found this briefing document which claims that 90% of UK income tax, from 2/3 of employees is collected via PAYE. And since 2013 PAYE reports and payments need to be made every month. A newly-arrived migrant worker with a job will have "contributed to the exchequer" within a month, and of course we have been over the statistics that EU migrants actually make more of a net contribution to govt funds than natives do. So where did the magic "4-year delay" figure come from, other than an out-of-a-hat election-sweetener for bigots and xenophobes?

I was nearly about to post some of my own thoughts on this Fiscal Charter guff, then realised Richard Murphy has again said it for me. Annoying man, what with being reasonable, and right, and all that.

His brief point on the debt being only payable through growth is important -- the debt is £1.5 trillion. Trillion. In a super-optimistic scenario where the govt manages to run a surplus of £15 bn every year, it would take more than a century to pay that off. It's not going to happen. Even substantial reduction would require decades of consistent budget surplus. And "optimistic" here is debateable -- "surplus" sounds good, but I wonder if people would be so keen on "responsibility" if Labour started pushing the actually quite right-wing/centrist message that a government surplus means the state taking more from them in tax than it pays back into the economy?

Saying that we need to fix the roof while the sun shines (a motherhood & apple pie truism) is disingenuous if taken, as clearly implied, to mean now -- we just announced a second quarter of negative inflation. Amazingly, the Tory policy of repeatedly saying that we are in strong recovery is being parrotted by the media just like the "Labour caused the crash" message, so plenty of people do actually believe that the UK is right now living the economic high life. And simultaneously believing that we need to cut public services at the same time as we are "booming" -- I sense a psychological parallel with self-flagellating pilgrims, loudly declaring penitence for our sinful overspending. That and the mistaken belief that it's some Other scroungers who are going to take the hit.

Anyway, effectively the national debt is so large that its repayment is orthogonal to the annual deficit: direct repayments (as if we are sending a cheque to... who? every month) are not a means by which it will ever be significantly reduced. I think an important factor in understanding the debt is that it is not like the UK took out a bank loan and has a single fixed rate of repayment in perpetuity. Instead, we acquire public debt by issuing gilt bonds to buyers (typically banks). We choose how many gilts to issue, their expiry dates, and the repayment terms. If the economy is growing, the UK is a very safe investment, and we can sell plenty of gilts despite offering paltry interest rates -- thus the size of the debt (i.e. number of active gilts) could be maintained, but the interest bill (currently £43 bn / yr) can be reduced. It's even better than that -- only about 25% of gilts are index-linked to inflation (cf. the Government Debt Management Office) so not only does economic growth allow issuing of gilts at more favourable rates, it allows the debt to be "inflated away". Provided that wages grow with that mild (few-percent) inflation, all is well. But this strategy requires an economic committment to growth, which is not what's being offered.

We can't aim to really reduce the debt through cuts. The state supports business. The state _does_ business. Cutting the services that support the people who do the work and generate economic activity is a great way to stagnate that activity. The logical end-point of cuts is for the government to do nothing and the whole economy to grind to a halt -- hurrah, no deficit, but boo, no _anything_. Just about every independent economist under the sun agrees that publicly-stimulated growth rather than cuts is the best way to address economic stagnation -- that's completely standard economic policy that everyone somehow forgot via a collective brain fart in the last 5 years.

I now have a horse of my own in this race, since anticipation of severe research council cuts after the comprehensive spending review (perhaps coupled with an expensive re-organisation of research councils cf. PPARC/CCLRC -> STFC, just for irony value) has led to STFC cutting particle physics rolling grant funding by 10%. Many excellent postdocs have already lost their funding, and the UK brain drain has begun -- what took many years to build up can be squandered in an instant. This would be bad enough if it were necessary -- that it's because of an economically illiterate policy being pursued for the personal power ambitions of the Conservative Party in general, and George Osborne in particular, is inexcusable.

Oh look, I did write something after all. Catharsis.

POSTSCRIPT: Oh look, another Murphy piece making the same points, but better.

It's been a "fun" (aka horrifying) few months for liberal watchers of British politics since the May election gave us a Tory majority for the first time in a generation. I've found myself reading an awful lot of economics articles, books, etc. in that time, hopefully not to the detriment of my day job, because from a physicist's perspective it's intriguing to see how any progress can be made in a discipline which studies a chaotic system without repeatability or control over variables, and with undeclared political bias ever present in interpretations.

Still, there does seem to be academic consensus that the Coalition's signature austerity policy, now set to go into overdrive in this untrammeled Tory administration, damaged growth rather than ensured it. False but superficially compelling comparisons to household budgeting and the situation in Greece, via a compliant media, ensured that austerity was the dominant narrative through the electioneering, and Labour's "austerity lite" sales pitch failed to convince. (Several good summary articles on this from very respectable economists: Simon Wren-Lewis, Paul Krugman, and Robert Skidelsky (again). The blogs of Wren-Lewis, Richard Murphy, Frances Coppola, and Steve Keen have also provided me with much food for thought this summer.)

The pop psychologist in me wonders if the attachment to the austerity narrative is particularly British. We are of course (clichedly) notorious for not just gallons of tea, never saying what we mean, and social awkwardness, but also for a national obsession with self-deprecation and defeatism. Which is not to say we're not proud to be who we are, at least deep down, but that quite a bit of that national pride is about not thinking too highly of ourselves... quite the paradox.

Believing -- even when confronted by good numerical evidence to the contrary -- that a) Labour overspent badly and caused the 2008 crash, and b) austerity was a necessary and successful response to that by a financially responsible government, seems to tick several boxes of Britishness.

First, it was Our Fault. We all know deep down that it's always our fault; it's almost relieving to have it laid out clearly by Serious People in charge. I don't know why this would be such a British trait -- surely guilt should primarily be the burden of institutionally Catholic countries, and we've been suppressing that for centuries. Maybe we're just jealous.

It wasn't the Hard Working People who did it, though. We all like to think that we're the HWPs being talked about. But unscrupulous cheats and foreigners taking advantage of our much-vaunted hospitality. Ooh, don't you just hate it when people take advantage, and being British we were too polite to tell them to stop: hurrah for the plain-speaking Tories ready to grasp the nettle!

And second, obviously you can't spend your way out of a hole. After all we know that our nice-but-dim Labour government had overspent so badly (George Osborne told us so, and he looks trustworthy), and we have collectively absorbed centuries of Protestant teaching about the moral failures of spendthrifts who can't balance their household budgets.

Let's ignore that -- as surely anyone who slightly thinks about it must realise -- experience of household budgeting has little to do with running a national economy. Most particularly an economy with its own fiat currency... but then it's definitely too good to be true that we could not just spend our way out of trouble but also print our own new money to do so! "Too good to be true" re. QE is a great rallying call for British fatalism; perfectly engineered to be wisely muttered over the beer-wet bartops that Nigel Farage used to so often be pictured guffawing over (at least until UKIP and others were shafted by our electoral system and he ceased to be the must-have feature of the daily news cycle for another year or so... there's little Faragian guffawing afoot now, I suspect).

I don't have the experience of other countries' politics (despite meaning to follow the Swiss and French versions when in situ) to know if this suspicion is really true: are the miserabilist British actually more convinced of our own culpability, and more sceptical of hopeful glimmers than other nations? No idea, but we certainly wear it well, even when digging ourselves into a self-flagellating austerity hole for the second electoral cycle in a row.

Like many, though, I've found the rise of Jeremy Corbyn to offer just the sort of hope that my national instincts want to squish. It's second-order hope, because it's still around 5 years before the party he now leads has a pop at being in charge and enacting the policies that he's been espowsing for the last few months. And he has yet to convince his parliamentary party, significantly right of the party membership, to follow him. And because the hostile media are surely going to land shambling-unpatriotic-back-to-the-80s-Michael-Foot-2.0 attacks any minute now, and while bollocks they could still be highly destructive. I'm also really not a fan of his foreign policy history -- exactly the reactionary liberalism that champions illiberal causes so well documented by Nick Cohen.

But Corbyn's economic policy at least is more in line with academic consensus than the current regime, who must be worried that simply by mentioning the unmentionable from a high profile position as leader of the opposition, the Overton Window of economic narrative will be dragged leftward. Economics matters because it's central to all the other business of state, and without the excuse of "necessary" austerity there is little motivation for the ideological service cuts that the Cons have presided over and are gleefully extending.

There will be a plan to discredit Corbyn -- and admittedly he has plenty of potential chinks in his armour. But for a grizzled old Trot he showed plenty of media management nous in his leadership campaign, and ran rings around the gang of allegedly "professional" politician drones that were ranged against him. It's all to play for -- if only the collective British psyche will tolerate discussion of hope for a little while longer.

It's been a couple of months since I posted here, partly because of holiday, partly work, and partly because what spare time I've had has been spent voraciously following the economic and political conversations that the suprisingly interesting Labour leadership election campaign has raised.

I suspect I'll spew my thoughts on the latter topic, and the general state we're in, at a not-much-later date. But today an easier task: a few thoughts on a residential "creativity" course that I did over the last couple of days.

All in all it wasn't bad. My thought on this sort of thing is that although many things that the University promote to us sound like the worst kind of management guff, there is often a kernel of useful content. Getting it might require a trade-off, i.e. sitting through 8 hours of infuriating nonsense to get the benefit of 5 mins of mild insight, but taking the occasional risk is, I think, preferable to decrying well-meaning "personal development" forever, and maybe missing out on something useful. As it happens, I had attended an "Effective Communication" course just a month ago, which was well worth the morning that I spent on it. Sometimes I even surprise my cynical self.

Two days residential course is a lot more than a morning, of course, but it was an easy call to make when I realised that this course would count as a more enjoyable version of the mandatory "Entrepreneurship" course that I need to attend. The first day had its good moments, but suffered -- or rather I suffered -- from endless hours of repeating the truism that creativity is in practice always about inventive, revealing combinations of existing things, rather than somehow popping 100% new ideas, based on not an iota of pre-existing knowledge, out of the vacuum. It's a reasonable point, and helpful for some, but is like an unconstructive mathematical proof: sure, that's true, but it has nothing to say about either what combinations are interesting, nor how to bias ourselves toward finding them. Day 2 was a bit better, with a nice exercise on a "toy model" topic... apparently we need to pay the training consultant more to get the "proper" course that works on real situations. Oh well. There's more to come, and I don't regret the two days of attempted self-improvement.

One of the things that struck me, among all the "unlearning" and challenging of preconceptions, was that every time my colleagues (in the broad sense -- I was the only physicist in the room) were given the opportunity to ask a free-form question, it was about grant applications, reviews, publications, and all the other paraphenalia of academic life. This isn't really surprising on the face of it, but the form of the questions caught me by surprise: there was rarely a sense of perspective, or awareness that there is value in many of the things that we do regardless of whether they lead to grants or publications. It was implicit that the only thing that matters is those superficial aspects of academic -- the cargo cult stuff -- that performance reviews and promotion metrics focus on. There was even a repeated question along the lines of "Shouldn't I just leave creativity for later, since I think grant reviewers want to see safe proposals at this stage". Very sad.

The question I ended up asking is "Why am I not equally myopic?" or more self-critically, "What do they know that I don't?". One part of the answer really is personal. My career hasn't had a very standard trajectory for a HEP experimentalist -- post-PhD I took 4 years "out" of experiment to work among theorists, in an environment where I was essentially my own boss for 90% of the time, and had great support from the likes of Jon Butterworth. This gave me a lot of freedom to establish a value system that was all about the science quality rather than the cargo-cult trappings of academia -- which of course I am framing to you as being the One True Way. And my natural inclination is anyway that rocking boats are more interesting than the plain sailing type. But while there are certainly plenty of experimental HEP'ers whose primary focus is strategic moves for rapid career progression -- conventionally followed by endless moaning about the unfairness of all the teaching and admin that they couldn't wait to inflict upon themselves -- I think on the whole our peculiar type of science protects us from the worst effects of modern academia's performance metrics on at least two fronts.

First up is the meaninglessness of citation counts in experimental HEP. In this strange world where one need only qualify as an ATLAS author once, and never again be asked to do any service (or indeed any) work to justify the constant flow of papers with your name on it, publication measures like raw citation counts, paper counts, or h-indices mean virtually nothing. Their strongest correlation is with longevity in the field, and in particular longevity on major running experiments. While this gives some advantage to those who spent the 2000s on running Tevatron experiments rather than in-development LHC ones, I think the main effect is a sort of scroll-blindness: everyone's numbers are so large and so similar that there is no power of differentiation. And when it comes to exercises like the REF, pretty much every UK HEP group points at the same major papers and has a half-decent case for doing so. Having filled our CVs up to the brim with collective publications, there is actually remarkable freedom on the resulting Fermi surface for us to focus on what we find interesting, without needing to daily obsess about The Paper. I'm also thoroughly looking forward to the demise of that outdated, cargo-cultish mode of academic communication -- and wealth transfer to Elsevier -- but that's for another day.

The second point in our favour is STFC's group consolidated grants, which are necessary for functional operation of very large projects over decades, as opposed to the 1 or 2 year peripatetic funding that is the norm elsewhere. Again there is individual freedom to be found in collectivism (christ, this is going to start reading like Maoist propaganda any minute now) -- most of us need not be overly concerned with grant chasing, particularly as there really aren't that many of them to chase. One poor biochemist I talked to said he'd put in 25 grant applications in the last 2 years -- I'm pretty sure there aren't even close to that many funding calls in total in UK/EU particle physics over that timescale. I can imagine that if your life becomes that dominated by application writing, then just like the people in the Bill Hicks skit you start to forget that it's just a ride. You forget that the reason you do this is not really the funding, or the promotion, or any of that crap, but the satisfaction of a job well done and of increasing our collective knowledge and wisdom.

Particle physics isn't a panacea, of course -- there is still deep unfairness over the number of excellent postdocs that we train, overwork, and then fail to provide permanent places for. And our huge collaborations have brought new modes of careerist gaming, and perverse incentives to do bad or at least substandard science. Grants are still chased, albeit with more emphasis on personal fellowships than project funding; and to my colleagues I'm sure I sound like a broken record when criticising their daily attendance of interminable ATLAS videoconference meetings* -- the motivation for which is something like "Jesus is coming; look busy" in the belief that being sub-co-coordinator of the Paper Clips Working Group is going to have some positive career impact. But despite all that, I think we've been strangely blessed by the administrative implications of our supersized science: a sort of academic asymptotic freedom. Long may it last.

[*] I'll maybe also moan about the appaling quality of ATLAS meetings at a later date. I'm just going to say here and now that I stopped attending them about 2 years ago, unless I specifically have a horse in that race. I've yet to notice any adverse impact, and I have a lot more headspace for physics thinking. Try it.

I mentioned recently about the painful transition of this website from the Ruby/Rails Radiant content management server to... well, anything that would actually work. Given its popularity, I have to assume that Ruby and Rails can be made to work well -- or that 1000s of development teams are herd-following idiots, but that can't be true, right? -- but my experience was a nightmare.

Mysterious Rakefiles, UI-disaster server commands, awful integration with system packages, god-awful outdated Radiant documentation, and changes with every release. In the end, an update of the base Ubuntu OS completely broke Radiant. I tried using Ruby Gems in all the ways I could find, and updated every package to the latest that Radiant thought it wanted but couldn't get it to run again. I tried making a new Radiant site and migrating the database via the advertised commands: it crashed. And in the end it seemed that Radiant's own declaration of package dependencies was inconsistent. This was just the final straw after several years of expecting a Rails epiphany, and dreading every time that I'd have to restart the server and somehow get the creaking mess up and running again.

Well, enough was enough. I'venow moved to using the Nikola static site generator instead and couldn't be happier: it's got a great command-line UI, it's totally clear what's going on, I can hack and extend it if I want to, and my data is forever in a human-readable, editable (even when offline!) format.

Radiant's page data is categorically not available in a human-readable format, so a significant part of the effort to get this site back to life was the need to write a script to access its article database, and dump out the pages in a form I could use. Fortunately the db is just an sqlite single-file database, and the table structure was pretty simple, so the dump script was easy. Here it is for posterity:

radiant2txt

#! /usr/bin/env python

"Convert a RadiantCMS SQLite3 db file into separate page and header text files"

import optparse, os
op = optparse.OptionParser()
op.add_option("-o", "--out", dest="OUTDIR", default="out")
opts, args = op.parse_args()

import sqlite3
conn = sqlite3.connect(args[0])
conn.row_factory = sqlite3.Row
c = conn.cursor()

import unicodedata
def norm(s):
    return unicodedata.normalize("NFD", s).encode("ascii", "ignore")

import datetime
def date(s):
    return datetime.datetime.strptime(s, "%Y-%m-%d %H:%M:%S").date().isoformat() if s else ""

import textwrap, re
class DocWrapper(textwrap.TextWrapper):
    """Wrap text in a document, processing each paragraph individually"""

    def __init__(self):
        self.tw = textwrap.TextWrapper(width=120, break_long_words=False)

    def wrap(self, text):
        """Override textwrap.TextWrapper to process 'text' properly when
        multiple paragraphs present"""
        para_edge = re.compile(r"(\n\s*\n)", re.MULTILINE)
        paragraphs = para_edge.split(text)
        wrapped_lines = []
        for para in paragraphs:
            if para.isspace():
                wrapped_lines.append('')
            else:
                wrapped_lines.extend(self.tw.wrap(para))
        return wrapped_lines

dw = DocWrapper()

for page in conn.execute("SELECT * FROM pages"):
    pagename = page["slug"] if page["slug"] != "/" else "index"
    outfile = os.path.join(opts.OUTDIR, "%s.md" % pagename)
    with open(outfile, "w") as f:
        f.write("<!-- \n")
        f.write(".. title: " + norm(page["title"]) + "\n")
        f.write(".. slug: " + pagename + "\n")
        if page["published_at"]:
            f.write(".. date: " + page["published_at"] + "\n")
        else:
            f.write(".. date: 2008-06-01 12:00:00\n")
        f.write(".. type: text\n")
        f.write(".. category: blog\n")
        f.write("-->")
        f.write("\n\n")
        for part in conn.execute("SELECT * FROM page_parts WHERE page_id = ? ORDER BY page_parts.name", (page["id"],)):
            text = dw.fill(norm(part["content"]))
            if text:
                f.write(text + "\n")

To get a bunch of pages out in the format I wanted (my site was using Markdown syntax, so the script writes out to a bunch of .md files), I ran this like:

./radiant2txt myradiantsite/db/radiant_live.sqlite.db -o out-nikola

A bit of manual hacking followed, but 95% of the job was done by the script above. Use if you like, but don't ask me for support; if you need something a bit different, hack it!

Well, I'm blogging again, and it seems to me that if I'm going to write a letter to my MP on a national issue, then I may as well wear my heart on my sleeve and make it an open letter. So here's the latest --- in fact the first I've written for a while, due to the replacement of my long-standing traditionalist/institutionalist MP with a hopefully more sympathetic model:

Attn: Owen Thompson MP Midlothian

Wednesday 10 June 2015

Dear Owen Thompson,

I'm writing to ask you to sign Parliamentary Early Day Motion 49, "Freedom of Information Legislation, publicly funded bodies and the royal family."

This EDM calls for two important things: 1) that commercial confidence not be a justification for secrecy on public sector contracts (after all, we all are the paying clients), and 2) that the Royal Family not be given special exemption from the freedom of information rules that govern all other publicly funded bodies (again, we are all paying for them and deserve to know what we get for our money).

The first point is, I hope, self-evident. One the second, I think it is worth noting that the recently published Prince Charles correspondence with ministers has shown how the heir to the throne, regardless of whether you agree with his comments, has abused his position of conventional neutrality on numerous political issues. He pressed ministers to favour his own interests and organisations, and the evidence is that most felt compelled to respond more substantially than they would to an "ordinary" citizen.

Extraordinarily, David Cameron claimed that there was an "important principle about the ability of senior members of the royal family to express their views to government confidentially" -- it's somehow democratically important than unelected aristocrats have special access to legislators despite that being constitutionally taboo?! And rather than respond constructively to the exposure, there is clearly a determination from the Conservative Government simply to hide the abuse from public view. This must be opposed, and indeed the existing exemption of the Royals from FoI requests (in response to the moves to publish Charles' letters) should be repealed. Please sign the EDM that calls for this.

Yours sincerely,

Dr Andy Buckley

And that's that.

A couple of years ago, I realised that actually quite a few people were using my PySLHA library and plotter, and that I should write it up for them to cite, that being the tail-wags-dog way that the academic world rolls. So I knocked something together.

While writing this, using a LaTeX class file of my own devising, I decided I wanted to render my Python code examples better than the venerable listings package can do. And I found minted, a clever LaTeX package which automatically runs Pygments via the LaTeX chell escape mechanism. Problem is, the arXiv doesn't allow -shell-escape running of LaTeX; I had to beg a favour to get my original version of the paper uploaded.

Now I'm coming up to a major new release of PySLHA, it seems worth updating that arXiv note, and maybe even trying to get it "properly" published for the usual ineffable reasons. And another minted special request isn't going to wash. But I still like its output. So I just figured out what it was doing, and fiddled together a teeny bash script that provides the same code snippets statically. I don't think this exists elsewhere, but it's not worth a proper code release, so here's the while thing in case someone finds it useful:

pygtex

#! /usr/bin/env bash

## Write a .sty file defining the commands used in each Verbatim code block (bit hacky)
echo "" | pygmentize -l python -f latex -P full=True | head -n -10 | grep -E -v "documentclass|inputenc" > pygtex.sty

## Make a Verbatim code block for each input code file, transforming foo.ext to foo-ext.tex
for inname in $@; do
    outname=$(echo "$inname" | sed -e 's/[\ \.]/-/g').tex
    pygmentize -f latex -P verboptions='frame=leftline,framesep=1.5ex,framerule=0.8pt,fontsize=\smaller' $inname > $outname
done

I called it pygtex; you can call it whatever you like. It can be called like pygtex *snippet.py (if you've made code snippets with that name pattern) and will write out a pygtex.sty file, and a .tex file for each snippet. Then include them in your doc like this:

1
2
3
4
5
6

\usepackage{pygtex}
...
\input{foo-snippet-py}
...
\input{bar-snippet-py}
...

Enjoy.

I've been very disappointed to see 38 Degrees, a people-power campaigning organisation whose petitions I've often signed, going down the data-blind anti-corporate route that blights the likes of Avaaz. Straying from their typical social justice agendas, 38 Degrees have decided to direct their ire at the government for considering a repeal of the EU-wide ban on neonicotinoid pesticides that's been in effect in Europe for the last year.

Read more…