After two short replies by Brooks and colleagues and Evans and colleagues that were published immediately after the original paper, nothing had happened for quite some time, although it was clear that many people had sent technical responses regarding that paper to Nature. Yesterday, however, Nature published two technical comments along with a response by He and Hubbel.

One comment is from Chris Thomas and Mark Williamson, who mainly reemphasizes some points that have also been made earlier by other people in the news and on blogs (see my previous post), namely that many, if not most current estimates of global species extinction rates are not exclusively based on SAR slopes, and that currently observed extinction rates are not generally lower than predicted.

The comment by Pereira and colleagues, “Geometry and scale in species–area relationships” concentrates on the geometry of habitat destruction that was assumed in the original paper. Their argument relates to a fact on which I already commented on in my earlier post: if species clumping is the reason for the difference between SAR and EAR, then the geometry of habitat destruction can not be inconsequential. For example, consider completely random habitat destruction, where one random individual after the other is removed – in this case, spatial clumping of species is completely inconsequential, each individual is removed with the same probability regardless of its position. Pereira et al. concentrate their reply on another geometry of habitat destruction: they concede that EAR is the correct model, assuming that SAR curves were generated nestedly, (outwards in the picture below), and assuming habitat destruction occurs with the same geometry. However, if the destruction would appear with a different geometry than the SAR/EAR sampling model, e.g. completely opposite (inwards in the picture), one would expect that inverse SAR estimates (inverse EAR) should be more appropriate.

Figure: Different geometries of habitat destructions, from Pereira et al.

The reply by He and Hubbel agrees to the points made by Chris Thomas, but rejects the arguments of Pereira et al. They raise serveral points, but I think the main one is that they simply think that one should assume the same geometry for habitat destruction than for statistical sampling. To avoid misinterpreting them, here’s the direct quote:

Pereira et al.1 commit a statistical error by confusing a specific configuration of landscape destruction with the statistical expectation. The SAR is a macroecological pattern defined as the expected number of species as a function of area. The word ‘always’ in the title of our paper2 refers to the fact that the expectation of extinction rate is always biased too high if one uses the backward power-law SAR method. One certainly cannot trust any single specific case of the extinction rate estimated in this manner to be reliable, and our result is a general proof that shows that the average extinction rate so estimated is always an overestimate.

I can understand this argument to a certain extent – clearly, if only one particular, very unlikely geometry of habitat destruction would change their results, this could be neglected. Yet, I don’t think it is likely that habit destruction is random in general. Rather, it clusters along roads, coastal zones etc. (e.g. Seabloom, 2002), so the geometry is usually non-random. I guess I would like to see more evidence still to be convinced that the geometry of habitat destruction is really inconsequential for estimating extinction rates.

I was interested in seeing whether the RNetLogo interface is fast enough for getting a lot of detailed data from NetLogo into R, so something like the state of each individual at each time step for a large number of individuals. As a test, I timed three options to get x and y coordinates of all individuals for 5000 individuals (for syntax see the manual):

>   ptm <- proc.time()
>   test1 <- NLGetAgentSet(c("xcor","ycor"),"turtles")
>   proc.time()- ptm
       User      System verstrichen 
       4.19        0.00        4.24 
>   
>   ptm <- proc.time()
>   test2 <- NLReport("[(list xcor ycor)] of turtles")
>   proc.time()- ptm
       User      System verstrichen 
       4.22        0.01        4.62 
>   
>   ptm <- proc.time()
>   test3 <- data.frame(NLReport("[xcor] of turtles"),NLReport("[ycor] of turtles"))
>   proc.time()- ptm
       User      System verstrichen 
       0.28        0.02        0.22 

and for 20.000 individuals:

>   ptm <- proc.time()
>   test1 <- NLGetAgentSet(c("xcor","ycor"),"turtles")
>   proc.time()- ptm
       User      System verstrichen 
      18.93        0.04       18.67 
>   
>   ptm <- proc.time()
>   test2 <- NLReport("[(list xcor ycor)] of turtles")
>   proc.time()- ptm
       User      System verstrichen 
      19.47        0.08       19.75 
>   
>   ptm <- proc.time()
>   test3 <- data.frame(NLReport("[xcor] of turtles"),NLReport("[ycor] of turtles"))
>   proc.time()- ptm
       User      System verstrichen 
       3.25        0.00        3.23 

So, considering that we speak about something like 2*N*8Bytes (double), it slow obviously, way slower than a call between R and C or R and Python would be, but I guess still acceptable for many applications. The third option, reporting each coordinate individually, is much faster than reporting them together, which is a little surprising according to usual programming logic because that way NetLogo has to go through it’s list of individuals twice. I could imagine that the list command of NetLogo is the culprit.

Anyways, simple reporters seemed the way to go, so I made a few further tests to see how this options scales with the number of individuals. Below you see the plot, the scaling on my laptop was roughly quadratic, unfortunately, not sure though whether this is because of the interface or because of the memory access of NetLogo, which seemed to do quite a bit of hard disk access for a larger number of Individuals.

ADDITION: Jan Thiele pointed out that something very close to the faster version 3 is already implemented in RNetLogo. Using the “as.data.frame=TRUE” command in NLGetAgentSet (see below, 20.000 individuals) yields runtimes very similar to version 3.

>   ptm <- proc.time()
>   test4 <- NLGetAgentSet(c("xcor","ycor"),"turtles", as.data.frame=TRUE, df.col.names=c("xcor","ycor"))
>   proc.time()- ptm
       User      System verstrichen 
       3.59        0.00        3.47 

As this list is already ordered, this option is probably usually preferable to version 3, where both x and y coordinates are in random order and therefore not necessarily belonging to the same agent when at the same position. I made a few checks, the scaling with the number of individuals seems to remain the same with this variant. Thanks to Jan for the comments!

It seems that power laws are being found virtually everywhere. However, what does that mean regarding their role in nature? To understand what all the fuzz is about, we have to recall that a power law is not only simply a convenient description of some empirical data – exponential, log-normal or other Weilbull distribution would often do the same job. Finding a power-law is so interesting because a “true” power-law may hint towards processes of self-organization such as self-organized criticality, preferential attachment or (although that is not really about criticality) optimal foraging such as in Levy flights (a good introduction to the mechanistic underpinning of power laws is Newman, 2005).

So, it is interesting to detect “real” power laws, and it has become something like a sport to find them in empirical data. The problem is that methods used by many studies are statistically questionable, and very often there is no mechanistic explanation at all that could suggest why a power law would be expected rather than, say, an exponential distribution. In particular, the tendency to find power laws is fueled by the still used, but very unfavorable methodology of linear regressions on log-log plots, although problems with this (bias, wrong significance levels) were repeatedly shown, as was the fact that log-log plots make nearly all decaying distributions look like a power law. A few people have taken it upon them in recent years to point out these statistical inconsistencies and also the lack of mechanistic explanations, for example Clauset et al. (2009), and its good to see that their warning voices finally made it from the more technical journals to the top, with the new publication of “Critical Truths About Power Laws” by Stumpf and Porter. As a side remark: Mason Porter also runs the power-law shop with fun T-shirts about power-law fits.

The arguments which are made in the article and the posts are somewhat more refined, relating not only to the profit-maximization, but also to some alleged dubious business practices, but I believe that the fundamental case remains the same: prices, contracts and access options are set by commercial publishers in a way to maximize their profits, which is, in the current system, contrary to the interest of scientists and the public that would like to maximize access to information.

And profits are remarkable: Elsevier, the largest academic publishing house, manages to create an impressive profit margin of 36% (substantially higher than the 28% of Apple which many view as insane). In a recent post , Mike Taylor makes the point that the profits of commercial publishers alone would be sufficient to publish all academic articles worldwide in an open-access system such as PLOSone, which charges around $1350 per article. And even this is a lot: one has to consider whether there is really a societal benefit in the same range basically only from the typesetting. If one is willing to drop layouting and editorial services as well, one could publish at even much lower costs. The arXiv’s FAQs, for example, state that their costs per article are smaller than 7$.

It seems that by now most people (including funding bodies and even some commercial publishers) agree on the need to revise the current publishing model in some way. Alas, getting there seems to be awfully difficult. On may think that the only thing one would need to do is to set up the same service that is currently provided as a payed-for open-access model such as the PLOS journals. Yet, despite the success of PLOS, their progress towards competitively excluding closed-access journals is veeeery slow. And new open access journals face not only the problem of gaining reputation and fighting problems with black sheep in their own rows, but also the problem of research budgets which, for the time being, seldom assign money for open-access publishing.

A much faster process of changed would probably be triggered if open-access models (even if commercial) could provide something that the closed, printed system can’t. Nature and PLOS have experimented quite a lot with web 2.0 type things such as blogs and comments, and other publishers have done similar steps. Yet, the current publishing model has proven remarkably immune to any changes; despite scientist being early adopters to the Internet, all more recent developments such as web 2.0, social media etc. have had hardly any impact on publishing and distributing scientific information (do you know anyone who clicks on the “like” button next to journal articles?). So, I don’t know where this leaves us … if someone could develop the science killer-app, a facebook for geeks, something that everyone wants to publish and comment in and that is open – that would be a game changer. A few random articles and websites that I found worth reading on that matter are this slide show by IanMulvany, this post by Nikolaus Kriegeskorte, and also the blog by Michael Nielsen, who’s new book I haven’t (yet) read though. As long as this does not happen, or a major top-down intervention occurs, it seems we’re stuck with doing very small steps towards open-access.

1. Generality – the correlation coefficient should be sensitive to a wide range of possible dependencies, including superpositions of functions.
2. Equitability – the score of the coefficient should be influenced by noise, but not by the form of the dependency between variables

These requirements sound sensible, but note that the consequential metric goes far beyond traditional measures of correlation, and rather towards what I would think of as a general pattern recognition algorithm that is sensitive to any type of systematic pattern between two variables (see the examples in Fig. 2 of the paper). One could probably argue about the second requirement, though, because there would be a certain sense in having more complicated relationships rated lower due to typical probability/parsimony arguments, but I can also see the argument against that.

Anyway, the authors go on showing that that the MIC (which is based on “gridding” the correlation space at different resolutions, finding the grid partitioning with the largest mutual information at each resolution, normalizing the mutual information values, and choosing the maximum value among all considered resolutions as the MIC) fulfills these requirements, and works well when applied to several real world datasets. There is a MINE Website with more information and code on this algorithm, and a blog entry by Michael Mitzenmacher which might also link to more information on the paper in the future.

Interesting, I guess, would be whether this method can be extended to higher-dimensional correlations.

ADDED LATER: Of interest may also be a post by Andrew Gelman, and a news article in nature.

As a prerequisite, we will use a few lines of code, very similar to a previous post on MCMC sampling. In the code, we create some test data in which a dependent variable y is in a more or less linear relationship with an independent variable x; define likelihood and priors for a linear model to be fit to the data; and implement a simple Metropolis-Hastings MCMC to sample from the posterior distribution of this model. It’s actually not important for the things that follow that you understand in detail what’s going on here, but for those who do not feel at ease about this, have a look at my previous post.

library(coda)

trueA <- 5
trueB <- 0
trueSd <- 10
sampleSize <- 31

x <- (-(sampleSize-1)/2):((sampleSize-1)/2)
y <-  trueA * x + trueB + rnorm(n=sampleSize,mean=0,sd=trueSd)

likelihood <- function(param){
	a = param[1]
	b = param[2]
	sd = param[3]
	
	pred = a*x + b
	singlelikelihoods = dnorm(y, mean = pred, sd = sd, log = T)
	sumll = sum(singlelikelihoods)
	return(sumll)	
}

prior <- function(param){
	a = param[1]
	b = param[2]
	sd = param[3]
	aprior = dunif(a, min=0, max=10, log = T)
	bprior = dnorm(b, sd = 5, log = T)
	sdprior = dunif(sd, min=0, max=30, log = T)
	return(aprior+bprior+sdprior)
}

proposalfunction <- function(param){
	return(rnorm(3,mean = param, sd= c(0.1,0.5,0.3)))
}

run_metropolis_MCMC <- function(startvalue, iterations){
	chain = array(dim = c(iterations+1,3))
	chain[1,] = startvalue
	for (i in 1:iterations){
		proposal = proposalfunction(chain[i,])
		
		probab = exp(likelihood(proposal)+ prior(proposal) - likelihood(chain[i,])- prior(chain[i,]))
		if (runif(1) < probab){
			chain[i+1,] = proposal
		}else{
			chain[i+1,] = chain[i,]
		}
	}
	return(mcmc(chain))
}

So, let’s run the MCMC:

chain = run_metropolis_MCMC(startvalue, 10000)

The run_metropolis_MCMC() function basically returns a posterior sample created by the MCMC algorithm as an array with one column for each parameter and as many rows as there are steps in the MCMC. This it is a little hard to see because the function return is already formatted as a coda mcmc object (the line “return(mcmc(chain))” at the end of the function transforms the chain into a coda object). Yet, you still see the structure either by str(chain), or by transforming the coda object back to a normal data-frame by data.frame(chain).

Some simple summaries of the chain

The advantage of having a coda object is that a lot of things that we typically want to do with the chain are already implemented, so for example we can simply summary() and plot() the outputs

summary(chain)
plot(chain)

which gives some useful information on the console and a plot that should look roughly like this:

Figure: Results of the plot() function of a coda object

I think the summary is self-explanatory (otherwise check out the help), but maybe a few words on the results of the plot() function: each row corresponds to one parameter, so there a are two plots for each parameter. The left plot is called a trace plot – it shows the values the parameter took during the runtime of the chain. The right plot is usually called a marginal density plot. Basically, it is the (smoothened) histogram of the values in the trace-plot, i.e. the distribution of the values of the parameter in the chain.

Marginal densities hide correlations

Marginal densities are an average over the values a parameter takes with all other parameters “marginalized”, i.e. other parameters having any values according to their posterior probabilities. Often, marginal densities are treated as the main output of a Bayesian analysis (e.g. by reporting their mean and sd of), but I strongly advice against this practice without further analysis. The reason is that marginal densities “hide” correlations between parameters, and if there are correlations, parameter uncertainties appear to be much greater in the marginals that they actually are. To check for pairwise correlations is quite easy – just use pairs on the MCMC chain:

pairs(data.frame(chain))

In our case, there should be no large correlations because I set up the example in that way, but we can easily achieve a correlation between slope and intercept by “unbalancing” our data, that is, having x-values that are not centered around 0. To see this, replace in the first large code fragment the creation of the test data by this line which creates non-centered x-values

x <- (-(sampleSize-1)/2):((sampleSize-1)/2) + 20

Running the MCMC again and checking for correlations should give you a quite different picture. I show an example below, plotted with a bit fancier plot function than pairs:

Figure: Marginal densities (diagonal), pairwise densities (lower panes) and correlation coefficien (upper panels) for the fit with unbalanced x-values

You can see the strong correlation between the first and the second parameter (slope and intercept), and you can also see that your marginal uncertainty for each parameter (on the diagonal, or in your plot() function) has increased. However, this is not because the fit is fundamentally more uncertain – the Bayesian analysis doesn’t care if you shift the x-values by 20 to the right. Unlike some other statistical techniques, the fundamental fit in terms of the chain has no problems with the correlations. However, it is problematic now to summarize the results of such an analysis e.g. in terms of marginal values. For example, it doesn’t make sense any more to say that the slope has a value of x +/- sd because this misses that point that for any given parameter of the intercept, the uncertainty of the slope is much smaller. For that reason, one should always check the correlations, and if possible, one should try to avoid correlations between parameters because this makes the analysis easier.

Note that we only checked for pairwise correlations here, there may still be higher order interactions that don’t show up in an analysis like that, so you may still be missing something. For that reason, the advice is to summarize the chain only when really necessary, otherwise things like the prior predictive distribution etc. should always be created by sampling directly from the chain.

Convergence diagnostics

Now, to the convergence: an MCMC creates a sample from the posterior distribution, and we usually want to know whether this sample is sufficiently close to the posterior to be used for analysis. There are several standard ways to check this, but I recommend the Gelman-Rubin diagnostic (check the coda help for other options that are implemented). Basically, Gelman-Rubin measures whether there is a significant difference between the variance within several chains and the variance between several chains by a value that is called “scale reduction factors”. To do this, we obviously need a second chain, and then simply run the commands:

chain2 = run_metropolis_MCMC(startvalue, 10000)
combinedchains = mcmc.list(chain, chain2) 
plot(combinedchains)
gelman.diag(combinedchains)
gelman.plot(combinedchains)

The plot should look something like this

Figure: Results of the gelman.plot() function

The gelman.diag gives you the scale reduction factors for each parameter. A factor of 1 means that between variance and within chain variance are equal, larger values mean that there is still a notable difference between chains. Often, it is said that everything below 1.05 is OK, but note that this is more a rule of thumb. The gelman,plot shows you the development of the scale-reduction over time (chain steps), which is useful to see whether a low chain reduction is also stable (sometimes, the factors go down and then up again, as you will see). Also, note that for any real analysis, you have to make sure to discard any bias that arises from the starting point of your chain (burn-in), typical values here are a few 1000-10000 steps. The gelman plot is also a nice tool to see roughly where this point is, that is, from which point on the chains seem roughly converged.

Improving convergence / mixing

So, what to do if there is no convergence yet? Of course, you can always run the MCMC longer, but the other option is to make it converge faster … the word that is used here is “mixing”, which basically means how well the algorithm jumps around in the parameter space … the mixing is affected by the choice of your proposal function. Two things can happen:

1. Your proposal function is narrow compared to the distribution we sample from – high acceptance rate, but we don’t get anywhere, bad mixing
2. Your proposal function is too wide compared to the distribution we sample from – low acceptance rate, most of the time we stay where we are

Figure: problems in the proposal function and how they show up in trace-plots

As displayed in the figure, these problems can be seen in the trace plots. Theoretical considerations show that an acceptance rate of 20-30% is optimal for typical target distributions, but this is in practice not so helpful because you can still have very bad mixing although being at this level by having the proposal of one parameter too narrow and the proposal of another parameter too wide. Better to look at the trace-plots of the individual parameters. Again, correlations are a problem, so if you have strong correlations in parameter space, you can get bad mixing. Using multivariate proposals that are adjusted to the correlation structure can help, but in general it is better to avoid correlations if possible. The good news at last: most of the more “professional” MCMC sampling software such as Jags or WinBugs will do these things automatically for you.

Google scholar launched a new service today, which is called “Google Scholar Citations” (see this post on the google scholar blog). Similar to ResearcherID, the commercial counterpart from Thomson Reuters (OK, google is also commercial, but you don’t have to pay), it allows authors to claim authorship of google scholar listed papers, and displays them on a central author page which looks like the one of Albert Einstein above (or check out mine if you want). Setting up the initial page was really ridiculously easy (well done google!), and they claim that new papers will be added automatically according to a statistical algorithm that attributes papers to authors based on previous decisions. One can also remove, correct or merge citations, see collaborators and citation metrics, and one can now search in google scholar for authors according to name or other attributes (e.g. all people with a label “ecology”).

Figure: Best-supported demographic models inferred by approximate Bayesian computation model-selection. From Lorentzen et al., copyright: Nature.