Attention conservation notice: Intellectuals gathering in Berkeley to argue about "knowledge" and "revolution". This looks like fun, and if I didn't have conflicting obligations I'd definitely be there. From Data to Knowledge: Machine-Learning with Real-time & Streaming Applications May 7-11 2012 On the Campus of the University of California, Berkeley We are experiencing a revolution in the capacity to quickly collect and transport large amounts of data. Not only has this revolution...

Attention conservation notice: Only of interest if you (1) like hearing people talk about statistics and machine learning, and (2) will be in Pittsburgh next week. I have been remiss about advertising upcoming talks. Mark Davenport, "To Adapt or Not To Adapt: The Power and Limits of Adaptivity for Sparse Estimation" Abstract: In recent years, the fields of signal processing, statistical inference, and machine learning have come under mounting pressure to accommodate massive amounts of...

In which extinct charismatic megafauna give us an excuse to practice basic programming, bootstrapping, and specification testing. Assignment, R Advanced Data Analysis from an Elementary Point of View

Non-parametric smoothers can be used to test parametric models. Forms of tests: differences in in-sample performance; differences in generalization performance; whether the parametric model's residuals have expectation zero everywhere. Constructing a test statistic based on in-sample performance. Using bootstrapping from the parametric model to find the null distribution of the test statistic. An example where the parametric model is correctly specified, and one where it is not. Cautions...

A change to the lecture schedule, by popular demand! R programs are built around functions: pieces of code that take inputs or arguments, do calculations on them, and give back outputs or return values. The most basic use of a function is to encapsulate something we've done in the terminal, so we can repeat it, or make it more flexible. To assure ourselves that the function does what we want it to do, we subject it to sanity-checks, or "write tests". To make functions more flexible, we use...

Attention conservation notice: Academics with blogs quibbling about obscure corners of applied statistics. Lurkers in e-mail point me to this pushback against the general pushback against power laws, and ask me to comment. It might be a mistake to do so, but I'm feeling under the weather and so splenetic, so I will. In our paper, we looked at 24 quantities which people claimed showed power law distributions. Of these, there were seven cases where we could flat-out reject a power law,...

Attention conservation notice: 1500 word pedagogical-statistical rant, with sarcasm, mathematical symbols, computer code, and a morally dubious affectation of detachment from the human suffering behind the numbers. Plus the pictures are boring. Does anyone know when the correlation coefficient is useful, as opposed to when it is used? If so, why not tell us? — Tukey (1954: 721) If you have taken any sort of statistics class at all, you have probably been exposed to the idea of the...

1. I'd like to say that you have no idea how long I have waited to read something like this piece by Michael Stumpf and Mason Porter in one of the glossy journals. But that would be a lie, because if you've been reading this for any length of time, you know that the answer is, long enough to be very tiresome about it. If the referees, and still more the editors, at those journals can be persuaded to pay attention, we will be on track for my mid-2007 hope that "in five to ten years even...

The "curse of dimensionality" limits the usefulness of fully non-parametric regression in problems with many variables: bias remains under control, but variance grows rapidly with dimensionality. Parametric models do not have this problem, but have bias and do not let us discover anything about the true function. Structured or constrained non-parametric regression compromises, by adding some bias so as to reduce variance. Additive models are an example, where each input variable has a...

In which spline regression becomes a matter of life and death in Chicago. Assignment Advanced Data Analysis from an Elementary Point of View