Offtopia — nothing personal

Back in the good old days of Yoom I had trained a neural density estimator on human body poses, and Michal Heker, Sefy Kagarlitsky and yours truly uploaded an arxiv paper about this. One trick described in the paper is that for proper density estimation in 6D rotation space, one needs to “de-Gram-Schmidt” the training data, similarly to the dequantization trick, but for points on a lower-dimensional manifold. However, the adoption of this trick was not smooth.

A coding agent is a dumb program that does what an LLM tells it to do; a mechanical executor. The LLM that drives the agent was trained on 1) linguistic correctness as an autoregressive text generator 2) human feedback. For human feedback, it was not trained to do “the right thing”, there is no way to train like that. It was trained for confirmation bias. That is, whatever one asks the agent to do, LLM instructs the agent to do that in such a way that he thinks the task was accomplished successfully. That means it is trained to “summarize” the work done in a way that you do not feel the urge to look at the code or throroughly check the results, relying on the agent’s diagnostic output instead. My personal experience is whenever opus/gpt/another LLM are allowed to “check itself” it always produces slop.

Nothing bad is happening to programmer jobs. there was a programming job bubble, and that bubble popped. That that the needle to pop the bubble was chatgpt and friends does not turn them into artificial intelligence.

I used to work with programmers of roughly my level, for years. a programmer of my level

I am going to job interviews, again. This time, a frequent request is: “Tell us about a failed project”. Of course, I never fail as a data scientist, how could I? A data science task involves a combination of domain knowledge and data, neither is held or produced by me, and a question someone else wants an answer to. All I do as a data scientist is encoding the domain knowledge as a model, updating the model’s latent variables based on the data, and computing a quantitative answer to the question. There are ways to ensure adequacy of the model, check convergence of inference, and express uncertainty of the answer. Just doing all these steps by the book ensures that there is absolutely no way to fail. Consider the task of classifying hand-written digits — although different models may have different accuracy, there is no way to ‘fail’ as long as one does things as taught. Or is there?

Thanks to the plague, we teach over Zoom, and have our lectures recorded. Many students do not attend in real time and instead replay the recordings at their convenience, and at 2x speed.

It is easy to label the students as superficial, but double speed replay has a perfectly valid though slightly embarrassing, for us the teachers, justification. When I was trained in public speaking, I was taught this basic technique for preparing a time-framed lecture:

arXiv | code

The ultimate Bayesian approach to learning from data is embodied by hierarchical models. In a hierarchical model, each observation or a group of observations $y_i$ corresponding to a single item in the data set is conditioned on a parameter $\theta_i$, and all parameters are conditioned on a hyperparameter $\tau$: \begin{equation} \begin{aligned} \tau & \sim H \\ \theta_i & \sim D(\tau) \\ y_i & \sim F(\theta_i) \end{aligned} \label{eqn:hier} \end{equation}

Probabilistic programs implement statistical models. Commonly, probabilistic programs follow the Bayesian generative pattern:

\begin{equation} \begin{aligned} x & \sim \mathrm{Prior} \\ y & \sim \mathrm{Conditional}(x) \end{aligned} \end{equation}

• A prior is imposed on the latent variable $x$.
• Then, observations $y$ are drawn from a distribution conditioned on $x$.

The program and the observations are passed to an inference algorithm which infers the posterior of latent variable $x$.

I taught a course on Bayesian data analysis, closely following the book by Andrew Gelman et al., but with the twist of using probabilistic programming, either Stan or Infergo, for all examples and exercises. However, it turned out that at least one important problem in the book is beyond the capabilities of Stan.

Gaussian processes are great for time series forecasting. The time series does not have to be regular — ‘missing data’ is not an issue. A kernel can be chosen to express trend, seasonality, various degrees of smoothness, non-stationarity. External predictors can be added as input dimensions. A prior can be chosen to provide a reasonable forecast when little or even no data is available.

Go gives the programmer introspection into every aspect of the language, and of a running program. But to one thing the programmer does not have access, and it is the goroutine identifier. Because the day the programmers know the goroutine identifier, they create goroutine-local storage through shared access and mutexes, and shall surely die.