That's the scene in pretty much every successful quant team.

An Engineering Puzzle

My favorite thing about investing is that novel research is valuable. Investing strategies have no moat. Cost of entry to public markets is zero.  

On the positive side, this means solid, rigorous novel research is a perpetual priority in investing, which means it’s almost always an intellectual challenge. However, that poses a tricky puzzle for the engineers trying to build and manage the underlying systems.

Typically engineers do a bunch of things to make manageable, reliable systems. They commonly use integration, encapsulation and generalization. A totally heterogeneous, non-compartmentalized system is difficult to manage, maintain and evolve. And if a system is not maintained, inertia accretes over time.  

But a rigid system can make it hard to do novel research. In normal business this is annoying but mostly acceptable. However, this is terrible for investing.  If you are a great systems designer, you are thinking — “Aha! Microservices solve this problem”. That is normally correct, but see the section below for why that is not usually the right solution here.

Great quant engineers have come up four different solutions that I’m aware of.

Four Solutions

Four Omega We allocate substantial resources for engineering so we can build a new high-quality platform whenever we need one. We integrate the platforms where they are similar, particularly in back-office or execution. We aggressively move to build super-flexible components that are flexible enough to share. This solution is the most expensive, but the flexibility and unique tools tend to delivery very good returns. 

Castle — Like the Four Omega solution, we build a new platform whenever we need one. However, to allow for both great engineering with a smaller engineering team, we aggressively de-commission older platforms and prune the strategies that those support. We are very flexible, and we support research that needs heavy engineering. This solution is also fairly expensive, but costs are fairly reasonable. The flexibility and strong tech tends to produce pretty strong returns, although it isn’t quite as consistent.

Silverman — We build a single very high-quality engineering platform for research and inference and design a custom language for researchers to use that is very flexible for the range of research avenues that we predict are most typical. We can pursue research for a range of topics that require heavy engineering but we avoid anything outside that range. This solution offers great tech, but is not very flexible. This results in modest but fairly consistent returns. The firm typically relies strong relationships, brand, and product innovation to compensate for those returns.

Scrappy — We use quick-and-dirty engineering. We stay flexible and can develop a truly novel alpha strategy as long as it can be done with fairly light engineering. We are flexible and fast, but competitors might degrade our alphas more easily, since they can also do so with light engineering. This solution leaves us flexible, but the technology is fairly weak. We expect periods of excellent returns, but those will often be short-lived as others can more easily replicate our strategies. We can use successful periods to fund the switch to one of the other solutions.  

A Kobayashi Maru

If we can build a high-quality platform that is also super flexible — we can increase our speed in Four Omega and can expand our research range in Silverman. It might also be useful for the Castle and Scrappy solution.  

Microservices

Many engineers increase flexibility and heterogeneity using micro-services. Very famously, the solution that keeps Amazon very flexible. The micro-services architecture solves the problem by generalized encapsulation. If you build a uniformly-specified API around everything, the inside of it doesn’t matter.

But for two reasons, micro-services are a poor fit for the core tasks of investing: research and inference.

Research, or finding new alpha, requires global access to data at the detailed level. Encapsulation doesn’t work because you need to see everything.

Inference (often just called “production”) involves plugging current data into model to provide an up-to-date prediction. Financial inference, for obvious reasons, has a high urgency and a near zero tolerance for error. To prevent errors we need pre-empty sanity checking and tools for very rapid investigation and debugging. Too much encapsulation blocks some sanity checking and makes it near impossible to do rapid debugging. The micro-services architecture also is slightly biased toward interactive-driven stuff than it is to batch or streaming.