The Abyss at the start of every software project

2 min readNov 26, 2023

SQL and relational databases were developed at a different time and age. It was a time when 1.5GB of HDD was “very large”. So, like ancient manuscripts, we wanted to store precious data, with data integrity in the most efficient way possible. When we succeeded at that, we needed a way to read this data, and businesses naturally wanted complex reports which ended up as complex SQL.

So far so good…

But, the problem was when we started using SQL everywhere for everything because admittedly we did a pretty good job with SQL as a language; although the databases could not keep up with it at scale. Even when technology progressed, data volumes increased exponentially and user load went up with time, we went the usual route because that is what we were all familiar with.

Unfortunately, it is relatively easy to get started with SQL and relational databases. Once you know you are trapped, it is too late already.

NoSQL databases are great for metadata; but not so great for storing important data where data consistency and referential integrity matter.

Some companies even start removing everything relational about a relational database in production, so things can keep working, leading to massive fights between the production DBAs and SQL Developers/ Architects

What are we doing wrong?

Not everything on a website is reports. Granted there are use cases where SQL is the easiest way to get the data you need — and Data Warehouses and Snowflake, etc have come up around that.

But look at everything else. Is SQL really the most scalable way to:

1. Simple reads and writes
2. Security implementations
3. Social Networking logic
4. Near/ Real-time anything

That is a lot of stuff as well. Things that do not easily fit into SQL — not because you can’t write the SQL for it; but because it does not scale well, and the performance is very poor.

You can buy a $50k server like some do, and move the problem to another day — but that may not be a viable option for everyone. And you can’t really buy $50k server per customer when you have 100 customers. Things get hard with shared infrastructure where one customer may be killing performance for every other customer on a server, and the list goes on & on.

Like I mentioned, the hardest time for an Architect is looking at this SQL Abyss right before the start of a complex enterprise application development project.

Reactor solves the problem. Arc makes it easy to implement our solution, even for existing projects.

#sql #databases #code #programming #product #scalability #performance #programming #design #software #startup #cloud #relationaldatabases




“If you see something that’s not right, not fair, not just, do something about it. Say something. Do something.” — Rep. John Lewis