It had been a while since we attended an event like this. Commit is worth it, the venue is great, the organizers work incredibly hard and the content is very interesting, at least the sessions we chose on this occasion.

Things we learned:

Learn more

Data mesh is trending.

Really? What exactly is that?

Despite my Copilot (I’m writing all this in VSCode) telling me it’s a data architecture, I don’t think it is.

Data mesh is an approach to data that tries to solve scalability and data governance problems in organizations.

For me it’s more of a working methodology than a data architecture, which will be whatever it needs to be.

Creating a data office with many sources, some in near real-time, event-oriented or not, with Datalakes, databases, geospatial engines, with ETL or ELTs, with machine learning models and their MLOps, with data governance control, security, dashboards, scheduled periodic reports and a long list of other things can be hell.

The same hell that unfolds before you if - in any software project - you don’t have an action strategy and haven’t analyzed the problems before taking off your clothes and jumping in the pool (is it empty?).

So, Data Mesh, seems to emerge when organizations realize that the project to become a “data driven” company has gotten very complicated, we don’t have the information we expected or it’s incorrect, and we’ve spent a fortune. The kicker is that we have no idea how to fix it or who’s to blame. “The data is to blame, Madam President”.

Basically Data Mesh proposes decentralizing data and assigning owners to each domain. It’s cool and we’ll definitely study it at taniwa to learn how to do it better, but it sounds like the usual: Think first and then put love into it, rather than the opposite.

ClickHouse:

This is actually technological and we didn’t know about it.

It’s a column-oriented database instead of row-oriented. That is, all values of the “price” column are stored together. With this strategy, aggregation queries are greatly accelerated, which are the most common in BI systems.

I suppose inserts will be penalized and it will be typical to have a Postgres and a ClickHouse each with their own tasks.

They also have a very interesting Materialized Views concept that allows you to have a view of a table that updates at certain intervals.

This is very useful, for example, for having a dashboard that updates every 5 minutes. New inserts are calculated separately but you benefit from, you save, the calculations done up to that moment.

In summary, ClickHouse is a high-performance analytics database that offers extremely fast speeds, scalability, real-time analytics, advanced SQL and compatibility with multiple data formats.

We’ll do a post about ClickHouse as soon as we try it out. It’s open source. Icing on the cake.

OpenSpace on efficient code and the software consulting business.

Both topics interest us a lot and that’s why we attended the open space for each of them. The sharing format of open spaces is great, many people speak instead of just one, you learn to listen, you learn not to be a bore and to contribute according to each person.

In the efficient code part, they talked about the importance of software architecture and the need to have a good design before starting to program. It was led by the LeanMind folks; we liked them.

In the software consulting business part, they talked about market volatility, how to update rates, etc. We contributed our experience at taniwa, which is what it is, but we’ve been in the market for 10 years now with zero turnover:

• We focus on understanding our clients’ business.
• Analyze first, design second and program last.
• Transparency in everything we do including rates, proposals and salaries.

The transparency part continues to be a problem in the sector, I think, but for us it’s key and we sense it contributes to us still being alive as a company.

Room for improvement.

The food was good, but everything wrapped in plastic packages. We’re not contributing to sustainability this way and at an event like Commit they should be a bit more ecological.