Why we needed a visual query builder
Toucan Toco is a web-based data storytelling platform. It allows sharing actionable insights with non-technical decision-makers. These users are not data experts, but they need to access the right data to take valuable decisions every day. Hence our main focus was first to provide them with the best user experience on the market. An intuitive experience that looks and feels like their favorite everyday apps.
Our next challenge was the need for the simplicity of app designers. App designers are people who build Toucan Toco applications for operational end-users. For them, we released the Toucan Toco Studio, which replaced a lot of configuration code. It was an amazing step ahead!
The only part where code remained in the Toucan Toco Studio was the data querying. Querying data in Toucan Toco is the most technical part of an app designer job, and doing it in code mode can be painful. Yet our mission is to bring fun and simplicity to the business intelligence world! So we were not satisfied with writing code for data queries. That is exactly why we needed WeaverBird. To bring autonomy, simplicity, and speed to our app designers!
So what is WeaverBird?
WeaverBird is an intuitive graphical interface to query your data. You can feel at home and play with your data table. You can get the right data for the right story in a few clicks. Everything you do gets logged, step by step, in a transformation pipeline:
- Like a recipe, every step is clear, easy, ordered and reproducible. As you already know, at Toucan Toco we love stories, so we wanted you to be able to read your queries like a book!
- You can go back in time and see what your meal looked like 3 steps before. Or you can edit or delete a step without impacting other steps. So it is much more flexible than a book, and can act as a fabulous debugging tool!
- Is your file not in good shape for the data visualization you need? You would need to write VBA code to do that in Excel right? No panic, we got your back! We help you to perform those complex reshaping operations (pivot/unpivot data).
- Do you need to combine several data sources? No problem: you can either append or join data.
- In the next release, we will tackle the tedious window computations (evolutions, percentage of the total, top/flop by group, etc.). You need them in every project and still, it causes you a serious headache in most tools. With WeaverBird, you will be able to perform such computations in a few clicks.
How cool is that?
Why go open-source?
Open-source is in our DNA
Since Toucan Toco was started, we decided to build our web-based product on open-source technologies (Vue.js, D3.js, Python, MongoDB, Docker, Nginx, Redis, etc.)
The fact is: we are glad to be part of this world! And we want to contribute to it. We know we can benefit a lot more from collaboration than from retention. The most robust software products are open-source. Because they can leverage the incredible community that gathers around it. People bring their skills, ideas, feedback and use cases from various contexts. That’s one reason why we decided to make the code open-source.
We want to build an extensible product!
The second reason is that we want WeaverBird to be extensible. What does it mean?
WeaverBird offers an intuitive interface to manipulate data. But behind the scene, we need to translate interactions into a query. In a language that the underlying database can understand. As of today, the first release of WeaverBird makes the translation in… Mongo query language! Remember our database of our standard stack is MongoDB? First come first served
Well, that’s fine. But what if your database is a SQL database? Or if you need to extract data from a REST API? Or if you need to use WeaverBird as a data preparation tool that will generate Python or Java code for instance? You still want the same intuitive interface, but you may want to plug it into different backends! That is our vision for WeaverBird: one user interface to rule them all (the backends…).
Going open-source, we open to a whole world of different contexts and query languages. If you want to use WeaverBird to query your favorite database, you can start contributing right now! Or you may want to watch the project for future improvements. Or even start the project for the support! You can do all those things by visiting our Github.
Open sourcing was an organizational shift
The third reason is organizational and concerns the development workflow.
We first raised a question before going hands-on. How much do we want our new code to depend on our existing codebase? Our answer to that question was quite straightforward. WeaverBird was foreseen as an independent component that we should import in our main product. This presented several advantages:
- We could afford to open-source the project as a standalone product.
- The code maintenance is clean and easy from the main Toucan Toco product standpoint. There are no more than a few lines to import the desired version of WeaverBird.
- The standalone project benefits from an independent development cycle.
The latter point deserves special interest. It was an opportunity to be more agile and quick, and to provide a field for testing new ways of working as a team. Indeed, we could pick and try the best practices of open-source project management on Github:
- Asynchronous communication through pull requests (PR) and issues.
- Every feature, enhancement or bug first goes through an issue we discuss and prioritize. We attach labels, project milestones and assignees to every issue. From there contributors can tackle them as they would do when picking a card in a Trello or Jira backlog.
- Every pull request refers to an issue. Merging the pull request closes the related issue. This way we log the discussion history and rationale of every new line of code. And a percentage of completion of every milestone displays on Github. This not-so-anecdotal feature fosters focus and motivation!
If you want to go further on asynchronous communication, you can read this article. It explains how Alan decided to put in place a no-meeting policy, relying on Github issues.
why Will you love WeaverBird?
If you are a Toucan Toco app designer or any other guy who needs to build data queries, WeaverBird brings you:
- Autonomy: No need to learn an underlying query language. Weaverbird takes care of everything. The only things you need to know are what data you need to create your charts, and how to use a mouse and keyboard!
- Simplicity: Even the most complex operations are just a few clicks ahead. And any query can be read like a book.
- Speed: No need to write code, in a few clicks you get the data you need. It is easy to go back and forth through your data transformation history. That makes your debugging way more efficient than in code mode!
If you are a developer, you will love the open-source project and the ability to extend it with new transformation steps and translations in new languages.
