The future of e-commerce
What would an e-commerce platform look like if we designed it with today’s technology?
For the past four years we’ve been developing all kinds of commerce software, including mobile commerce apps, Magento API extensions, in-store applications, funny legacy software integrations, and everything in between.
It’s interesting that, in spite of the current trend of serverless and microservices architectures and the introduction of modern and faster languages like Go and Swift, more than a half of the e-commerce installed base is built over the classic LAMP stack, featuring monolithic architectures.
At the time of this writing, for instance, Magento is the e-commerce platform with the biggest slice of market share, and there are very big players using it. Thanks to its flexibility and extensibility it can pretty much support any imaginable use case in the e-commerce world. That doesn’t come for free, though: Magento internals are really hard to understand, from both the user and development perspectives, so if the default Magento installation is not enough for you (which is the case for most mid-size+ companies), you’ll invariably need the help of specialized Magento developers and have to rely on third-party extensions and services. In addition, achieving good performance and scaling or upgrading Magento can be really challenging quests for your team.
We’ll save a rundown of the strengths and weaknesses of current e-commerce platforms for another post, though. In this article we’d like to share our vision of how a modern commerce system would look based on our experience in this field, discussions we’ve been having with people all around the world, and a great big dose of imagination.
Our dream commerce system
Most people fail in life not because they aim too high and miss but because they aim too low and hit.
— Les Brown
Microservices architecture
Nowadays we could say that commerce applications have a “macroservices” architecture. In order for a retail company to work, it needs to coordinate a number of complex software pieces. As an example, a typical mid-size company can use Magento for its storefront, an ERP backstage to manage suppliers, fulfill orders, and consolidate financial details, and a bunch of scripts and third-party tools integrated along with them.
We think that breaking those “macroservices” into smaller ones is especially suitable for the commerce domain because it can be easily broken into smaller subdomains that have entities by themselves. For example, a typical e-commerce application would have an online storefront, an order management system, and a fulfillment system to optimize deliveries among other subsystems. These three subsystems can usually work independently, and, when they need to communicate, they can work as a chain using the data produced in one module as the input for the next one.
We also believe that we need to think about the software systems for a commerce company as a whole — there’s too much clunky proprietary closed software in the market that doesn’t play well with others. We think that the industry needs to switch to a more open approach and we are positive that a microservices approach would be a catalyst for a progressive change in that direction, where businesses would be able to keep using their current systems and progressively replace some of their functionalities as simpler modules are developed.
Splitting the main submodules of commerce applications into microservices could yield many benefits:
- Smaller, decoupled and domain-focused systems are easier to maintain than a huge, monolithic, and generalist codebase.
- The subsystems can be deployed and scaled independently, so you could have a lot of machines dedicated to the high-traffic storefront and just a few for your warehouse operations. You can also update one service without the risk of affecting the others.
- The system is easier to extend and customize. Well-known APIs between services can be used to create new modules or replace existing ones, without the other modules even noticing. Those APIs make them also easier to integrate with other systems.
- The system becomes independent of a specific technology, so you can use whichever technology best suits each specific problem.
Event-Driven; Reactive!
The commerce domain is real-time by nature: customers are visiting our physical stores and using our web and mobile apps, our employees are processing orders, and providers are delivering new goods, everything at the same time. Being able to react to these events is usually hard for most companies, sometimes impossible, requiring costly integrations and probably a dedicated IT team.
One of the hardest challenges we’ve found in commerce systems is keeping everything in sync and consolidated among all the systems in the company. Every company is unique, with ERPs, online catalogs, or point of sale software, all of them customized for the company’s very specific needs and all using a wide range of third-party services and extensions. It’s not surprising that it’s almost impossible to find solutions that work right out of the box.
This is not a problem exclusive to the commerce domain. Actually, it’s a more general problem: how to keep coherence in a complex system that is working in a real-time environment. We’ve learned that Linkedin had exactly the same problem and they found that moving to an event-driven architecture helped them to decouple their systems and make things easier to manage. Lately, we’ve also found ourselves using solutions like RabbitMQ or Apache Kafka to solve some specific sync problems and the overall feeling from our development team is that they want more of that.
Front-end development, on the other hand, needs to deal with many real-time events. We’ve been watching event-driven architectures like ReactiveX, reactive cocoa, and especially Facebook’s React becoming mainstream, making the industry pay more attention to initiatives like the Reactive Manifesto.
Reactive systems go a step beyond event-driven systems: they see the event stream as something that can be transformed into new event streams, so the system is defined in terms of events and transformations instead of data models. We think that this idea can really make the difference when designing commerce systems, and produce simpler and more decoupled subsystems.
Schema-less data and heterogeneous databases
If there’s something that all commerce companies have in common it’s that they’re all different. While from a high-level view they all look very similar, they have different policies in the way they do things and small but potentially significant differences in the data they manage. For instance, there are companies that have a single photo for each product in their catalog, while others have dozens for each version of the same product; there are companies that allow guest purchases and others that don’t. These small details have a bigger impact at the API/Integration level than you might expect, which is one of the reasons why integrations require that amount of work.
Most current commerce solutions rely on relational schema-driven databases and, in order to mitigate this problem, they have configurable database structures that are quite complex, and really hard to work with. In recent years there has been an explosion of NoSQL databases, which have influenced the way the SQL databases have evolved in turn. We believe that switching to a more open schema-less approach could significantly simplify critical parts of commerce software while keeping its flexibility.
We don’t think of NoSQL databases as a replacement for relational databases, but they’re definitely tools to consider when designing modern systems, because their different approaches may help to solve some specific problems. Using a microservice event-driven architecture makes it easier to use the best tool for each case, as each subsystem can have its own set of databases and use the event store to communicate with other parts of the system.
Event Sourcing and CQRS
Another of the big challenges in the commerce domain is traceability, the ability to know what happened in the software system or the business so decisions can be made or problems fixed. Most current systems focus on modeling the commerce domain, but they don’t usually feature a structured log of operations, so it’s common for some information to get lost in the process.
Event sourcing is a technique where a log of committed events is stored and then used to build the system state. This has many implications:
- It’s easy to replay events to rebuild current or previous states. This makes finding and fixing the origin of problems easier.
- Subsystems become more robust; if for some reason a subsystem loses its state, it can be easily recreated from the event log.
- The event store is an append-only database, making it easier to scale and manage.
- By making a copy of the production event store, you can introduce fake events to predict how the system will behave in certain circumstances.
In any given event-sourced system it is important to share a standardized approach to generating events, but every system can read the event source and build its own versions of the data in a different way. This is a technique known as CQRS (Command Query Responsibility Segregation), which can lead to more independence between subsystems, and data models better suited for the specific problem the submodule needs to solve.
Data, data, and data
Saving all the events in an event store has an interesting consequence: you have access to all the raw data from your company activities. That means that you can build projections of your data, use data science techniques to study it, and create machine learning or mathematical models. For instance, we’ve found data science and AI to be especially powerful in commerce applications to solve problems like:
- Automating recurrent decisions like stock balancing, when to order new goods from suppliers, or when and where to promote a new product.
- Detecting possible problems before they happen and sending alerts to avert them.
- Improved recommendation systems that take into account a wider range of parameters.
- Optimization of operations in general.
Thanks to the event stream and the reactive architecture, data-driven modules can be easily introduced without affecting other subsystems.
User interface independence
Retail companies need user interfaces that match their brands, to enable their customers to identify them and have a great experience. They also need more than a single UI: it’s not enough today to have just a website and mobile apps for the major platforms. We’re seeing more and more companies with in-store apps, digital dressing rooms, or smartwatch integrations, and they’re entering new fields like the internet of things or virtual reality.
We believe that a modern commerce framework must be designed independently from the UI and focus on the domain logic. UI templates are great for starting companies or low-budget projects, but if you have a great API and use modern tools, it’s usually harder to adapt a configurable UI to your needs than to start a new UI from scratch. In the case of customer-facing applications, it’s usually better to avoid templates if you want to achieve a great user experience. As a modern system, we want to avoid the need for re-engineering the backend in the future, even when augmented reality replaces all our devices.
Open source!
This is the last and most important point. If there’s something we’ve learned working on commerce applications over the last few years, it’s that privative software just makes things harder. Commercial “enterprise” systems tend to make the data captive and are usually non–interoperable: in many cases, the only way to make things work is to pay a fortune to the software supplier to make a custom integration. We believe that we need more open standards and improved interoperability, and we believe the only way to achieve that is to take the open-source path.
Update: Let’s make it real!
Since I wrote this article in 2016, we’ve been putting these ideas into action, testing all kinds of alternatives on different real-world applications. In 2019, pooling together the knowledge acquired, we unveiled the Booster Framework, an ongoing open-source project to easily build event-sourced backend applications on Serverless technologies like AWS Lambda and DynamoDB. If you try it or want to collaborate, leave us a message on our Discord channel!
