The Challenge
When Commonstock started building out the social network for investors, they knew they needed a highly reliable messaging bus. At first, the team used foreign data wrappers in a relational database management system (Postgres) but found that approach to be highly inefficient because the wrappers were tightly coupled.
When the company expanded from a monolith to having multiple services that needed to communicate with each other, they needed a new consistent way to maintain state. This would enable them to stream messages and data in real time to end users, integrate with microservices easily, and flexibly scale up and down.
To drive value for its users, Commonstock searched for a software-as-a-service streaming solution that would enable the team to focus on developing the social platform.
Commonstock conducted extensive research on Apache Kafka, a popular event streaming and messaging platform with a managed service, to see if it could be their one-stop solution. However, after hosting it on AWS, the team found that Kafka lacked crucial features they needed.
“There were a lot of things Kafka did not do out of the box that we had to think about,” says Michael Smith, Director of Engineering at Commonstock.
A major problem was scaling. Scaling was difficult on Kafka because we would need to scale both the broker and the storage at the same time. “It would be such a waste to scale up the brokers for more storage for things that we’re not going to use often, but that we still needed,” Smith said.
Kafka also didn’t offer encryption between client and storage. Smith explained, “With Kafka, it felt like there was a lot more work to do, and it felt like a thousand paper cuts.” That was when they came across Apache Pulsar.
The Solution
The team was immediately impressed by Pulsar’s ability to scale brokers and storage independently, which would allow them to scale up and down dynamically without downtimes.
Pulsar’s tiered storage system was also attractive. Their old data would get shuffled out of Amazon S3 cloud objective storage, but would still be available through the same API.
These features made Pulsar the seamless streaming choice, especially during times when the team would need to spin up a new service and a new consumer and consume from the front of the queue.
However, Commonstock faced another hurdle. At the time, there wasn’t a managed solution for Pulsar. “Even though Pulsar was the better option, we were still leaning toward Kafka because some managed solution was better than none,” Smith said.
But this changed when they were introduced to DataStax Astra Streaming: a multi-cloud messaging and event streaming platform powered by Apache Pulsar.
Using Astra Streaming, Commonstock can now quickly create Pulsar instances, manage clusters, scale across cloud regions, and manage Pulsar resources like topics, connectors, functions, and subscriptions.
“Astra Streaming was perfect for us because it was very straightforward to get it up and running without taking up too much of the developers’ time, which is critical to us as a startup. This means we can spend more time on infrastructure,” Smith said.
Now, Commonstock uses Astra Streaming as a highly reliable messaging bus. Pulsar’s recommendation system consumes messages coming in from different services and regions on Commonstock’s social feed and determines what’s relevant and what’s not in real-time.
For instance, if a user follows someone on the platform, and the user likes a piece of content, it gets fed into the algorithm and recommended to all the followers on the social feed.
With the help of Pulsar and Astra Streaming, Commonstock transformed the tightly coupled data wrappers into a more loosely coupled process.
“Our end goal is not to have one service communicating with another service. Because when there is a dependency, if one service goes down, the other would too. But since Pulsar is just consuming messages, it’s not going to bring down any other services even if the messages stop,” said Smith.
The Results
Commonstock is more deeply integrating Astra Streaming into their infrastructure as they continue to expand their services. Currently, they’re planning to use Astra Streaming extensively to write new services that consume the old messages that are being generated, or to generate new messages or topics entirely.
DataStax’s customer service was a crucial factor in onboarding Pulsar, according to Smith. When they had to migrate their clusters twice, DataStax’s team of Pulsar experts was there to help.“The customer-service aspect of it was super important. We had to decide between spending our developer dollars on supporting a critical piece of software where we might not have as much in-depth knowledge, or spend it on something that’s business critical,” Smith said of his decision to adopt Astra Streaming.
Smith added that, “DataStax helped us get infinite retention and were very flexible in helping us host Pulsar instances in different regions. They really went above and beyond to accommodate our needs, to the point where we didn’t have to worry about anything.”
Since then, Astra Streaming has become a critical technology in the infrastructure that fuels the Commonstock platform, streaming real-time data to its users without downtime.
Smith has concluded that “DataStax is definitely the Pulsar expert so we can now focus on creating value for our business and customers.”
