Combining Segment with Salesforce

Hello and welcome to another X Force Data Summit presentation. Today we have a data presentation living up to our name and it's by Tomas, who's a data analyst at Bitrise and he's talking about combining segment data with Salesforce and he tells me that this is a case study and that he'll go through the good, bad and maybe the ugly of what happened when he tried to do that. So here's Thomas.

Hello, nice to meet you.

This is, as Leonardo mentioned, will be a case study about how we implemented integration of segment data to Salesforce. So basically it's about how we, send these events information and what we've learned from this project.

So what is this that is first?

If you are in a mobile domain that maybe Bitrise rings a bell. But anyway, what you should know is it's a continuous integration and delivery solution specifically for mobile platforms. What makes Bitrise unique is the extensive support of mobile platforms and mobile development tools.

You can build your projects on iOS, Android, summary and Flutter React Native. You can add several integrations doing multiple tests, testing, building, code signing, deployment. Basically, you link, let Bitrise access your code base and GitHub or Bitbucket, and then events on these platforms triggered, builds and specific text and workflows on Bitrise.

And last year, end of last year, we started to scale up our sales team and there's this huge product came up that we should have only one platform for the salespeople and this should be Salesforce as soon as possible.

So we need these basic information like email, first and last name, company name, and some data specific customer attributes, number of build events, last repository I did, organization owners and some forty seven others.

Currently, but this we are still counting. It's an ongoing project. There are new ideas coming in that will be recognized that some other information is needed like subscription related information from site or report regularly.

And we have even tracking already implemented in Segment.

If you are not familiar with this too, but you've got a web application or mobile application, you should first check it out. It's a customer data infrastructure service provider. You can track your events, you can easily integrate specific services and you can enrich your data with Clearbit as well.

So this is how the interface look like in Segment. It's super simple. You've got your data sources and then you map them to your destinations. For example, here we've got JavaScript libraries, backend ruby libraries, sending information to postgres, big query or using this information, segmenting users in Google Analytics or amplitude, enriching information in support systems like Zendesk or Intercom. And what you can see here or that you can't see here is Salesforce. Segment actually provides a solution, a kind of destination for Salesforce leads. You can load any kind of data to Salesforce from your services.

And when this need from sales arose that we had two options that we investigated. One is a kind of development, but actually it was not on the the roadmap of developers or engineering teams. What you can do is that you are implementing this integration in your code or set up service in your web application that sends information.

At first it was only specifically focused on leads that we should generate leads from the user data. For example, somebody signs up that we should have this information in Salesforce, we should have additional information coming from the website that whether they have started to use our product or not, hold a big day, enter the trial or something like that. And usually these implementations happen with reasonable engineering resources that you set up kind of web service for that and stream this data to some of the Salesforce APIs.

But we had already had Segment and we had the Salesforce integration in Segment. But as you could see in my former chart, this is not the solution that we opted for since it's not that straightforward as it looks like the first time it looked like.

There are multiple other codes like page views or identifying the group of the user belongs to, but let's focus on two kinds of calls from the segment SDK.

One is the so called track code. You can imagine it's a method which tracks the events happening on on.

Mostly on the back end, the website back end is the most important source, not only for segment, but any other information because Bitrise is all about integration. So you link your code base on GitHub, then when there's a new pull request, it can trigger specific events on Bitrise, and then that's why we are concentrating mostly on the back end events.

For example, here that somebody finished a build on Bitrise and then there are other properties that we are sending with them.

Like how many concurrency they had, what was the ID of the build, what was the repository's ID, who was the owner of that repository, it was it a GitHub repository or BigBucket in this specific example, it's BigBucket. And the other code is the identify which as it says, is tying the users to the event.

And this is this is the method that can be loaded as a lead from from segment to Salesforce out of the box. Unfortunately, since we've got more than one hundred kinds of back ends on back end events and several others from the front end, this is simply too much, too many requests. There's no such API that can handle millions of concurrent API calls at the same time.

And also there could be other data sources that we should link or enrich with these kinds of identify codes. For example, subscription data that we should know that what is the customer ID in Stripe. Did the user have invoice? What was the total amount of that invoice?

And things like that. So the original idea was not to use engineering resources, but have the data analysts, in this case me, just link segment to Salesforce. But as I mentioned, and this was not the way forward, and that's why we added XPlanet or ETL service to the package and the current final data flow looks something like what I'm presenting here, that there are information coming from Bitrise sending to Segment. These are the events, the user information.

Segment also collects information from Stripe or Subscription Manager, the ones that I mentioned totally, invoices, customer IDs, so on. These all put to Google BigQuery and Xplan B is combining these two data sources, one from Segment to BigQuery and the other one other information from Bitrise, and it is Xplan B loading this data to Salesforce.

This lets us not only load leads, but also concentrate on converted objects, accounts, and contacts as well when updating these leads. So basically, we implemented two packages and I'm explaining one is inserting new leads from recent registrations. The second one is updating the existing leads context and account.

Both are scheduled daily.

Xpeng is good at is batch processing, so it's using the bulk API of Salesforce so you can throw multiple records at the same time on Salesforce. And again, that we thought that this will be a solution. This will be simple. We are just writing some SQL code and then the data will flow.

But you should know that Bitrise, there are two kinds of entities. You can have user, but you registered on the application, but you can belong to multiple organizations as well. But the organizations behave some kind of similarly users in a way that they can own repository. They can have builds again.

So this is is much more complex. So the first iteration we've tried is linking, having different sources and explaining and linking them in this kind of octopus way that there was a monster series of joints and components. It was super hard to debug.

The package exponential, the complexity of the package exponentially grew when when we added new data sources.

And it's not a fourth of Explanti, it's a fourth of how we scope the project and how we misunderstood the scope. That first it was just only the leads, it was only a couple of information, that's first name, last name, email, so on. Then we recognized that we've got other data sources, there was new ideas coming from new newly hired sales representatives and so on and that's ended up fifty different attributes and several different sources. So we had to refactor all these packages.

And basically, we use these kind of joints or logic of data integration is all written in SQL. This is what data people talk. This is simple. This is easy.

You can easily debug. You can hand it over to other analysts.

Two years ago, when we started to use XPlanity, it was a very important aspect or feature that we was looking for in or at that point of time, future ETL, is that it should support SQL out of the box.

And because of these cases that simply this is the language all data people speak, this is simply efficient and you can iterate over so we can easily add new attributes or remove them. For example, when we are recognizing that it shouldn't the update package shouldn't overwrite existing names or email addresses because there could be other sources in Salesforce enriching this information.

But also there are limitations of this solution that we go forward. As I mentioned, are, Expert is very good in batch processing, but not supporting data streaming. There could be specific cases where you need real or almost real time data, and this is not a case or this is not part of the scope of this, that was not the scope of the project.

But there could be specific product verified leads, for example, that we see existing paying user is investigating enterprise packages and we should already notify or immediately notify the sales representative that can reach out in that specific region.

And this is something that cannot be done with the current setup. They can have all the new leads, all the new registered users, all the latest information updated on existing leads and contacts and accounts each day, but we cannot trigger assignment rules. For example, we know, for example, from the billing address that this specific organisation is in Europe or US or any other part of the world, but and we should assign this lead to the sales representative of that region, but unfortunately we can't do that because the bug API, since it's processing several records at the same time, can't investigate them one by one. What is the specific property of the billing address to assign that specific lead to the corresponding sales representative.

What you can see here now is a kind of solution with I had to admit that we haven't implemented it yet. It's just still under investigation or POC that how we can stream specific events to Salesforce or trigger specific tasks based on the events of what segment collects.

So still segment is collecting information from stripe specific events on the the back end and front end that other sources and then what segment can do as, for example, in this case we are using GCP, but I suppose that you can set up similar solutions on Azure or Amazon Web Services as well.

The segment can trigger pops up sending a message to Google Cloud Functions and this Cloud Function can process this information, look up different additional attributes in Google BigQuery or Postgres or other sources, and then storing the stream information in BigQuery and also sending it to Salesforce immediately. So this would speed up, the whole data processing. On the other hand, I wouldn't recommend something that for for example, today, each night we are processing or updating one hundred thousand of users, more than one hundred thousand of users. This is not something for that. This is for example, I have said product qualified lead. This can be a solution for specific events when you have you need immediate sales response.

It's up to the business that what can or should trigger this data workflow and SaaS outreach.

Here what you can say as how simple it would be, it's a code in Python. This is a basic function sending information in two ways. One is a simple API creating the lead one by one, and the other one that you can have multiple ones, you can call the bulk API as well. So you can just add the simple Salesforce or other wrapper to your GCP cloud function and it the pops up message will trigger that. For example, here we are sending the email and this email will trigger or having this event in pops up with the email, we trigger the lead creation.

But you can also assign tasks for specific sales representatives or you can trigger assignment rules if you are not using the Bulk API.

As I mentioned, we made some mistakes in this process, and this leads us to see where we've learned what we should learn in the hard way.

It's good to see that it was and still we succeeded in creating quite complex data solutions with segmenting, explaining without reasonable engineering resources. And so you can really easily, not only prototype, put it into practice as well.

But if you can do something or you see something is feasible, it doesn't mean that it's good for you or you should do that. And there's no moral reasons behind it. It's just simply against your own interests that you can can link.

Segment directly to Salesforce, but you shouldn't do or I wouldn't recommend or I can only recommend that in this case when you've got a limited number of events coming through. But as I mentioned that we had hundreds of events on the back end of thousands, hundreds of thousands of users, this is something that that Salesforce API is not supporting.

You should do your homework. You should do your research. This is something that we really like, that it was a small scope that we should populate leads. There should be a couple of attributes and then it's just grow and grow and grow each iteration. The scope would grow that then we need to add STRIDE then we need to add trial information and so on.

That leads to multiple refactoration of the code and the solution and expanded packages so it would be better and you should do the research. You should understand that what is feasible with specific tools, what is not feasible, what is the business requirement. That should be very specific and it shouldn't be on the technical level. It should be that what sales people need on Salesforce.

And when you understand that, then you can tailor your solution towards that and not towards vice versa that they've got this kind of tool set. So we are putting something together and it will be super good and serving. I hope we will it will serve that. And unfortunately, is not the case.

So XpenTi can save a lot of engineering hours, but it's not able to make miracles that still you have to write code, you should maintain code, you should think it over, you should follow specific engineering practices, Most of these services marketing themselves, it's just a drag and drop interface, you click it through and it will be magic will happen. But unfortunately, this is not the case or this is just in specific cases. But when you've got a complex business and as your team grows, as your organization grows, as your business gets more complex, it can't be your solution. We have to have custom business logic implemented in code in this case, but in this case it puts SQL and it's a good that expanding support that.

The same goes with segment that it's very good that it can integrate specific services, it can send back the data that you've corrected, You can send it to data warehouses of any kind, Redshift, Google BigQuery, Postgres, so on. It can send these information, the user data out of the box to insert com to send that which works very well. But not for Salesforce because it's it has a pretty complex API. It has multiple endpoints and it has.

This kind of mapping of identity, identify calls to Salesforce leads, it's not working out of the box.

It's pretty limited.

Thanks, Thomas.

That was that's a that was an interesting overview of a very you seem to have a very complex data ecosystem there at Bitrise, especially capturing all those all those build events in your tool and pulling it into segment and basically basing looks like you base sales decisions on that.

Are you the were you the person who implemented all this or how big was the team that implemented your X Plenty project?

I should admit that our team is super under hired or how it's said. Understaffed?

Understaffed. Thank you. Super understaffed relatively not only to this project, but there are other business questions coming in. You should build your reports in Amplitude Metabase and other BI solutions. Basically, I was working on this solution along and the team is just only two persons we are currently hiring and there are super experienced people in the finance. Hopefully this will change in the future.

But that's why we opted for this solution, that you can see that this ecosystem contains multiple services and this is this was always the core approach in choosing or opting for any solution that we shouldn't do something because it's fancy. For example, in data engineering at some point in time it was apache spark then now it's airflow so you can hire multiple data engineers you can set out different workflows and pipelines and so on But you shouldn't do that. And I mentioned that it's something that you can do. It doesn't mean that you should do that. If there are a simpler solution, cheaper solution and especially cheaper in terms of time, then you should go for that.

But at some point of time, definitely as as the business gets more and more complex, we need to consider having a specific person maintaining these solutions as well.

And so you you learned X Plenty tool on your own or did you did our support people help you build your pipelines or how did how did that happen?

Mostly by myself, you know, it's it's like learning languages. Like after the third or fourth one, it's sad I can't speak so many languages. It's sad that after the third or the fourth it's much easier to pick up the new one. So it was this way, I suppose, with the data tools as well that you see several ones and then you can easily pick up. Not to mention that XpennT is super easy to use.

Really. And we have pretty good experience with the support people as well.

So whenever we start, we have immediate support to And the and the sort of the looks like that that little piece of Python you show that's going to do kind of a, you know, data streaming or I don't know, I guess for want of a better word of like, it's like a trigger.

It's like a trigger process, right? Certain business things happen.

Are are you putting that together, too, or do you have are your software engineers doing that or how is how is that being built?

Not me again, but you know the Python and this is why it's good that these these cloud functions usually support Python. The other language besides SQL that simply not only data engineers but data analytics speaks as well.

But this is the technical aspect. But I think that the real responsibility of data analysts and business analysts is scope out the business question. And then it's up to you that there are so many tools that so many solutions equally good or you can it can fit your organization it can fit your team skill set if you know python then you can go for a tool that speaks Python. But what is really important is to understand the business, have a good communication between the different domains, different teams to understand the sales need is, in this case, Salesforce, but the same goes for marketing and the other teams as well.

So I think that this would be the harder part or this would be more important in this project. Having a tool, it's what I presented here is that how easy it is, it couldn't be a bomb and drop to put it together. But actually there's a team work behind to set up the scope to have something final that really supports your colleagues, not just having data because we are we are a data driven company. It's It's not that case.

It should be specific data in the specific format and the proper time as well.

Yep, that makes sense.

Some people say you need to have not only have to have the know how, but you have to have the know whether know whether to do something, know whether it's worth doing.

Thank you so much. Thanks again for the for the presentation. Thanks for sharing your experience with X Plenty, with Salesforce, with Segment and at Bitrise.

Appreciate your time.

My pleasure. Thank you.

‍