When we built our first machine learned model in Spark to predict pros’ interests in customer requests, we loaded pros’ historical engagement features directly into an online Golang application, which used the features to determine which pros are potentially interested in a given request. This approach had a few limitations:
- Other applications could not reuse the model.
- The feature loading process was not scalable due to the ever-increasing number of features.
- The interoperability between Scala (for offline Spark jobs) and Golang (for online applications) led to a substantial amount of duplicated code for common data models,