Select Page

The scenario:

Our data ingestion team is in charge of collecting data from myriad data sources including APIs, flat files, and log files.  Then, we must transform the data and finally store it in different Relational/NoSql databases. Our AI engine later uses this data to make accurate recommendations.

The first approach:

We first developed a Java-based “One size fits all” solution. This service used a complicated if-else approach to determine which database to use to persist the data’s metadata, based on the corresponding data source. Though this approach worked for a small number of data sets, it became unmanageable as the number of data sets increased.

The second approach:

We decided to split our monolithic solution into several micro-services. Each micro-service reads from a single data source and stores data in a single database. This approach helped us to have a clean process to store metadata and worked for some time.2016-11-04-13_22_10-highly-parametrizable-micro-service1-doc-protected-view-microsoft-word
The new requirement:

After some time, new data requirements existed, forcing each micro-service to persist the processed data into multiple databases.

2016-11-04-13_22_21-highly-parametrizable-micro-service1-doc-protected-view-microsoft-word

Refining our solution:

Our first approach, a naïve one, was to simply create a code replica of each micro-service with a different configuration data set. This approach could not scale, because it would force us to create a Git repository for each micro-service.  With each service’s logic beginning to drift apart, the code could not be easily maintained.

Using SpringBoot, we took advantage of SpringBoot’s beans (@ConditionalOnProperty) and configuration classes (@Configuration), making only one compact and clean bean at the start time of each configured database, instead of creating a complicated if-else service.

The building blocks:

First we added all available data source configs in our configuration file:
application.yml


spring:
datasource:
avata-mysql:
url: databse 1
username: you-password
password: you-password
driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver
avata-sqlserver:
url: xxx 2
username: you-password
password: you-password
driver-class-name: com.microsoft.sqlserver.jdbc.SQLServerDriver


To that same configuration file we added a single field specifying which data source to use:

databases:
engine: mysql #mssql #postgres etc


We created a configuration class, creating a single bean dependent on the database specified on the application.yml file

@Configuration
public class DatabaseDialectConfig {
@Bean(name = “databaseDialectSetupService”)
@ConditionalOnProperty(prefix = “databases”, name = “engine”, havingValue = “mssql”)
public DatabaseDialectSetupService databaseDialectServiceMssql() {
return new MSSQLDialectServiceImpl();
}

@Bean(name = “databaseDialectSetupService”)
@ConditionalOnProperty(prefix = “databases”, name = “engine”, havingValue = “mysql”)
public DatabaseDialectSetupService databaseDialectServiceMysql() {
return new MySQLDialectServiceImpl();
}

@Bean(name = “databaseDialectSetupService”)
@ConditionalOnProperty(prefix = “databases”, name = “engine”, havingValue = “postgres”)
public DatabaseDialectSetupService databaseDialectServicePostgres() {
return new PostgresDialectServiceImpl();
}

}


Now we use a configuration class that builds different data source beans based on the application yml.
Note that each bean has a unique identification name:

@Configuration
public class DatabaseConfig {
@Bean(name = “primaryDataSource”)
@ConfigurationProperties(prefix=”spring.datasource.avata-mysql “)
public DataSource primaryDataSource() {
return DataSourceBuilder.create().build();
}
@Primary
@Bean(name = “secondaryDataSource”)
@ConfigurationProperties(prefix=”spring.datasource.avata-sqlserver “)
public DataSource secondaryDataSource() {
return DataSourceBuilder.create().build();
}
@Bean(name = “clientJdbcTemplate”)
public JdbcTemplate jdbcTemplatePrimary(@Qualifier(“primaryDataSource”) DataSource primaryDataSource) {
return new JdbcTemplate(primaryDataSource);
}
@Primary
@Bean(name = “avataJdbcTemplate”)
public JdbcTemplate jdbcTemplateSecondary(@Qualifier(“secondaryDataSource”)        DataSource secondaryDataSource) {
return new JdbcTemplate(secondaryDataSource);
}
}


Finally, each database dialect service was created:

@Service
public class MSSQLDialectServiceImpl implements DatabaseDialectSetupService {
@Autowired
@Qualifier(“avataJdbcTemplate”)
JdbcTemplate avataJdbcTemplate;

}

Conclusion:

Use of SpringBoot with conditionalOnProperty beans allows teams to reuse the same codebase in different scenarios, with almost negligible modifications at the configuration file.