I am trying to accomplish a simple task of:
- Read a text file that contains SQL queries separated by ";".
- Perform each query in a separate transaction
- Be able to restart/resume the execution from the first query that failed.
I tried to do this with spring-batch.(version 2.1.8.RELEASE) The problems I am facing are as follows:
I failed to configure a simple out-of-the-box ItemReader that would read several lines up to ";" and aggregate them before passing to the ItemWriter.
I do not need a FieldSetMapper I simply do not have no fields. My input is a text file containing SQL queries. Each query can take 1 or more lines and the queries are separated by a semicolon. Seems like it is impossible to define an LineMapper inside a ItemReader without specifying a FieldSetMapper to a complex bean (I tried to set a BeanWrapperFieldSetMapper to map to java.lang.String but failed due the exception described below).
Question 1: why do I need a FieldSetMapper with the prototypeBeanName set to a complex object if all I want is to append all the lines in a single String before passing it to the ItemWriter? When I configure the fieldSetMapper's property prototypeBeanName to refer to the java.lang.String I get an Exception saying that the number of names is not equal to the number of values. I debugged the spring-batch code and found that there are 2 values mapped to a single name: name=%SOME_NAME% values={%MY_SQL_QUERY%, ";"} and the Exception is thrown by AbstractLineTokenizer.tokenize() line 123
Question 2: Can the line aggregation be achieved by using out-of-the box spring-batch ItemReaders if yes, how they need to be configured? Do I miss something in my configuration? (see below)
configuration:
<bean id="basicParamsIncrementer" class="com.linking.core.exec.control.BasicJobParamsIncrementer" />
<batch:job id="rwBatchJob" restartable="true" incrementer="basicParamsIncrementer">
<batch:step id="processBatchStep">
<batch:tasklet transaction-manager="transactionManager" >
<batch:chunk reader="batchFileItemReader" writer="sqlBatchFileWriter"
commit-interval="1" />
</batch:tasklet>
</batch:step>
</batch:job>
<bean id="batchFileItemReader" class="org.springframework.batch.item.file.FlatFileItemReader">
<property name="resource"
value="classpath:/bat.sql.txt" />
<property name="recordSeparatorPolicy" ref="separatorPolicy" />
<property name="lineMapper">
<bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
<property name="lineTokenizer" ref="sqlTokenizer" />
<property name="fieldSetMapper" ref="sqlFieldSetMapper" />
</bean>
</property>
</bean>
<bean id="separatorPolicy" class="com.linking.core.SemiColonRecordSeparatorPolicy" />
<bean id="sqlTokenizer" class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
<property name="delimiter" value=";" />
<property name="names" value="sql,eol"/>
</bean>
<bean id="sqlFieldSetMapper" class="org.springframework.batch.item.file.mapping.BeanWrapperFieldSetMapper">
<property name="prototypeBeanName" value="sql" />
</bean>
<bean id="sql" class="com.linking.core.model.SQL" scope="prototype" />
com.linking.core.model.SQL is a simple class having 2 members of type String: sql (for the query) and eol (for the ';') It is used to overcome the situation that the AbstractItemReader returns all the lines up to the separator (as a String) and also the separator (in my case ";") as values.
Question 3: In case the job that runs a bulk of SQL queries using just a single step reader-writer approach fails in the middle how can I resume/restart the job from the 1st failed query? Should I do it programatically by analyzing last run or can it be achieved by context configuration of the job-step and the related out-of-the-box spring-batch beans?