Table Of Contents

Previous topic

Contact us

Next topic

About Us

This Page

Upgrading InterMine

InterMine 2.0 is a disruptive release and is not backwards compatible. This means that databases, webapps and code from previous releases will need to be updated to work with the new InterMine release. Below are detailed instructions on how to migrate your InterMine to the new build system.

Warning

If you have custom InterMine code, your changes will likely not work as expected after the upgrade. Please contact us and we can help you migrate your edits to the new system.

Please contact us if you have any questions or concerns! We have a mailing list or you can contact us directly via email or our discord channel (chat.intermine.org). If you are having difficulties, we can also arrange a skype call to walk through any problems together. Please make sure your code is public, e.g. GitHub, so we can help test!

Gradle

InterMine now uses Gradle to manage dependencies and to build and run InterMine. Please see Gradle Quick Start for useful Gradle commands and Gradle FAQs for help with common questions and errors.

See the Gradle blog post for details as to why we made this change.

Maven

You will need Maven installed. We use Maven to manage mine-specific InterMine dependencies, including your mine-specific data parsers.

# for Ubuntu
sudo apt-get install maven

You do not need to install Gradle locally. Instead, use the Gradle wrapper provided.

Remove InterMine code

Previously you had to download and compile InterMine. Now, instead, you’ll be using the compiled InterMine JARs available via Maven. This means you should remove all InterMine code from your mine repositories. Your mine repositories should only contain your mine (webapp and dbmodel) and your mine’s custom data sources.

If you have your mine and bio/sources in your InterMine checkout, instead of in their own repository, you’ll have to separate them out.

What you want to end up with:

Options to separate out your mine repo:

  1. You can copy over your directories directly. Don’t do this! You’ll lose your history.
# don't do this
~/git $ cp intermine/flymine flymine; cd flymine
~/git/flymine $ git init; git add *; git commit -am "initial commit"
  1. Instead, use git filter-branch command. Follow the directions on how to move a directory to a new repository and keep your history in GitHub.

You should not have any core InterMine code locally.

New directory structure

InterMine has switched to use the standard Maven directory structure.

src/main/java
src/main/resources
src/test/java
src/test/resources

You will have to run two migration scripts to move your current mine over to this new layout – one script for your mine and one for your mine’s data parsers. The migration scripts are located in the intermine-scripts repository.

~/git $ git clone https://github.com/intermine/intermine-scripts.git

Migrate Mine webapp to New directory structure

Run “migrateMine” script to move your mine over to the new directory system. You might want to create a new gradle branch for testing.

~/git/intermine-scripts/gradle-migration/mine $ ./migrateMine.sh ~/git/flymine

Migrate Data Sources to New directory structure

Run the “migrateBioSources” script to move your sources over to the new directory system.

~/git/intermine-scripts/gradle-migration/data-sources $ ./migrateBioSources.sh ~/git/flymine-bio-sources

Run this command to put your sources on the classpath and therefore available to the database build:

# not part of the upgradle process. You will install every time you make a change
~/git/flymine-bio-sources $ ./gradlew install --stacktrace

This task builds the JARs and places them on your classpath in ~/.m2/repository.

Note the command is ./gradlew instead of gradle. Use the provided Gradle wrapper instead of locally installed Gradle.

You will have to install your sources every time you update the source code to update the JAR being used by the build.

Previously the data model was merged from all data sources’ additions XML file. This is no longer true. Since each source is in its own JAR now, the data model is self-contained. Therefore if you reference a class in your data parser, it must be present in the additions file. Alternatively, you can specify a single data model file that will be merged into each source:

// [in bio/sources/build.gradle]
// uncomment to specify an extra additions file for your bio-sources
// this file will be merged with the additions file for each data source
// and included in each source JAR.
//bioSourceDBModelConfig {
//    extraAdditionsFile = "MY-MINE_additions.xml"
//}

Update config

  1. Remove <property name=”source.location” location=”../bio/sources/”/> from your project XML file
  2. Set GRADLE_OPTS instead of ANT_OPTS
    • Use the same parameters.
    • Append -Dorg.gradle.daemon=false to prevent daemons from being used.
  3. Update project XML for some sources
    • SO source’s location has been updated to be: <property name=”src.data.file” location=”so.obo” />
    • Protein2ipr source has a new attribute: <property name=”osAlias” value=”os.production”/>
    • intermine-items-xml-file isn’t a valid value for “type” anymore. Use the project name instead.
    • src.data.dir can only have a location attribute. src.data.dir cannot have a value attribute.
  4. InterPro data file needs to be updated. The file incorrectly references interpro.dtd when you should have the full path instead.
    • Update interpro.xml
    • <!DOCTYPE interprodb SYSTEM “ftp://ftp.ebi.ac.uk/pub/databases/interpro/interpro.dtd”>
    • I asked InterPro to fix but they said no. Maybe you could ask too?
  5. Update each data source’s additions file to be correct. Alternatively you can use the extraAdditionsFile (see previous section).
  6. PostprocessUtil.java moved to the bio package, so you maybe have to update your import to be import org.intermine.bio.util.PostProcessUtil;.

Please see Gradle Quick Start for details on Gradle and common Gradle commands and Gradle FAQs for help with common questions and errors.

Data Model

  • Syntenic Regions have been added to the data model
  • Protein.molecularWeight is now a Float instead of an Integer
  • GO evidence codes now have a name and URL
  • OntologyAnnotation can now annotate any InterMine object, as long as that class inherits Annotatable
  • Sequence Ontolgy has been updated to the latest version
  • Organism.taxonId is a String instead of an Integer.

See the Model Changes blog post for details.

You have may to update your data sources and queries to match the new data model.

Dependencies

Software dependency requirements have been updated to the latest versions. This is so we can get rid of legacy code and make use of new features.

Java SDK 8
Tomcat 8.5.x
Postgres 9.3+

You will get errors if you use older versions. e.g. If you use Java 7, you will get this error: Caused by: java.security.NoSuchProviderException: no such provider: SunEC

API changes

We are making some non-backwards compatible changes to our API. These three end points have a parameter called xml which holds the XML query. We are going to rename this parameter to be query (as we now accept JSON queries!) to match the syntax of all the other end points.

/query/upload
/template/upload
/user/queries (POST)

Please update any code that references these end points.


Pre-InterMine 2.0 Upgrade Instructions

To pull changes in your local repository and merge them into your working files:

$ git pull upstream

If you don’t have a git repo yet, see Get the Software for details.

If you host a copy of the CDN, then you should also pull in changes from that repository.

Upgrade to InterMine 1.6

The core model of InterMine has changed in release 1.1 so you may encounter more errors than usual.

update integration keys
You may need to update your integration keys if they are using a class or field that’s been changed.
update custom converter
If you are storing data using a class or field that’s been changed, you will have to change your code to use the new model. See below for the complete list of model changes.
template queries
You will have to update your templates to use the new model
interaction viewer
The cytoscape tool uses the new model - will not work until you build a database with the new code
class old new
Interaction gene1 participant1
gene2 participant2
relationshipType (Term) relationshipType (String)
InteractionDetail allInteractors (Gene) allInteractors (Interactor)
Interactor stoichiometry
InteractionDetail.role1 role
InteractionDetail.type type
class old new
ProteinDomain proteins proteinDomainRegions
Protein proteinDomains proteinDomainRegions
ProteinDomainRegion start
end
identifier
database

Upgrade to InterMine 1.4

There are no model changes, but we’ve added some new features that require an update.

We’ve added a new fancy connection pool, you should see a performance improvement. However you do need to update some configuration files.

Postgres config file

The number of database connections required will depend on your usage. 100 connections is the default and should be okay for production webapps. However each webapp reserves 20 connections so on your dev machines it may be wise to raise the maximum quite a bit.

postgresql.conf

max_connections=250

$MINE properties files

in your $MINE directory:

default.intermine.integrate.properties

set

db.production.datasource.maxConnections=20

db.common-tgt-items.datasource.maxConnections=5

and for each database replace

db.production.datasource.class=org.postgresql.ds.PGPoolingDataSource

(or any other db pooling class)

with these 2 lines

db.production.datasource.class=com.zaxxer.hikari.HikariDataSource db.production.datasource.dataSourceClassName=org.postgresql.ds.PGSimpleDataSource

default.intermine.webapp.properties

set

db.production.datasource.maxConnections=20

and for each database replace

db.production.datasource.class=org.postgresql.ds.PGPoolingDataSource

(or any other db pooling class)

with these 2 lines

db.production.datasource.class=com.zaxxer.hikari.HikariDataSource db.production.datasource.dataSourceClassName=org.postgresql.ds.PGSimpleDataSource

Any other data source you use should be set to five connections, raised to ten if you encounter problems, e.g. the build failing with an error like so:

Error message

Caused by: org.postgresql.util.PSQLException: FATAL: connection limit exceeded for non-superusers

Or this (See #912)

Error message

Unable to get sub-ObjectStore for Translating ObjectStore

See HikariCP and InterMine settings for details.

InterMine-model Refactor

The metadata package has moved from to InterMine-model. If you have custom data sources that use InterMine Utils, you may have to update your code to reflect the new location. Your IDE should be able to do this for you.

Tomcat

Add clearReferencesStopTimerThreads to your $TOMCAT/conf/context.xml file, so it should look like so:

<Context sessionCookiePath="/" useHttpOnly="false" clearReferencesStopTimerThreads="true">
...
</Context>

Upgrade to InterMine 1.3.x

This code will work with any webapp and database created with InterMine 1.3+.

Upgrade to InterMine 1.3

  • Remove all duplicate entries from web.xml
  • Model changes:
    • DataSet now has a publication reference
    • AnnotationExtension has been moved from GOAnnotation to GOEvidence.

Also, we have changed our GO parser a bit. Each line in a gene annotation file now corresponds with an Evidence object. In prior releases, each Evidence object was unique, e.g. only a single evidence code per gene / GO term pair.

Upgrade to InterMine 1.2.1

If you have your own home page (begin.jsp), you must manually make this change: 501e221

This is a fix for the keyword search - when users submit a blank search form, see Issue #329

There are no model or configuration changes in this release.

Upgrade to InterMine 1.2

The core data model has not been changed, so you should be able to release a webapp using InterMine 1.2 code without making any changes.

Upgrade to InterMine 1.1

The core model of InterMine has changed in release 1.1 so you may encounter more errors than usual.

update integration keys
You may need to update your integration keys if they are using a class or field that’s been changed.
update custom converter
If you are storing data using a class or field that’s been changed, you will have to change your code to use the new model. See below for the complete list of model changes.
template queries
You will have to update your templates to use the new model
interaction viewer
Widget uses the new model - will not work until you build a database with the new code

Model Changes

Updated to latest version of Sequence Ontology, 2.5

old new
Comment.text Comment.description
Gene.ncbiGeneNumber
Gene.description
Gene.briefDescription

Interactions

class old new
Interaction gene gene1
interactingGenes gene2
type details.type
role details.role1
details.role2
name details.name
shortName
InteractionRegion primaryIdentifier
name

Gene Ontology

class old new
GOAnnotation withText evidence.withText
with evidence.with
annotationExtension
OntologyTerm crossReferences [1]
[1]used for Uberon

Identifiers

We have several [wiki:Homologue new homologue data converters] available in this InterMine release. However, some of these new data sources use Ensembl IDs. If you want to load the model organism database identifier instead (important for interoperation with other InterMines), you should use the Entrez Gene ID resolver:

  1. Download the identifier file - ftp://ftp.ncbi.nih.gov/gene/DATA/gene_info.gz
  2. Unzip the file
  3. Add the path to properties file:
# in ~/.intermine/MINE_NAME.properties
resolver.entrez.file=/DATA_DIR/ncbi/gene_info

Configuration Updates

Web services uses the webapp.baseurl property to run queries, so be sure this is the valid URL for your mine. Otherwise you will get an “Unable to construct query” error on the query results page.

# in ~/.intermine/MINE_NAME.properties
# used by web services for running queries, needs to be valid
webapp.baseurl=http://localhost:8080