Friday, October 16, 2015

Extract my data from runtastic and upload to strava

Background

So I started using Strava a few months ago after one of the super nice employees at Sunlight Bike Shop in Glenwood Springs, CO showed me some local routes on his phone (he also owned a big dummy). He was using Strava and it looked really nice.

I had been having some problems with Runtastic RoadBike Pro, where it was slowing down every time I saved a new activity. It was so bad that it was taking 2 minutes to save anything, so I was already looking for a better solution.

I really like Strava and have been looking for ways to get the past 3 years of my data out of Runtastic, with limited results. This was the easiest way I could figure out how to do this. I did see some JS scripts and here to get the files out of the runtastic UI, but I didn't want to click the "OK" button to download 275 TCX files.


Disclaimer and high level overview

This is not for non-technical folk, you will be extracting an encrypted backup from your phone, unencrypting it, finding the sqlite DB, getting some data out of the DB, extracting your session cookie from runtastic after logging in with your browser, getting the files using curl, slicing them up into directories with 25 or less TCX files, then uploading them to Strava.

Overall you should expect to spend 1-2 hours, but the end result is all of your data will be in one place!


My setup is as follows

  1. Nexus 6 with the Runtastic Roadbike APP installed, and all of the activities downloaded
  2. Ubuntu laptop running 15.10 (I'm sure any recent version is fine, or any other linux or Mac)
  3. adb installed (sudo apt-get install android-tools-adb)
  4. ant installed (sudo apt-get install ant)
  5. sqlite installed (sudo apt-get install sqlite)
  6. openjdk 8 installed (sudo apt-get install openjdk-8-jdk)
  7. USB cable
  8. Developer options turned on 
  9. USB debugging enabled
  10. Authorized my computer to USB debug my phone

Instructions

Here are some (mostly) step by step instructions, although this does assume that you have a few things installed (or can install them) and are running on a linux machine with a USB cable to your phone.

Extract the DB from your phone, my phone required setting a password.

 adb backup -f runtastic-data.ab -noapk 'com.runtastic.android.roadbike.pro'  


Unencrypt the backup 

I used https://github.com/nelenkov/android-backup-extractor for this. The ant build was the easiest, and the openJDK 8 already had the encryption libs that are referenced. Just edit the build.xml file, change it to the version you have and type 'ant'.
 ABE_PASSWD=<your password> java -jar abe.jar unpack ../runtastic-data.ab ../runtastic-data.tar  


Get all of your IDs from the runtastic DB from your phone

 sqlite3 runtastic.sqlite 'select serverSessionID from session where serverSessionID <> -1 order by startTime' > runtastic-ids.txt  


Login to runtastic

Pull your cookie from a request to their site. Put it in an ENV variable for the download script.
 export RT_COOKIE='Cookie:<the whole thing pulled from a GET request'  
Get your username and set that in another ENV variable. Your username is in the URL right after '/users/'. Mine was "Torleiv-Flatebo"
 export RT_NAME="Torleiv-Flatebo"  


Runtastic Download Script

Save the below into a file called runtastic-download-script.sh
 curl -s -w "%{filename_effective} %{http_code}\n" -O -J "https://www.runtastic.com/en/users/$RT_NAME/sport-sessions/$1.tcx" -H "$RT_COOKIE"


The magic happens

Cat out the IDs and run the download script, watch as each file is downloaded. It took about 5-10 minutes to get all of my 275 activities.
 mkdir tcx
 cd tcx
 cat ../runtastic-ids.txt | xargs -n 1 sh ../runtastic-download-script.sh   
 runtastic_20120628_1755_Race Cycling.tcx 200  
 runtastic_20120630_1106_Race Cycling.tcx 200  


Prep files for upload to Strava (optional if more than 25 activities)

 mkdir 1 2 3 4 5 6 7 8 9 10 11  
 ls -1 *.tcx | head -25 | sed 's/.*/"&"/' | xargs -n1 mv -t 1  
 ls -1 *.tcx | head -25 | sed 's/.*/"&"/' | xargs -n1 mv -t 2  
 etc.  


Upload to Strava

Now you have all of the tcx files, you can upload them to strava using their website. You are limited to 25 files per upload, so I just put them all in separate dirs (see above).

I only had about 275 tcx files, so I figured it would be easier to use their UI, since you can tag and edit the activities in a screen after you select 25 of the files. For me I like to tag the activities to be a commute, or pick the bike that I am using for that ride, so their UI was preferable.

If you have more activities from runtastic you can use their API to upload.