Friday, October 16, 2015

Extract my data from runtastic and upload to strava

Background

So I started using Strava a few months ago after one of the super nice employees at Sunlight Bike Shop in Glenwood Springs, CO showed me some local routes on his phone (he also owned a big dummy). He was using Strava and it looked really nice.

I had been having some problems with Runtastic RoadBike Pro, where it was slowing down every time I saved a new activity. It was so bad that it was taking 2 minutes to save anything, so I was already looking for a better solution.

I really like Strava and have been looking for ways to get the past 3 years of my data out of Runtastic, with limited results. This was the easiest way I could figure out how to do this. I did see some JS scripts and here to get the files out of the runtastic UI, but I didn't want to click the "OK" button to download 275 TCX files.


Disclaimer and high level overview

This is not for non-technical folk, you will be extracting an encrypted backup from your phone, unencrypting it, finding the sqlite DB, getting some data out of the DB, extracting your session cookie from runtastic after logging in with your browser, getting the files using curl, slicing them up into directories with 25 or less TCX files, then uploading them to Strava.

Overall you should expect to spend 1-2 hours, but the end result is all of your data will be in one place!


My setup is as follows

  1. Nexus 6 with the Runtastic Roadbike APP installed, and all of the activities downloaded
  2. Ubuntu laptop running 15.10 (I'm sure any recent version is fine, or any other linux or Mac)
  3. adb installed (sudo apt-get install android-tools-adb)
  4. ant installed (sudo apt-get install ant)
  5. sqlite installed (sudo apt-get install sqlite)
  6. openjdk 8 installed (sudo apt-get install openjdk-8-jdk)
  7. USB cable
  8. Developer options turned on 
  9. USB debugging enabled
  10. Authorized my computer to USB debug my phone

Instructions

Here are some (mostly) step by step instructions, although this does assume that you have a few things installed (or can install them) and are running on a linux machine with a USB cable to your phone.

Extract the DB from your phone, my phone required setting a password.

 adb backup -f runtastic-data.ab -noapk 'com.runtastic.android.roadbike.pro'  


Unencrypt the backup 

I used https://github.com/nelenkov/android-backup-extractor for this. The ant build was the easiest, and the openJDK 8 already had the encryption libs that are referenced. Just edit the build.xml file, change it to the version you have and type 'ant'.
 ABE_PASSWD=<your password> java -jar abe.jar unpack ../runtastic-data.ab ../runtastic-data.tar  


Get all of your IDs from the runtastic DB from your phone

 sqlite3 runtastic.sqlite 'select serverSessionID from session where serverSessionID <> -1 order by startTime' > runtastic-ids.txt  


Login to runtastic

Pull your cookie from a request to their site. Put it in an ENV variable for the download script.
 export RT_COOKIE='Cookie:<the whole thing pulled from a GET request'  
Get your username and set that in another ENV variable. Your username is in the URL right after '/users/'. Mine was "Torleiv-Flatebo"
 export RT_NAME="Torleiv-Flatebo"  


Runtastic Download Script

Save the below into a file called runtastic-download-script.sh
 curl -s -w "%{filename_effective} %{http_code}\n" -O -J "https://www.runtastic.com/en/users/$RT_NAME/sport-sessions/$1.tcx" -H "$RT_COOKIE"


The magic happens

Cat out the IDs and run the download script, watch as each file is downloaded. It took about 5-10 minutes to get all of my 275 activities.
 mkdir tcx
 cd tcx
 cat ../runtastic-ids.txt | xargs -n 1 sh ../runtastic-download-script.sh   
 runtastic_20120628_1755_Race Cycling.tcx 200  
 runtastic_20120630_1106_Race Cycling.tcx 200  


Prep files for upload to Strava (optional if more than 25 activities)

 mkdir 1 2 3 4 5 6 7 8 9 10 11  
 ls -1 *.tcx | head -25 | sed 's/.*/"&"/' | xargs -n1 mv -t 1  
 ls -1 *.tcx | head -25 | sed 's/.*/"&"/' | xargs -n1 mv -t 2  
 etc.  


Upload to Strava

Now you have all of the tcx files, you can upload them to strava using their website. You are limited to 25 files per upload, so I just put them all in separate dirs (see above).

I only had about 275 tcx files, so I figured it would be easier to use their UI, since you can tag and edit the activities in a screen after you select 25 of the files. For me I like to tag the activities to be a commute, or pick the bike that I am using for that ride, so their UI was preferable.

If you have more activities from runtastic you can use their API to upload.

Sunday, March 2, 2014

Cassandra: 'Segmentation fault: 11' when trying to run any command in cqlsh

Trying to get cassandra running on my Mavericks laptop, I ran into an issue right away. Other posts mentioned JVM versions, but I upgraded that and made sure my brew stuff was all up to date.

There were no errors at all in the cassandra server logs, so I figured it must be something with Python, since cqlsh was running inside of Python.

Any time I would run a command (other than help) in cqlsh, I would get a "Segmentation fault: 11", with this in the details:

Process:         Python [35348]
Path:            /Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python
Identifier:      Python
Version:         2.7.3 (2.7.3)
<snip>
Crashed Thread:  0  Dispatch queue: com.apple.main-thread

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_INVALID_ADDRESS at 0x0000000000000000


This turned out to be a bug in Python < 2.7.6. An upgrade to Python 2.7.6 solved the issue.

A note on Python's site referred to
Python 2.7.6 was released on November 10, 2013. This is a 2.7 series bugfix release. Most importantly, it resolves an issue that caused the interactive prompt to crash on OS X 10.9. It also includes numerous bugfixes over 2.7.5. 

Saturday, November 9, 2013

Monthly hackathon or unconference in Mpls or St. Paul

I want to hear from people who are interested in doing the City Camp MN and Open Twin Cities idea more often primarily in a hackathon format.

Doesn't have to be amazing or awesome every month, just a group of creative and interested and civic minded people getting together once per month or so.

I can help to organize and find space for this.

Especially interested in having non-coders attend, developers aren't the best with idea creation and collaboration between non-technical and techies is always the best.

I know there are people looking for projects to code on, and I know there are people looking for help moving their ideas forward.

Post in the comments if you are interested in this and I will contact you or post on this blog as this moves forward.

#hackmonthlymn

Tuesday, June 5, 2012

My favorite XML editor and viewer for linux


mlview

http://packages.ubuntu.com/hardy/mlview


I always forget this because I don't use it enough. It's pretty fast, but can be a little slow on very large documents.


Friday, December 23, 2011

Increase memory size for SQLDeveloper

For some reason, they make it very difficult to increase the memory size of SQLDeveloper, you can't even pass in the options on the command line.

I'm not sure about Mac or Windows, but on Linux, it's pretty easy, just edit your ide/bin/ide.conf file and change the lines "AddVMOption -Xmx640M" and "AddVMOption -Xms128M" lines to something larger (or smaller). You can see what I have mine set to below.

Here is mine after I changed it:

tor@XXX:~/sqldeveloper$ pwd
/home/tor/sqldeveloper
tor@XXX:~/sqldeveloper$ cat ./ide/bin/ide.conf
#-----------------------------------------------------------------------------
#
# Oracle IDE Configuration File
# Copyright 2000-2008 Oracle Corporation.
# All Rights Reserved.
#
#-----------------------------------------------------------------------------
IncludeConfFile ../../ide/bin/jdk.conf
AddJavaLibFile ../../ide/lib/ide-boot.jar
SetMainClass oracle.ide.boot.Launcher
#
# If you are getting the 'Low Memory Warning' Message Dialog while running
# JDeveloper, please increase the -Xmx value below from the default 768M to
# something greater, like 1024M or 1250M. If after increasing the value,
# JDeveloper is no longer starting up because it fails to create a virtual
# machine, then please reduce the modified -Xmx value.
#
AddVMOption -Xmx2048M
AddVMOption -Xms512M

#
# Turn off verifications since the included classes are already verified
# by the compiler. This will reduce startup time significantly. On
# some Linux Systems, using -Xverify:none will cause a SIGABRT, if you
# get this, try removing this option.
#
AddVMOption -Xverify:none
#
# On some Windows Terminal Server installations, relocation errors of
# system DLLs can sometimes occur when using OJVM. Use this option to
# specify a heap base address for OJVM to resolve this problem. This
# option only works with OJVM and cannot be used with any HotSpot JVM.
#
# AddVMOption -Xheapbase100000000

Sunday, October 30, 2011

Time Machine on Mac OS 10.6.8 failing backing up to an Ubuntu netatalk server with "(Error) IndexSDB in si_writeBackDBO:Error 2 storing dbo"

I ran into some problems about four months ago with being able to back up my 10.6 Mac to an Ubuntu netatalk server. I just started failing, with a generic error message on the UI. All other netatalk services and shares were working fine.

It was getting to the point where it looked like it was working, but once it actually began backing up, it would get to about 100kb, and then fail with this message in the system log on the Mac:

Oct 29 16:37:48 my-macbook mds[38]:
(/Volumes/Backup of MyMac/.Spotlight-V100/Store-V1/Stores/XXX-XXX-XXX-XXX-XXX)
(Error) IndexSDB in si_writeBackDBO:Error 2 storing dbo(9770a3,430,182,4b076d10a69e9,0,1d1686,0)

I was able to follow the instructions copied below from a comment on this page:

Obviously I didn't follow them exactly, since I have an Ubuntu server and not a Mac OSX server, but it now turns out that my Mac now creates the sparsebundle file for me! I basically deleted the whole folder (sparsebundle that I had created for the previous version of Mac OS) that I was using for the netatalk share, created an empty one with 777 perms, added the .com.apple.timemachine.supported file, and my Mac did the rest!

Note: The backup did fail once or twice after following these instructions, the third time it started working all by itself, all I did was retry.

10.6: Set up Time Machine on networked AFP volume
Authored by: Henk Poley on Feb 27, '11 01:52:53AM
Maybe I'm doing something wrong.. but the only thing I have to do is:

1. Create an AFP share on an OS X machine (set permissions, user accounts & passwords as you like)
2. Run `touch .com.apple.timemachine.supported` from Terminal.app on that share
3. Mount the AFP share by clicking around in Finder
4. Point Time Machine to the disk

It then goes off to create a .sparsebundle, mounts that and does a backup.

I may have run the magic `defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1` way back when, so you may have to run that too.

Both machines run 10.6.6, but I (maybe miss-)remember doing it like that on Leopard too.

(Before anyone says this doesn't work anymore, I'm doing an initial backup using that method right now)
Edited on Feb 27, '11 01:59:53AM by Henk Poley

Tuesday, August 10, 2010

SQuirreL and Oracle ORA-01867

Global Preference (v3.1)

I have had a specific error plaguing me for some time now, when using the SQuirreL SQL client to access an Oracle Scheme with a database link to another Scheme.  The query might look something like this:

SELECT * FROM SOMETABLE@OTHERSCHEME

I would consistently get an ORA-01867 error. Every forum I searched seemed to point back to DATE or TIMESTAMP handling. Playing with the Global Preferences, I found that the DATE handling option titled Interpret DATE columns as TIMESTAMP must be un-selected. -- Chad

Wednesday, June 2, 2010

How to set chrome as your default browser for Thunderbird on Ubuntu

From this bug report:

it appears that in Thunderbird 3 the protocol handlers are ignored in favor of the
settings in the "Attachment" preferences. Indeed, once I set the http and https
"content types" to xdg-open, Thunderbird correctly opened http and https links in
Chrome.

Sunday, March 14, 2010

Export from iMovie to play on Blackberry Bold 9700

Want to export a video from iMovie to play on your Blackberry Bold 9700? Tired of showing your videos on the youtube player? No problem!

For the impatient: use MPEG4, 25fps, auto key frames and restrict to 1248 kbits/sec at 480x270 letterbox. Audio is AAC codec, Mono, 44.1, and a target bitrate of 64k.

A 3:30 video renders to about 33MB.

First choose 'Export using Quicktime' from the menu



Then you will see a dialog like this. Choose options



After that you should see the setting for the video formatting and audio. Choose the top 'Settings' button.



Now you will get a dialog for the video format. Use MPEG4, 25fps, auto key frames and restrict to 1248 kbits/sec. Now close this dialog, and hit the 'size' button.



Choose a custom size, and enter 480x270, preserve aspect using letterbox, and deinterlace. Close this dialog, and choose the sound settings button.



Use an AAC codec, Mono, 44.1, and a target bitrate of 64k. Close this dialog.



Now give your video a name, and hit save

Thursday, February 11, 2010

Squirrel not starting fix

If OOM kills squirrel hard, and squirrel will not restart giving a Null Pointer error, you can follow this thread below by removing the SQLAliases23_treeStructure.xml file from your $HOME/.squirrel dir. It is a transient file.

Stack trace:

tor@unwise:~/SQuirreL SQL Client$ ./squirrel-sql.sh
Exception in thread "main" java.lang.reflect.InvocationTargetException
at java.awt.EventQueue.invokeAndWait(EventQueue.java:997)
at javax.swing.SwingUtilities.invokeAndWait(SwingUtilities.java:1323)
at net.sourceforge.squirrel_sql.fw.gui.GUIUtils.processOnSwingEventThread(GUIUtils.java:350)
at net.sourceforge.squirrel_sql.client.gui.WindowManager.(WindowManager.java:161)
at net.sourceforge.squirrel_sql.client.Application.executeStartupTasks(Application.java:753)
at net.sourceforge.squirrel_sql.client.Application.startup(Application.java:246)
at net.sourceforge.squirrel_sql.client.Main.main(Main.java:60)
Caused by: java.lang.RuntimeException: java.lang.NullPointerException
at net.sourceforge.squirrel_sql.client.gui.db.JTreeAliasesListImpl.initTree(JTreeAliasesListImpl.java:233)
at net.sourceforge.squirrel_sql.client.gui.db.JTreeAliasesListImpl.(JTreeAliasesListImpl.java:101)
at net.sourceforge.squirrel_sql.client.gui.db.AliasesList.(AliasesList.java:44)
at net.sourceforge.squirrel_sql.client.gui.WindowManager.createAliasesListUI(WindowManager.java:877)
at net.sourceforge.squirrel_sql.client.gui.WindowManager.initialize(WindowManager.java:861)
at net.sourceforge.squirrel_sql.client.gui.WindowManager.access$200(WindowManager.java:77)
at net.sourceforge.squirrel_sql.client.gui.WindowManager$1.run(WindowManager.java:165)
at java.awt.event.InvocationEvent.dispatch(InvocationEvent.java:199)
at java.awt.EventQueue.dispatchEvent(EventQueue.java:597)
at net.sourceforge.squirrel_sql.client.Application$1.dispatchEvent(Application.java:195)
at java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:269)
at java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:184)
at java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:174)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:169)
at java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:161)
at java.awt.EventDispatchThread.run(EventDispatchThread.java:122)
Caused by: java.lang.NullPointerException
at net.sourceforge.squirrel_sql.client.gui.db.AliasFolderState.applyExpansionAndSelectionToNode(AliasFolderState.java:144)
at net.sourceforge.squirrel_sql.client.gui.db.AliasFolderState.applyExpansionAndSelection(AliasFolderState.java:160)
at net.sourceforge.squirrel_sql.client.gui.db.JTreeAliasesListImpl.applyAliasFolderState(JTreeAliasesListImpl.java:264)
at net.sourceforge.squirrel_sql.client.gui.db.JTreeAliasesListImpl.initTree(JTreeAliasesListImpl.java:220)
... 15 more