L2tp IPSec VPN with Mikrotik

I recently discovered the VPN feature on my Android phone and though let me try this to set up a VPN with my home gateway, which is a MikroTik router.

I followed a few guides on the internet, but this just did not work, Android just gave me a short "does not work" message and that's it. So I dropped the ball again.

Now I am going to talk at a conference and I thought, I could set up the scenario on my BigBox at home instead of the Laptop. But how do I expose that? Well, perhaps I should use a VPN. So I googled "MikroTik macOS VPN" and found a few pages that also repeated what I had set up before. I tried again, no luck. Did some packet sniffing on the router. No luck.

And then I got a thought: what if my provider filters udp port 500? Googled around, does not seem so. But it brought me to another idea:

If you look at the above setup, you can see that my MikroTik is not directly connected to the internet, but there is a provider router in-between. And this guy also has a firewall and is filtering out ports. In the past I had poked some holes into this (but I log forgot since), but udp port 500 and others needed for IPsec were not forwarded to the MikroTik.

Once I enabled forwarding of these ports, both VPN on macOS and Android started to work like a charm. :-)

One thing I still had to enable was proxy-arp as described in the above guide to be able to reach not only the router, but also my real target host on my LAN.

I have after the good setup played a bit with the configuration (and not put it under version control and everything failed with "no IKEv1 peer config for". Solution was that I had a local address in the /ip ipsec peer and this meant that MikroTik would only allow for connections to that specific IP, which failed.

Another good guides on the topic:

• https://itimagination.com/mikrotik-l2tp-ipsec-dedicated-vpn-appliance-setup/
• https://manuth.life/l2tpipsec-vpn-server-mikrotik-routeros/
• MikroTik docs https://jcutrer.com/howto/networking/mikrotik/ios-ikev2-vpn-mikrotik
• Troubleshooting guide

New Android tooling with gradle: Order matters

I am trying to convert RHQPocket over to use the new Android build system with Gradle.
The documentation is comprehensive, but as always does not exactly match what I have in front of me.

After moving the sources around I started generating a build.gradle file. I added my external libs to the dependencies but the build did not succeed with a lot of trial and error.

At the end with the help of googling and StackOverflow I found out that order matters a lot in the build file:

First comes the section about plugins for the build system itself:

buildscript {
    repositories {
        mavenCentral()
    }

    dependencies {
       classpath 'com.android.tools.build:gradle:0.4.2'
    }
}

Settings herein only apply to the build system, but not to the project to be built.

After this we can use those plugins (the android one)

apply plugin: 'android'
apply plugin: 'idea'

And only when those plugins are loaded, we can start talking about the project itself and add dependencies

repositories {
        mavenCentral()
}
dependencies {
	compile 'org.codehaus.jackson:jackson-core-asl:1.9.12'
	compile 'org.codehaus.jackson:jackson-mapper-asl:1.9.12'
}

android {
    compileSdkVersion 17
    buildToolsVersion "17.0"
}

I guess for someone experienced with Gradle, this is no news, but it took me quite some time to figure out especially as the documentation mentions all the parts but not on a larger example.

The file layout now looks like this:

One caveat is that the external libraries are not automagically yet loaded into the IDE - one still has to manually add them in the project structure wizard.

The last thing is then to mark build/source/r/debug as source directory to also get completion for the R.* entries.

Android and locations - a sad storry

So I am e.g. browsing Google Maps and have a GPS fix. Now I switch to Ingress or c:geo from the recently used apps list. And what happens?

GPS-fix icon in the upper left corner turns off, app changes and I have to wait between 6 and 15sec to get a new fix even if I did not move a centimeter.

Worse is that the position during the switch-over is sometimes kilometers off, as the app uses the location from the network, which depends on the size of the network cell and the location of the next tower. If you are in a location based app, that switchover can even cause additional network traffic as e.g. maps needs to load a new tile or g:geo does the "nearby caches" search in a non-interesting area.

The technical background is the application lifecycle in Android and the recommendations to safe battery just don't mix.
When app one (e.g. maps) is brought to background, the GPS receiver is switched off, then the switchover happens and only after the onResume() has finished, it is turned on again. Unfortunately the Android system itself seems not to cache the GPS info for a little time and offers that to the next app as a potential current location, but just falls back on network location.

A solution could be for the Android system to cache the GPS location and add an uncertainty linear to the age of the fix. So while the location + uncertainty is smaller than the distance from the location obtained from network, the GPS location would be used. And then after the mentioned 6-15 sec, the real GPS data would take over.

RHQPocket updated

I have worked over the last days on improving RHQPocket, an Android application that serves as an example client for RHQ. Most of the features of RHQPocket will work (best) with a current version of the RHQ server (from current master).

The following is a recording I made that shows the current state.

Update on RHQPocket from Heiko W. Rupp on Vimeo.

The video has been recorded with QuickTime player with the running emulator. I first tried this with the "classical" emulator, but this used 100% cpu (1 core) and with recording turned on, this was so slow that simple actions that are normally immediate, would take several seconds.

My next try was to film my real handset, but too much environmental light and reflexions made this a bad experience. After I heard that others had successfully installed the hardware accelerated GPU and the virtualized x86 image for the emulator I went that route and the CPU usage from the emulator went from 100% before to around 15-20% (home screen), so I went that route for recording.

You can find the full source code on GitHub - contributions and feedback are very welcome.
If you don't want to build from source, you can install the APK from here.

Two nice days at Android DevCamp Stuttgart 2

In Stuttgart, we had an Android Developer Camp organized by the local Google Technology User Group.

This was a two day camp with workshops and a hackathon on the first day and a bar camp style set of sessions on the second one.

Some of the almost 200 plush Androids

On the first day I went to the Hackathon and worked on Android 3 aka Honeycomb migration issues together with Fridger from Openintents. On the second day I was in a few sessions about app mashups, home screen widgets, git and also gave a session that was talking about what I have done in the Android 2->3 migration so far. My hope was to learn from some participants, but it looks like tablets are not yet that popular. Slides of my session are here.

Samsung, who were one of the main sponsors of the event, sent some developer advocates with new 8.9 and 10.1 tablets to play with, which was nice too.

Food was good and the Androids were plenty, so this was a good event. Thanks to all sponsors and the organization team.

And I thought JBossAS had long stack traces ...

05-04 21:49:32.865: ERROR/AndroidRuntime(10036): FATAL EXCEPTION: main        java.lang.IndexOutOfBoundsException: Invalid index 12, size is 12        at java.util.ArrayList.throwIndexOutOfBoundsException(ArrayList.java:255)        at java.util.ArrayList.get(ArrayList.java:308)        at de.bsd.zwitscher.StatusAdapter.getView(StatusAdapter.java:79)        at android.widget.AbsListView.obtainView(AbsListView.java:1943)        at android.widget.ListView.makeAndAddView(ListView.java:1756)        at android.widget.ListView.fillDown(ListView.java:656)        at android.widget.ListView.fillSpecific(ListView.java:1314)        at android.widget.ListView.layoutChildren(ListView.java:1587)        at android.widget.AbsListView.onLayout(AbsListView.java:1794)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.LinearLayout.setChildFrame(LinearLayout.java:1542)        at android.widget.LinearLayout.layoutHorizontal(LinearLayout.java:1527)        at android.widget.LinearLayout.onLayout(LinearLayout.java:1316)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.FrameLayout.onLayout(FrameLayout.java:400)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.LinearLayout.setChildFrame(LinearLayout.java:1542)        at android.widget.LinearLayout.layoutVertical(LinearLayout.java:1403)        at android.widget.LinearLayout.onLayout(LinearLayout.java:1314)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.FrameLayout.onLayout(FrameLayout.java:400)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.FrameLayout.onLayout(FrameLayout.java:400)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.LinearLayout.setChildFrame(LinearLayout.java:1542)        at android.widget.LinearLayout.layoutVertical(LinearLayout.java:1403)        at android.widget.LinearLayout.onLayout(LinearLayout.java:1314)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.FrameLayout.onLayout(FrameLayout.java:400)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.FrameLayout.onLayout(FrameLayout.java:400)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.LinearLayout.setChildFrame(LinearLayout.java:1542)        at android.widget.LinearLayout.layoutVertical(LinearLayout.java:1403)        at android.widget.LinearLayout.onLayout(LinearLayout.java:1314)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.widget.FrameLayout.onLayout(FrameLayout.java:400)        at android.view.View.layout(View.java:9330)        at android.view.ViewGroup.layout(ViewGroup.java:3795)        at android.view.ViewRoot.performTraversals(ViewRoot.java:1201)        at android.view.ViewRoot.handleMessage(ViewRoot.java:1944)        at android.os.Handler.dispatchMessage(Handler.java:99)        at android.os.Looper.loop(Looper.java:126)        at android.app.ActivityThread.main(ActivityThread.java:3997)        at java.lang.reflect.Method.invokeNative(Native Method)        at java.lang.reflect.Method.invoke(Method.java:491)        at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:841)        at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:599)        at dalvik.system.NativeStart.main(Native Method)

 

 

A pitfall in PendingIntent (with solution)

The Android documentation has a nice overview chapter about how to notifiy the user with status bar notifications.

The example text works quite nicely and the user gets informed and can then call back into the application. But when working on Zwitscher it did not work as intended by me. But lets start slowly.

Setting up a notifiction goes along the lines of (taken from the developer guide):

Intent notificationIntent = new Intent(this, MyClass.class);
PendingIntent contentIntent = 
  PendingIntent.getActivity(this, 0, notificationIntent, 0);
notification.setLatestEventInfo(context, "Title", 
    "something went wrong", contentIntent);

where a PendingIntent is set up as a "pointer" and stored by the system so that when the user selects the notification in the status bar the target activity specified in the notificationIntent can be called.

Now sometimes you want to attach some additional data to the intent to be delivered - like a longer explanation why your action failed. You would go like:

Intent notificationIntent = new Intent(context,MyClass.class);
 notificationIntent.putExtra("key","value");        
 notificationIntent.putExtra("key2",someCounter++);

to add the payload. And in MyClass you would get the data via

Intent intent = getIntent();        
Bundle bundle = intent.getExtras();        
String head = bundle.getString("key");        
Integer body = bundle.getInt("key2");

Now when the notification fires,  the intent is created and attached to the PendingIntent and this shows up in the status bar

User then selects the status bar to see the longer message and presses this area to see the full details. This means that the system delivers "out of the blue" the created Intent message and thus starts MyClass-activity, which then pulls the payload from the intent.

When you do this a few times in a row you will see that the passed counter (someCounter) does increase in your sending activity, but that the receiver always shows the initial value. Canceling the notification in the sender does not help here.

This comes from the fact, that the system does not assume that only because we pass a new Intent object to the pending intent, we want this new intent object to be delivered and just keeps the old (initial) pending intent alive.

To get the desired semantics, we need to pass a flag to tell the system:

PendingIntent pintent = 
   PendingIntent.getActivity(context,0,intent,PendingIntent.FLAG_CANCEL_CURRENT);

This flag (FLAG_CANCEL_CURRENT) tells the system that the old pending intent is no longer valid and it should cancel (=remove) it and then create a fresh one for us. There are more possible flags, which are described on the javadoc page for PendingIntent.

 

You can see a full example in the Zwitscher source code on github in the

• UpdateStatusTask::createNotifitcation method (sender)
• ErrorDisplayActivity::onCreate() method (receiver)

And remember that Zwitscher is live on the Android market - download and try it :-)

Zwitscher is on the market

Zwitscher, my Twitter client for Android(*) is now available in the Android-Market.

 

While v0.50 has still rough edges it is what I am basically using as my main and only Twitter client on my mobile. I've completely deleted the official Twitter app :-)

Zwitscher is open source and hosted on GitHub.

Using formatted text (in code) on Android

Putting text into a TextView on Android is easy - you just use TextView.setText(text) unlike other overloaded methods with the same name, you can not just use a String with Html formatting to format the passed text.

In order to do so, you either need to obtain the text from a resource file, use Html.fromHtml(String html) helper method or create a SpannableString that can then be passed to the method.

Html.fromHtml() is actually very convenient:

String s = "This is a <b>fat</b> string";

TextView tv = .. ;

tv.setText(Html.fromHtml(s));

But while working on Zwitscher I saw that Html.fromHtml() is actually very expensive to use, as internally it is using the whole arsenal of XML parsing.

Plan B is using SpannableString...

Creating such a text is a bit complicated as you need to deal with individual Spans and so on, which I found inconvenient.

So I've introduced a SpannableBuilder helper class to make this process easier.

Usage is as follows:

SpannableBuilder builder = new SpannableBuilder(context);

builder.append(status.getRetweetedStatus().getUser().getName(),Typeface.BOLD)      
      .appendSpace()
      .append(R.string.resent_by, Typeface.NORMAL)                
      .appendSpace()                
      .append(status.getUser().getName(), Typeface.BOLD);

textView.setText(builder.toString());

Using this helper class does not trigger any XML parsing and is thus a lot faster than Html.fromHtml().

So far only methods that deal with TypeFaces are implemented, but extending the class for modifications of the background would be easy.

Using the camera (from your programs) on Android is easy ...

... if you know how to do it.

If you don't know it, it can be a PITA as I have e.g. described at the Android-Tech-Talk in Stuttgart.

For Zwitscher I was for quite some time trying to enable it to take pictures and upload them to pictures services like with other Twitter clients. So I started looking at the documentation and found a How To entry which basically points to the documentation for the Camera.  So I've been trying around with the preview stuff and was googling like crazy and so on, but this turned out to be too complicated for me to follow in the short term.

Yesterday I was reading about Intents in the Documentation and what Google Intents are available. This brought me to the idea of investigating if the camera could also be called that way.

So I've googled for "android camera app intent" and found this forum post which explains how to do it.

Basically start the camera via:

Intent intent = new Intent(android.provider.MediaStore.ACTION_IMAGE_CAPTURE);

startActivityForResult(intent, 1);

and then later when the picture has been taken fetch the picture in the onActivityResult() callback:

@Override

public void onActivityResult(int requestCode, int resultCode,Intent data) {

super.onActivityResult(requestCode, resultCode, data);

if(requestCode==1&& resultCode==RESULT_OK) {

Bitmap bitmap = (Bitmap) data.getExtras().get("data");

}

}

This takes a small picture suitable for e.g. Twitter.

To take a larger picture you need to tell the intent where to store a larger picture and pick it up from that location.

intent.putExtra(android.provider.MediaStore.EXTRA_OUTPUT, path)

So you see - if you know all this, it is fairly easy to start the camera, take a picture and then use it later. No dealing with PreviewHolder and all that stuff.

 

Mein erster Android-Vortrag - und wie geht nun es weiter?

Gestern habe ich bei der STUGTUG in Zusammenarbeit mit der JUGS beim Android TechTalk meinen ersten Vortrag zum Thema gehalten.

Ich wurde gefragt, ob ich so über meine ersten Erfahrungen erzählen könne, was ich dann auch gemacht habe - nachdem ich mich an Hand von konkreten Beispielen in eine Materie einarbeiten muss habe ich eben von den Erfahrungen mit meiner ersten App, Zwitscher, berichtet (Folien gibt es hier und hier das Video). Peter Hoffmann hat einen Review geschrieben, wie auch Benny. Alle Videos sind in diesem Album zusammengefasst.

Die Auswertung der Feedback-Bögen gibt ein eher gemischtes Bild. Von "Sehr gut" bis "Kann ich nichts mit anfangen, weil ..."  "... ich nicht weiss was eine Activity ist", "... ich das schon seit Jahren mache" war alles dabei. Der Notendurchschnitt lag bei 2,8.

Mein Fehler war hier offensichtlich, für Anfänger in der Android-Programmierung schon zu tief in der Materie gewesen zu sein, während für die "alten Hasen" nicht so viel Neues dabei war.

Deswegen habe ich mir jetzt mal vorgenommen, möglichst bald einen Workshop für Einsteiger anzubieten, bei dem eine kleine Applikation von Null aus aufgebaut werden soll. Der Workshop wird dabei nicht nur "Frontalunterricht" sein, sondern die Teilnehmer sollen / müssen selbst Hand anlegen (Java SE Kenntnisse müssen vorhanden sein).

Hierzu nun ein paar Fragen:

• Wird so was (= Einsteiger-Workshop) überhaupt gewünscht?
• Was ist ein möglicher Zeitrahmen (am Abend, Samstag, Wochenende, Weihnachtsfeiertag, ...)
• Was darf es kosten? Der Workshop ist non-Profit, aber Räume, WLAN, Getränke und Brötchen / Pizza kosten einfach Geld
• Was für eine App soll es werden?

Ich habe auch eine Idee für einen umfangreicheren Workshop im Hinterkopf, möchte dazu erst mal noch nichts verraten ;-)

 

 

Android and the title bar progress indicator

 

I just spent quite some time trying to figure out how to get a tiny little spinning progress thingy in the title bar of my application. And to be honest, while there is a lot of documentation out there, it is far from trivial to finally implement it.

The best "how to" is just  few weeks old and the author also wanted to have such a thing.

Basically steps are: define a custom layout for the title bar, ask the system to allow replacing the title bar, set the content layout, only then set the custom title bar layout and then obtain a reference to the ProgressBar object.

As the post above is quite extensive, I just want to emphasis on the latter points:

 

protected void onCreate(Bundle savedInstanceState) {

super.onCreate(savedInstanceState);

   requestWindowFeature(Window.FEATURE_CUSTOM_TITLE); // 1

   setContentView(R.layout.single_tweet);        // 2

   getWindow().setFeatureInt(Window.FEATURE_CUSTOM_TITLE,

                              R.layout.window_title); // 3

   pg = (ProgressBar) findViewById(R.id.title_progress_bar); // 4

 

Comments for the lines above:

1. Ask the system to allow for a custom window title
2. Set the content view of the whole window
3. set the layout for the custom title
4. Obtain the reference to the progress bar

If this order is not honored, pg, the reference to the ProgressBar will be null.

The layout for the title is shown in above post, so I'll not repeat it here.

The second important aspect is the usage of the ProgressBar. The main thread of execution in Android is the UI thread. And when a callback is running, the UI is not updating while such a callback is running. So, the following is not starting the progress indicator:

   public void myButtonPressedCallback(View v) {

      pg.setVisibility(ProgressBar.VISIBLE);

      doSomeThingLongRunning();

      pg.setVisibility(ProgressBar.INVISIBLE);

}

The state of the progress indicator will only be set after myButton..() has returned. In oder to correclty handle this, you need to e.g. use an AsyncTask:

 

    private class DownloadImageTask extends AsyncTask<User, Void,Bitmap> {

        protected void onPreExecute() {

            super.onPreExecute();

            pg.setVisibility(ProgressBar.VISIBLE);

        }

        protected Bitmap doInBackground(User... users) {

          doSomeThingLongRunning();

        }

        protected void onPostExecute(Bitmap result) {

           pg.setVisibility(ProgressBar.INVISIBLE);

        }

 

   }

In this case, the ProgressBar is made visible in onPreExecute() (which runs in the UI thread. Then the background thread is started to do the long running computation and after this has finished, onPostExecute() runs again in the UI thread, where you can 'switch off' the ProgressBar again.

 

Android app lifecycle or "why do I sometimes get an empty screen?"

 

Many examples on the net for Android programs use something like this:

  public class MyActivity extends Activity {

    protected void onCreate(Bundle savedState) {

         super.onCreate(savedState);
         doSomeSetup(); // e.g. login to a remote site
         dispatchToOtherActivity();
    }
  }

This is usually fine, but in some situations you end up with an application that just shows an empty screen. You may wonder what happens here.

Actually if you have a look the Android Application Lifecycle , you will see that onCreate() will only be called when the application is 'cold started'.

If the application is just resumed, onCreate() is not called again and and thus the dispatch to the next activity never happens. The application just sits on the empty screen.

There is an easy solution: onResume() is always called when the application comes to foreground again no matter if the app was cold started or just came to foreground again. So the above code could also read:

public class MyActivity extends Activity {

   protected void onCreate(Bundle savedState) {

         super.onCreate(savedState);
         doSomeSetup(); // e.g. login to a remote site
   }
   protected void onResume() {

         super.onResume();
         if (!loggedIn()) 
                doSomeSetup()
         dispatchToOtherActivity();
   }
}

Now on cold start, the login is done in onCreate() and then control goes to onResume() that dispatches to the next activity. On warm start, onResume() is directly called, that can check if the login is still valid.