Building, running, and debugging Android source

There is a lot of confusion surrounding the work flow in the Android source tree, so allow me to simplify:

1. Follow the initial instructions for downloading the source at:
   
   http://source.android.com/download
   
   
2. Set up your environment to build the engineering build for the generic device and generic product. This is similar to the SDK, but with a few pieces missing.
   
$ source build/envsetup.sh
$ lunch 1


3. To build for the first time:

$ make

   If you have a multi-core system, you can build with make -jN where N is twice the number of cores on your machine. This should speed up the first build considerably.
   
   
4. To launch the emulator from your build:

$ ./out/host/<your-machine-type>/bin/emulator

   On my system <your-machine-type> is linux-x86.
   
   NOTE: The emulator knows where to find system and data images as a result of running lunch 1 above. This sets the environment variable ANDROID_PRODUCT_OUT to point to the target directory. For this example, it should be out/target/product/generic/.
   
   
5. If you wish to make changes to the source code, there are handy utilities that have been exposed to your environment by source build/envsetup.sh above. For example, if you modify the Email app and just want to rebuild it:
   
   $ mmm packages/apps/Email
   
   
6. To see your changes in the emulator you can run:
   
   $ adb remount
$ adb sync
   
   Which will copy the regenerated Email.apk file into the emulator's /system/app folder, triggering the PackageManager to automatically reinstall it.
   
   
7. Or if you change framework resources in frameworks/base/core/res/res/ you could regenerate framework-res.apk with:
   
   $ mmm frameworks/base/core/res
   
   Or if you modified even the framework itself you could run:
   
   $ ONE_SHOT_MAKEFILE="frameworks/base/Android.mk" make framework
   
   This is a special variation of mmm which is used to build frameworks/base/core/java.
   
   To sync these changes you must restart the running framework and sync, as with this handy sequence:
   
   $ adb remount
$ adb shell stop
$ adb sync
$ adb shell start


8. Finally, to debug your changes you can use the DDMS tool to select a process for debug and then attach Eclipse to it. If you have the Eclipse Android Development plugin installed, there is a special DDMS perspective which you can use to choose the process for debug. To attach Eclipse to it, see these instructions:
   
   http://source.android.com/using-eclipse
   
   This document also describes how to use Eclipse for development. Any IDE should work with the proper finagling though. Just note that the IDE won't really be an integrated environment: the final output of APKs, system.img, and even the generation of R.java files will have to be done by make!
   
   A note about the processes in Android:
   
   
   • system_process houses all things under frameworks/base/services. This includes the PackageManagerService, StatusBarService, etc. It has many, many threads (one for each service, and then one main UI thread), so be wary when debugging.
   • com.android.acore hosts Launcher (home), Contacts, etc. You can determine the apps/providers that run here by looking for android:process="android.process.acore" in the various AndroidManifest.xml files in packages/.
     
   
   Also remember that the "framework" (under frameworks/base/core/java) is not hosted by any one process. It is a library used by most processes, so to debug code there you can usually use a simple demo app that takes advantage of whatever you changed and debug that app's process. A useful trick for setting up your debug connection is to call Debug.waitForDebugger() during some startup part of an application or system service.
   
   

Push services: Implementing persistent mobile TCP connections

As a result of my work on IMAP IDLE support in Android's default mail application, I have been experimenting with various strategies for implementing long-lived services and persistent connections that operate efficiently in a variety of circumstances. Several quirks about Android and mobile devices in general arose that could be of value to anyone implementing similar services.

For most protocols, you will need to implement some type of client-initiated keep alive at the application layer. For my purposes with IMAP I simply complied with the RFC and elected to leave IDLE mode then re-enter after 28 minutes of inactivity. On Android, you must use the AlarmManager service to wake the CPU for this task. You might be tempted to use a Handler for timing or even a simple thread with a looped sleep() however it should be noted that unless your application otherwise holds a WakeLock you cannot rely on any timing mechanism other than the AlarmManager. Once the screen goes blank, the CPU may sleep and once it does other timing mechanisms will block until the CPU wakes up again, regardless of any timeout paramters you supply.

After running my test for several days I noticed Android was mysteriously killing processes, claiming that the services implemented in them have "died", then restarting them just a few minutes later. No call to the service's onDestroy method will occur, and even on service restart you will only see a call to onCreate and not onStart. In order to compensate for this you are expected to store your state persistently and check for a discrepency during onCreate and then invoke startService for yourself if necessary. The SharedPreferences system can be handy for this.

Source code for a functional demonstration on this topic can be found at my android-random project page, under the module TestKeepAlive.

Working on IMAP IDLE support for Android's Email app

For all those screaming about the lack of push e-mail updates for regular IMAP accounts with the basic Email app in Android, I have been digging into the code for the past few days and am preparing a patch for IMAP IDLE support. For those that do not know, IMAP IDLE allows a client to maintain a persistent, light connection to the server that is then notified of new messages as they arrive. Many IMAP servers support this extension, including Microsoft Exchange which would thus allow push e-mail for those that need corporate e-mail options on the G1.

I will post an APK for folks to try once I have the support working well. Stay tuned...

Generate Callback Listener Helpers from AIDL

As I redesign Five I find myself tweaking/expanding the callback listeners in my interface which are handled through the RemoteCallbackList class in the service implementation. This class is great, however for each call code must be written similar to:

int N = mCallbacks.beginBroadcast();

for (int i = 0; i < N; i++) {
  try {
    mCallbacks.getBroadcastItem(i).onSomeEvent();
  } catch (RemoteException e) {}
}

mCallbacks.finishBroadcast();

This can quickly pollute the service implementation, and even if you factor these calls out to a separate class it can be a pain to create and maintain them. So, I came up with a solution in the form of a Perl script which takes as input an AIDL file defining the callback interface and outputs a class implementing the extended RemoteCallbackList. To use, simply type:

$ ./aidl-cbliststub.pl < IFooListener.aidl > IFooListenerCallbackList.java

Download: aidl-cblistsub.pl

Android Instrumentation Example

As Android approaches maturity with its recent 0.9r1 release, I began to think its time that my main project updates to match. I decided to take another look at Android's instrumentation and unit testing features to build robust tests for my critical services and activities. Unfortunately, there isn't anything new in the way of documentation however there is now is an example we can use in ApiDemos.

Looking at the new ApiDemos we see that there is a new tests directory which contains a second AndroidManifest.xml with no user activities defined. This is important as this structure allows us to have two separate APKs, one for production and one for testing, so our main distribution and build doesn't need to be polluted. Unfortunately, this approach is not compatible with the Eclipse plugin, and so we must define everything in an external build environment.

For convenience, I packaged the complete ApiDemos project with my Ant build environment here: ApiDemos-instrumentation-build.tar.gz. To build, start the emulator and run the following commands:

$ adb shell rm /data/app/ApiDemos.apk
$ ant install
$ cd tests && ant install

It is essential above that you remove the old ApiDemos.apk as the instrumentation APK must be signed with the same key as the package it tests.

Once we've got both APKs installed we can begin running our unit tests. There is no UI in Android for this, however it can be invoked more conveniently through adb as such:

$ adb shell am instrument -w com.android.samples.tests/android.test.InstrumentationTestRunner

This will run all defined tests and print a dot character for each successful test, F for failure, or E for invocation error. See the javadoc in tests/src/com/android/samples/AllTests.java for more sample command lines to run individual test cases or specific tests within them.

My main project, Five, has recently been updated to include unit tests in the five-client component, with more test coverage to follow in the next few weeks. For a non-trivial test case, see my CacheServiceTest.

NOTE: Previously, I was using the masa Android plugin for Maven, however it has not yet been updated to support 0.9r1. I feel that using Maven for this would have been cleaner, but the Ant approach is sufficient for this simple demonstration. Once updated, I will return here and finish this example with a more sophisticated build environment to automate testing.

Android on the HTC Vogue

As many of you know, the Vogue can run Android quite nicely with support for sending/reciving SMS, incoming and outgoing calls, GPRS, touch screen, etc thanks to dzo over at Xda-Developers. See his posts and materials here: http://it029000.massey.ac.nz/vogue/.

Several folks from freenode/#android have access to this device, myself included, and have been developing tools and applications to further explore the platform on real hardware. Those tools are being hosted at the android-random project page. There is a simple file manager (Glance), a much-improved threaded text messaging app (Messages), and perhaps most importantly of all a RemoteLogcat tool that allows us to observe the running devices logcat (normally accessible through adb logcat) by piping it to a server on the public Internet.

Recently, I created an emulator skin that can be used to simulate this device's resolution during development. Simply download vogue-skin.tar and unpack it into $ANDROID_SDK/tools/lib/images/skins along with the other default skins. Then fire up the emulator as:

$ emulator -skin vogue

The skin has no decorations, just a simple 240x320px layout, but this should give you a pretty good idea how different Android can be on this device.

Interruptible I/O example using HttpClient

Recently I found myself considering some of the gotchas of threaded, blocking I/O as I've been using it on Android. Specifically, how do we gracefully handle interruptions demanded by the user or the system to free resources? After some thought there seems to be three basic strategies with Java:

• 1. Use non-blocking I/O, which is generally clumsy and unintuitive for most Java engineers.
  
• 2. Cancel blocking I/O threads by simply setting a stop flag and discarding the reference to the thread. The thread will clean itself up after an arbitrary length of time up to its transmit or connect timeout.
• 3. Close the socket owning the input stream on which the blocking thread is waiting.

It seems that #2 is the most popular choice on Android however I would like to make a case for #3 as a cleaner method of tidying resources on user request. With this approach, we can be certain to quickly close any open files, relinquish database, object, or file locks, and allow the thread to clean up its resources quickly. As it turns out, using HttpClient makes this approach relatively painless, but there are a few gotchas in this pattern that we must watch out for.

So, to get started we need to create our StoppableDownloadThread class which allows us to encapsulate our interrupt logic.

public class StoppableDownloadThread extends Thread
{
  private String mURL;
 
  private HttpGet mMethod = null;
 
  /* Volatile stop flag used to coordinate state between the two
   * threads involved in this example. */
  protected volatile boolean mStopped = false;
 
  /* Synchronizes access to mMethod to prevent an unlikely race
   * condition when stopDownload() is called before mMethod has
    been committed. */
  private Object lock = new Object();
 
  public StoppableDownloadThread(String url)
  {
     mURL = url;
  }
}

This simply outlines our basic strategy for stopping and synchronization. A simple volatile boolean flag and a monitor lock to share the HttpGet handle should do just fine. Now let's continue with the implementation of the run() method:

public void run()
{
  HttpClient cli = new DefaultHttpClient();
  HttpGet method;
 
  try {
     method = new HttpGet(mURL);
  } catch (URISyntaxException e) {
     e.printStackTrace();
     return;
  }
 
  /* It's important that we pause here to check if we've been stopped
   * already.  Otherwise, we would happily progress, seemingly ignoring
   * the stop request. */
  if (mStopped == true)
     return;
 
  synchronized(lock) {
     mMethod = method;
  }
 
  HttpResponse resp = null;
  HttpEntity ent = null;
  InputStream in = null;
 
  try {
     resp = cli.execute(mMethod);
 
     if (mStopped == true)
        return;
 
     StatusLine status = resp.getStatusLine();
 
     if ((ent = resp.getEntity()) != null)
     {
        long len;
        if ((len = ent.getContentLength()) >= 0)
           mHandler.sendSetLength(len);
 
        in = ent.getContent();
 
        byte[] b = new byte[2048];
        int n;
        long bytes = 0;
 
        /* Note that for most applications, sending a handler message
         * after each read() would be unnecessary.  Instead, a timed
         * approach should be utilized to send a message at most every
         * x seconds. */
        while ((n = in.read(b)) >= 0)
        {
           bytes += n;
           System.out.println("Read " + bytes + " bytes...");
        }
     }
  } catch (Exception e) {
     /* We expect a SocketException on cancellation.  Any other type of
      * exception that occurs during cancellation is ignored regardless
      * as there would be no need to handle it. */
     if (mStopped == false)
        e.printStackTrace();
  } finally {
     if (in != null)
        try { in.close(); } catch (IOException e) {}
 
     synchronized(lock) {
        mMethod = null;
     }
 
     /* Close the socket (if it's still open) and cleanup. */
     cli.getConnectionManager().shutdown();
  }
}

This is a pretty standard example of an HTTP GET using HttpClient4, however do note that we have strategically placed checks against our stop flag to avoid leaving the download thread in an inconsistent state when it's being cancelled. We're not done yet though as we still need to implement the stop part of the interface so that our main thread (or any other thread) can abort the download thread:

public void stopDownload()
{
   if (mStopped == true)
      return;

   /* Flag to instruct the downloading thread to halt at the next
    * opportunity. */
   mStopped = true;

  /* Interrupt the blocking thread.  This won't break out of a blocking
    * I/O request, but will break out of a wait or sleep call.  While in
    * this case we know that no such condition is possible, it is always a
    * good idea to include an interrupt to avoid assumptions about the
    * thread in question. */
   interrupt();

  /* A synchronized lock is necessary to avoid catching mMethod in
    * an uncommitted state from the download thread. */
   synchronized(lock) {
      /* This closes the socket handling our blocking I/O, which will
       * interrupt the request immediately.  This is not the same as
       * closing the InputStream yieled by HttpEntity#getContent, as the
       * stream is synchronized in such a way that would starve our main
       * thread. */
      if (mMethod != null)
         mMethod.abort();
   }
}

This completes our basic interface, but we still don't have a usable example here. There's no communication between our download thread back to the user in any meaningful way as would be required for an Android application. For that I have modified the above code slightly and introduced an Android layer in the form of a working demo. Source code for the full example: CancelHttpGet.tar.gz.