Content

1. State of affairs, or the end of Kugo
2. What it is about
   1. Trying to find a new ROM on the interwebs
   2. Down the rabbit hole
3. AOSP for Kugo
   1. Android 8 Oreo
      1. Building it
      2. Comparison to the stock ROM
         1. Working features
         2. Limitations
      3. Getting GApps onto the phone
         1. With the userdebug build
         2. With the engineering build
   2. Android 9 Pie
      1. Building it
      2. First impressions

State of affairs, or the end of Kugo

At some time in 2022 my Sony Xperia X Compact, running my build of Anroid 9 did stop to work with any SIM card. Unlocking the SIM via the PIN worked, but no signal. I bought another phone in order to have a working phone, while Kugo was in this state. But as the pressure to solve the problem was gone with the new phone, I never dived deeper into the problem.

Thus, Kugo is dead, and will not receive any further updates. The links below, pointing to guides by Sony do not work anymore. Sony, however, still offers some guides for building Android for various phones. So if you want to go down that road, you'll still might find some of the problems ahead of you, and solutions for them, below.

What it is about

I own a Sony Xperia X Compact (model F5321, codename "Kugo"), which I like very much. It is not too large (actually it could still be a bit smaller) and Sony didn't go with the "Apple uses glas, so we all use glass"-mentality and instead has a plastic cover for the display. Yes plastic might scratch easier, but I find it easier not to place my phone in the same pocket as my keys than not dropping my phone every once in a while.

There is only one problem with Kugo which bugs me more and more: At the end of 2018, only two years after it was released, Kugo received the last OTA update of its firmware. As I'm writing it is the end of 2021, this makes three years of unpatched security holes in my pocket.

Of course I could go for a new phone, but I totaly hate the idea of ditching a perfectly fine phone - the battery still works, the computing power is good enough for me - just because Sony decided that two years is the life span of a phone. So I decided to replace Kugo's stock ROM by something fresher.

Trying to find a new ROM on the interwebs

More years ago than I want to admit to myself, I had my first contact with android moding. That time I replaced the stock ROM of an HTC Desire with CyanogenMod. After that I had a row of mobiles which didn't last me as long (because spare parts became rare, or where not available in good quaility, too fast). Hence, I never ever again wanted to replace a stock ROM with something else until now.

This is why I first looked for CyanogenMod only to find it being replaced by LineageOS, and LineageOS not supporting Kugo. There are some ROMs for Kugo on xda-developers.com, even LineageOS. But since the last time that I replaced a stock ROM, I got a little bit older and older me isn't that easy with flashing my phone with some unofficial ROM someone has build who I don't know. Someone who might not have the best of intentions, who might have added some lines of code...

Down the rabbit hole

Hence, I asked myself if I can't build LineageOS for Kugo on my own. I toyed a bit with the idea, started to read a bit, doubted the possibilities, and then found that Sony supports building AOSP for some devices, including Kugo. So instead of directly trying to build LineageOs, I decided to build AOSP, starting as close to the stock ROM as possible, comparing both ROMs feature-wise and going up in versions as far as possible.

On this page I will document my journey and all problems which I encountered, such that others may find it easier to follow this way.

AOSP for Kugo

Sony does offer AOSP configurations for some devices. This includes guides for building various AOSP versions, prebuilt kernels, guides to find a UART port on some devices (unfortuantely not for Kugo). Sony also offers a tool for flashing the stock ROM onto devices, which I yet (2021-09-19) haven't tested, due to it being a windows tool.

I didn't want to risk loosing my phone due to bricking it in the proceeding. Hence, I bought a slightly broken one (does not detect SIM card) on ebay, which I use for testing, before flashing my actual one.

Sony does document differences in funcionality between the stock ROM and the AOSP builds, that is they document the functionality in one big table. This table does raise some questions. I list the differences in this table below and comment some of them, however, for many of the raised questions some test will have to be performed.

So, the differences documented by Sony contain some fancy marketing names. I guess that some features of the stock ROM are not given by Sony as open source software, in order to keep an advantage on the market. However, some may be simple marketing BS anyways. This becomes obvious with the camera.

With the camera the stock AOSP ROM only lists 20MP, while the stock ROM is listed with 32MP and some fancy marketing names. The missing 3MP are a bit strange and I do not know yfileet what to think of that. But the list of features of the camera with the stock ROM also lists "1/2.3 Exmor R™ for mobile image sensor", which seems to be the marketing name of the lens, which obviously does not change with the software. (The same holds true for the secondary camera).

A further feature of the camera, which - according to Sony - does not exist with AOSP, is a "Predictive Hybrid Autofocus". My guess is, that this is an autofocus algorithm, which uses data from the phone's ToF (Time of Flight) sensor in combination with the last focus setting (hybrid) in order to analyse the movement of an object, such that it can predict it's next position and hence, focus setting of the lens.

During a bit of browsing in Sony's github repos I found a module which was named "ToF_Autofocus" or something like that, so I think that there is at least an open source algorithm for using the ToF sensor for autofocus puposes. Regarding the ToF-Sensor itself I think it's just a simple one (not a ToF image sensor, just one reading).

The documented differences further state that with AOSP one can only have ISO 800, this I guess is just a limitation of the camera app which comes with AOSP. I'd expect that the camera driver can set the image sensor's gains (and thus ISO). Also the camera app which comes with AOSP does have some HDR mode, which Sony lists as feature of the stock ROM but not of AOSP.

Android 8 Oreo

My original plan was to first build version 8.0.0, as this was the last Android version which was delivered by Sony as an update for Kugo. The idea behind this was to get a good comparision between the stock ROM's features vs. the AOSP build's features. However, I had to found that the build of version 8.0.0 is broken for Kugo, as explained in detail below. Hence, the first version which I build successfully is version 8.1.0, revision 52.

Building it

I build this version on a Thinkpad X240 (Intel(R) Core(TM) i7-4600U CPU @ 2.10GHz, 8GiB RAM), runing arch linux. I went through some errors, which I fixed on the way, hence I can't tell you a build time for it. As explained above, I started with version 8.0.0, following this guide, and changed to 8.0.1 following this guide after hitting build errors due to incompatibilities as explained below. I started using zsh, as it is my shell of choice, but changed to bash in the proceeding.

The following list contains all problems into which I ran in chronological ordering.

1. Invalid parameter name, when selecting the device in source build/envsetup.sh && lunch
   This is an incompatibility with zsh. The solution can be found here: setopt shwordsplit and export LC_ALL=C. The latter is also needed in bash, because otherwise the build will fail in loadlocale.c:130 due to cnt < (sizeof (_nl_value_type_LC_TIME) / sizeof (_nl_value_type_LC_TIME[0])) being false.
2. Build calls some python 2 code. Solved that by setting up a virtual environment with python 2 using virtualenv and virtualenvwrapper:
   pip install virtualenv
   pip install virtualenvwrapper
   export WORKON_HOME=~/pyEnvs
   mkdir -p $WORKON_HOME
   source /home/julian/.local/bin/virtualenvwrapper.sh
   mkvirtualenv -p=/usr/bin/python2.7 android
   workon android
   
3. Errors due to "missing java.lang"
   • Changing from zsh to bash didn't fix the problem. Used bash from then one anyways.
   • Cleaning the project (m clean) finally fixed it.
4. prebuilts/misc/linux-x86/bison/bison: No such file or directory, despite it existing. This was due to missing 32bit support. The solution (on arch linux) consists of the following steps:
   1. Enable multilib support in the pacman configuration (/etc/pacman.conf) by uncommenting the [multilib] section, as explained here.
   2. Install 32bit dependencies: sudo pacman -S lib32-glibc lib32-ncurses lib32-gcc-libs
5. Cannot connect to Jack server, despite jack-diagnose not giving any error at all, preceeded by warnings that a proprietary format (PKCS12) is being used.
   Found the solution here. I.e. enable TLSv1 and TLSV1.1 support in java by removing them from dk.tls.disabledAlgorithms in /etc/java-8-openjdk/security/.
6. Build error: vendor/oss/fingerprint/egistec/legacy/EgisOperationLoops.cpp:158:29: error: no matching constructor for initialization of 'const hidl_vec<uint8_t>' (aka 'const hidl_vec<unsigned char>').

   This error took me a while to figure out (or rather to accept it). The fingerprint reader driver does use a constructor for hidl_vec which takes two arguments. This constructor however does not exist with Android 8.0.0_r30. If I'm not mistaken, than it does not exist in any revision of Android 8.0.0 (well, it's only a revision, right?). It does however exist in Android 8.1.0. So the problem is, that the fingerprint reader driver implementation which comes with the Android 8.0.0_r30 tag of Sony's code, does use a newer hidl version and, hence, is incompatible with Android 8.0.0_r30. Hence, I switched to Android 8.1.0 at this point.

   During switching from 8.0.0 to 8.1.0, I first encountered errors during repo init. In order to get rid of them, I first had to check out the 8.1.0 branch of Sony's code in local_manifests.

7. Java heap out of size.
   Reducing the number of cpu cores used for building did not fix this. I finally used the solution given here, i.e: $ export _JAVA_OPTIONS="-Xmx4g" and then build on two cores again.
8. Finally I found that the build process uses rsync, which I hadn't installed and, thus, ran into an error which was easily fixed by installing it.

Comparison to the stock ROM

I have flashed my test phone only today (2019-09-19), so tests are still ongoing, and might take longer, as I have some busier weeks ahead.

So far I found that the main camera has a bit duller colors than the one with my main phone. Unfortunately I didn't compare the cameras of my test and my main phone prior to flashing the test phone. Hence, I'm not sure if this is software or hardware. It might be that the stock ROM comes with some ICC profiles for the camera, if color-management is a thing on Android, of which I'm not sure. However, I think that it's most likely that the stock ROM's camera app does push the saturation of the images' colors, in order to match actual or expected customer expectations.

Working features

The following features are definetly working with the AOSP ROM:

• Fingerprint reader
• Bluetooth
• Both cameras
• Vibration
• WiFi
• GPS
• Acceleration sensor

Limitations

I found the following limtations with the AOSP ROM:

• Developer options menu not working.
  

  Crashes the Settings app. This only happens in the engineering build. With the engineering build the logging level "wtf" will make apps crash, on the userdebug build those level will simply be logged. With the developer options menu I found this in the logcat output, where the developer menu tried to load the oem_lock service which is missing with the AOSP build. That brought me to this github issue, where the exact problem gets described.

• Phone sometimes not waking up from standby on pressing the power button.
  

  With this one I'm not sure if this is due to my test phone, as I yet (2021-09-19) haven't tested the AOSP ROM with my actual daily-use mobile.

• No service menu.
  

  With the stock ROM one can enter a serive menu by dialling *#*#7378423#*#* (*#*#SERVICE#*#*). This menu unfortunately does not exist with AOSP. This is unfortunate, because with said service menu one can test various subsystems of the phone (camera, display, various sensors, etc.). I yet (2019-09-19) haven't found a replacement.

• Haven't found a way to obtain screenshots yet.

  Maybe I just didn't use the key combo (volume-down plus power) correctly, but I was able to take screenshots with the userdebug build.

• Camera related stuff
  The differnces in functionality documented by Sony list some differences concering the camera. Among them is that the main camera can take images with 23MP resolution in the stock ROM and only 20MP in the AOSP ROM. In the stock ROM I indeed found the option of 23MP in the Sony camera app (and also found that I'm only taking 8MP images, which a) is way enough, and b) If I want to take serious images I use a proper camera). The highest resolution setting which I could find in the camera app which comes with AOSP was 17MP, but frankly speaking I couldn't care less. Further with some Open Camera APP i could set an ISO level of 4800, which is more than the 800 that where anounced in Sony's list of differences in functionality.

Getting GApps onto the phone

With the userdebug build

While analyzing the crashing developer options menu, I found that the engineering build is not meant for daily usage but for, well, engineering. The actual build for daily usage is the userdebug build. With that, however, i couldn't add gapps the way I did with the engineering build. Every change which I made to the system partition, either via adb shell, or by modyfing the image file prior to flashing, vanished during rebooting. I found that this is due to Android using the device-mapper-verity kernel feature, which checks the integrity of the image. This can not only detect changes, but also features forward error correction for recovering the original state.

After trying out some ideas and briefly toying with the idea to deactivate verity in the userdebug build, I finally decided to give the Open GApps project a try. The open GApss can be integrated into the aosp build, which is the option which I chose. The variant which I've chosen is nano.

With the open GApps integrated into the build I stumbled across a few problems, the first one being that the git lfs part of the documentation didn't worked for me. I then found the solution here. The second one came with a strange error message when the build system tried to build the system image:

Could not build FEC data! Error:
Out of space? the tree size of out/target/product/kugo/system is (MB):
1377 out/target/product/kugo/system
The max is 5910 MB.

So, the image is smaller than the maximum size and hence too large? After trying numerous things I found that the build sometimes worked and sometimes didn't and finally I saw in the kernel messages, that a process which computes the forward error correction information had been killed, due to my little thinkpad running out of RAM. So if you get this strange message: You may want to close some applications in order to save RAM when building on a machine with "small" RAM.

Finally, after flashing my new build my phone was hanging in a bootloop. Luckily I found that adb logcat is already possible during the boot process. Here I saw the following message.

04-08 00:23:41.787 1966 1966 E AndroidRuntime: *** FATAL EXCEPTION IN SYSTEM PROCESS: main
04-08 00:23:41.787 1966 1966 E AndroidRuntime: java.lang.RuntimeException: There must be exactly one installer; found [ResolveInfo{558d8 com.android.packageinstaller/.InstallStart m=0x608000}, ResolveInfo{268d431 com.google.android.packageinstaller/com.android.packageinstaller.InstallStart m=0x608000}]

So the problem was, that with open GApps there came the google package installer in adition to the one which comes with the AOSP build. I fixed that by adding GAPPS_FORCE_PACKAGE_OVERRIDES := true to device/sony/kugo/device.mk.

With the engineering build

For the time being I'd just like to get a newer Android version. In the future I might also try to keep google a bit more out of my phone. But for now I wanted to have the PlayStore in order to easily install some apps. Hence, adding GApps was one of the next logical steps.

There are again some dubious sides where you can download GApp bundles. There's also the Open GApps project. Also I found a description for pulling the GApps from Official Google nexus images and decided to try it this way.

Initially this way failed with multiple errors. This might be the case because I initially only copied the .apk files which where mentioned in the stackoverflow answer, not realizing that, in Android 8.1.0, with PhoneSky there is also an .apk.prof file and with prebuiltGMSCore there is also an app_chimera folder which contains further .apk files. Also a colleague of mine hinted to me that it might be that the GApps have to be placed in the ROM prior to booting it the first time (thanks Bekim ˆ-ˆ). The following way finally worked for me:

1. Get the original Google Nexus image for the correct Android version from here
2. Convert the image to a filesystem image using android-simg2img and mount the system image.
3. If you already booted the AOSP ROM, reflash it. With the freshly flashed phone, boot it into the recovery system (Kugo: Volume down plus power button).
4. In the recovery system: mount the system partition
5. Copy the following from the mounted system partition to the phone using adb push whatToCopy /sdcard/:
   • priv-app/GoogleServicesFramework
   • priv-app/Phonesky
   • priv-app/PrebuiltGmsCore
   • etc/permissions/privapp-permissions-google.xml
6. Remount the system partition writable by calling adb remount and open an adb shell.
7. In the shell, move the stuff which we have copied from the Google Nexus image to the phone to it's correct location, chmod 755 all the copied directories and chmod 644 all copied .apk files.
8. Reboot the phone (adb reboot).
9. Activate WLAN, grant the GApps all permisions (not really sure if this step is needed).
10. You now have the PlayStore. Enjoy :)