An experiment: Sony Xperia S and AOSP

Over time, AOSP has added files related to various hardware targets.
We started with just a few scripts on a web page (1.0), then we had a
git project (Cupcake), then we released some of the exact source files
that were used on retail flagship devices (Froyo), then distributed
proprietary binaries (Gingerbread), then we were able to run on
PandaBoard (Ice Cream Sandwich).

For a new challenge, I'd like to try to go one step further, and to
target some hardware beyond the usual categories. I've added a git
project for the Sony LT26, i.e. Xperia S. This seems like a good
target: it's a powerful current GSM device, with an unlockable
bootloader, from a manufacturer that has always been very friendly to
AOSP.

That git project is currently empty. I'm open to suggestions about the
best way to populate it. I think I'll start by putting together a
skeleton set of makefiles, followed by a kernel. Contributions are
strongly encouraged, and there should be more freedom than usual to
submit experimental changes since that won't impact the devices that
Google is most directly involved in.

I don't know how far that'll go, and there are so many unknowns that
the only way to know is to try it.

As usual, please be very careful about handling any proprietary files,
for Xperia S or any other device. Don't copy them, use them, or
distribute them without an appropriate license. Obviously, don't
upload them to AOSP if you don't own them. When in doubt, please ask
ahead of time, it's easier to answer an email than to fix things in an
emergency.

TL;DR: For a long while this'll probably only be useful for low-level
engineering, but won't be usable as a daily driver.

In theory, AOSP is designed such that it should be possible to plug in
the files related to additional hardware targets. In practice, that
has never happened.

The short-term goal is to investigate the difference between theory
and practice, on a favorable real-world example, and to see whether
the result is worth the effort. The long-term goal is to try to
eliminate whatever hurdles we find, both the expected and the
unexpected ones.

I expect that we'll need at least a few iterations before that can
become usable as a day-to-day phone, and it's unclear whether that'll
ever happen. There's a reasonable chance that it could be useful for
people working at a lower level, literally several layers below the UI
and applications that everyone is familiar with. If you've seen the
classic Android Architecture diagram, we're going to be in the kernel
and just above it.

I don't want to use the word "supported" because there's no way to
agree on a single unambiguous definition.

JBQ

On Tue, Aug 14, 2012 at 12:43 PM, Artem Russakovskii
<arch...@gmail.com> wrote:
> Fantastic. So what's the high level goal here, in layman's terms? Does the
> future hold direct support in AOSP for building more than just Nexus
> devices? Are you thinking of targeting only a certain subset of devices
> (say, ones that satisfy a list of parameters - GSM, unlockable bootloader,
> etc)? What about the lack of proprietary drivers (though even the Nexus
> phones have this problem, and they're supported by AOSP)?
> --
> You received this message because you are subscribed to the "Android
> Building" mailing list.
> To post to this group, send email to android-...@googlegroups.com> To unsubscribe from this group, send email to
> android-buildi...@googlegroups.com> For more options, visit this group at
> http://groups.google.com/group/android-building?hl=en

Beware that there's no actual kernel (I've put an empty file in its
place). My next step is to get the real kernel sources in AOSP and to
build my own kernel.

I think the other urgent step is to get a proper BoardConfig.mk in
there. You probably have something suitable.

I'm not incredibly worried about recovery (yet). I guess I'm biased,
since I only work on devices with unlocked bootloaders, so I'm used to
relying solely on fastboot.

JBQ

On Wed, Aug 15, 2012 at 4:47 AM, Andreas Makris
<andreas...@gmail.com> wrote:
> Hey JBQ,
>
> nice challenge you put up here :D
> I am one of the guys maintaining Sony Devices on CM tree. So this device is
> very common to me. OFC it is a powerful device, but it has some tricky stuff
> in it.
> For example the recovery is not this useable than a recovery on a Nexus
> device, so we trigger the recovery boot from boot.img and not from
> bootloader itself.
> Some more special stuff is going on here, but all is possible to handle ;)
>
> So start the skeleton, I am happy to help getting this up and running on
> AOSP.
>
> Best Regards
> Andreas aka. Bin4ry

Well, assuming that we can get to the same level of hardware support
(which is challenging on the AOSP side), we'd obviously have a
slightly different feature set.

I expect that the biggest difference might be more on the philosophy:
while CM is more likely to focus on stability, being based on a stable
tagged release, my work in AOSP is more likely to aim for the bleeding
edge, by using the master branch.

CM is "cherry picked" from various sources including but not limited to improvements donated by community. Since I do not have Xperia S, I can't say what exactly, but as a CM nightly user, I can say it would be the right "bleeding edges" AOSP desperately needs.

For one thing, CM kernel would likely be more open to change then Sony's unmodified

In all seriousness, we're made enough progress already to get us to
the point where we're facing the harder problems:
-making changes in the common Android tree that aren't actually
necessary for the devices that Google works on.
-handling the proprietary files.
-restoring the device to its factory state.

I see that as good news. That means that we've already taken care of a
number of smaller problems. Those also don't surprise me, from my
experience managing other devices in AOSP, and so far we haven't hit
any unforeseen hurdle.

JBQ

On Sat, Aug 18, 2012 at 10:16 PM, Tom Gascoigne <t...@gascoigne.me> wrote:
> An interesting idea. Depending on the success of this project, is there any
> chance we might see more non-nexus devices being supported in AOSP in
> future? There are a number of people (myself included) who are maintaining
> AOSP builds for various devices, and I for one would love to contribute back
> to the project.
>
> Tom

To be extremely clear, I'm definitely not in the business of forcing
OEMs to release source code. This part of the discussion ends here.

JBQ

On Sun, Aug 19, 2012 at 1:00 AM, RS <rajes...@gmail.com> wrote:
> Short version:
> JBQ, kudos. Coax manufacturers to release more driver sources.
>
> Long version:
> JBQ, incredible work with AOSP and Nexus already. Kudos.
>
> Now about this experiment, don't we all know from past that the real trouble
> is manufacturers' support for drivers.
> Whenever we move across a major kernel change, it becomes a nightmare to
> hack the old (binary) drivers in without the sources.
>
> Options:
> 1) Utopia: encourage manufacturers to submit sources to such drivers on AOSP
> and let them use "Google f/w update guaranteed" tag through CTS as long as a
> fresh firmware compiled from AOSP passes CTS on their device.
>
> 2) Real-world: Coax them to release as much to AOSP but have an internal
> Google branch where they could share sources with Google just enough of it
> to help rebuild drivers on major kernel change.
>
> Why hang on Sony's word today that they'll assign a couple of engineers to
> provide you updated drivers .... pulling the plug is just a non-tech dumb
> manager's whim away in a corporate. Google might be different but how do you
> see Sony, HTC, Moto-google-rola, LG, ... with dwindling android revenue to
> be reliable?
>
> Convince Samsung, the only non-nexus success on android to open-source to
> the extent possible and others might follow suit.
>
> If they submit bug-fixes well and good otherwise at least the old stuff will
> run on new android flavors.
>
> Thanks,
> RajS

just to make u aware, sony bootloader have some issue and sometime boot fail
we have guessed about certain boot.img size broke boot, change compression
or add a filler file to make the image smaller or bigger fix the issue

also boot partition is very big (20MB) and i would recommend to use
uncompressed
kernel image

On Thu, Sep 6, 2012 at 12:59 AM, Akhil Pachaury <akhi...@gmail.com> wrote:
> Thanks for information, but i need help regarding binary installation, I
> download the binary zip file for sony site, how i have to unlock boot
> downloader. Kinldy tell me steps that considering to install those file so
> that i can install AOSP in my sony xperia S.

On Thu, Nov 8, 2012 at 1:06 AM, Jim <zong...@gmail.com> wrote:
>
> Hello,
>
> I am new in this workgroup, and I am available to make some tests.
>
> For the moment,
> - I have retrieved the android-4.0.4_r2.1.
> - I have copied kernel 6.1.A.0.452 files from Sony (kernel, vendor, external
> directories).
> - I have copied device/sony/lt26 files from
> https://android.googlesource.com/device/sony/lt26.git> - I have copied SW_binaries_for_Xperia_S_v1 files into vendor directory
>
> Now I would like to build the project, but I do not have so much experience
> in android build...
> I ran build/envsetup.sh, and I can see that device/sony/lt26/vendorsetup.sh
> is included.
> But when I run "lunch" and I select full_lt26-userdebug in the list, I have
> the following error message:
>
> build/core/product_config.mk:193: ***
> _nic.PRODUCTS.[[device/sony/lt26/full_lt26.mk]]:
> "frameworks/native/build/phone-xhdpi-1024-dalvik-heap.mk" does not exist.
> Stop.
>
> ** Don't have a product spec for: 'full_lt26'
> ** Do you have the right repo manifest?
>
> Does anybody could tell me how I can solve the issue ? Does my overall
> method correct ?
>
> Thanks a lot !

This indicates that you haven't actually synced and checked out
the current master of external/e2fsprogs, because the reference
to LOCAL_MODULE_STEM that make complains about was removed in
commit dc3069a about a year ago. The most recent commit in that
git should be 5886dc5, and that's the commit you should have
checked out. Try this (and post the output):

> Thanks a lot for your support, and sorry for the late answer, I was
> not there during the last three days.
> I think I have already the latest commit. Here is the git log:
>
> external/e2fsprogs]$ git log
> commit 5886dc5cdcccd3d09a208d41d8c23748c25a2a22
> Merge: 0a02698 fada366
> Author: Theodore Ts'o
> Date: Tue Jun 12 12:53:18 2012 -0700
>
> am fada3660: e2fsck: correctly propagate error from journal to
> superblock
>
> * commit 'fada366033e80c119867eb303e8b48a8e027a9be':
> e2fsck: correctly propagate error from journal to superblock
>
>
> I have executed on last Thursday a repo sync on master branch, so I
> guess I have the latest commits almost every where.

If you look in the Android.mk files below external/e2fsprogs, is
LOCAL_MODULE_STEM set for any shared library module (actually,
anything but executable or host executable)? Show the output of the
following commands (run from the e2fsprogs root):

It looks like your source workspace is okay. I don't have time to help
you any further. You should get to the bottom of which module it's
actually complaining about (e.g. by commenting out parts of the
makefiles, piece by piece until it stops yelling) and take it from
there.

6.1.A.0.452 is ICS-based and won't work out of the box with the
JB-based master branch. Unless you really know what you're doing
I don't recommend trying it.

> This delivery contains several directories in /external,
> including external/e2fsprogs
> I will make new tests, and try to include directories one by one.

What do you mean, did you get the build problems when including the
source code from Sony's 6.1.A.0.45[234] drop? That would explain the
build error but wouldn't explain why Git reported external/e2fsprogs
to be clean and have the correct commit checked out.

No, the master branch contains Jellybean and some other stuff. You can
assume that the master branch is always ahead of anything on release
branches.

> Here is what I want to do:
> Be able to build a full android image, with the associated kernel, and
> flash it in my Xperia S.
>
> For the kernel sources, I only found ICS (6.1.A.0.452). So I guess it
> is easier to make this exercise with ICS.
> Then I first tried to use branch android-4.0.4_r2.1 to be sure to have
> ICS, but I understood that the lt26i is not supported in this branch.
> Jean-Baptiste then told me to simply retrieve the master branch, that
> is in fact ICS. Is it right ?

The master branch is closer to Jellybean than to ICS. You need the
master branch to get the device configuration, but the code that Sony's
released is for ICS. Yes, this makes things harder. You either have to
retrofit the LT26 setup from the master branch to an ICS branch (sounds
pretty easy) or use the ICS-based Sony software on top of Jellybean.

> The compilation of this matser branch is ok in lt26i device mode
> (if I download vendor/sony/lt26 git).
> However, I also made the test to copy external and vendor files
> from 6.1.A.0.452 delivery, and then the compilation issue occurs.

You can't just copy the files. You'd have to figure out what Sony has
done in those directories and apply the same patches to a Jellybean
tree. They've changed about 80 files in the Webkit tree but I don't
think they've made any significant changes anywhere and probably none
that are necessary for the phone to boot. The reason those directories
are included in the archive is because they're under GPL or LGPL.

> (And the error does not disapear if I only change the
> external/e2fsprogs to the original one).

> Something is still not very clear for me: the dependence between the
> kernel and the android source code.
>
> I have the kernel source code 6.1.A.0.452. I understood that it was ICS.

Correct.

> So I guess that if I switch to android master branch that is JB,
> I will need the JB kernel.
>
> Am I right ?

An ICS kernel might work, it depends on the actual kernel changes
between ICS and JB. It might be possible to apply the changes between
the stock ICS kernel and Sony's ICS kernel onto the stock JB kernel,
but that probably wouldn't be a trivial task.

> Where could I find this kernel ?

Sony hasn't released any JB-based kernel sources. I think they've
announced that JB updates for their phones will be released during
Q1 2013, so I wouldn't expect kernel sources to be publicly available
prior to that.

Someone mentioned that Intel need a similar patch, so it would make sense to look at the raised concerns and try to get this patch into the tree.

For now you can cherry-pick this patch (minor conflict though).

This will make it possible to flash the partitions and power on the device. Unfortunately the graphics stack is updated, so with the current set of gralloc and adreno binaries we're missing https://android-review.googlesource.com/#/c/42485/ and the "legacy" variant of SurfaceComposerClient::getDisplayInfo().

By cherry picking #42485 and reverting (the revert) 38b657265ccc5ae45bd7860a68b0d9373b47a2f3 you can get the device to boot.

We're looking at sorting these things out in a proper way so that we can publish updated source and binaries.