[RESEARCH] External touch screen instead of AA head unit

Search This thread

mufunyo

Senior Member
Jul 28, 2012
70
31
Seeing videos of people integrating Raspberry Pis into their cars with OpenAuto got me interested in Android Auto. There is one thing I don't like about it, however: you're just adding another device to pass on audio/video/touch to an external display, from the phone. Why not cut out the middle man?

Back in the good old days of my Xperia S, phones had a dedicated micro-HDMI output. Delicous 60fps 720p (and even 1080p, with the UI drawn at 720p, but for instance video playback in full 1080p), and a free charging/OTG port. However, nowadays, we're stuck with MHL, and unless you have a Samsung phone, which has a proprietary connector, you can't use MHL and OTG at the same time, nor does MHL properly support touch functionality (only in theory), so we need the OTG.

DisplayLink to the rescue. We're going to sacrifice a lot of display smoothness/responsiveness here unless you have a modern phone with a USB 3.0 Type-C port (so you can use a newer high performance DisplayLink adapter), but we can use an OTG hub to get both HID touch functionality and HDMI through the DisplayLink adapter at the same time. More on this later. Also, I originally bought a $6 USB to HDMI adapter off eBay thinking it would be DisplayLink (or a compatible off-brand clone), but it turns out to be the really dodgy Fresco Logic FL2000, which is so cheap because it does none of the clever things that DisplayLink adapters do, and instead just spits out full resolution frames as fast as it can, which is completely incompatible with USB 2.0 or low power devices like phones. I found an affordable HP DisplayLink DVI adapter second hand from a Chinese seller that works.

On to the next hurdle: charging while using OTG. This is an interesting one, as it's not something I really gave a lot of thought initially. I mean surely just using a powered hub and giving the phone 5V over its micro-USB port would work, right? Well, it's a bit more complex than that, but Sony used to have an OTG dock that could charge, so I'm confident once I get the right OTG hub, it will work fine on my Z5. The one I got off eBay wouldn't do anything but charge, and when I opened it up, I saw it doesn't even use the 5th OTG pin, which would explain why it didn't work. I soldered a regular Type-A plug onto it and used my Sony OTG adapter to test, and I can get either charging or OTG, so something a bit less hacky is required. I ordered the Acasis H027, so when it arrives, we'll see if that works.

Touch: I haven't tested it yet. I have an HDMI touch screen in storage at my brother's house, so next time I visit I'll see if I can get touches to register on the phone. Based on my Googling most people on the internet seem to have gotten this to work fine, although there is no touch calibration on Android as far as I'm aware, like there is on Windows.

Portrait mode: this is my personal pet project and what I've been struggling with the most so far. It would be by far the easiest to just use a touch screen in landscape mode, but hear me out. Because HDMI (touch) monitors are cheap anyway, and in the 7 inch to 24 inch size range, bigger usually means cheaper, I want Tesla-style portrait mode. In portrait mode we can fit more screen estate in the centre console of the car than in landscape mode, and a bigger screen = bigger text = quicker glancing = safer driving. And it also happens to look extremely cool. If you have a big car you can probably fit a 20 or 24 inch screen, but I think I'm going for 13 inch. However. Using the DisplayLink Presenter app or the DisplayLink Desktop demo app which as far as I can tell does exactly the same things, I cannot for the life of me get native portrait mode to work. Android insists on pillarboxing portrait mode, and no amount of forcing rotation, setting build.props like ro.sf.hwrotation or persist.demo.hdmirotation works to change its mind. More worryingly, it seems that for the HDMI rotation prop to work, you need to also set persist.demo.singledisplay, which prevents apps from accessing secondary displays, which means DisplayLink Presenter, which itself is an app, cannot mirror the screen output any more and you get a sad single stripe of garbage pixels on an otherwise completely black screen. I probably need help here from an XDA developer, to hack the DisplayLink app, or to develop an Xposed module that causes all apps to draw themselves in portrait mode while the system is actually in landscape mode, or something along those lines. I've tried most things in my power that I could think of, and since my phone is unlocked, rooted, magisk'd and xposed, that is actually quite a lot, but nothing helped. So, uh, help?

Lastly there is DPI, which is one of the easier hurdles that I did solve. Android Auto (in phone mode, so not connected to any head unit) has a ridiculously huge UI on my phone's native dpi, and while I understand the reasons for it, with a larger external touch screen attached it just becomes unreasonably huge. With Tasker set to run "wm density 240" the entire Android UI becomes a lot more suited to a large screen, and even though Android Auto is still pretty huge compared to other apps, it's what I would consider reasonable.

More to come!
 
  • Like
Reactions: V8 Cowboy

mufunyo

Senior Member
Jul 28, 2012
70
31
So, quick update:

While the screen portrait mode issues were "simple" to fix (although root was definitely required), the touch orientation issues as well as charging-during-OTG require kernel modifications to be fixed. I managed to compile a modified kernel with charging-during-OTG support thanks to @nlra 's work on that front, but I couldn't get the new image to boot.

A few things happened in the mean time:
- I discovered scrcpy
- I got an Xperia XZ3 (which I haven't rooted yet)

Scrcpy seems to be basically what Android Auto does, but for the whole Android desktop instead of only one app. I kind of don't like it because it involves adding a computer in between the display and the phone again (probably a Raspberry Pi), but the advantages are so huge it's basically the only realistic option right now.

Scrcpy:
- Basically always runs at 60fps, even on USB 2.0
- Handles portrait/landscape gracefully
- Integrates display, touch, (audio in a future version), and charging in a single connection
- Doesn't require root (although automatically setting the Android resolution to 1920x1080 and keeping the display on at 0% brightness are things that can probably only be accomplished with Tasker, which requires root)

So basically this simplifies and moves the project further immensely, however there are still some blocking issues right now. Touch screens only work on Windows because in addition to generating touch events, Windows also generates fallback mouse click events for touches, something that Linux doesn't do, and because there is no formal touch screen support in scrcpy, multi-touch doesn't work at all. Audio support also seems to be in an experimental state currently, and is not enabled for regular builds.

I hope ROM1V will eventually implement touch screen support (it's been in his GitHub issue tracker since March) as I have enough work to do as it is. I will focus on the hardware part (Raspi, cabling, VESA mount etc.) first and if by that time touch support is still missing I'll take a crack at it myself. Thankfully scrcpy is built on SDL which I'm fairly familiar with, although I've never worked with the touch input API before.
 

dzid_

Member
Jan 29, 2011
27
1
For the charging you need hub with usb-c PowerDelivery passthrough and HDMI. I am testing ones with less power consumption right now. I am not sure why you weren't able to use it.
I think people would be better of using following app for changing resolution, etc. using SecondScreen (I think root is required to run HDMI in portait mode, because it is grayed out for me).

I am doing the same project, but I keep the screen 1600x1200 horizontally and use apps in split view mode. I don't want root. I was thinking of using SamsungDEX for it, but the menus are too small.
The good thing is though, that on Samsung it is possible to create two-apps split screen pair (e.g. google map + music) as a launcher shortcut (using Good Lock (MultiStar plugin) from Galaxy Store). Unfortunately, I don't think it is possible to automate launching two apps in split screen automatically, nor create a split-pair shortcut on other launchers.
It would be good to have some multiwindow manager since I also wanted my BMW-tuning/logging gauges app to run in a floating window on top or minimized to a floating icon. It is possible naturally but it is a lot of manual clicking :/

One more thing that I don't think will be possible, is to completely turn off the phone screen. With the screen on, Note8 doesn't do fast charging. Can the screen be off with scrcpy somehow, I don't think?
 

Top Liked Posts

  • There are no posts matching your filters.
  • 1
    Seeing videos of people integrating Raspberry Pis into their cars with OpenAuto got me interested in Android Auto. There is one thing I don't like about it, however: you're just adding another device to pass on audio/video/touch to an external display, from the phone. Why not cut out the middle man?

    Back in the good old days of my Xperia S, phones had a dedicated micro-HDMI output. Delicous 60fps 720p (and even 1080p, with the UI drawn at 720p, but for instance video playback in full 1080p), and a free charging/OTG port. However, nowadays, we're stuck with MHL, and unless you have a Samsung phone, which has a proprietary connector, you can't use MHL and OTG at the same time, nor does MHL properly support touch functionality (only in theory), so we need the OTG.

    DisplayLink to the rescue. We're going to sacrifice a lot of display smoothness/responsiveness here unless you have a modern phone with a USB 3.0 Type-C port (so you can use a newer high performance DisplayLink adapter), but we can use an OTG hub to get both HID touch functionality and HDMI through the DisplayLink adapter at the same time. More on this later. Also, I originally bought a $6 USB to HDMI adapter off eBay thinking it would be DisplayLink (or a compatible off-brand clone), but it turns out to be the really dodgy Fresco Logic FL2000, which is so cheap because it does none of the clever things that DisplayLink adapters do, and instead just spits out full resolution frames as fast as it can, which is completely incompatible with USB 2.0 or low power devices like phones. I found an affordable HP DisplayLink DVI adapter second hand from a Chinese seller that works.

    On to the next hurdle: charging while using OTG. This is an interesting one, as it's not something I really gave a lot of thought initially. I mean surely just using a powered hub and giving the phone 5V over its micro-USB port would work, right? Well, it's a bit more complex than that, but Sony used to have an OTG dock that could charge, so I'm confident once I get the right OTG hub, it will work fine on my Z5. The one I got off eBay wouldn't do anything but charge, and when I opened it up, I saw it doesn't even use the 5th OTG pin, which would explain why it didn't work. I soldered a regular Type-A plug onto it and used my Sony OTG adapter to test, and I can get either charging or OTG, so something a bit less hacky is required. I ordered the Acasis H027, so when it arrives, we'll see if that works.

    Touch: I haven't tested it yet. I have an HDMI touch screen in storage at my brother's house, so next time I visit I'll see if I can get touches to register on the phone. Based on my Googling most people on the internet seem to have gotten this to work fine, although there is no touch calibration on Android as far as I'm aware, like there is on Windows.

    Portrait mode: this is my personal pet project and what I've been struggling with the most so far. It would be by far the easiest to just use a touch screen in landscape mode, but hear me out. Because HDMI (touch) monitors are cheap anyway, and in the 7 inch to 24 inch size range, bigger usually means cheaper, I want Tesla-style portrait mode. In portrait mode we can fit more screen estate in the centre console of the car than in landscape mode, and a bigger screen = bigger text = quicker glancing = safer driving. And it also happens to look extremely cool. If you have a big car you can probably fit a 20 or 24 inch screen, but I think I'm going for 13 inch. However. Using the DisplayLink Presenter app or the DisplayLink Desktop demo app which as far as I can tell does exactly the same things, I cannot for the life of me get native portrait mode to work. Android insists on pillarboxing portrait mode, and no amount of forcing rotation, setting build.props like ro.sf.hwrotation or persist.demo.hdmirotation works to change its mind. More worryingly, it seems that for the HDMI rotation prop to work, you need to also set persist.demo.singledisplay, which prevents apps from accessing secondary displays, which means DisplayLink Presenter, which itself is an app, cannot mirror the screen output any more and you get a sad single stripe of garbage pixels on an otherwise completely black screen. I probably need help here from an XDA developer, to hack the DisplayLink app, or to develop an Xposed module that causes all apps to draw themselves in portrait mode while the system is actually in landscape mode, or something along those lines. I've tried most things in my power that I could think of, and since my phone is unlocked, rooted, magisk'd and xposed, that is actually quite a lot, but nothing helped. So, uh, help?

    Lastly there is DPI, which is one of the easier hurdles that I did solve. Android Auto (in phone mode, so not connected to any head unit) has a ridiculously huge UI on my phone's native dpi, and while I understand the reasons for it, with a larger external touch screen attached it just becomes unreasonably huge. With Tasker set to run "wm density 240" the entire Android UI becomes a lot more suited to a large screen, and even though Android Auto is still pretty huge compared to other apps, it's what I would consider reasonable.

    More to come!