KODI with - amazon voice search :D (proof of concept)

[is0-mick]

Hi Guys,
I managed to root my FireTV a few days back, and yesterday decided to look at the voice search to see if I could use it for other things.

After reversing some code, I found the actual voice search is handled by com.amazon.vizzini.apk

The SearchOverlay.class has this piece of code, which calls back to the amazon fireTV UI SearchResultsActivity upon completion with the search result returned as a string.

I replaced the fire TV UI with my own code, which receives the search text, and then sends a JSON rpc to KODI's web interface, and brings KODI to the front after the search is completed.

Here is a video of it in action.
http://youtu.be/hpgKci_gJYY

android studio project
http://uptobox.com/ccykod7zua1l
mirror
http://www107.zippyshare.com/v/mAhl3UuM/file.html


***** I have a FireTV v1 updated to fire os 5 ******
I have no idea if this will work on older versions of software.

To make it work you will require ROOT, and you will have to uninstall or move the existing fireTV amazon UI.
Reason for this is that the vizinni.apk calls back to
localIntent.setComponent(new ComponentName("com.amazon.tv.launcher", "com.amazon.tv.launcher.ui.SearchResultsActivity"));

So your activity has to be in that package, and called SearchResultsActivity.
The only other way to possibly get round this would be to modify the vizinni.apk to call a different package instead, but then voice search wouldn't work on the amazon UI anyway. I wasn't really bothered about the amazon UI working myself which is why I did it the way I did.

I just moved the original system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk to /system then installed my code via android studio.

mount -o rw,remount /system
mv /system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk /system/

You will also need to change KODI's settings to enable web interface control on port 8080.
On my setup it was .. system.. services.. webserver.. and tick the box that says "Allow control of kodi via http" and make sure the port is set to 8080.

edit: 07/07/2016
I did start work a few months back on an Xposed module (works but not 100%), I was going to add a settings page but I think there was issues with xposed on android version that the fire tv runs on, due to permissions etc.

You can set a prefix in the code (currently hard coded to KODI) so if you say " kodi star wars" it would pass the param of "star wars" to kodi.
If you just say "star wars" without the prefix it would pass this to the normal amazon UI.

If anyone wants to take it further, it's attached on the link below.

http://uptobox.com/ensqll4a7r65




Mick

 

[harlekinrains]

Thats actually pretty great.

Conventional wisdom was that both voice recognition and response "results packaging" was done server side - and the only return query you could get were preexisting amazon database references - not the result of the initial voice recognition.

I've actually read that multiple places - but as it turns out, no one had bothered reversing the process I guess.

Major props and thank god for overlays containing strings... Finally - that microphone might not be so useless after all..

Next step - implement it to interface with google search. (Weather in ...)
edit: Actually - there are three interesting usecases I can think of on top of my head...

1. Launch other Apps (maybe even with "fixed Keywords (App 1 f.e. would work edit: "Number 1" delivers better results. )" if App names arent known to Amazon) - so thats text>launcher app - see if sphinx02 has any interest in coding that as part of Firestarter..
2. google search (Whats the weather in.. ) text>google search app (if possible)
3. direct text input as seen in the POC video
edit 4. forward to Amazon - not to "break" their implementation

This could be realized by a quick 4 way select screen (just confirm with the direction button) after you select the string in the overlay.

 

[YoColdRain]

great find and implementation. would love see more expansion as above post mentions. Are you planning to share fire tv UI code?

 

[harlekinrains]

I was playing with voice search strings and found that Amazon tends to strip out "Google" in front of search queries -so instead of "Google how is the Weather tomorrow" only "How is the weather tomorrow" will get returned.

But - Alexa as a trigger word will be returned fine, so - I vote for using "Alexa" as a trigger word to forward all search queries to the google search app.

Ok - maybe not - but conceptually, this would be a great "work of art".

 

[Taconut]

An even better idea than Harklekinrains' would be to check the foreground app and do different things based on that. For example, if Kodi is open it could send the intent to Kodi. If the Fire TV launcher is open it could fall back to default functionality, etc. Simulating keystrokes could also cover 99% of the other applications. I'm excited for this. It really makes the Fire TV so much less of a novelty.

 

[is0-mick]

I've just updated the original post with a copy of the android studio project, and a few more details.

Mick

 

[dafunkydan]

Great mate,thanks! gonna check it out soon!
One question (as im not firm with intents and stuff): aint there a possibility to listen for intents sent to the amazon ui and catch them? Personally, i dont care too much about it right now as im using nothing but kodi, but maybe some day when wanting to use prime or similar stuff?

 

[is0-mick]

Great mate,thanks! gonna check it out soon!
One question (as im not firm with intents and stuff): aint there a possibility to listen for intents sent to the amazon ui and catch them? Personally, i dont care too much about it right now as im using nothing but kodi, but maybe some day when wanting to use prime or similar stuff?

I don't think so as the actual code in the vinizzi apk is as follows:
Intent localIntent = new Intent();
localIntent.setComponent(new ComponentName("com.amazon.tv.launcher", "com.amazon.tv.launcher.ui.SearchResultsActivity"));
localIntent.putExtra("identifier", null);
localIntent.addFlags(402653184);
localIntent.putExtra("term", str);
localIntent.putExtra("text", str);
localIntent.putExtra("source", "VOICE");
localContext.startActivity(localIntent);

So basically when it receives the voice search response from amazon's server, it's starting the activity com.amazon.tv.launcher.ui.SearchResultsActivity from package com.amazon.tv.launcher

Mick

 

[harlekinrains]

Integrating the search attempt based on the previous foreground app would mean that "voice search" could never "initate a new attempt". This would prevent the "lets just ask google, or lets launch an app - impulse use" of the feature. Still - in the long term it might turn out to be the right approach - f.e. if

"People tend to use the the google search only lets say "half a dozen times", and in the majority of cases just want text input in Kodi" Also - without having a select screen with "four (dont make it too many) predefined "use options"" discoverability is pretty non existent. People would have to read readmes to find out which interactions are supported.

Also Amazon wants you to "be able to always reach the Amazon content search from anywhere" - so, political implications.

Also - each time a new app would want to integrate the voice to text feature - they would have to contact the devs of this project - if you dont use "just text input" as a default in the "use the previously open app as an indicator of intent" approach. So make sure you default to "just text input" in that case. (Amazon launcher > forward query to Amazon (do not break functionality), ....)

Dont implement it as a mixed approach though as "Kodi is open most of the time" probably for most people - and the intent (message) gets "confusing".
--

What you probably shouldnt do - regardless, is to use "trigger words" as an "indicator of intent" - because Amazon can start blocking them. "Number 1" is probably generic enough so they wont try to block that - but in principal, they can. Also Amazons "voice to text" engine is optimized for "short phrases" - the longer your input query gets the more prone to errors their results become. Leave the actual "voice input" as "natural" as possible (dont embed logic there). Imho.

 

[kratosjohn]

I'm in the USA and uptobox.com is not available in our country.

I want to give the source a look and see what I can do to contribute.
Can you put it on github or somewhere else?

Much appreciated.

 

[is0-mick]

I'm in the USA and uptobox.com is not available in our country.

I want to give the source a look and see what I can do to contribute.
Can you put it on github or somewhere else?

Much appreciated.

I've just uploaded it to zippyshare too. Link added to original post.

Mick

 

[BagiMT]

It would be nice if this could be generic way of entering text in any app, similar to the FTV Remote App that has keyboard support. I am surprised Amazon has not done this already. If you are in a text field voice search would fill in the text, else it can continue with normal Alexa functionality letting you " always reach the Amazon content search from anywhere". Should be an easy solution for Amazon to implement, not sure if it could be added with the progress here, and it would really make the voice control so much more useful.

 

[dafunkydan]

Hey Mick, as im still very exited about that (so frustating to enter searchphrases with a dpad...), i thought about what your great find could develop to. I hope u dont get me wrong, as i unfortunatly cant contribute any programming (if u need something made with tasker let me know ), its not that i want to demand anything - rather share my thoughts or ideas...
- I think its a good thought not to 'blow it up' by adding many keywords, make it more complicated, and avoid amzn to lock features down. even if it may be an abstract fear. I think with a well-structured menu in kodi, all apps one will need to start are just 2-3 clicks away.
- if i got it right, xposed could hook into every module and alter it, right? so basically it should be possible to avoid to exchange the amazon ui and/or alter the vizinni.apk,but just modify the SearchOverlay.class? wouldnt it be (on top) the most convenient way to decide wether one is on e.g. Amazon Prime, or on kodi, and either hand over the result to the original module, or hand it over to kodi?

So, please understand my post as a mixture between sugestions and questions, not as a demanding 'please make it how i want it asap'. Im really excited about your finding, appreciate your sharing, and want to contribute what i can do - unfortunatly its rather thoughts and suggestions. Cheers mate!

 

[watermouf]

Great Work !!! I totally understand this is for amazon Fire products only. My question is would we be able to port for generic Android TV box like nVidia Shield which also has mic capability and comes with root support.

 

[biglo]

Good job @is0-mick it's great to see you accomplish something that Netflix doesn't even want to be bother with on their app.

 

[fach1708]

No you dont understand. You havent even read or understood the first posting - but you have bought another device and now want others to move in your direction on your behalf.

Seriously mate if XDA bothers you so much, you need to take a break from it for your own sanity!

 

[YoColdRain]

Seriously mate if XDA bothers you so much, you need to take a break from it for your own sanity!

No, I really don't think he needs to take break, his argument is completely understandable. This is not a Shield forum (btw before we make judgements, I own both devices). We need to get this fixed for one device, before even mentioning whether other devices are an option.

 

[I_did_it_just_tmrrow]

To make it work you will require ROOT, and you will have to uninstall or move the existing fireTV amazon UI.
Reason for this is that the vizinni.apk calls back to
localIntent.setComponent(new ComponentName("com.amazon.tv.launcher", "com.amazon.tv.launcher.ui.SearchResultsActivity"));

So your activity has to be in that package, and called SearchResultsActivity.
The only other way to possibly get round this would be to modify the vizinni.apk to call a different package instead, but then voice search wouldn't work on the amazon UI anyway. I wasn't really bothered about the amazon UI working myself which is why I did it the way I did.

Is it not usefull for this to create a xposed mod? I thought xposed was made for this...
Perhaps you should talk with rbox in his thread about a integration into version 1.5 of his mods, look here.

I just moved the original system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk to /system then installed my code via android studio.

mount -o rw,remount /system
mv /system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk /system/

So does that mean, while you are using your proof-of-concept you can not use the normal Launcher-UI?

Really great your work! I hope we got more
Sadly I am with my FireTV-Stick (hardware-Rooted-superSU) and my FireTV box gen1 (hardware-Rooted-superSU+custom-recovery+unlocked-bl)on the old FW. I wait for custom roms.

Greetings by Idijt

EDIT:
I forgot to ask you something. Did you got the kind of search-request back?
Can you see if amazon-voice-rgn knows if it is a app or a video or a moviestar?

 

[carpenter940]

This looks pretty cool since I don't like typing the name of the movie in the search box... now I can just speak it.
As a novice I have one question, You wrote:

I just moved the original system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk to /system
Mick

The code I downloaded was in .RAR format not .apk. Do I need to covert it to .apk or rename it to .apk before replacing the original?

I've not rooted my device yet so I haven't looked at the file structure, but I'd be willing to root to get this feature. OR is the adding of this code better done by someone other than a novice?

Thanks

 

[Animux]

The code I downloaded was in .RAR format not .apk. Do I need to covert it to .apk or rename it to .apk before replacing the original?

It's the source code. You have to compile it and create the apk with AndroidStudio.