KODI with - amazon voice search :D (proof of concept)

Search This thread

is0-mick

Member
Jul 12, 2005
19
27
Hi Guys,
I managed to root my FireTV a few days back, and yesterday decided to look at the voice search to see if I could use it for other things.

After reversing some code, I found the actual voice search is handled by com.amazon.vizzini.apk

The SearchOverlay.class has this piece of code, which calls back to the amazon fireTV UI SearchResultsActivity upon completion with the search result returned as a string.

I replaced the fire TV UI with my own code, which receives the search text, and then sends a JSON rpc to KODI's web interface, and brings KODI to the front after the search is completed.

Here is a video of it in action.
http://youtu.be/hpgKci_gJYY

android studio project
http://uptobox.com/ccykod7zua1l
mirror
http://www107.zippyshare.com/v/mAhl3UuM/file.html


***** I have a FireTV v1 updated to fire os 5 ******
I have no idea if this will work on older versions of software.

To make it work you will require ROOT, and you will have to uninstall or move the existing fireTV amazon UI.
Reason for this is that the vizinni.apk calls back to
localIntent.setComponent(new ComponentName("com.amazon.tv.launcher", "com.amazon.tv.launcher.ui.SearchResultsActivity"));

So your activity has to be in that package, and called SearchResultsActivity.
The only other way to possibly get round this would be to modify the vizinni.apk to call a different package instead, but then voice search wouldn't work on the amazon UI anyway. I wasn't really bothered about the amazon UI working myself which is why I did it the way I did.

I just moved the original system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk to /system then installed my code via android studio.

mount -o rw,remount /system
mv /system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk /system/

You will also need to change KODI's settings to enable web interface control on port 8080.
On my setup it was .. system.. services.. webserver.. and tick the box that says "Allow control of kodi via http" and make sure the port is set to 8080.

edit: 07/07/2016
I did start work a few months back on an Xposed module (works but not 100%), I was going to add a settings page but I think there was issues with xposed on android version that the fire tv runs on, due to permissions etc.

You can set a prefix in the code (currently hard coded to KODI) so if you say " kodi star wars" it would pass the param of "star wars" to kodi.
If you just say "star wars" without the prefix it would pass this to the normal amazon UI.

If anyone wants to take it further, it's attached on the link below.

http://uptobox.com/ensqll4a7r65




Mick
 
Last edited:

harlekinrains

Senior Member
May 12, 2011
113
47
Thats actually pretty great.

Conventional wisdom was that both voice recognition and response "results packaging" was done server side - and the only return query you could get were preexisting amazon database references - not the result of the initial voice recognition.

I've actually read that multiple places - but as it turns out, no one had bothered reversing the process I guess. :)

Major props and thank god for overlays containing strings... ;) Finally - that microphone might not be so useless after all.. ;)

Next step - implement it to interface with google search. ;) (Weather in ...)
edit: Actually - there are three interesting usecases I can think of on top of my head...

1. Launch other Apps (maybe even with "fixed Keywords (App 1 f.e. would work edit: "Number 1" delivers better results. :) )" if App names arent known to Amazon) - so thats text>launcher app - see if sphinx02 has any interest in coding that as part of Firestarter.. ;)
2. google search (Whats the weather in.. ) text>google search app (if possible)
3. direct text input as seen in the POC video
edit 4. forward to Amazon - not to "break" their implementation

This could be realized by a quick 4 way select screen (just confirm with the direction button) after you select the string in the overlay.
 
Last edited:
  • Like
Reactions: is0-mick

harlekinrains

Senior Member
May 12, 2011
113
47
I was playing with voice search strings and found that Amazon tends to strip out "Google" in front of search queries -so instead of "Google how is the Weather tomorrow" only "How is the weather tomorrow" will get returned.

But - Alexa as a trigger word will be returned fine, so - I vote for using "Alexa" as a trigger word to forward all search queries to the google search app. ;)

Ok - maybe not - but conceptually, this would be a great "work of art". ;)
 
Last edited:

Taconut

Member
Dec 20, 2014
19
0
An even better idea than Harklekinrains' would be to check the foreground app and do different things based on that. For example, if Kodi is open it could send the intent to Kodi. If the Fire TV launcher is open it could fall back to default functionality, etc. Simulating keystrokes could also cover 99% of the other applications. I'm excited for this. It really makes the Fire TV so much less of a novelty.
 

dafunkydan

Member
Feb 4, 2008
47
5
Great mate,thanks! :) gonna check it out soon!
One question (as im not firm with intents and stuff): aint there a possibility to listen for intents sent to the amazon ui and catch them? Personally, i dont care too much about it right now as im using nothing but kodi, but maybe some day when wanting to use prime or similar stuff?
 
  • Like
Reactions: is0-mick

is0-mick

Member
Jul 12, 2005
19
27
Great mate,thanks! :) gonna check it out soon!
One question (as im not firm with intents and stuff): aint there a possibility to listen for intents sent to the amazon ui and catch them? Personally, i dont care too much about it right now as im using nothing but kodi, but maybe some day when wanting to use prime or similar stuff?

I don't think so as the actual code in the vinizzi apk is as follows:
Intent localIntent = new Intent();
localIntent.setComponent(new ComponentName("com.amazon.tv.launcher", "com.amazon.tv.launcher.ui.SearchResultsActivity"));
localIntent.putExtra("identifier", null);
localIntent.addFlags(402653184);
localIntent.putExtra("term", str);
localIntent.putExtra("text", str);
localIntent.putExtra("source", "VOICE");
localContext.startActivity(localIntent);

So basically when it receives the voice search response from amazon's server, it's starting the activity com.amazon.tv.launcher.ui.SearchResultsActivity from package com.amazon.tv.launcher

Mick
 

harlekinrains

Senior Member
May 12, 2011
113
47
Integrating the search attempt based on the previous foreground app would mean that "voice search" could never "initate a new attempt". This would prevent the "lets just ask google, or lets launch an app - impulse use" of the feature. Still - in the long term it might turn out to be the right approach - f.e. if

"People tend to use the the google search only lets say "half a dozen times", and in the majority of cases just want text input in Kodi" Also - without having a select screen with "four (dont make it too many) predefined "use options"" discoverability is pretty non existent. People would have to read readmes to find out which interactions are supported.

Also Amazon wants you to "be able to always reach the Amazon content search from anywhere" - so, political implications.

Also - each time a new app would want to integrate the voice to text feature - they would have to contact the devs of this project - if you dont use "just text input" as a default in the "use the previously open app as an indicator of intent" approach. So make sure you default to "just text input" in that case. (Amazon launcher > forward query to Amazon (do not break functionality), ....)

Dont implement it as a mixed approach though as "Kodi is open most of the time" probably for most people - and the intent (message) gets "confusing".
--

What you probably shouldnt do - regardless, is to use "trigger words" as an "indicator of intent" - because Amazon can start blocking them. "Number 1" is probably generic enough so they wont try to block that - but in principal, they can. Also Amazons "voice to text" engine is optimized for "short phrases" - the longer your input query gets the more prone to errors their results become. Leave the actual "voice input" as "natural" as possible (dont embed logic there). Imho.
 
Last edited:
  • Like
Reactions: dafunkydan

kratosjohn

Member
Apr 2, 2009
23
10
I'm in the USA and uptobox.com is not available in our country.

I want to give the source a look and see what I can do to contribute.
Can you put it on github or somewhere else?

Much appreciated.
 

BagiMT

Senior Member
Nov 30, 2015
69
20
It would be nice if this could be generic way of entering text in any app, similar to the FTV Remote App that has keyboard support. I am surprised Amazon has not done this already. If you are in a text field voice search would fill in the text, else it can continue with normal Alexa functionality letting you " always reach the Amazon content search from anywhere". Should be an easy solution for Amazon to implement, not sure if it could be added with the progress here, and it would really make the voice control so much more useful.
 

dafunkydan

Member
Feb 4, 2008
47
5
Hey Mick, as im still very exited about that (so frustating to enter searchphrases with a dpad...), i thought about what your great find could develop to. I hope u dont get me wrong, as i unfortunatly cant contribute any programming (if u need something made with tasker let me know :D), its not that i want to demand anything - rather share my thoughts or ideas...
- I think its a good thought not to 'blow it up' by adding many keywords, make it more complicated, and avoid amzn to lock features down. even if it may be an abstract fear. I think with a well-structured menu in kodi, all apps one will need to start are just 2-3 clicks away.
- if i got it right, xposed could hook into every module and alter it, right? so basically it should be possible to avoid to exchange the amazon ui and/or alter the vizinni.apk,but just modify the SearchOverlay.class? wouldnt it be (on top) the most convenient way to decide wether one is on e.g. Amazon Prime, or on kodi, and either hand over the result to the original module, or hand it over to kodi?

So, please understand my post as a mixture between sugestions and questions, not as a demanding 'please make it how i want it asap'. Im really excited about your finding, appreciate your sharing, and want to contribute what i can do - unfortunatly its rather thoughts and suggestions. Cheers mate!
 

watermouf

New member
Mar 1, 2016
1
1
Great Work !!! I totally understand this is for amazon Fire products only. My question is would we be able to port for generic Android TV box like nVidia Shield which also has mic capability and comes with root support.
 
  • Like
Reactions: is0-mick

fach1708

Senior Member
Oct 25, 2010
489
99
No you dont understand. You havent even read or understood the first posting - but you have bought another device and now want others to move in your direction on your behalf.

Seriously mate if XDA bothers you so much, you need to take a break from it for your own sanity!
 

YoColdRain

Senior Member
Jul 1, 2010
149
29
Seriously mate if XDA bothers you so much, you need to take a break from it for your own sanity!

No, I really don't think he needs to take break, his argument is completely understandable. This is not a Shield forum (btw before we make judgements, I own both devices). We need to get this fixed for one device, before even mentioning whether other devices are an option.
 

I_did_it_just_tmrrow

Senior Member
Feb 27, 2011
344
61
To make it work you will require ROOT, and you will have to uninstall or move the existing fireTV amazon UI.
Reason for this is that the vizinni.apk calls back to
localIntent.setComponent(new ComponentName("com.amazon.tv.launcher", "com.amazon.tv.launcher.ui.SearchResultsActivity"));

So your activity has to be in that package, and called SearchResultsActivity.
The only other way to possibly get round this would be to modify the vizinni.apk to call a different package instead, but then voice search wouldn't work on the amazon UI anyway. I wasn't really bothered about the amazon UI working myself which is why I did it the way I did.
Is it not usefull for this to create a xposed mod? I thought xposed was made for this...
Perhaps you should talk with rbox in his thread about a integration into version 1.5 of his mods, look here.
I just moved the original system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk to /system then installed my code via android studio.

mount -o rw,remount /system
mv /system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk /system/
So does that mean, while you are using your proof-of-concept you can not use the normal Launcher-UI?

Really great your work! I hope we got more :D
Sadly I am with my FireTV-Stick (hardware-Rooted-superSU) and my FireTV box gen1 (hardware-Rooted-superSU+custom-recovery+unlocked-bl)on the old FW. I wait for custom roms.

Greetings by Idijt

EDIT:
I forgot to ask you something. Did you got the kind of search-request back?
Can you see if amazon-voice-rgn knows if it is a app or a video or a moviestar?
 

carpenter940

Member
Feb 28, 2009
30
3
This looks pretty cool since I don't like typing the name of the movie in the search box... now I can just speak it.
As a novice I have one question, You wrote:
I just moved the original system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk to /system
Mick
The code I downloaded was in .RAR format not .apk. Do I need to covert it to .apk or rename it to .apk before replacing the original?

I've not rooted my device yet so I haven't looked at the file structure, but I'd be willing to root to get this feature. OR is the adding of this code better done by someone other than a novice?

Thanks
 

Top Liked Posts

  • There are no posts matching your filters.
  • 11
    Hi Guys,
    I managed to root my FireTV a few days back, and yesterday decided to look at the voice search to see if I could use it for other things.

    After reversing some code, I found the actual voice search is handled by com.amazon.vizzini.apk

    The SearchOverlay.class has this piece of code, which calls back to the amazon fireTV UI SearchResultsActivity upon completion with the search result returned as a string.

    I replaced the fire TV UI with my own code, which receives the search text, and then sends a JSON rpc to KODI's web interface, and brings KODI to the front after the search is completed.

    Here is a video of it in action.
    http://youtu.be/hpgKci_gJYY

    android studio project
    http://uptobox.com/ccykod7zua1l
    mirror
    http://www107.zippyshare.com/v/mAhl3UuM/file.html


    ***** I have a FireTV v1 updated to fire os 5 ******
    I have no idea if this will work on older versions of software.

    To make it work you will require ROOT, and you will have to uninstall or move the existing fireTV amazon UI.
    Reason for this is that the vizinni.apk calls back to
    localIntent.setComponent(new ComponentName("com.amazon.tv.launcher", "com.amazon.tv.launcher.ui.SearchResultsActivity"));

    So your activity has to be in that package, and called SearchResultsActivity.
    The only other way to possibly get round this would be to modify the vizinni.apk to call a different package instead, but then voice search wouldn't work on the amazon UI anyway. I wasn't really bothered about the amazon UI working myself which is why I did it the way I did.

    I just moved the original system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk to /system then installed my code via android studio.

    mount -o rw,remount /system
    mv /system/priv-app/com.amazon.tv.launcher/com.amazon.tv.launcher.apk /system/

    You will also need to change KODI's settings to enable web interface control on port 8080.
    On my setup it was .. system.. services.. webserver.. and tick the box that says "Allow control of kodi via http" and make sure the port is set to 8080.

    edit: 07/07/2016
    I did start work a few months back on an Xposed module (works but not 100%), I was going to add a settings page but I think there was issues with xposed on android version that the fire tv runs on, due to permissions etc.

    You can set a prefix in the code (currently hard coded to KODI) so if you say " kodi star wars" it would pass the param of "star wars" to kodi.
    If you just say "star wars" without the prefix it would pass this to the normal amazon UI.

    If anyone wants to take it further, it's attached on the link below.

    http://uptobox.com/ensqll4a7r65




    Mick
    3
    I managed to get the xposed framework on, although I did think I'd bricked the thing at one point. ( I do have one of the adapters maximus64 made, so I could still recover by hardware if needed).

    I actually extracted the .zip file, and looked at the script that was supposed to be installed via custom recovery, and did it all manually lol.
    At first I had a black screen, but figured out it was because I didn't chcon ''u: object_r:zygote_exec:s0' app_process32 and a couple of other files.

    I think I've had enough for today, but at least I now have xposed on there to mess with :)

    Mick
    2
    So, just been reading about the xposed framework, and had a quick look at rbox's AlternateLauncher code.

    Looks like if I had the xposed framework installed I could use findAndHookMethod, but from what I've read the framework needs to be installed from a custom recovery? If someone could clarify thanks. (I have a FireTV 1 with FireOS 5)

    Mick
    2
    I've been trying a slightly different approach.

    My idea was to modify vizzini.apk to start an activity in my own program, and leave the original Amazon UI untouched.
    I was then thinking of trying to find the current foreground application and then either send key events or paste data to it from the clipboard.

    I seem to have hit a snag at the first hurdle though :(
    I modified vizzini.apk, but now it can't access the controller to perform a read due to what I think is the permissions requiring signature level protection.
    I installed my modified version into /system/priv-app but after re-signing it, it now has a different signature to the rest of the stuff.

    Logcat output:

    <snip>
    03-04 10:14:46.070: W/PackageManager(900): Not granting permission com.amazon.device.permission.READ_CONTROLLERS to package com.amazon.vizzini (protectionLevel=130 flags=0x4008be45)
    03-04 10:14:46.070: W/PackageManager(900): Not granting permission com.amazon.identity.permission.GENERIC_IPC to package com.amazon.vizzini (protectionLevel=130 flags=0x4008be45)
    <snip>

    So not exactly sure if there is a way around this? Anyone any thoughts?
    Mick
    1
    Thats actually pretty great.

    Conventional wisdom was that both voice recognition and response "results packaging" was done server side - and the only return query you could get were preexisting amazon database references - not the result of the initial voice recognition.

    I've actually read that multiple places - but as it turns out, no one had bothered reversing the process I guess. :)

    Major props and thank god for overlays containing strings... ;) Finally - that microphone might not be so useless after all.. ;)

    Next step - implement it to interface with google search. ;) (Weather in ...)
    edit: Actually - there are three interesting usecases I can think of on top of my head...

    1. Launch other Apps (maybe even with "fixed Keywords (App 1 f.e. would work edit: "Number 1" delivers better results. :) )" if App names arent known to Amazon) - so thats text>launcher app - see if sphinx02 has any interest in coding that as part of Firestarter.. ;)
    2. google search (Whats the weather in.. ) text>google search app (if possible)
    3. direct text input as seen in the POC video
    edit 4. forward to Amazon - not to "break" their implementation

    This could be realized by a quick 4 way select screen (just confirm with the direction button) after you select the string in the overlay.
Our Apps
Get our official app!
The best way to access XDA on your phone
Nav Gestures
Add swipe gestures to any Android
One Handed Mode
Eases uses one hand with your phone