Camera-based Motion Tracking

[gregnapola]

Hello everyone!

I'm new on this forum and I've only read TONS of useful information till now...Few days ago I bought an HTC Touch device and immediately I flashed its ROM with a new one found in this *** WONDERFUL *** site :p

Well, in the past I developed some (stupid) applications for Windows in several languages; then I discover the "pocket-pc" world and began to have to do with it! One night I was reading about a feature of a developement-version of a well known browser and WOOOOOOOW!!!! "You can perform several actions, like scrolling, panning, zooming...only by moving your phone!" The software use image-processing algorithms to recognize your gestures ...only using the built-in camera!!!! No accelometers needed...

Something like this: http://www.youtube.com/watch?v=I3pVaEVHpvI

Now, image-processing algorithms are not too difficult to implement...but how can I get access to the built-in camera on my Htc ?!? I tried to search all over the internet, and in this forum too...but I found only few things about HTCCameraAPI and many other programs and their dlls. I cannot believe there's no way to have access to the camera!!!
:confused:
HELP please!!! this "project" could be very interesting!

[mr_deimos]

One thing is sure - it _can_ be done, and it can work really good, even on weak hardware - look at this:
http://rivia.net/Fishopolis.aspx
I just tried the demo and i was amazed how well it worked on my wizard (especially considering its crappy camera), even in less than perfect lighting conditions (incadescent light). No overclocking required.
Also, from what i've seen WM7 is going to use the camera-based gestures too. So, anyone willing to beat M$ to it and develop a camera-based ftouchflo-like app? :D

[thundershadow14]

I've looked at the libs, and I've even dabbled in the source, and this is my next project. Like I said, we know it can be done, it's just a matter of doing it. Obviously though, I've got to finish off PocketMelody etc. But it's my next move, once no one beats me to it ;). And personally, I hope someone does! It's not THAT hard.

[frenchglen]

Welcome to the forums gregnapola!

This website hepled me choose my very first PPC Phone 18 months ago and helped it become the best little device I own. It just keeps getting better and better with tools such as FTouchFlo, PocketMelody and Picture Flow, and I look forward to this feature onMY HTC Hermes!

Shows how much really can be done with this WM platform, all you need is developers with imagination and talent (and all the other things that go along with it like coffe and donations and and respect on the forums :)).

Thanks guys.

[Desiflare999]

maybe we would have some1 develop a program that automatically changes portrait mode into landscape by jus rotating the camera..hey u never kno..this is XDA-DEVELOPERS..:D

[hebrewhammer]

Sounds awesome and would definitly love to see this done. Just a quick question about it. Would it be a big battery drain to have this running all the time?

[thundershadow14]

Sounds awesome and would definitly love to see this done. Just a quick question about it. Would it be a big battery drain to have this running all the time?

Considering that you're not rendering anything to the screen, it's at least safe to assume you'll be using A LOT less power than if you were walking around with your camera application on all day :).

[gregnapola]

Thank you for your replies...

I've just installed the demo version of "Rivia Fishopolis" to see how well can it work but it doesn't run it because of an unknow security-error :(

Now, I found a lot of useful camera-based applications, but they'll come in the future - I'm talking about WM7 gesture-based control, NetFront 3.5 or EyeMobile Games by GestureTek...but all I really need is the way to capture frames by my Elf camera; code samples in any Visual Studio language with HTCCameraAPI, or any other lib... :confused:

[christiankirana]

any update?

[teixeirabmb]

I'd also love to substitute my D-Pad for a Camera-based Motion Tracking system ;)

No Dev takes this challenge?

[cf222111]

Thank you for your replies...

I've just installed the demo version of "Rivia Fishopolis" to see how well can it work but it doesn't run it because of an unknow security-error :(

Now, I found a lot of useful camera-based applications, but they'll come in the future - I'm talking about WM7 gesture-based control, NetFront 3.5 or EyeMobile Games by GestureTek...but all I really need is the way to capture frames by my Elf camera; code samples in any Visual Studio language with HTCCameraAPI, or any other lib... :confused:

I resolved your error assigning owner information (all). But camera feature seems to not work on my trinity...also the d-pad can't move the cursor...is there anyone with same problem?
thanks

[shandar]

Would it be possible to have it access the front camera too and switch of the LCD when you're talking on the phone? Something like what the iPhone does to save battery.

[gregnapola]

any update?

The only update is that my elf was stolen about a week ago...and so...I cannot do anything at all!!! :(:(:( So sad....

[noellenchris]

The only update is that my elf was stolen about a week ago...and so...I cannot do anything at all!!! :(:(:( So sad....

That's really stinks!! I hope you can get another one. Sorry to hear the news.

Chris

[docnas]

Im trying to use Fishopolis on my phone ( an HTC Wizard) but i keep on getting some kind of weird security error.

Any ideas why im getting that error and how to fix it?

[noellenchris]

doesn't seem to work with the camera on the Hermes...hmmm

[vetvito]

Looks like we will have to wait until 2009 for WM7.

[mariner_heart]

wouldn't it consume a lot more battery if camera is always on.
just my thought

[JudgeDredd67]

I think this is a cool idea and I think I mentioned this somewhere else in this forum, but if this could be done, it would be cool if one could use the camera of there phone, link it via bluetooth and use it as a mouse for a laptop.

Just thinking outloud. :)

[mugglesquop]

lol, some interesting ideas.

Docnas, refer to post by cf222111:

I resolved your error assigning owner information (all).

Mariner_heart, refer to post by thundershadow_14:

Considering that you're not rendering anything to the screen, it's at least safe to assume you'll be using A LOT less power than if you were walking around with your camera application on all day .

[docnas]

I resolved your error assigning owner information (all). But camera feature seems to not work on my trinity...also the d-pad can't move the cursor...is there anyone with same problem?
thanks

i saw the post but couldnt figure out what he meant by assigning the owner information (all) . But using my vast intellect( hinting at my own stupidity for not realizing it earlier) i think he meant that i have to fill in all the fields of the owner information, will try it now thanks

EDIT : YaY it works thanks!

RE-EDIT : HOLY CR@P i just looked at my post and saw im a "Senior Member". How the hell did that happen??

[PLESTIUC]

I think this is a cool idea and I think I mentioned this somewhere else in this forum, but if this could be done, it would be cool if one could use the camera of there phone, link it via bluetooth and use it as a mouse for a laptop.

Just thinking outloud. :)

sure, why not use it for a hammer instead ? :D

[JudgeDredd67]

sure, why not use it for a hammer instead ? :D

My TyTN doesn't seem built heavy enough to be a hammer:rolleyes:

[iztoogood]

just see what is said here...
maybe will it be a native fuinction of wm7

.../...
it will use the camera on the phone to detect motions and create appropriate actions
.../...

http://www.osnews.com/story/19121/Windows_Mobile_7_To_Focus_on_Touch_Motion_Gestures

[GiZZy]

So has anyone found a way to create this for WM6/6.1 ?
Will be nice to have configurable gestures like button shortcuts but i'm just throwing ideas if its possible.

[hongleon]

i will have a look it when i get back home.
the algorithm for motion detection is not hard, but there's a bit of codes need to be done for the motion gesture detection.
What i've done previously with a pc webcam was just a overlay image mark motion to detect thing like...ghosts (but havn't seen one yet LOL), using Delphi
however, i didn't see an exact good usage of motion detection at this point, WM7 certainly make some use of it, but all unknown till it's release

[hongleon]

had a look at it
firstly, picture capture is fine, but it only allow one picture to be captured, something's missing (i am caputre directly though the device using a threaded timer)

secondly, my photo comparing algorithm is a bit weak, take about 2-3 sec to calculate two photos...still trying to grab a better one

[PLESTIUC]

Hello everyone!

I'm new on this forum and I've only read TONS of useful information till now...Few days ago I bought an HTC Touch device and immediately I flashed its ROM with a new one found in this *** WONDERFUL *** site :p

Well, in the past I developed some (stupid) applications for Windows in several languages; then I discover the "pocket-pc" world and began to have to do with it! One night I was reading about a feature of a developement-version of a well known browser and WOOOOOOOW!!!! "You can perform several actions, like scrolling, panning, zooming...only by moving your phone!" The software use image-processing algorithms to recognize your gestures ...only using the built-in camera!!!! No accelometers needed...

Something like this: http://www.youtube.com/watch?v=I3pVaEVHpvI

Now, image-processing algorithms are not too difficult to implement...but how can I get access to the built-in camera on my Htc ?!? I tried to search all over the internet, and in this forum too...but I found only few things about HTCCameraAPI and many other programs and their dlls. I cannot believe there's no way to have access to the camera!!!
:confused:
HELP please!!! this "project" could be very interesting!

what do you think about and application to rotate the screen automaticaly, like the iPhone, according to the way you hold the phone in your hand (landscape or portrait) ?
possible ?

[avarilone]

what do you think about and application to rotate the screen automaticaly, like the iPhone, according to the way you hold the phone in your hand (landscape or portrait) ?
possible ?

I believe the iPhone uses an accelerometer to automatically shift from portrait to landscape mode. So, if the force of gravity is detected on the vertical axis, it is in portrait, and if gravity is detected on the horizontal axis, it shifts to landscape, etc. I'm sure many of you on these boards know how this works, however I'm sure a few don't, which is the reason for the brief explanation.

I'm not a software engineer, however I personally think that without the use of an accelerometer, the automatic portrait/landscape feature could not be developed in a way that would be very reliable and usable. Then again, I think that the use of phone gestures as demonstrated by the youtube video (http://www.youtube.com/watch?v=I3pVaEVHpvI) posted earlier in this thread are just ridiculous overkill. Why would you want your phone to scroll by tipping it forward and backwards? Sure its an interesting software project, but it is actually useful or practical? I personally would rather just use a finger gesture on the screen, or the good old scroll bar.

Back on topic, I think that the concept of using image comparing software which receives data from the camera in order to switch from portrait to landscape is just not practical. This is a process that I can already do by pushing a button or simply flipping my keyboard open (HTC 6800). However, I strongly encourage someone to create it and prove me wrong. :)

[Strikah9999]

I like the idea, it even seems Microsoft is liking the idea, I have read somewhere about WM7/8, they will also have motion detection. But instead of an accelerometer, they will use the camera like here. It could be just a rumor though. Nevertheless, about the portrait to landscape, I guess it would be able to do that if the cam is able to detect scrolling etc.

[mr_deimos]

I really don't think that MS will use just the camera to detect motion - accelerometers are really cheap and very small nowdays, so there isn't any good reason against putting one in a phone. The downside of accelerometers is that they measure only linear acceleration - can detect weather you're shaking the phone, can estimate phone's position (even if the phone isn't moving, there's still gravitational acceleration always affecting it, that has a known orientation. The direction in which the average measured acceleration is biggest time is down), can detect if the phone is falling down (if the measured acceleration is zero), but the orientation it's still just an estimate - everything works fine if the phone is stationary. But when it's shaking or moving around it's simply impossible to precisely measure its tilt - for example to use it for scrolling web pages. Gyros are used for that (actually they measure angular acceleration not tilt, but tilt can be calculated from it quite precisely), and they're a bit more expensive than accelerometers, so i guess that this is where camera image recognition comes in - when you tilt your phone, the image seen trough the camera shifts a lot, when you just move it sideways, it moves only slightly, so it doesn't have much influence on the tilt measurement.

And i really don't think that switching landscape/portrait is possible with just a camera - when you look at a picture you can usually easily tel which way is down and which way is up on the picture, but let's not forget that our phones lack the intelligence needed to recognize this. Sure, you could measure how much the phone rotated without too much problems, but how could you tell in what position the rotation started - was the phone held vertically? And maybe it was held diagonally? The only situation i could think of where camera-based phone angle measurement would work is pointing the phone at the horizon during the day. But then again - just pressing a hardware button to rotate the screen is a bit easier than activating a recognition program, walking up to the window, pointing it at the horizon and rotating :P
I've made a few simple image recognition programs and a task that is extremely simple for a human - like distinguishing a number of separate points (like a few light sources) on an image, or recognizing object's shape based on an image is a real pain to program.
The whole image recognition stuff may seem quite simple - but only until you actually try doing it.

[jessegreene]

I like the idea, it even seems Microsoft is liking the idea, I have read somewhere about WM7/8, they will also have motion detection. But instead of an accelerometer, they will use the camera like here. It could be just a rumor though. Nevertheless, about the portrait to landscape, I guess it would be able to do that if the cam is able to detect scrolling etc.

Just a thougt:

Wouldn't having the CCD chip for the camera turned on all the time drain power?
Would this type of technology work in the dark?
Not sure if this will work!?!

[ErGo_404]

Microsoft told about the use of the camera as motion sensors, but it was only in case of, I'm pretty sure wm7 will have drivers for accelerometers, but if wm7 has a function which needs it, and if the phone doesn't actually include an accelerometer, wm7 will use the camera instead.

[Strikah9999]

Heya again,

I dunno how MS will be solving the darkness problem, but I found it on this site:
http://microsoft.blognewschannel.com/archives/2008/01/06/exclusive-windows-mobile-7-to-focus-on-touch-and-motion-gestures/

I believe it is the second paragraph

[gregnapola]

Why would you want your phone to scroll by tipping it forward and backwards? Sure its an interesting software project, but it is actually useful or practical? I personally would rather just use a finger gesture on the screen, or the good old scroll bar.

Back on topic, I think that the concept of using image comparing software which receives data from the camera in order to switch from portrait to landscape is just not practical. This is a process that I can already do by pushing a button or simply flipping my keyboard open (HTC 6800). However, I strongly encourage someone to create it and prove me wrong.


Hi folks,

I've just purchased another WM6 phone, but it's not an HTC Touch :( ...It's an Lg Ks20 and I'm still waiting for it - So I hope to begin as soon as possible to "study" the device and its hw... ;)

I read all your answers and ... my intention is not to create a must-have , otherwise apply a simple software to web-browsers, image-viewers and so on, not only to GAMES or a simple screen-rotation... but only for fun, always...(to be honest my aim is to demonstrate that what i-phone, can also be do with any other phone :D )
And as I currently see that there's not a free-software wich can do this things, I think it wouldn't be a bad idea to develop my own one... :)

And ...yes...I think it's possible; you don't need to capture 640x480 pictures to perform a motion-detection; I think that for a 200 Mhz processor is not such a hard work...

[christiankirana]

any update? samsung had already applied this thing on their Omnia...

[mil_2007]

really cool thread!
thank you guys!

fishopolis doesn't work on my polaris:mad:

cheers

miL

[piper_indebt]

Fishopolis is working on touch elfin wm6.1 but the camera function is not doing anything, even the D pad doesnt work, Bummer. any updates guys? makes me think/compare the power of symbian devices, just thinking :)