[WORKING] Software Multi-touch gamepad on Blackstone [DEVS HELP TO REFINE?]

Search This thread

Kizmet

Senior Member
Jan 18, 2008
216
0
For progress on the development of a gamepad for blackstone please go to the end of the thread.






was pondering the problem of playing non-touchscreen games on the hd and the lack of control options.
In a basic form what is needed is 8 way direction control and a number of buttons that can be pressed simultaneously:

There are three options that I think are worth investigating:

1) general g pad controls simulated by g sensor plus additional buttons created on a SIP or shell for the games.

2) a software shell that places a touchscreen dpad on one side of the landscape screen and have the light sensor and front camera on the other side as buttons. A keypress would be simulated when the ssensor is covered by a finger. Perhaps the back camera too? This would allow many games to be played, I think.

3) Calculated multitouch. When a resistive screen is multitouched I think this produces an intermediate result between the two touches.

If you created a shell that had precise positions of all the buttons worked out multitouches could be mathematically calculated and then sent as simulated keypresses... n effect creating a virtual and invisible keypad in the middle that only becomes active when more than one button is pressed.

For example pressing up and the (a) button creates a virtual touch in the red circles etc.

If actual touches are intercepted and only virtual touches sent as keypresses this could also work.

attachment.php


My question is for the technical people out there who know much more about this device than me... is there any obvious reason you can see that one or all of these methods would not work?
 

Attachments

  • multi.jpg
    multi.jpg
    16.6 KB · Views: 19,054
Last edited:

Kizmet

Senior Member
Jan 18, 2008
216
0
Theoretically combinations could allow for great flexibility... for example - in a first person shooter - look direction could be controlled by the g sensor, actual body direction controlled by a virtual d pad and firing and jumping by simulated key presses.

Or in a driving simulator left right accelerate and brake could be controlled by the g sensor and a gearstick and other controls by virtual buttons.

Very versatile.
 

cymax

Member
May 30, 2007
40
4
Anyway all that games are 640x480 at most so we have 160 pixels to put software DPAD for playing them (instead of doing scaling) and I think that in the future when 800x480 games will come out there will be better solution to play those games ;). All we need is someone who is gonna write the codes for that software DPAD to fill rest of the screen.
 

Kizmet

Senior Member
Jan 18, 2008
216
0
This is my point... a solution to access the many great vga games. However the control system is problematic... hence this post to determine the best solution. A standard sip will not do.
 
Last edited:

Kizmet

Senior Member
Jan 18, 2008
216
0
I'm working on it :)

The current version has a fully functional dpad and mouse.
I'm intending to make a specialised version for gaming, with just a dpad and maybe a couple of "A" and "B" type action buttons, probably customisable.

V
Looks great, mate. For your gaming version will it integrate with wvgafix?
 

sschrupp

Senior Member
Jul 27, 2008
371
10
Iowa
vijay555, I may have just fell in love with you! We're getting closer and closer to being able to play emulator games, flash games, and all sorts of other games! Wheee!
 

sschrupp

Senior Member
Jul 27, 2008
371
10
Iowa
Looks great, mate. For your gaming version will it integrate with wvgafix?

It would definitely be cool if that black space at the bottom that's not used when switching to VGA/QVGA could be used by the pad.

Just out of curiosity, vijay, does the app you're working on simulate multi-touch like Kizmet is suggesting here? (Kudos to you, Kizmet, for illustrating this idea so wonderfully by the way!)
 

vijay555

Retired Moderator
Jun 4, 2005
5,789
68
Witch Space
www.vijay555.com
I'm investigating WVGAFix right now - I've been out of the scene for a while, and so much new stuff has come out, so it's the first time I've seen this app.

I currently use a different solution which achieves a similar result to WVGAFix, but in a different way. When VJVirtualMouseWinFX loads, the screen will shrink up to make space for my control zone, in a similar way to WVGAFix.
However, WVGAFix is very interesting, and possibly a better solution for fullscreen games, but it may interfere with receiving "mouse clicks" in the area of the screen that is made black, as that black area doesn't "exist" as far as the OS is concerned.

Another problem is that WVGAFix won't work neatly with landscape orientations - unless you really want the DPAD control zone to be stuck on the right of the screen while playing in landscape. As I said elsewhere, I'm investigating some other neater layouts when in landscape.

I may try to find a neat hybrid solution between WVGAFix and my method.

The multitouch concept IS incorporated in another version - a dpad only version specialised for games (as you want more space so you don't accidentally click the mouse).

The multitouch DOES work, but is not as comprehensive as the recent G1 hack for Android multitouch (well, as far as I know, I don't have a G1). Calculated multitouch on Windows Mobile works in some situations, but for instance the iPhone "pinch" gesture cannot be well emulated. However, "pinching" with one finger while leaving the other one still DOES work, so multitouch is about 80% functional.

Also, simultaneously rotating two digits around a static centre point (eg Minority Report) also doesn't work well, but that's not a very comfortable gesture anyway!

V
 

Kizmet

Senior Member
Jan 18, 2008
216
0
I'm investigating WVGAFix right now - I've been out of the scene for a while, and so much new stuff has come out, so it's the first time I've seen this app.

I currently use a different solution which achieves a similar result to WVGAFix, but in a different way. When VJVirtualMouseWinFX loads, the screen will shrink up to make space for my control zone, in a similar way to WVGAFix.
However, WVGAFix is very interesting, and possibly a better solution for fullscreen games, but it may interfere with receiving "mouse clicks" in the area of the screen that is made black, as that black area doesn't "exist" as far as the OS is concerned.

Another problem is that WVGAFix won't work neatly with landscape orientations - unless you really want the DPAD control zone to be stuck on the right of the screen while playing in landscape. As I said elsewhere, I'm investigating some other neater layouts when in landscape.

I may try to find a neat hybrid solution between WVGAFix and my method.

The multitouch concept IS incorporated in another version - a dpad only version specialised for games (as you want more space so you don't accidentally click the mouse).

The multitouch DOES work, but is not as comprehensive as the recent G1 hack for Android multitouch (well, as far as I know, I don't have a G1). Calculated multitouch on Windows Mobile works in some situations, but for instance the iPhone "pinch" gesture cannot be well emulated. However, "pinching" with one finger while leaving the other one still DOES work, so multitouch is about 80% functional.

Also, simultaneously rotating two digits around a static centre point (eg Minority Report) also doesn't work well, but that's not a very comfortable gesture anyway!

V

Exactly... calculated multitouch works on half the distane between the touched points.... a pinch gesture would essentially provide the same result all through the pinch.

But in limited control circumstances it can work to simulate a dpad press and simultaneous firing press.

I think your version of shrinking the screen may actually be more flexible than wvgafix... because it would allow you theoretically to have a shell on either side of a landscape screen.. right?

This is what is needed.. dpad on the left buttons on the right.

The spare space of 160 pixels may not be enough... so if the screen can be shrunk slightly then that may allow for better controls.

Alternatively.. using the sensors (light and camera) as simultaneous buttons would give two available buttons with very little coding. But the question is how much battery does that drain and are the sensors sensitive enough?
 

vijay555

Retired Moderator
Jun 4, 2005
5,789
68
Witch Space
www.vijay555.com
Kizmet: yes exactly re the pinch.
WVGAFix's solution is good for making other software compatible with WVGA, but my solution is generally compatible with normal Windows Mobile software.

I think regarding fullscreen games, I envisage placing a dpad on one side of the screen, the A and B buttons on the other in the bottom corner, and these can be physically overlaid over whatever's on the screen. I think that feels quite comfortable.

V
 

Kizmet

Senior Member
Jan 18, 2008
216
0
that sounds good. That may clash with some games and obscure something, though... perhaps semi-transparent controls may work?
 

vijay555

Retired Moderator
Jun 4, 2005
5,789
68
Witch Space
www.vijay555.com
@Kizmet: I have considered semi transparent, but it's a big amount of work.
Windows Mobile doesn't support layered/alpha blended windows well, so the blending has to be done manually. This is CPU intensive, and so is likely to slow the game down.

And even then, it's not very easy to achieve - VJVirtualMouseWinFX supported an alpha blended mouse pointer, but it was too crufty to rely on, and not worth the work, so I removed this feature.

And of course - if your thumbs are covering the control pads, you can't see through them anyway on to the screen!

I'll have to test how well, if at all, this can be achieved on a fullscreen high motion game.

I have some other ideas how to implement the control mechanism, but I think we should all think of any other ideas of how to implement it.

Personally, I'd rather lose say a 100x100 pixel square in each corner, than 100x480 rectangle on the left and right of the screen, so I think an overlaid control pad in each corner makes most sense.

Unfortunately this screenshot doesn't demonstrate this well, but it's a mockup to give you an idea of what I'm thinking.
dpadmockup.jpg

V
 

sschrupp

Senior Member
Jul 27, 2008
371
10
Iowa
I'm a little confused. If we had a game that would normally take up all 800x480 of the screen are you talking about keeping the game that size and then placing the controls on top of it (which would hide the image behind the controls) or are you saying you can shrink the application's display giving unused screen space on either side that can be used for the controls?

Keeping in mind a game controller that would have a d-pad and left bumper on the left side, and 4 buttons and a right bumper on the right side I would say the shrunken display in landscape would be great.

Too bad that transparent controls wouldn't work though since there's some wasted screen space. Although that's not a concern I suppose since most games/emulators/etc wouldn't use the full screen anyhow.
 

vijay555

Retired Moderator
Jun 4, 2005
5,789
68
Witch Space
www.vijay555.com
@sschrupp: obviously if all games are made in WVGA, then that would be great, but as some games are only designed to support a maximum of VGA, it means that we have blank space anyway, that's being wasted.
The suggestion is to find a way to make use of the blank screen, perhaps by placing the controls in the blank areas.

However, what I'm saying, is that whether the game is QVGA, VGA or WVGA, somehow, if you want to emulate a DPAD etc over the game, it needs to occupy some screen space, obviously.

Whether we want to overlay those controls on top of the full screen game, obscurring the screen underneath, or shrink the game screen, either way we have to live with some inconvenience.

Personally, given as many games will either be VGA, or will have some "chrome" already, ie will already have some non game space on screen, personally I think we just stick the dpad controls in some definable position on screen and hope we're not covering much (if any) of the actual game.

V
 

Kizmet

Senior Member
Jan 18, 2008
216
0
I'm a little confused. If we had a game that would normally take up all 800x480 of the screen are you talking about keeping the game that size and then placing the controls on top of it (which would hide the image behind the controls) or are you saying you can shrink the application's display giving unused screen space on either side that can be used for the controls?

In this instance we don't need to worry about games that are already 800x480 - they will have been designed for devices with predominant touchscreens... so the control system will be part of the game. But don't hold your breath for too many releases like that. Without buttons the type of games available is limited.

The need arises to allow some way for the hd to be able to play games already available for similar devices - these are typically in vga 640x480 but need hardware keys the blackstone doesn't have.

So the question is how do we best utilize the screen space and how do we best emulate buttons?

Answer these two questions without slowing down the performance and then you will have the killer shell that will make the blackstone an awesome gaming machine as well as multimedia device.
 
Last edited:

Kizmet

Senior Member
Jan 18, 2008
216
0
@Kizmet: I have considered semi transparent, but it's a big amount of work.
Windows Mobile doesn't support layered/alpha blended windows well, so the blending has to be done manually. This is CPU intensive, and so is likely to slow the game down.

And even then, it's not very easy to achieve - VJVirtualMouseWinFX supported an alpha blended mouse pointer, but it was too crufty to rely on, and not worth the work, so I removed this feature.

And of course - if your thumbs are covering the control pads, you can't see through them anyway on to the screen!

I'll have to test how well, if at all, this can be achieved on a fullscreen high motion game.

Although I think blended buttons would be lovely to look at.. as you say it's not very practical.

The essence is speed here, I think. Speed and simplicity.

I have some other ideas how to implement the control mechanism, but I think we should all think of any other ideas of how to implement it.

Personally, I'd rather lose say a 100x100 pixel square in each corner, than 100x480 rectangle on the left and right of the screen, so I think an overlaid control pad in each corner makes most sense.

I think we're just going to need more space than that.

The control can't be too small otherwise it will be fiddly and hard to use.

I did a couple of mock ups of two of the alternatives:

Calculated Multitouch:

attachment.php


A shell program that places a bar of 100 px on the left and 60px on the right... with buttons set onto it.
Multitouches can be calculated and only touches registering at positions 1,2 or 3 (for example) would be sent to the program by the shell as button presses.

An 8 way controller with 2 buttons would need 16 virtual centre points to be calculated (with some margin for error) they must not overlap but they can be tight against each other as they will not be used.

Although it's called calculated - there is actually no need for calculation on the fly - positons would be stored in a simple look-up table. And thus it could be coded to be fast!

It's also very flexible since there is theroretically no reason you couldn't have more buttons.

However only two touches can be registered simultaneously.

Sensor Method

This is more of a quick fix to me since it only allows for two buttons... but if the light sensor and camera are up to it this is a faster and easier method:

attachment.php


The control centre on the left is 160 px wide and the two sensors held to the right.

Covering a sensor with a fingertip would register as a button press as would covering the camera.

The question here is can the light sensor be polled fast enough? Does it drain battery too fast and is it sensitive enough?

Same q's for the front camera... in fact the back camera could also be used if needed.

This method is slightly less flexible but potentially faster and more accurate. The larger dpad this allows would be better for blokes with large fingers, like me. :)

Now.. take either of these solutions and throw in a gsensor button simulator that calibrates on touching the d pad and then gives you 4 more directions with accelerations and you have a powerful game control system. Powerful enough to handle complex games like driving/air sims and first person shooters.

:)
 

Attachments

  • blackstone2.jpg
    blackstone2.jpg
    30 KB · Views: 1,867
  • blackstone.jpg
    blackstone.jpg
    27 KB · Views: 1,860
Last edited: