Building a real Thermal Imaging addon for Android Smartphones

Search This thread

CommanderROR

Senior Member
May 4, 2008
627
109
Hi everyone,

Thermal imaging has been a long-time dream for me, and I guess for many other as well. However, even the most simple TI cameras tend to be too expensive to justify if you want then just for private use.


It seems like their might be a way out of that now...

A Company called Melexis has released a cheap and low-resolution Thermopile Array (Melexis MLX 90620), that could be the ideal base for an extremely basic Thermal Camera. It's 16x4 Pixel Sensor won't win any prizes for building inspection or detailed thermal analysis, but it is cheap (if you buy a single unit it's around 60$ from Digikey or Futureelectronics, should be in stock this week at Futureelectronics according to Melexis) and comes complete with built-in signal amp, RAM, EEPROM and digital I2C output.

I came across this idea through the Kickstarter Project "Sensordrone" that gives you an easy way to hook up Sensors to your Smartphone via Bluetooth. They are planning to build this TI Sensor as an addon Module to their Sensordrone, but they need somebody to help build the app that turns the raw data coming from the sensor into colourful pixels... ;)
Connecting the Thermopile Array to the Sensordrone is as easy as connecting the 4 pins to the Addon Connector pins. Adding a simple Silicon Diode to reduce the power from 3.3v down to the recommended 2.6v is the only additional step.

You could also hook the Thermopile Array up to a standard Arduino or the Android ADK board I guess if you don't want to go for the Sensordrone.

So it all comes down to Software, and this is where I hope somebody in this fantastic community might help. If you would like to see this happen and (unlike me) have the required coding skills, just go ahead, contact the Sensorcon Team (developers of the Sensordrone) or build your own ADK kit with Thermal Imaging.

TecSpecs for the Melexis MLX90620 can be found on their website together with a demo video. The Sensordrone can be found on Kickstarter (just enter the name) and specs for their unit are available there or on the Sensorcon forums.


I hope somebody takes this up and makes my (and I hope many others') long time dream real! Once the cheap and low-res version works, going for the more high-resolution Heimann Thermopile arrays with 32x31 and 64x62 might be the next step to an affordable Smartphone Thermocam.
 
  • Like
Reactions: got556

Mardaso

Member
Sep 13, 2010
31
9
I had the same idea and ordered the sensor a few days ago.
Got a message from Future electronics that they get the sensor on the 23rd and than it will take 5 to 7 days for me to receive it.
So I will have to wait a little bit longer and than start programming.
I have some ideas to combine it with the phone camera to make it a real thermographic camera.
I don't know if i will use the Sensordrone or a standalone bluetooth solution. With the later I have more control over the firmware.
 
  • Like
Reactions: CommanderROR

CommanderROR

Senior Member
May 4, 2008
627
109
Thanks for replying Mardaso.

The idea of combining it with the smartphone cmos camera is great, I had thought about that as well, but am not sure how it could be achieved. You'd either have to synchronize the FOV, which is pretty much impossible given the parameters of the Melexis module or you'd have to track the area the sensor is "scanning" with the smartphone and overlay the image then. This might work with LEDs or laser dots, but it would be difficult.
But maybe I'm missing something here and it might not be that complicated after all.

My idea for increasing the effective resolution was in using motion sensors and "image stitching" to create a larger and more detailed image with the low-resolution sensor. This would work much like the "panoramic" feature in Android smartphones. It works perfectly for my Nexus. This might be rather challenging to program though...but maybe also worth a try.

I think this would maybe all be relevant for later on if/when the basic functionality is there... ;)

I'm hoping you'll give the Sensordrone a try since I ordered one and would like to see Theraml Imaging on it ASAP...or if you go with a standalone Bluetooth module, would you provide your sourcecodes for the app so that it can be easily adapted for use with the Sensordrone?
The main advantage of the Sensordrone from a dev. point of view is that it takes care of the low-level stuff for you. You'd get the required raw data from the sensor easily and could concentrate on the features of the app...

Maybe you could contact Sensorcon, the creators of the Sensordrone, and see whether you'd like to work with them on this project. They are really nice guys. Are you going to do the initial work on an Arduino based ADK btw?
Whatever you do, I'm glad that you're picking this up. Your work will be useful in bringing a "world's first thermal imaging device" to the Smartphone. Since the Sensorcon folks are also in contact with Heimann Sensors for a later-to-come higher resolution version of the the TI module there is definitely a future for this and your software could be the basis for all that...:good:

I'd like to offer my assisitance, but since I am absolutle yno good at programming I'll probably not be able to do anything useful...:rolleyes:
 

CommanderROR

Senior Member
May 4, 2008
627
109
Bluetooth should not be a problem. Even BT 2.1 has about 3Mbps capacity. With the low resolution of the Thermopile arrays that is a non-issue.


@Mardaso

which FOV option die you order? 40 degree or 60?
 
Last edited:

Mardaso

Member
Sep 13, 2010
31
9
Bluetooth should not be a problem. Even BT 2.1 has about 3Mbps capacity. With the low resolution of the Thermopile arrays that is a non-issue.


@Mardaso

which FOV option die you order? 40 degree or 60?

I ordered: MLX90620ESF-BAB-000-TU so the 60 degree version. They had only this one listed and the evaluation board.
I want to start programming asap. So when the sensors arrive I will start first with a controller (arduino or pic(32)) sending data serial to a PC.
After that go for bluetooth and an android app.

When looking at the data sheet there is a lot of calculation to be done before you get the actual temperature. These calculations could be handled by the Pic32 processor. But we could always use the sensordrone as a gateway for the serial data.

I want to use the motion sensors from the phone to "stitch" the data.
Maybe we will need some kind of holder to align the thermo sensor and the camera pictures so you won't need an extra laser or manual alignment. Just some thoughts.
 

CommanderROR

Senior Member
May 4, 2008
627
109
Futureelectronics seems to have the 40 deg sensor listed now. I'm not sure, but I think that might be better for image stitching...not sure though.

Aligning the sensor with the phone for motion sensor useage...I'd probably go with a bit of double-sided sticky tape... ;)
At least for initial trials that could work well. It would not overlay the "real" image with the Thermal image but would create a more high-res thermal image. That might be easier as a start.

About the calculations...I read that data sheet too, but I'm not sure what is done in the sensor itself and what you have to do in software. It's rather confusing in some passages... ;)
As far as I can tell, calibration values are stored in the EEPROM and can be changed (but I would leave those well alone) and the 65 data packages are written to RAM from where they can be read. 64 temp readings for T(Object) and 1 for T(Case). I'll try to figure out what I can from the Spec sheet, but it's not exaclty my field of expertise...^^
If it gets really complicated you could always ask Melexis for advice. They are really helpful.

The Output from the Sensor should be handled as I2C, that would be easiest and would also be the way the Sensordrone receives data. I'm thinking about getting an Arduino ADK board and a breadboard so that I can at least help with testing, but I'll wait a bit...not sure whether it's worth the money if the Sensordrone comes in October and can do the same and more...



I just reread the spec-sheet, and it seems that the calculations really are not done internally but have to be performed each time you read data from the sensor.
I'd suggest doing those calculations in the app rather than in the microcontroller since that would make things fully compatible with the Sensordrone which as far as I know only passed the data through.

Btw...have you thought about refresh rates? The default is 1Hz (1fps) which is ok for tryouts, but probably not very satisfying for actual use. I' suggest something like 12Hz for general use, maybe more for "painting" a larger image with the "stitching" method. However, I do not know how much the accuracy suffers then...
 
Last edited:

CommanderROR

Senior Member
May 4, 2008
627
109
OK...the Sensors and Evaluation Boards are now listed "in Stock", so you should receive yours soon.

I'm looking forward to following your progress.

If I understand correctly, you'd need to initiate a PTAT read and a Full Frame read to get the raw data from RAM. In addition, you'd have to read the Constants from EEPROM since you need them for calculation.
One constant is set to 25°C, I'm not sure whether the others have to be read each or if they can be stored somewhere in the app and read from there after the first initialization (since they are constants they should not change).

Values are sotred in two's format, so you have to convert them to "real" numbers before using them...my god this is complicated. I think I'll make lunch and leave the calculating bit to somebody who knows what they are doing... ;)
 
Last edited:

cybersteveUK

Member
Jul 24, 2012
6
5
MLX90620 progress

Its worse than that!

Firstly, the chip is designed to run at 2.6V, despite what is says on some (though not all) of the pages of the manual. So,

you will need to get 2.6V from somewhere - I used a LM317 variable voltage regulator. Then you will need level changers

between the 2.6 and your 3.3 or 5v of your microprocessor.

The first thing to do is to extract the contents of the eeprom that is built into the chip. Not difficult, because it is a

22AA02 Mircochip device.

I was disappointed to find that I couldn't get all the information from the EEPROM which is supposed to be there - notably

the temperature gradient coefficient (TGC) and all the "Ta dependence of IR pixels offset" and the "individual sensitivity

coefficients" which all came out as 0! As the other variables came out, I don't think this is a fault of my code.


I have no confidence in the values I did get out either.... The first thing you need to do is to get the temperature of the

chip itself (Ta), as the 64 pixel readings are relative to that. You read a value called PTAT from the Melexis and run it

through some maths. I did that and got crazy values, even when I put the configuration into a spreadsheet. So, I have put

my Melexis in a temperature controlled oven and taken a number of readings from room temperature up to 60 degrees C to

create my own formula.

The next job is to read the 64 channels and the 'compensation pixel' value. I failed to get these out in one I2C session,

but that might be my code. I get the data pixel by pixel at the moment.

If you had confidence in the configuration variables, you could then start to run this data through the 'maths' that is

well documented in the manual (section 7.4.2). There is no way that I can get that maths to run in my target MCU - when

was the last time you tried to calculate 2 to power of 42 in an 8-bit micro!

Not the easiest thing to program for!



Futureelectronics seems to have the 40 deg sensor listed now. I'm not sure, but I think that might be better for image stitching...not sure though.

Aligning the sensor with the phone for motion sensor useage...I'd probably go with a bit of double-sided sticky tape... ;)
At least for initial trials that could work well. It would not overlay the "real" image with the Thermal image but would create a more high-res thermal image. That might be easier as a start.

About the calculations...I read that data sheet too, but I'm not sure what is done in the sensor itself and what you have to do in software. It's rather confusing in some passages... ;)
As far as I can tell, calibration values are stored in the EEPROM and can be changed (but I would leave those well alone) and the 65 data packages are written to RAM from where they can be read. 64 temp readings for T(Object) and 1 for T(Case). I'll try to figure out what I can from the Spec sheet, but it's not exaclty my field of expertise...^^
If it gets really complicated you could always ask Melexis for advice. They are really helpful.

The Output from the Sensor should be handled as I2C, that would be easiest and would also be the way the Sensordrone receives data. I'm thinking about getting an Arduino ADK board and a breadboard so that I can at least help with testing, but I'll wait a bit...not sure whether it's worth the money if the Sensordrone comes in October and can do the same and more...



I just reread the spec-sheet, and it seems that the calculations really are not done internally but have to be performed each time you read data from the sensor.
I'd suggest doing those calculations in the app rather than in the microcontroller since that would make things fully compatible with the Sensordrone which as far as I know only passed the data through.

Btw...have you thought about refresh rates? The default is 1Hz (1fps) which is ok for tryouts, but probably not very satisfying for actual use. I' suggest something like 12Hz for general use, maybe more for "painting" a larger image with the "stitching" method. However, I do not know how much the accuracy suffers then...
 

CommanderROR

Senior Member
May 4, 2008
627
109
Thanks for your input!

getting the voltage down from 3.3v to 2.6 should be easy with at silicon diode, they have a drop of 0.7v built-in and cost less than 1$.
Getting it from 5v would require more work. According to the spec-sheet the sensor should work at voltages between 2.4 and 3.6v (if I remember correctly) but I'd definitely try to run it at it's optimal voltage of 2.6v

The math is indeed strange, there is one step I don't understand where they convert something and I can't figure out what it is they are converting too and from...but I'm not good at math anyway so I guess there'll be somebody out there who gets this...I'll keep looking at it though.

mardaso is the one who plans to write the app for this, if you'd be willing to join in then I'm sure this could go better and smoother. I won't be much help, but I'll try my best to assist... ;)

edit:

This is what I could not work out:

from page 16 in the spec-sheet: 256.26 + 120 = 6776 decimal value
The example hex values convert to 26 and 120 without trouble...but I have no idea how the result of the addition yields 6776

edit2: I figured it out..sort of.
run the results, for examle 6776 trhough this converter http://www.statman.info/conversions/hexadecimal.html and you'll see something interesting about the values... ;)
Doesn't really make sense to me, but it does work...
 
Last edited:

CommanderROR

Senior Member
May 4, 2008
627
109
@cybersteveUK

Could you post the values you get from the sensor and eeprom? Then we can all go through the data together and try to find out how to translate it into useable temp readings.
Thanks.

Sent from my Galaxy Nexus using xda premium
 

amnipar

New member
Jul 31, 2012
2
0
another sensordrone backer and mlx90620 owner

Hi there,
I just registered to this forum to let you know that I wish to participate in this effort too. I am also a sensordrone backer on kickstarter, and I just received my melexis mlx90620 sensor. To start with, I plan to use it with my pandaboard, which has more than enough power to handle the math. When I get the sensor working, I will try to register the sensor data with a camera to overlay the thermal data on the rgb image.

I don't own an android phone yet, but I'm planning to get a Galaxy S3 later. For using the sensor with the android phone, I guess I would start by connecting it with the sensordrone via some simple electronics and using sensordrone's bluetooth connection to send the data to the phone for processing. Later a standalone small bluetooth module that could be attached to the phone would be nice.. Registering the sensor data with the camera image would require that the module stays somewhat stationary in relation to the phone's camera.

I don't know yet when I have time to test the module, hopefully in the next couple of weeks. Meanwhile, should we set up a project and some version control in some place like github? It would be good to have some place to store the data we gather, documentation, and the code we write.
 

CommanderROR

Senior Member
May 4, 2008
627
109
Welcome on board!
Feel free to start a project on google code or similar. So far cybersteve seems to have put the most work into this, but he has not replied in this thread apart from his initial post, so I don't know if he is willing to cooperate.
 

Mardaso

Member
Sep 13, 2010
31
9
update: Did some first testing. Ram register values looking good but eeprom readings are NOK.

@cybersteveUK Did you get any further? My eeprom readings give all 249(dec) 2s complement => -7 except address 0 = 160 and address 87 = 49. Not something I expected. Tomorrow I will take the logic analyser to see if the I2C eeprom code is ok.
 
Last edited:

Mardaso

Member
Sep 13, 2010
31
9
Good and bad news.
The good news is that the code is OK and the bad news is that the eeprom of one sensor is not.
Tried it with the second sensor and all readings are as expected.
At least I can start coding but I have to see if Melexis will exchange the other sensor.
 

CommanderROR

Senior Member
May 4, 2008
627
109
Good and bad news.
The good news is that the code is OK and the bad news is that the eeprom of one sensor is not.
Tried it with the second sensor and all readings are as expected.
At least I can start coding but I have to see if Melexis will exchange the other sensor.


Getting it exchanged should not be a problem. Futureelectronics should take it back without a problem.
I wonder if cybersteveUK also got a damaged sensor since his readings seemed to be off as well and what that says about the quality if this first batch...
 

KRAZYADROIDMASTER

Senior Member
Feb 1, 2011
1,449
80
One you get this project working, Are you going to make a YOUTUBE video of it working???

Thanks in advanced.

Sent from my LG-P999 using xda premium
 

Top Liked Posts

  • There are no posts matching your filters.
  • 2
    MLX90620 progress

    Its worse than that!

    Firstly, the chip is designed to run at 2.6V, despite what is says on some (though not all) of the pages of the manual. So,

    you will need to get 2.6V from somewhere - I used a LM317 variable voltage regulator. Then you will need level changers

    between the 2.6 and your 3.3 or 5v of your microprocessor.

    The first thing to do is to extract the contents of the eeprom that is built into the chip. Not difficult, because it is a

    22AA02 Mircochip device.

    I was disappointed to find that I couldn't get all the information from the EEPROM which is supposed to be there - notably

    the temperature gradient coefficient (TGC) and all the "Ta dependence of IR pixels offset" and the "individual sensitivity

    coefficients" which all came out as 0! As the other variables came out, I don't think this is a fault of my code.


    I have no confidence in the values I did get out either.... The first thing you need to do is to get the temperature of the

    chip itself (Ta), as the 64 pixel readings are relative to that. You read a value called PTAT from the Melexis and run it

    through some maths. I did that and got crazy values, even when I put the configuration into a spreadsheet. So, I have put

    my Melexis in a temperature controlled oven and taken a number of readings from room temperature up to 60 degrees C to

    create my own formula.

    The next job is to read the 64 channels and the 'compensation pixel' value. I failed to get these out in one I2C session,

    but that might be my code. I get the data pixel by pixel at the moment.

    If you had confidence in the configuration variables, you could then start to run this data through the 'maths' that is

    well documented in the manual (section 7.4.2). There is no way that I can get that maths to run in my target MCU - when

    was the last time you tried to calculate 2 to power of 42 in an 8-bit micro!

    Not the easiest thing to program for!



    Futureelectronics seems to have the 40 deg sensor listed now. I'm not sure, but I think that might be better for image stitching...not sure though.

    Aligning the sensor with the phone for motion sensor useage...I'd probably go with a bit of double-sided sticky tape... ;)
    At least for initial trials that could work well. It would not overlay the "real" image with the Thermal image but would create a more high-res thermal image. That might be easier as a start.

    About the calculations...I read that data sheet too, but I'm not sure what is done in the sensor itself and what you have to do in software. It's rather confusing in some passages... ;)
    As far as I can tell, calibration values are stored in the EEPROM and can be changed (but I would leave those well alone) and the 65 data packages are written to RAM from where they can be read. 64 temp readings for T(Object) and 1 for T(Case). I'll try to figure out what I can from the Spec sheet, but it's not exaclty my field of expertise...^^
    If it gets really complicated you could always ask Melexis for advice. They are really helpful.

    The Output from the Sensor should be handled as I2C, that would be easiest and would also be the way the Sensordrone receives data. I'm thinking about getting an Arduino ADK board and a breadboard so that I can at least help with testing, but I'll wait a bit...not sure whether it's worth the money if the Sensordrone comes in October and can do the same and more...



    I just reread the spec-sheet, and it seems that the calculations really are not done internally but have to be performed each time you read data from the sensor.
    I'd suggest doing those calculations in the app rather than in the microcontroller since that would make things fully compatible with the Sensordrone which as far as I know only passed the data through.

    Btw...have you thought about refresh rates? The default is 1Hz (1fps) which is ok for tryouts, but probably not very satisfying for actual use. I' suggest something like 12Hz for general use, maybe more for "painting" a larger image with the "stitching" method. However, I do not know how much the accuracy suffers then...
    2
    This week I made a new software version.
    Now with sensor data overlay. I need to reduce the lag between the sensor data and the camera video.
    http://youtu.be/6KE1TqAbOGo
    2
    Update:
    After some weeks when I had my holidays, I started with fresh ideas and made some progress on the Android app.
    The phone camera is functioning and so is interpolation as you can see on the attached screen shot from a hot cup of water.

    Last week I got the new 40 degrees sensor but didn't had a change to test it.
    But I think the 40 degrees sensor will be best suited to my needs.
    We will see.
    1
    On Kickstarter you can buy a standalone version that my software will support.
    When I can make a deal with Sensordrone the software will support that one also.

    @CommanderROR: I need a beta tester for the Android app and sensor, contact me if you're interested.
    1
    I had the same idea and ordered the sensor a few days ago.
    Got a message from Future electronics that they get the sensor on the 23rd and than it will take 5 to 7 days for me to receive it.
    So I will have to wait a little bit longer and than start programming.
    I have some ideas to combine it with the phone camera to make it a real thermographic camera.
    I don't know if i will use the Sensordrone or a standalone bluetooth solution. With the later I have more control over the firmware.