DISCLAIMER: This post is solely for academic purposes. Do not try to intercept a text or call as it is generally illegal in most if not all countries. Don't play with the licensed radio frequencies.
The question is: Can we use an Android phone, without any external radio receiver, to intercept a GSM call or text not destined to our phone? How?
Overview: GSM calls and texts use mostly insecure networks, protocols and encryption algorithms, all over the air.
This means that calls and texts can be intercepted and deciphered. This has been demonstrated at various security conferences and it is documented carrier-by-carrier at gsmmap.org.
Source: Decrypting GSM phone calls (Karsten Nohl)
Prequisites: To hack GSM call or text we need:
1. Processing power to run the A5/1 cracking software.
I don't know if there is any port on Android(ARM) platforms but that's probably not a real issue.
2. A programmable radio receiver to have raw access to GSM uplink and downlink frequencies digital data. That's where we DON'T want to use an external radio module, and use the phone built-in radio module.
Let's have a look at the different issues behind this question:
A. Can the phone GSM modem listen to the uplinks (phone to GSM network) of other phones?
It is normally built to listen to the GSM downlinks. But whatever, we can already intercept much with the downlink.
Moreover, antennas may use some sort of beamforming that may require the hacker phone to be in a specific zone, if using a passive intercept technique.
B. What piece of software "filters" the GSM data not destined to the phone ?
First, we need to understand how the radio data is accessed on Android.
Source: Radio Layer Interface (Android Open Source Project, Kandroid)
The GSM filtering (in terms of frequency selection or data dismiss) should either occur at the baseband level or at the RIL level. Otherwise, that would mean it's handled directly in the radio chipset (and I don't think we can do much in this latter case...).
The RIL communicates with the baseband with AT commands (specs here). These AT commands seem too be to high level commands to treat raw data streams.
So I guess the suspect is the baseband firmware but I may be wrong.
C. Can we hack the baseband to access raw GSM data not destined to the phone?
Technically, yes, it's a file flashable with ODIN. (The RIL can be flashed too).
But I've not seen on this forum any special activity on custom baseband development (it's always official baseband firmware).
The issue is that the baseband is hardware-specific and it is closed-source: "Every mobile device that is connected to a cellular network runs some kind of baseband processor with highly proprietary and closed-source firmware." (source).
Attempts to hack official baseband firmwares to develop custom baseband firmwares is still only an emerging concept, at the specification study level.
Regarding open-source software, note that "Airprobe has, for most users, since been replaced by the cheaper Osmocom phones". OsmocomBB is an Free Software / Open Source GSM Baseband software implementation. It intends to completely replace the need for a proprietary GSM baseband software". However the list of OsmocomBB compatible phones is very limited.
The help and knowledge of xda community would be much appreciated to progress on this topic
[EDIT] Interesting links:
Decompiling baseband firmware?
HackRF external transceiver (~300$)
[FAQ] The Baseband (Optimus 2x) by sudden36
Monitor mode for Broadcom WiFi Chipsets by Omri Ildis, Yuval Ofir and Ruby Feinstein (check their RECon PPTX presentation with footnotes to see how they reverse engineered the WiFi chipset firmware based on ARM)