here is a guide on how to fix Google Play Store to stop showing the compatibility error when using different screen density than the stock ones.
so you need to edit 2 apk's:
GoogleServicesFramework.apk
Vending.apk
let's start with GoogleServicesFramework.apk
first thing you need to do is to decompile it using apktool
apktool d GoogleServicesFramework.apk
now open
/GoogleServicesFramework/smali/com/google/android/gsf/checkin/CheckinRequestBuilder.smali
with notepad++ or whatever you favorite editor is.
search for the following line
iget v3, v0, Landroid/util/DisplayMetrics;->densityDpi:I
and replace it with
const/16 v3, 0xf0
now instead of requesting current screens DPI density, it will use the hardcoded value of 240 DPI (f0 in hex).
save the changes and you can recompile the application.
now decompile Vending.apk
open /Vending/smali/com/google/android/finsky/api/model/Document.smali
search for
.method public isAvailable(Lcom/google/android/finsky/api/model/DfeToc;Lcom/google/android/finsky/library/Library

Z
delete the entire method and replace it with
.method public isAvailable(Lcom/google/android/finsky/api/model/DfeToc;Lcom/google/android/finsky/library/Library

Z
.locals 10
.parameter "dfeToc"
.parameter "library"
.prologue
const/4 v0, 0x1
return v0
.end method
save the changes
open /Vending/smali/com/google/android/finsky/utils/DeviceConfigurationHelper.smali
search for
iget v0, v12, Landroid/util/DisplayMetrics;->densityDpi:I
and replace it with
const/16 v0, 0xf0
save the changes
open /Vending/smali/com/google/android/finsky/api/DfeApiContext.smali
search for
.method private getSmallestScreenWidthDp(Landroid/content/Context

Ljava/lang/String;
delete the entire method and replace it with
.method private getSmallestScreenWidthDp(Landroid/content/Context

Ljava/lang/String;
.locals 8
.parameter "context"
.prologue
invoke-virtual {p1}, Landroid/content/Context;->getResources()Landroid/content/res/Resources;
move-result-object v5
invoke-virtual {v5}, Landroid/content/res/Resources;->getDisplayMetrics()Landroid/util/DisplayMetrics;
move-result-object v3
.local v3, metrics:Landroid/util/DisplayMetrics;
iget v5, v3, Landroid/util/DisplayMetrics;->widthPixels:I
int-to-float v5, v5
iget v6, v3, Landroid/util/DisplayMetrics;->density:F
const/high16 v6, 0x3fc0
div-float/2addr v5, v6
float-to-int v4, v5
.local v4, widthDp:I
iget v5, v3, Landroid/util/DisplayMetrics;->heightPixels:I
int-to-float v5, v5
iget v6, v3, Landroid/util/DisplayMetrics;->density:F
const/high16 v6, 0x4000
div-float/2addr v5, v6
float-to-int v2, v5
.local v2, heightDp:I
invoke-static {v4, v2}, Ljava/lang/Math;->min(II)I
move-result v5
sput v5, Lcom/google/android/finsky/api/DfeApiContext;->sCachedSmallestScreenWidthDp:I
.end local v2 #heightDp:I
.end local v3 #metrics:Landroid/util/DisplayMetrics;
.end local v4 #widthDp:I
invoke-static {v5}, Ljava/lang/String;->valueOf(I)Ljava/lang/String;
move-result-object v5
return-object v5
.end method
save and your done. you can recompile the application