Intermediate: Integration of landmark recognition feature in tourism apps (ML Kit-React Native)

ask011

Member
Dec 24, 2020
17
1
3
Overview

Did you ever gone through your vacation photos and asked yourself: What is the name of this place I visited in India? Who created this monument I saw in France? Landmark recognition can help! This technology can predict landmark labels directly from image pixels, to help people better understand and organize their photo collections.

Landmark recognition can be used in tourism scenarios. The landmark recognition service enables you to obtain the landmark name, landmark longitude and latitude, and confidence of the input image. A higher confidence indicates that the landmark in the input image is more likely to be recognized. Based on the recognized information, you can create more personalized app experience for users.



In this article, I will show how user can get the landmark information using ML Kit Plugin.
Integrate this service into a travel app so that images taken by users are detected by ML Plugin to return the landmark name and address, and the app can provide the brief introduction and tour suggestions based on the returned information.

Create Project in Huawei Developer Console

Before you start developing an app, configure app information in App Gallery Connect.

Register as a Developer

Before you get started, you must register as a Huawei developer and complete identity verification on HUAWEI Developers. For details, refer to Registration and Verification.

Create an App

Follow the instructions to create an app Creating an App Gallery Connect Project and Adding an App to the Project. Set the data storage location to Germany

Adding an App to the Project. Set the data storage location to Germany

React Native setup
Requirements

  • Huawei phone with HMS 4.0.0.300 or later.
  • React Native environment with Android Studio, NodeJs and Visual Studio code.
Dependencies
  • Gradle Version: 6.3
  • Gradle Plugin Version: 3.5.2
  • React-native-hms-ml gradle dependency
  • React Native CLI: 2.0.1
1. Environment setup, refer below link.

2. Create project by using this command.
Code:
react-native init project name
3. You can install react native command line interface on npm, using the install -g react-native-cli command as shown below.
Code:
npm install –g react-native-cli
Generating a Signing Certificate Fingerprint
Signing certificate fingerprint is required to authenticate your app to Huawei Mobile Services. Make sure JDK is installed. To create one, navigate to JDK directory’s bin folder and open a terminal in this directory. Execute the following command:
Code:
keytool -genkey -keystore <application_project_dir>\android\app\<signing_certificate_fingerprint_filename>.jks -storepass <store_password> -alias <alias> -keypass <key_password> -keysize 2048 -keyalg RSA -validity 36500
This command creates the keystore file in application_project_dir/android/app
The next step is obtain the SHA256 key which is needed for authenticating your app to Huawei services, for the key store file. To obtain it, enter following command in terminal:
Code:
keytool -list -v -keystore <application_project_dir>\android\app\<signing_certificate_fingerprint_filename>.jks
After an authentication, the SHA256 key will be revealed as shown below.



Adding SHA256 Key to the Huawei project in App Gallery

Copy the SHA256 key and visit AppGalleryConnect/ <your_ML_project>/General Information. Paste it to the field SHA-256 certificate fingerprint.



Enable the ML kit from ManageAPIs.
Download the agconnect-services.jsonfrom App Gallery and place the file in android/app directory from your React Native Project.

Follow the steps to integrate the ML plugin to your React Native Application.

Integrate the HMS-ML plugin
Code:
npm i @hmscore/react-native-hms-ml
Download the Plugin from the Download Link

Download ReactNative ML Plugin under node_modules/@hmscore of your React Native project, as shown in the directory tree below:
Code:
project-dir
    |_ node_modules
        |_ ...
        |_ @hmscore
            |_ ...
            |_ react-native-hms-ml
            |_ ...
        |_ ...
Navigate to android/app/build.gradle directory in your React Native project. Follow the steps:

Add the AGC Plugin dependency.
Code:
apply plugin: 'com.huawei.agconnect'
Add to dependencies in android/app/build.gradle:
Code:
implementation project(':react-native-hms-ml')

Navigate to App level android/build.gradle directory in your React Native project. Follow the steps:
Add to buildscript/repositories
Code:
maven {url 'http://developer.huawei.com/repo/'}
Add to buildscript/dependencies
Code:
classpath 'com.huawei.agconnect:agcp:1.3.1.300’'3)
Navigate to android/settings.gradle and add the following:
Code:
include ':react-native-hms-ml'
project(':react-native-hms-ml').projectDir = new File(rootProject.projectDir, '../node_modules/@hmscore/react-native-hms-ml/android')
Use case
Huawei ML kit’s HMSLandmarkRecognition API
can be integrate for different applications and to return the landmark name and address, and the app can provide the brief introduction and tour suggestions based on the returned information.

Add below under AndroidManifest.xml file.
XML:
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.ACCESS_NETWORK_STATE" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-permission android:name="android.permission.INTERNET" />
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.READ_EXTERNAL_STORAGE" />
<uses-permission android:name="android.permission.ACCESS_WIFI_STATE" />
 <application
      <meta-data
     android:name="com.huawei.hms.ml.DEPENDENCY"
     android:value="dsc"/>
</ application>

Set API Key:
Before using HUAWEI ML in your app, set Api key first.
  • Copy the api_key value in your agconnect-services.json file.
  • Call setApiKey with the copied value.
JavaScript:
HMSApplication.setApiKey("api_key").then((res) => {console.log(res);})
catch((err) => {console.log(err);})
Analyze Frame
Using HMSLandmarkRecognition.asyncAnalyzeFrame() recognizes landmarks in images asynchronously.
JavaScript:
async asyncAnalyzeFrame() {
    try {
    var result = await HMSLandmarkRecognition.asyncAnalyzeFrame(true, this.getFrameConfiguration(), this.getLandmarkAnalyzerSetting());
      console.log(result);
      if (result.status == HMSApplication.SUCCESS) {
        result.result.forEach(element => {
          this.state.landmark.push(element.landMark);
          this.state.possibility.push(element.possibility);
          this.state.url.push('https://en.wikipedia.org/wiki/'+element.landMark)
          long = [];
          lat = [];
          element.coordinates.forEach(ll => {
            long.push(ll.longitude);
            lat.push(ll.latitude);
          })
          this.state.coordinates.push(lat, long);
        });
        this.setState({
          landMark: this.state.landmark,
          possibility: this.state.possibility,
          coordinates: this.state.coordinates,
          url:this.state.url,
        });
      }
      else {
        ToastAndroid.showWithGravity(result.message, ToastAndroid.SHORT, ToastAndroid.CENTER);
      }
    } catch (e) {
      console.error(e);
    }
  }
Final Code:
JavaScript:
import React from 'react';
import {
  Text,
  View,
  TextInput,
  ScrollView,
  TouchableOpacity,
  Image,
  ToastAndroid,
  SafeAreaView
} from 'react-native';
import { styles } from '@hmscore/react-native-hms-ml/example/src/Styles';
import { HMSLandmarkRecognition, HMSApplication } from '@hmscore/react-native-hms-ml';
import { showImagePicker } from '@hmscore/react-native-hms-ml/example/src/HmsOtherServices/Helper';
import { WebView } from 'react-native-webview';
 
export default class App extends React.Component {
  componentDidMount() { }
 
  componentWillUnmount() { }
 
  constructor(props) {
    super(props);
    this.state = {
      imageUri: '',
      landmark: [],
      coordinates: [],
      possibility: [],
      url:[]
    };
  }
 
  getLandmarkAnalyzerSetting = () => {
    return { largestNumOfReturns: 10, patternType: HMSLandmarkRecognition.STEADY_PATTERN };
  }
 
  getFrameConfiguration = () => {
    return { filePath: this.state.imageUri };
  }
 
  async asyncAnalyzeFrame() {
    try {
      var result = await HMSLandmarkRecognition.asyncAnalyzeFrame(true, this.getFrameConfiguration(), this.getLandmarkAnalyzerSetting());
      console.log(result);
      if (result.status == HMSApplication.SUCCESS) {
        result.result.forEach(element => {
          this.state.landmark.push(element.landMark);
          this.state.possibility.push(element.possibility);
          this.state.url.push('https://en.wikipedia.org/wiki/'+element.landMark)
          long = [];
          lat = [];
          element.coordinates.forEach(ll => {
            long.push(ll.longitude);
            lat.push(ll.latitude);
          })
          this.state.coordinates.push(lat, long);
        });
        this.setState({
          landMark: this.state.landmark,
          possibility: this.state.possibility,
          coordinates: this.state.coordinates,
          url:this.state.url,
        });
      }
      else {
        ToastAndroid.showWithGravity(result.message, ToastAndroid.SHORT, ToastAndroid.CENTER);
      }
    } catch (e) {
      console.error(e);
    }
  }
 
  startAnalyze() {
    this.setState({
      landmark: [],
      possibility: [],
      coordinates: [],
      url:[],
    })
    this.asyncAnalyzeFrame();
  }
 
  render() {
    console.log(this.state.url.toString());
    return (
      <ScrollView style={styles.bg}>
        <View style={styles.containerCenter}>
          <TouchableOpacity onPress={() => { showImagePicker().then((result) => this.setState({ imageUri: result })) }}>
            <Image style={styles.imageSelectView} source={this.state.imageUri == '' ? require('@hmscore/react-native-hms-ml/example/assets/image.png') : { uri: this.state.imageUri }} />
          </TouchableOpacity>
        </View>
        <Text style={styles.h1}>pick the image and explore the information about place</Text>
        <View style={styles.basicButton}>
          <TouchableOpacity
            style={styles.startButton}
            onPress={this.startAnalyze.bind(this)}
            disabled={this.state.imageUri == '' ? true : false} >
            <Text style={styles.startButtonLabel}> Check Place </Text>
          </TouchableOpacity>
        </View>
      
        <Text style={{fontSize: 20}}> {this.state.landmark.toString()} </Text>
        <View style={{flex: 1}}>
     <WebView
      source={{uri: this.state.url.toString()}}
      style={{marginTop: 20,height:1500}}
      javaScriptEnabled={true}
  domStorageEnabled={true}
  startInLoadingState={true}
  scalesPageToFit={true}
    />
 
  </View>
      </ScrollView>
      
    );
  }
}

Run the application (Generating the Signed Apk):
1. Open project directory path in Command prompt.
2. Navigate to android directory and run the below command for signing the Apk.
Code:
gradlew assembleRelease
Output:



Tips and Tricks:

  • Download latest HMS ReactNativeML plugin.
  • Copy the api_key value in your agconnect-services.json file and set API key.
  • Images in PNG, JPG, JPEG, and BMP formats are supported. GIF images are not supported.
  • For project cleaning, navigate to android directory and run the below command.
Code:
gradlew clean
Conclusion:
In this article, we have learnt to integrate ML kit in React native project.
This service into a travel apps, so that images taken by users and detected by ML Plugin to return the landmark information, and the app can provide the brief introduction and tour suggestions to user.

Reference
https://developer.huawei.com/consum...-Guides/landmark-recognition-0000001050726194