Virtual sightseeing by walking at home : For Android App development
You might have been forced to live at home under the influence of the Coronavirus, and you may be under a lot of stress.
I think the two most common causes of stress are lack of exercise and refraining from going out.
There are many people who suffer from chronic fatigue and cannot enjoy leisure activities such as sightseeing, traveling, climbing, and walking.
With that in mind, this time, I’d like to share with you the following tips for relieving the stress of lack of exercise and self-restraint from going out
Virtual sightseeing by walking home.
I’d like to introduce you to the android application development for that.
Introduction
This time, you’ll need the following device and app.
- Android smartphone
- Google Chromecast (device)
- Google Home (app)
- TV (general TV for home use)
Android app code
This code refers to google/simple-pedometer and googlemaps/android-samples.
1. What is virtual tourism in home walking?
Here’s how it works.
1. walk on the spot with a smartphone in your hand or in your pocket
2. smartphone app detects your walking movements and number of steps
3. panoramic images of the scenic spot are projected on TV in conjunction with the movement
When you walk around with a TV showing a sightseeing spot, you would feel as if you are walking around a sightseeing spot.
I’d like to elaborate a bit more on this.
About walking that you could at home
One of the ways to get rid of a lack of exercise is to get some aerobic exercise. In fact, could you actually get a cardio workout just by walking at home?
Professor Kenichi Nemoto of Matsumoto University’s Graduate School introduces effective methods in “Stepping on the Spot”.
Raise your thighs slowly and high.
Waving your arms in a way that the fingertips of your hands are firmly extended and pulled far back.
After stamping on my feet in this way for about five minutes, I felt my sweat oozing from my feet and my metabolism improving. It seems to work well as an aerobic exercise.
By sensing this movement with your smartphone, it seems that you can walk more effectively when you don’t get enough exercise.
About virtual tourism
We use google street view as a tool for virtual tourism.
A 360-degree panoramic image is used, so you can experience a real sightseeing experience through the screen.
For example: Mt.Fuji
However, it is not suitable for this application as it requires mouse and finger operation (touch and scrolling), so it is not suitable for this application as it is.
It is necessary to create a system in which the screen moves in conjunction with the movement of the walk and the number of steps.
2. How to develop an app for virtual tourism on home walks
Now, I’d like to explain how to develop the app from here.
The key points in the development of this application are
- Detects walking movements and number of steps with a smartphone
- Detects movements such as turning to the side and looking around
- Move the panoramic image (google street view) according to the movement and number of steps.
This will be a I will explain the above three points in detail.
Detecting your walking movements and the number of steps you take by smartphone.
Smartphones are equipped with sensors that can detect a variety of movements. It is equipped with a mechanism that can be moved up and down, rotated, and sensed when it is moved.
In this case, we will use a combination of up-and-down and left-right movements to sense the movement of walking.
Specifically, I’d like to get into the mechanics of the “pedometer”.
Popular smartphone pedometer apps are made to only sense the movement when you walk. Simple vibrations alone are not sensed as a walking motion and the number of steps taken are not counted.
This is where the smartphone’s accelerometer plays a role. The acceleration sensor is a system that detects upward and downward movement, left and rightward movement, and forward and backward movement (depth or front). These are combined to detect and count the movements that are closest to walking.
The detailed implementation is described later.
Detecting movements such as turning to the side and looking around
With the pedometer mechanism, it can only sense the forward motion of walking. It does not detect any movements such as turning to the side or looking around.
Therefore, we can use gyroscopic sensors to detect such movements. A gyro sensor is a sensor that senses rotation and orientation.
By using the gyro sensor, the smartphone can be rotated to turn left or right, and it can sense the movement of looking around.
Moving google street view according to the movement and number of steps.
The movement sensed by the accelerometer and gyro sensor must be linked to the panoramic image.
By moving forward and rotating the vehicle according to the movement of walking and the number of steps, the system conveys a series of movements such as turning left, right, and looking over 360 degrees.
From here, we’ll show you how to implement it while actually looking at the program.
Pre-requisites
- android studio 3.6.3
- android 9.0
- API Level 28
- Latest Android Build Tools
- Google Play services
3. Implementation of the pedometer
For the pedometer program, I will introduce it by referring to google/simple-pedometer.
As mentioned earlier, you can count the motion of walking with an accelerometer.
Now let’s explain how to detect the behavior.
A. Acquisition of 3-axis data for the accelerometer
The accelerometer senses movement on three axes: x-axis (lateral movement), y-axis (vertical movement), and z-axis (forward and backward movement). It judges the movement close to the vibration of walking based on the movement of the three axes.
B. Calculate the total value of the 3-axis vector
In order to sense the motion of walking, it is necessary to capture the combined motion of the x, y, and z axes.
When you actually put the smartphone in your pocket and walk around, you can see that the vibrations are transmitted to the smartphone from up and down, left and right, back and forth. You will not be limited to just one movement of any one.
Therefore, we calculate the sum of the three axis vectors so that we can determine which axis of the xyz axis acceleration (hereinafter referred to as the three axis vector) is reacting.
The sum of the three axis vectors can be calculated using the following formula.
C. Noise Removal
The accelerometer is responsive to even the slightest vibration.
If it detects all the slightest vibration, it will not be able to accurately measure the number of steps walked.
Such unwanted vibrations are called noise, and it is necessary to remove them.
Here are some ways to eliminate noise
- Low-pass filter
- Whether the average value every n times (the total vector average of the three axes) exceeds the threshold
A Low-pass filter is a method of removing noise using the following formula
Y(t) = a*Y(t-1) + (1-a)*x(t)
Y = filtered value, x = raw data, t = time axis, a = constantAlthough it is a commonly used method, this time we will use the following method.
Whether the average value every n times (the total vector average of the three axes) exceeds the threshold
In this method, the average value of the sum of the three axis vectors is obtained every n times, and if the average value does not exceed a certain threshold, it is judged as noise and removed.
The n times is the number of times that the acceleration sensor reacts, but the larger the number, the less noise it picks up.
If the average value calculated here exceeds a certain value (threshold), it is counted as one step of walking.
In this case, we want to sense the movement of walking, which is effective for lack of exercise, so we set the threshold high enough to pick up larger movements.
D. Delay
Finally, I will explain about the delay.
In the action of walking, there is a certain interval of time between counting one step and counting the next one.
In order to sense that interval, a delay is introduced.
In the delay logic, after sensing a step, it is implemented in such a way that it does not react to the vibration until a certain amount of time has elapsed.
Let’s take a look at the code.
Get an instance of the accelerometer.
“onPause” is the process of when the app goes into the background. De-register the accelerometer to prevent battery drainage.
“onResume” is what happens when the app resumes from the background. Register the accelerometer again.
“onSensorChanged” is the process of processing the accelerometer when it reacts. simpleStepDetector.updateAccel is called to determine the walking behavior and the number of steps.
I’ll go over it in the comments, point by point.
Point A
This area is averaged every n times for noise reduction. Note that this is not the vector sum, but the acceleration for each of the three axes.
Point B
Here, the total value of the 3-axis vector is calculated.
Point C
Here we check whether the sum of the vectors exceeds the threshold value. Delays are also taken into account. STEP_THRESHOLD is set to high to pick up more movement to detect effective walking for lack of exercise.
This code also uses a variety of other methods to process acceleration data to accurately sense walking movements and the number of steps taken.
Finally, it counts the number of steps taken when it senses the movement of the walk.
4.About the implementation of turning to the side and looking around
Turning to the side and looking around uses a gyro sensor that senses rotation.
A. Acquisition of rotation data around 3 axes of the gyro sensor
The gyro sensor senses the rotation around the three axes.
Rotation of the x-axis center
Rotation of the y-axis center
Rotation of the z-axis center
In this case, when the smartphone is rotated vertically (when it is rotated in the center of the y-axis), it will be judged as turning to the side and looking around.
B. Calculate the total value of the 3-axis vector
Like the acceleration sensor, the gyro sensor calculates the sum of the three axes of rotation, and the sum of the three axis vectors can be calculated using the following formula.
C. Noise Removal
Here, as with the accelerometer, we would like to determine whether the average value (the total vector average of the three axes) every n times exceeds the threshold value in order to eliminate noise.
For more information, see the previous section on removing noise from accelerometers.
D. Delay
As with the accelerometer, the delay is taken into account here.
Let’s take a look at the code from here.
The source code doesn’t exist in google/simple-pedometer, so I made it myself. The idea is the same as the accelerometer.
Adds a gyro sensor to SimplePedometerActivity.java.
Register the gyro sensor with onResume.
LotateDetector.updateGyro will be called to detect horizontal looking and ambient looking behavior.
Create a new file LotateDetector.java. The base is almost the same as LotateDetector.java, but we add the right or left direction sensing. When the smartphone is held vertically (rotated around the center of the y-axis), it senses when you turn it counterclockwise (y-axis rotation > 0) to the left, and when you turn it clockwise, it senses when you turn it to the right.
The motion of looking around can be achieved by continuing to rotate in the same direction.
Finally, this section describes what to do when the above action is detected.
5.Implementation for moving google street view according to movement and number of steps.
From here, I will explain the code that links the movement detected by the smartphone with google street view.
A. Enable Maps SDK for android API
Enable the API of Maps SDK for android to run google street view on android.
https://console.cloud.google.com/
B.call google street view
Next, call google street view on the app.
There is a sample source, and I will explain it along here.
Follow these steps to add the obtained API key to your program.
https://github.com/googlemaps/android-samples/blob/master/ApiDemos/java/README.md
2.Create a file in the ApiDemos/java directory called secure.properties (this file should NOT be under version control to protect your API key)
3.Add a single line to ApiDemos/java/secure.properties that looks like MAPS_API_KEY=YOUR_API_KEY, where YOUR_API_KEY is the API key you obtained in the first step
4.Build and run
After that, we will call google street view.
C. Go straight ahead along the arrow
If you look at google street view, you’ll see that there are arrows. Click here to go straight in the direction of the arrow.
The code for this behavior is as follows.
It finds an arrow in the direction the camera is pointing and advances in the direction of that arrow.
When it senses walking, you can call the above code according to the number of steps, and it will work in conjunction with GOOGLE STREET VIEW to move forward.
D. Turn to the side, look around
If you look at the google street view, you may see several arrows with forks in the road. If you want to go to an arrow other than a straight line, you need to turn to the side.
To turn sideways, it senses which direction to turn in and how much to rotate, and reflects that information in the google street view.
Set the rotation angle to minus when facing left, and to plus when facing right.
After turning to the side, you can move forward to the arrow in the direction you’re facing.
When the above program is combined with the gyro sensor, it is possible to turn the smartphone to the side and look around (rotate 360 degrees) by rotating the smartphone around the Y-axis.
E.Fixing your smartphone to the side screen
Fix your smartphone’s screen to the side for projection on the TV. This is because if you cast to TV in portrait mode, the screen will not match the ratio of the TV screen.
Set android:screenOrientation=”landscapescape” in AndroidManifest.xml.
The implementation of the code is now complete.
6. Cast to TV with Chromecast and google home
So far, I’ve been able to link walking and google street view on my smartphone app.
Next, use your Chromecast and google home to reflect it on your TV screen.
Read more about Chromecast and google home here.
Learn more about casting to the screen here.
7. Try virtual tourism by walking at home!
Now you are ready to go.
So let’s try it.
- By google Home, Cast your smartphone screen to your TV.
- Launch the app and make sure that the app screen is cast to the TV.
3. Hold your smartphone or put it in your pants pocket and walk to the TV by “Stepping on the Spot” manner.
When walking, keep the following things in mind.
Raise your thighs slowly and high.
Waving your arms in a way that the fingertips of your hands are firmly extended and pulled far back
4.When your smartphone detects your walking motion, the TV screen is also linked to the walk.
5.If you want to turn to the side, stop and rotate your smartphone. Turn counterclockwise when you want to turn left, and clockwise when you want to turn right.
6.The TV screen is also linked with the rotation. Keep spinning in the same direction so that you can see your surroundings. Once you’ve turned in the direction you want to go, walk and move forward.
How did you like it?
I hope this has helped you to relieve the stress of a life of self-discipline.
Thank you so much for reading.
