Alright, so you've got a shiny new Moto G5 plus, which supposedly has a similar Sony imx sensor as the one found on flagships like the Galaxy S7 and Pixel2. Yet, you are somehow disappointed by the camera's quality.
YOU ARE NOT ALONE!
Turns out the camera sensor on this thing is actually pretty darned good, and the f/1.7 lens is no slouch, however it is being held back by poorly designed software from Moto/Lenovo.
Before I got into the solutions, let me specifically point out the problems I found with the camera so we're on the same page (click to reveal content, or just skip the fixes below):
1) Overzealous Sharpening and aggressive Noise Reduction are the biggest culprits. It almost looks like a bad filter is being applied to your images, turning fine details into mush and contrasty lines like text into something bizarre and artificial looking. The camera is doing this as part of the internal capture process, so this happens no matter what application it is taken in.
2) Highlights clipping earlier than it should. I'd say this might be a function of the camera's sensor more than the hardware, but analyzing RAW data from the sensor (more on that later) in my admittedly unscientific half-hazard naked eye, I'd go out on a limb and say it seems like we're not being shown all the DR the camera is able to see. To the layman: this means brights turn completely white effectively overexposing parts of the image that should otherwise still look OK.
3) The preview/viewfinder mode is even worse at #1 than the captured image is. That is, the preview mode (the video feed you see before you capture the full resolution image) is still applying the sharpening and NR at the same level as the captured image, however since the resolution is lower for the preview, it looks more exaggerated and ugly. This means two things: A) Its hard to tell what the image you take will look like, as the preview is ugly (and you have to hope the final will look better), and B) some apps use the preview feed as their capture method. For example, snapchat and augmented reality apps like Pokemon Go. It just looks terrible.
The good news is that there is a lot that can be done to improve and fix the camera.
The bad news is that almost all of it requires modifying system files. That means those with locked bootloaders (Amazon ad users or those who are afraid of voiding their warranties) might be out of luck.
You don't necessarily need root access, although some of this can be more easily performed with root. I needed to unlock my bootloader and performed most of this within TWRP, but chose to keep my phone otherwise unrooted. I'm going to split this into sections, fixing and improving.
A) FIXING THE IMAGE PROCESSING ISSUES
If you want to remove/circumvent the aggressive sharpening and NR issues (so that apps like snapchat or other camera applications don't look over-processed, including the stock Moto cam), try the following:
There are two sets of files to replace in your system. Chromatix and LIBMM. NOTE: I don't recommend the LIBMM / binning part. Binning is combining pixels together into an averaged single pixel value, effectively reducing resolution to combat noise on the sensor. In other words, instead of having a large possibly noisy image, you get a smaller less noisy picture. But you can always choose to resize the image in post after you take it with the same resulting effect, so I don't see why this would be desirable to do. I recommend just applying the chromatix mod to remove the sharpening and NR. If you're comfortable using TWRP, there's actually a flashable zip in post #9, which is what I personally did (again, remember I'm not rooted, so I needed to flash/overwrite/etc within TWRP).
After applying this mod, even the stock camera app looks better, and apps that use the live preview (Snapchat, Pokemons AR view, etc) don't have the shadow ghosting from excessive sharpening anymore either. You might want to apply some sharpening and NR in post yourself if the image requires it, but thats always something you can choose but at least you have control over it and even without doing that it looks far better and more natural than stock to me.
2) Enable camera2 API. This doesn't have a direct effect on quality, but will allow other apps to have more direct access to the camera. If you want to improve beyond the results of #1 above, you'll need this.
The stock rom, surprisingly, has Camera2 API included, however it is disabled by the build.prop file. Again, no idea why Moto/Lenovo chose to do this, but the good news is you can enable it again easily. To do this, you need to add the following line to the build.prop
Or if rooted, via terminal / shell:
su persist.camera.HAL3.enabled 1
Once this is enabled, you can use apps such as the modified freedcam to shoot RAW, as found here: https://forum.xda-developers.com/g5-...-root-t3582392
I found RAW from this app particularly difficult to work with, so I'm keeping it for experimentation purposes but for practical real-world usage, I'd recommend the next item below.
B) IMPROVING YOUR PHOTOS (or, how to install Google Camera!)
Section A was all about how to avoid the crummy filters that get automatically applied to the camera. But your device is capable of even BETTER photos by using Google's magic HDR+ algorithm (the special sauce that makes the Pixel phones so awesome). Here's how to do that:
1) If you haven't already enabled the Camera2 API, you need to do that first (info in section A above).
2) Get a working version of Google Camera with HDR+ for your phone. The last available 32-bit version with HDR+ is Gcam 4.2, so if you're on the stock ROM like I am, you're stuck with only the 32-bit versions (all the new versions with the bells and whistles are 64-bit only). The image quality is remarkably better, both in sharpness and dynamic range in almost any light. I have to admit, I'm in love with the images from this, and have even printed some for my wall.
UPDATE: New version from Savitar on post #44!
Problem is, our cameras use a different focus system than phones like the Nexus and Pixel. So, in photo mode, it doesn't really work correctly. Switching over to video mode, however, DOES work (likely using contrast detection focus, which is slower but more universal, especially for video focus). So the workaround is to swipe over to video mode, focus, swipe back to photos and press-and-hold to lock focus. It's annoying, but HDR+ is that good that I still try to work with it much of the time.
For times that it would be frustrating or not necessary, I'll just resort to the stock camera app or Footej camera, since the chromatix mod makes them mostly usable now. Just not ZOMG AMAZING like the Google HDR+ does.
OPTIONAL #3) Install a 64-bit custom Rom and use the latest Google Camera HDR+ app floating around (v4.4). The recent versions B-S-G released based on the Pixel camera has options to change camera modes in settings, some combination of which supposedly focus properly. You will likely get a more stable camera app experience with this as it requires less modification to work, but you'll have to be committed enough to go off stock.
Check out the app and suggested settings to try here:
Thanks @shanks125 for the info!
Update 1/3/18: I just heard there is a port of the new portrait mode feature from the Pixel2 that is sort of working on 64-bit roms, but I think some modifications will need to be made to avoid focus drift like previous modded gcam versions.
There is also a 32-bit version of the new v4.4 that @Aby lad came up with in this thread:
In order to make it more compatible with other devices, it looks like he has disabled the Hexagon DSP, which I think is what makes HDR+ work, so while we can now change settings and focus, HDR+ isn't working. The app still has some Google goodies like Sphere and Blur mode, but without HDR+ the real benefit of Google camera is lost anyway. Might as well use stock with chromatix mod at that point.
Hope some of you find this helpful/useful!
UPDATE 12/28/17: New GCam with 6P features on post #14.