AR Sneakers - from the box to your feet in Augmented Reality

7 min readMar 20, 2021

How to make wearable sneakers for Snapchat?

Virtual clothes and footwear are in full swing.

Snapchat as a platform has gone the furthest because it has the most developed ML and body recognition system with which you can recognize not only the face, hands, but also the feet or the whole body.

Surely during the lockdown you looked sadly in online catalogs with new sneaker models and thought about the day when stores will open so you can try them out and of course buy them. Given the precarious situation, time passed and you were still longingly looking at photos of sneakers on your favorite web shop trying to imagine how they would fit you.

You don’t have to imagine that anymore. With the help of Snapchat, you can actually try on sneakers in your own home and make it easier to decide to buy.

Also, a preview of the 3D model of sneakers will give you an insight into all the details on the shoes to make your decision easier.

To be able to do it yourself, read this article to the end. If you are not able to do the whole procedure yourself, be sure to call send me a message to help you do it.

The procedure I will describe consists of several parts.

Photographing an object (Sneakers)
Photo processing (Adobe LightRoom)
Convert photos to 3D model (Capture Reality)
Model processing in 3D program (Blender, MeshMixer)
Optimization and preparation for AR (Mantis LOD Editor)
Creating AR in Lens Studio or Spark AR (Snapchat/Instagram)
View on Snapchat or Instagram

Photogrammetry

First step is photogrammetry. In the previous AR Food tutorial you could read what it is all about, but let me repeat briefly. With a series of photos of sneakers from all sides, it is necessary to completely cover the model, which we will turn into a 3D model in the next phase. It is important that the transitions between the corners are smooth so that the conversion tool (Reality Capture) has no problems in recognition and that everything fits together perfectly, which in the end means that you have as little work as possible to repair the model and for later processing.

As last time I used my Samsung S20, 1440 x 3200 pixels, 20: 9 ratio (~ 563 ppi density), 12 MP for taking pictures. Other equipment was a light ring from AliExpress and a table LED lamp (with paper for diffused light emission).

I mounted cell phone on a tripod with a light ring, and the sneaker was rotated on IKEA’s SNUDDA (Lazy Susan) painted in black.

In this process I found about this special ultra mega black paint called Vanta black so I investigate more and found even blacker paint called BLK 3.0 and I bought some on Ebay. BLK 3.0 is the worlds blackest black acrylic paint. You can’t see topography or shadows on it it has “almost” no light reflection.

https://culturehustle.com/products/black-3-0-the-worlds-blackest-black-acrylic-paint-150ml

Key features of BLK 3.0

Absorbs up to 99% of visible light
Works with most surfaces (wood, paper, canvas, plastic, metal, plaster etc…)
Thins with water
Lightfast and archival
No need for cooking, vacuum chambers, or weapons-grade scientists

The sneaker I photographed was around 35 cm in diameter, I’m size 40 EU, 6.½ US. After setting everything up the sneaker was placed in the middle of the tray (Lazy Susan) and I made few full circles to check everything fits inside the frame on my mobile screen. The photographs were shot in Samsung Pro mode so later photos can be processed better.

Photos were taken in portrait mode and after each photo the tray was moved about 10 degrees clockwise.

For each full circle (360 °), I took approximately 25–35 photos. After one round my camera and moved it up to better cover top view of the sneaker.

Later I tilted sneaker to see a bottom and inside, so I did six rounds.When I had enough photos from all sides, they were imported into Adobe LightRoom for processing. I cropped photo parts I don’t need.

Also, I removed shadows and fixed the color of the object to look more realistic.

3D model

For this step I used Reality Capture because as I mentioned before I got best results with it.

When a 3D model was created you can export 3D model in all needed formats for further processing.

I exported to .obj and .fbx and after that model was opened in Blender where I further cleaned it. I have used Meshmixer for retopology.

After Meshmixer, the model goes again in Blender and after that is exported to the .fbx format, which is the least optimized version of the 3d model.

Snapchat and Instagram supports a maximum effect size of 4MB, and for that reason it is important to stay under 4MB or even lower.

In Blender there is an Add-on for Spark AR called Spark AR Toolkit for Blender.

Use Spark AR Toolkit to:

Make your 3D objects more lightweight
Set the height and the pivot point position
Export quickly and easily

After Blender optimisation we have a final 3D model in .fbx format. If you still aren’t satisfied with model size you can use an application called Mantis LOD Editor for further optimization.

You can check different options in Mantis LOD Editor and when you found best quality and size, it is time to same your mesh.

Now your 3D model is ready to go into Lens Studio for Snapchat or Spark AR for Instagram/Facebook. Before we go to Lens Studio, we need to align sneaker with foot position. For that we will need Foot tracking template fromLens Studio

https://lensstudio.snapchat.com/templates/ml/ml-templates-library/.

Lens Studio

In the file you download go to this location: foot-tracking -> Public -> Example Assets and here you’ll found a file called foot.fbx.

Blender sneakers align with feet 3d model

We need this 3D model so we can perfectly align our sneaker 3D model with legs (size and position).

When you’re satisfied with legs and shoes position you can Publish your Lens.

Enter name of your Lens, add preview image and video, icon and tags.

After that you need to wait for approval and then you can share your virtual sneakers with the world.

Scan this code below directly from Snapchat.

Texture

The model texture should be optimized, either you are using it for Snapchat or Instagram/Facebook. You can reduce image size in pixels from 4096x4096px to 2048x2048px. Try to achieve lowest file size and the best quality.

Adobe Photoshop

After processing the texture in Photoshop, you can go a step further with optimization and use online tool called ShortPixel. ShortPixel will make the extra optimization so your image is even smaller size almost without losing quality.

ShortPixel