Developer Blog

Today was a productive day. The key lesson: the models learn best from exactly the kind of images they will see in real use - and I now have plenty of those, because every pile photographed on the phone and checked by hand becomes training material.

Segmentation: whole photos instead of single cut-outs

The first stage of the pipeline finds the bricks in the photo and separates them pixel by pixel. Until now, retraining used cut-outs showing exactly one brick each. The training metric even looked good, but in practice the model collapsed: it learned "the image is a brick" and drew large boxes around whole groups of parts on real piles.

Now I export every checked pile as whole photos with all bricks marked - in the first run 90 photos with around 2,000 parts, with three piles held back as a test set. Measured against the manual correction:

• Recall (bricks found): 86 % → 99 %
• Precision (hits without false detections): 91 % → 94 %
• Boxes spanning several parts: 3 → 1

On top of that, the app now processes the photos at a higher resolution (640 instead of 512 px). With both changes it reaches 96.5 % recall on the phone, with practically no false detections. The price is runtime: 16 to 22 seconds per photo on a mid-range phone. There is still work to do here.

Recognition: real cut-outs as reference examples

The second stage determines which part exactly it is. Each cut-out is turned into a feature vector and compared against a catalogue. Since yesterday the phone has been running a retrained recognition model. Today I also added every cut-out from checked piles to the training and to the catalogue. The catalogue now holds around 13,000 entries, roughly a quarter of them real photos and the rest rendered views.

On four test piles that were never part of the training (222 individual views, 81 bricks):

• base model: 77 % per view, 83 % per brick
• retrained (yesterday): 88 % per view, 89 % per brick
• retrained (today): 89 % per view, 93 % per brick

Most of the gain comes from the real examples in the catalogue, not from the training itself. To be fair, I have to add one caveat: the test piles consisted of the same physical bricks as some of the catalogue examples. With completely new parts the gain will be smaller.

Cleaning up the parts catalogue

The error analysis showed that some of the apparent misrecognitions were caused by inconsistencies in the parts catalogue. A few parts are listed there under two numbers because they were renamed or merged over time. In these cases the app had identified the part correctly, but the evaluation counted the differing number as an error. Such aliases are now mapped consistently to one canonical number in the catalogue, the training and the evaluation.

Parts the app only knows from renders

The most exciting question of the day: what happens with parts for which there is not a single real photo? To find out I photographed three piles of gears. Results per brick:

• the two large gears: 16 out of 16 correct - from renders alone
• small gear and an inconspicuous little part: often mixed up with similar parts
• overall: 28 out of 43, and 41 out of 43 with real cut-outs from the other two piles

So distinctive parts can already be recognised from renders alone. Small and similar parts need real photos.

Tools

The tool on my PC that I use to correct the phone photos now uses the same recognition as the app: 88 % instead of 47 % correct suggestions per view. It also rebuilds the app's catalogue at the push of a button whenever new checked piles or new parts come in.

Next steps

Cut the runtime on the phone, test recognition on a completely new set of parts, and teach the app to tell me while taking photos when the distance is right and there is enough light.

Today was about two questions: does the app actually see the bricks sharply enough to tell them apart? And how good is it really - measured rather than estimated?

High-resolution photos instead of preview frames

While going through the latest test photos I noticed that the app was not saving real photos at all, but frames from the camera preview stream at a very low resolution. Now it takes a real photo for each of the three views:

• Resolution: 480 × 864 → 1080 × 1920 pixels
• Brick cut-out, median: about 37 → 88 pixels across
• Bricks found in all three views and matched to each other: clearly more than before

Recognition now works with much more detail. Runtime on the phone, however, is still far from the target.

The first hand-checked test pile

To stop guessing, I went through a complete pile by hand for the first time: 39 bricks in three views each, i.e. 117 individual views labelled with the correct part number. This finally gives an objective benchmark for the app:

• bricks found per view: 79.5 %
• correct part number per view: 46 %
• correct part number per brick (vote across three views): 51 % (19 out of 37)
• matching of the three views to the same brick: 95 % correct

The result is sobering but revealing. The processing pipeline on the PC performs just as well on the same photos, so the problem lies in the recognition itself, not in the port to the phone. And almost all errors are size mix-ups: a 2×2 brick recognised as 2×3, a 2×4 plate as a smaller plate. More reference examples alone barely help - the most frequently confused brick already had almost 50 real examples in the catalogue.

A digital tape measure

The obvious idea: measure the size directly. From the position of the phone in each shot, the footprint of a brick on the table can be estimated. Two hurdles showed up:

• The camera's distance reading is useless as a scale; it was off by more than a factor of two. Instead the measurement calibrates itself per pile: most bricks are two studs wide, which gives a known reference.
• Along the viewing direction, the height of a brick distorts the measurement. Only the extent across the viewing direction is reliable. From the three angles, the footprint can still be reconstructed.

Used as a filter, the tape measure rules out parts whose footprint does not fit. In a simulation on the checked pile, the hit rate per brick rose from 19 to 24 out of 37, without a single brick getting worse. Height - brick or flat plate - could not be determined this way, though.

The cross-check on the phone

In the evening two more checked piles followed, one in poor and one in good light, this time measured directly on the phone. There, the tape measure did not live up to the simulation:

• With the filter, 1 and 2 fewer bricks were recognised correctly than without it.
• The size measurement from the phone's masks is off by 5 to 6 mm, too inaccurate for a filter threshold of 4 mm.
• More light improves the measurement, but not enough.

So the tape measure is switched off for now and only runs along in logging mode. The lesson: a simulation on a single pile is no proof - the cross-check on the device is part of the job.

The two new piles also show where the bigger lever is. The app turned 17 bricks into 34 to 36 objects because it did not merge the three views cleanly. The newly checked cut-outs went into the catalogue as reference examples, adding 1 to 2 correctly recognised bricks on each of the three piles.

Next steps

Improve how the phone matches the three views, and find a different signal than size measurement for telling a brick from a plate.

Brickscanner photographs every pile from three directions: from the front, from the left and from the right. That helps a lot, because a brick that looks like another one from the front often gives itself away from an angle.

For that to work, though, the app has to know which brick on the left photo belongs to which one on the middle and the right photo. I built a new matching method for this: it compares appearance, position and colour, and makes sure two different bricks are never lumped together by mistake. A red and a blue brick, for example, can no longer count as the same one.

This week was all about speed: recognition on the phone was taking far too long per photo. After some measuring and testing, I tracked down the biggest bottleneck and cut it out - recognition time dropped noticeably as a result. While I was at it, I also double-checked that accuracy did not suffer: it did not. Next up is improving the hit rate itself.

For the first time the entire recognition runs directly on the smartphone - no internet, no server, no cloud. The app takes the photos, finds the bricks in them and stores the results right on the device.

The first attempt was still very slow, though. A phone simply has far less computing power than a computer with a graphics card. So I now let the app prepare the recognition in the background as soon as it starts, and slimmed things down in a few places. It is not fast yet, but it works - and that was the most important step.

A key step toward mobile: I exported the recognition engine into a format that also runs on smartphones. That brings the fully offline, cloud-free app a big step closer.

I swapped out the recognition engine under the hood so Brickscanner can later be released as an app without any legal roadblocks. Nothing changes for you - except that the road to the app is now clear.

To spot and fix mistakes quickly, I built myself a small tool: a simple web interface where I can review results, correct them, and let the app learn directly from those corrections.

Real-life photos turned out trickier than expected: shadows, reflections, overexposed shots. I taught the app to handle these better and to learn from corrections, so it keeps getting a little smarter.

The first full run actually worked: photo in, brick recognised, matching model suggested. A big milestone, even though it is far from perfect yet. From here on it is all about fine-tuning.

It all started with one question: can an app recognise loose bricks from a photo? I photographed the first bricks, built a small catalogue and put together an early test version of the recognition engine. That was the beginning of Brickscanner.