Today was about two questions: does the app actually see the bricks sharply enough to tell them apart? And how good is it really - measured rather than estimated?

High-resolution photos instead of preview frames

While going through the latest test photos I noticed that the app was not saving real photos at all, but frames from the camera preview stream at a very low resolution. Now it takes a real photo for each of the three views:

  • Resolution: 480 × 864 → 1080 × 1920 pixels
  • Brick cut-out, median: about 37 → 88 pixels across
  • Bricks found in all three views and matched to each other: clearly more than before

Recognition now works with much more detail. Runtime on the phone, however, is still far from the target.

The first hand-checked test pile

To stop guessing, I went through a complete pile by hand for the first time: 39 bricks in three views each, i.e. 117 individual views labelled with the correct part number. This finally gives an objective benchmark for the app:

  • bricks found per view: 79.5 %
  • correct part number per view: 46 %
  • correct part number per brick (vote across three views): 51 % (19 out of 37)
  • matching of the three views to the same brick: 95 % correct

The result is sobering but revealing. The processing pipeline on the PC performs just as well on the same photos, so the problem lies in the recognition itself, not in the port to the phone. And almost all errors are size mix-ups: a 2×2 brick recognised as 2×3, a 2×4 plate as a smaller plate. More reference examples alone barely help - the most frequently confused brick already had almost 50 real examples in the catalogue.

A digital tape measure

The obvious idea: measure the size directly. From the position of the phone in each shot, the footprint of a brick on the table can be estimated. Two hurdles showed up:

  • The camera's distance reading is useless as a scale; it was off by more than a factor of two. Instead the measurement calibrates itself per pile: most bricks are two studs wide, which gives a known reference.
  • Along the viewing direction, the height of a brick distorts the measurement. Only the extent across the viewing direction is reliable. From the three angles, the footprint can still be reconstructed.

Used as a filter, the tape measure rules out parts whose footprint does not fit. In a simulation on the checked pile, the hit rate per brick rose from 19 to 24 out of 37, without a single brick getting worse. Height - brick or flat plate - could not be determined this way, though.

The cross-check on the phone

In the evening two more checked piles followed, one in poor and one in good light, this time measured directly on the phone. There, the tape measure did not live up to the simulation:

  • With the filter, 1 and 2 fewer bricks were recognised correctly than without it.
  • The size measurement from the phone's masks is off by 5 to 6 mm, too inaccurate for a filter threshold of 4 mm.
  • More light improves the measurement, but not enough.

So the tape measure is switched off for now and only runs along in logging mode. The lesson: a simulation on a single pile is no proof - the cross-check on the device is part of the job.

The two new piles also show where the bigger lever is. The app turned 17 bricks into 34 to 36 objects because it did not merge the three views cleanly. The newly checked cut-outs went into the catalogue as reference examples, adding 1 to 2 correctly recognised bricks on each of the three piles.

Next steps

Improve how the phone matches the three views, and find a different signal than size measurement for telling a brick from a plate.