Interesting idea. One note: sometimes after I connect two items it'll reselect the combined item, sometimes it won't. I'd prefer if it never did, but at least if it was consistent I could get used to having to click "Deselect" each time...
EDIT: Also, would be helpful if the items where I've created a category were somehow grouped (column on the left?) so when I find a third item, it's easier to find the ones I'd already joined.
I think the click logic may be a little twitchy. My trackball button worked okay, but my footpedal button frequently did what you describe here - possibly due to a longer-duration button press?
And yes, that would be helpful, but would remove part of the memory challenge. Maybe an optional feature?
EDIT: Also, would be helpful if the items where I've created a category were somehow grouped (column on the left?) so when I find a third item, it's easier to find the ones I'd already joined.