Ah, I forgot to clarify about the four conditions: we designed the the study with two stimulus conditions, which we administered to two groups of subjects, which is why the IRF chart shows four conditions. So, that matrix accurately represents the stimulus presentation.
So yes, it seems we did pack the stimuli in there. I presume that this could have contributed to the problem... is it because it did not provide the model with the means for an optimal baseline estimate? When randomizing stimuli, what would you recommend as a desired percentage of TRs to estimate the baseline?