I have a fun problem to solve. Take a look at the GIF of a scene I'm animating:

The white area will soon be a pile of leaves on the ground. My character scoops them up and lifts some of them off the ground.
Now, I'm ready to just keyframe this in some way or another, but before I do that I want to consider using Physics (or Particles) to accomplish it. Especially because it would get a nice effect of a few leaves spilling off from his hands as he lifts.
Information on using physics for anything even close to this is hard to find (or old), so I wanted to ask. It seems I cannot have a keyframed animation affect a physics group in any way. My only option for "animating" anything "myself" is to use Physics Nudge repeatedly. So far, I am imagining a pile of leaves where each leaf "starts asleep". I then have some invisible bowl-shaped objects roughly the size and shape of this character's arms get "nudged" towards the leaves so the bowl-things pick the leaves up. I'd have to do this carefully so they match the keyframed bone animation I've already made here....
Is there a better way than that?