libero "punch set"
 

this was brought up for discussion at local HS chapter meeting.

if a libero on or in front of the attack line uses closed fist instead of the normal overhead finger action pass, does this avoid the illegal back row attack violation if the ball is next attacked above the net.

Yes.

NCAA Rule, emphasis added:

12.1.2.5 A teammate may not complete an attack-hit when, at the moment of the
attack-hit, the ball is entirely above the top of the net and the ball is coming
from an overhand finger pass by a libero in the front zone or the opponent’s
free zone.

NFHS is similar 9-5-6.b

there was some debate as to whether it counted as finger action apparently at another local's meeting.