As usual, my two cents worth, I could be wrong (it happens sometimes) but it may not be the fiber wad itself that's closing up your pattern, it could be the slight increase in pressure caused by the restrictive nature of the fiber wad placed in front of the shot charge.
When I got my first French Fusil I had it custom built by a guy whose 'vous name was Sweet William, he was a well known gun builder up and down the east coast, I ordered the flintlock fowler and gave him a deposit at the spring Original North East Rendezvous and picked it up a year later at the next one, it was a beautiful copy of a "D" model Fusil de Chasse, but to my surprise it didn't shoot as good as it looked.
About a year and allot of trying later I sold it and purchased another, and then a third, all of them shot low, I had to cover the target with the barrel in order to hit the black with Ball or Shot, talk about disappointment, I was about ready to give up on smoothbore flinters, about the same time (late '70's) I had a close on line friend from Priest river Idaho by the name of Bill Odem, Bill went by the name of Three Toes and had a web site called Bill's world of Buckskinning.
Bill spent more time in his hunting shirt and moccasins than he did in his jeans and cowboy shirt, and more time hunting and woods running with his flintlock trade gun than any other he owned, so one day I confided in him and told him about my smoothbore situation, He asked me what I used for a load and shot charge, I told him that I'd tried different loads but usually about 90 grns. of Goex 2-F, an over powder wad, 1-1/4 oz. of shot, and an over powder wad, or in the case of a round ball, same powder and charge weight and a patched round ball.
Bill told me to try his pet load, but to follow it to the letter, for a load of shot, 85 grns. of 3-F, one over powder card wad, one lubed fiber wad, one thin over powder card wad, the 1-1/4 ~ 1-1/2 ounce of shot and one over shot card wad.
For round ball, use the same powder charge, over powder card wad, lubed fiber wad, and a patched round ball, I tried his loads and I cut the bulls eye every time at 25 yds from a rest with the ball, and my shot pattern was as right on and as good as any of my modified choke shotguns at 30 ~ 35 yds.
When I told Bill about my surprise at my new found smoothbore accuracy, he sent me a LOL and explained that allot of people have the same problem with their smoothbores, he did also starting out until an old greybeard schooled him on the finer points of smoothbore fowlers and the importance of maintaining a certain level of load pressure, too little pressure and you shoot low, too much and you blow shot pattern.
My favorite smoothbore flinter is my .62 cal./20 gage original Centermark Fusil de Chasse, it's as good if not quite as pretty as my custom flinters, but it's not afraid of the woods, if you get my drift.
I think there's a possibility that the fiber wad being in front of the shot charge offers a bit more resistance and ups the pressure just enough to increase velocity and tighten up your group, If you want to experiment a little, try using 85 grns. of 3-F powder, over powder wad, fiber wad, thin card wad, shot, and top with a thick over powder wad or two thin over shot wads, I'll bet you get as good or better a shot pattern as when using the fiber wad in reverse.
Another way to tighten your group is to use a .20 ga. plastic shot cup, cut the foot off the shot cup, load it right over the over fiber wad, pour in you powder charge, top with an over shot wad.
It's ok to use a 1 ounce 20 ga. shot cup with a 1-1/2 ounce shot charge.