Use Case for Table Grouping:
Is it possible to do table grouping with related list item so that the related list tables are grouped by product category, for example?

App Expert Reply:
You can merge the related list and sort by the product category.  You could also use the filter option if you wanted to merge a related list for each product category. Details on the filter and sorting feature are shown in this related article.

Use Case for Sub-Groping in Table:
Would like to “Sub-grouping Totals” and not just sort them per the sort feature in template builder.

App Expert Reply: 
Our A5 Documents app just merges fields that exist. So you will need to create the sub groupings functions within Salesforce so you can merge those groups and totals. For example, you would add a field like group id on opplineitem object. Or group number, or its based on some product.category. Then you would need to create roll up fields on the opportunity to calculate the sub totals of the oppline items for specific groups. Then with A5 documents you would use the filter feature to merge the opplineitem related list for only items that equal that group and merge the sub total field on the opportunity.
As far as tagging groups there are many options and depends on the clients use case. Perhaps its manual, perhaps is creating a group field or category on the Product then the formula field on opplineitem object to reference the product field. Or apex trigger if needed. If you use something like Salesforce CPQ they have a grouping feature for example to leverage. For more assistance in creating a solution for sub-grouping, we offer premium app support to assist you.