Custom AOV are presented in the AOV Selector's Custom section, and can be selected for output to an image layer just like the built-in AOVs. A custom AOVs is represented by shading nodes of a specific type; the node's name defines the AOV name.
To create a custom AOV, you can create an AOV Group node and add as many AOV layers you desire for that node. Then simply pass connect a shader to any of the AOV layers you have created, and connect the OutColor output to the material's aovGroup input. You can edit the name of the AOVs on the parameter's editor, and those changes will be updated on the dlAOVGroup network node synchronously. The AOV will then appear in the AOV Selector.
Note that the material has to be connected to a renderable object for the AOV to show up in the AOV Selector.