According to the API documentation of LookupOp, if there are as many bands (channels) in the LookupTable as there are color components (not including alpha), LookupOp applies each of the bands of the LookupTable to the corresponding color channel.
Hence if you'd like to filter only one channel, just construct a multi-channel LookupTable in which the irrelevant channels have identity mappings (i.e., output the same values as inputs). For example,
byte[][] table = new byte[3][256];
for (int i = 0; i < 256; i++) {
table[0][i] = mapRed(i); // Your red channel mapping here
table[1][i] = i; // Don't alter green channel
table[2][i] = i; // Don't alter blue channel
}
ByteLookupTable lookupTable = new ByteLookupTable(0, table);
...