Interests
What I want to work on in a PhD
My main interest is evaluation and model selection: the assumptions behind a scoring rule and how a ranking changes when we change its criteria.
I would like to study cost-sensitive selection, with explicit costs for false positives and false negatives; evaluation under class imbalance, with attention to rare events; and language model evaluation and routing, where correctness, latency, cost, and safety must be considered together.
TOS, a relative metric for model selection
First-author publication, IEEE RAAICON 2021
TOS: A Relative Metric Approach for Model Selection in Machine Learning Solutions
R. Shaha, D. Talukder, M. A. Iqbal and M. M. Haque. 2021 IEEE International Conference on Robotics, Automation, Artificial-Intelligence and Internet-of-Things (RAAICON), pages 26 to 31, 2021.
TOS combines standardized accuracy and error measures. Both are standardized across the candidate models being compared:
In the landslide comparison, the two measures are cross-validated accuracy and false-negative rate. Specifying both makes the selection criterion explicit. It does not, by itself, establish the appropriate cost of either error for a deployment.
Origin: the landslide thesis
Undergraduate research at CUET
My undergraduate thesis built a landslide susceptibility model for the Rangamati Hill Tracts (a district within Chattogram division, Bangladesh). When it came time to choose between the candidate models, several of them sat within one standard deviation of each other on accuracy. That made a choice based only on the highest mean accuracy difficult to justify.
Accuracy also counts a missed landslide and a false alarm equally. I wanted the comparison to account explicitly for missed landslides, which led to the TOS paper and my continuing interest in evaluation.
Limitation: candidate-pool dependence
How the comparison set affects the ranking
TOS is not independent of irrelevant alternatives
Because the z-scores are computed across the pool, adding a candidate that can never win may still reorder the candidates above it. The score is a property of the field as well as of the models.
On the thesis data, adding a most-frequent-class baseline leaves the winner unchanged but swaps two mid-ranked candidates. A ranking produced this way is therefore only meaningful when reported alongside the pool it was computed over.
The candidate-pool experiment reproduces this change. I want to study which ranking properties a relative criterion should preserve and when its dependence on the pool is acceptable.
Current direction
I’m interested in selection rules that accept an explicit error-cost ratio and in measuring their stability as the candidate set changes. A related direction is language model routing: choosing a model for each query under quality, latency, and cost constraints.