One of my primary areas of interest is player action recognition. By using advanced computer vision techniques and machine learning algorithms, I aim to identify and analyze specific player actions on the ice. This could be used in player evaluation techniques as input to analyze their performance and provide coaches and teams with valuable insights into player performance, such as identifying areas for improvement or optimizing their game strategies.

I’m really excited about the work I’ve been doing on goalie pose estimation, too. We’re working on predicting goalie, net, and equipment poses with high accuracy, which will be super valuable for improving goalie action recognition and performance evaluation.

Another area of research I’m interested in, and I’ve worked on during my Master’s is reinforcement learning. I want to improve the generalization ability of RL agents so that they can adapt to new environments and situations more effectively.

Before starting my Ph.D., I worked in the industry doing business analytics for a great company. It was a fantastic experience, and it’s given me a unique perspective on the intersection of academia and industry.

Overall, I’m really passionate about using data and technology to drive our understanding of the world. I’m excited to keep making significant contributions to the field of sports analytics and beyond!

Email: marjan.shahi@uwaterloo.ca

David Clausi
Alexander Wong

Pose Estimation, Action Recognition

From Jan. 2010, Amir is collaborating with the Action Lab at Centre for Cognitive Neuroscience, Wilfrid Laurier University, ON. Prior to his PhD, Amir worked for five years (May 2001- Aug. 2006) on hardware and software design for advanced control processes and robot vision for factory automation. He worked in a job sharing projects with several German, French, Italian, and Korean companies. Amir took several training courses and holds certificates in the field of computer vision and control processes from world-class companies such as Siemens, FrigoFrance, Nagel, ISRA, and Pepperl+Fuchs.

From Aug. 2012 until present, Amir is an Adj. Assistant Professor at the University of Waterloo, continuing his academic endeavor with the focus on Computer Vision and Pattern Recognition. He is actively collaborating with other faculties and local industries.

Email: hshabani@engmail.uwaterloo.ca

So far, we have been mainly focused on improvement of different components of a standard discriminative bottom-up framework (such as widely used Bag-of-Words approach (Fig. 1)) for action recognition in video. We have three main contributions on local salient motion feature detection, action representation, and action classification. Fo references on our improvements on robust salient feature detection refer to [1], [2], [3], [4].  Our contributions on more descriptive action representation and better classification are under review and will appear here as they are publishing. Stay tuned 😉  

Fig.1: Standard Bag-of-Words framework for human action recognition. This section focuses on how to improve the detection of spatio-temporal salient features.

Improved Spatio-temporal Salient Feature Detection for Action Recognition             

Existing salient feature (or key point) detectors use a non-causal symmetric temporal filter such as Gabor. In contrast, the biological vision and the recent publications are in favor of efficient temporal filtering using asymmetric filters. We have thus designed three causal and asymmetric temporal filters of asymmetric sinc, Poisson, and truncated exponential [1]. The complex form of these multi-resolution filters are biologically consistent with the human visual system to obtain a phase-insensitive and contrast-polarity insensitive motion map[2]. Figure (1) shows our salient feature detection framework using the introduced complex temporal filters. 

Here, we provide some results for the motion map (R) and feature detection using our novel asymmetric sinc filtering. For more details, please refer to [1].

Multi-resolution motion map:

To detect local video events with different spatial an temporal scales, the motion map should be computed at multiple resolutions. This video clip shows the motion map at nine different spatio-temporal scales (\sigma, \tau). The top-left shows the motion map at the finest scale of (2,2) and the bottom-right shows the motion map at the coarsest scale of (4,4).

Watch video on YouTube

Fig. 2: Motion maps at nine different spatio-temporal resolutions.

Spatio-temporal salient features:

This video clip shows the salient features detected at nine different spatio-temporal scales (\sigma, \tau). The corresposning fetaures are highlighted by red box on the original video. Below to each video, the local volume representing the salient features are on and the rest of non-salient regions are off. The top-left shows the results at the finest scale of (2,2) and the bottom-right shows the results at the coarsest scale of (4,4).

Watch video on YouTube

 Fig. 3: Salient features detected at nine different spatio-temporal resolutions. The symmetric Gabor filter and our three asymmetric temporal filters are tested comprehensively under three scenarios: precision of the detected salient features, reproducibility under geometric deformations, and human action classification. For more details, please refer to [1].

1- Precision and robustness tests

Precision requires salient features to be from the foreground. Robustness requires re-detection of the same features under different geometric deformation such as a view/scale change or affine transformation. As Table (1) shows the asymmetric temporal filtering has better performance than symmetric Gabor filtering for both the precision test and the robustness tests. Among asymmetric filters, our novel asymmetric sinc performs the best.

Table 1: Experimental results showing better performance of asymmetric sinc filtering.

2- Action classification

To evaluate the quality of the features detected using different temporal filters, we used the baseline discriminative bag-of-words classification framework for action recognition. Our experiments on three different benchmark datasets of the Weizmann, the KTH, and the University of Central Florida (UCF) sports dataset show that (A) salient features detected using asymmetric filters perform better than those detected using a symmetric Gabor filter. (B) The salient features detected using our novel asymmetric sinc filter provide the highest classification accuracy on all datasets [1]. 

Table 2: Higher average classification accuracy is obtained using salient features detected by a complex, asymmetric sinc filtering.

Comparison of structured-based and motion-based salient features:

We have recently compared the structured-based spatio-temporal salient features (3D Harris, 3D Hessian, 3D KLT) with the motion-based salient features detected using the best asymmetric temporal filter (i.e., asymmetric sinc filter) and the symmetric Gabor filters. This evaluation is performed in a common BOW framework on several benchmark human action recognition including: KTH, UCF sports under two different protocols of nine/ten categories, and the challenging Hollywood Human Actions (HOHA I) datasets. As Table 3 shows, in all three datasets, the motion-based features provide higher classification accuracy than the structured-based features.

More specifically, among all of these sparse feature detectors, the asymmetric motion features perform the best as they capture a wide range of motions from asymmetric to symmetric. With much less computation time and memory usage, these sparse features provide higher classification accuracy than the dense sampling as well.

Table 3: Motion-based salient features are more informative in encoding human actions than structure-based features.

Fig. 4: 2D projection of different spatio-temporal salient features on sample frames from diving action from UCF sports dataset. Among all the detectors, the asymmetric motion features are more localized on the moving body limbs with much less false positives from the background.

Comparison with the state-of-the-art methods:

Table 4 presents the classification rate of using asymmetric motion features and other published methods on three different datasets. As can be seen, the asymmetric motion features provide the highest accuracy on both the UCF and HOHA datasets. On the KTH dataset, our 93:7% accuracy is comparable with 94:2% [29] accuracy obtained using joint dense trajectories and dense sampling which require much more computation time and memory compared to our sparse features. In a comparable setting with Wang et. al [BMVC 2009], the asymmetric motion features perform better than other salient features and dense sampling. This is in contrast to the previous observations about better performance of dense sampling/trajectories, showing the importance of effective temporal filtering for robust and quality salient feature detection.

Table 4: Asymmetric motion features can better encode the human actions and hence, they provide the highest classification accuracy. These features compete with computationally-expensive dense sampling and dense trajectories.

 
CONCLUSION

We introduced three asymmetric temporal filters for motion-based feature detection which provide more precise and more robust salient features than the widely used symmetric Gabor filter. Moreover, these features provide higher classification accuracy than symmetric motion features in a standard base-line discriminative framework. We also compared different salient structured-based and motion-based feature detectors in a common discriminative framework for action classification. Based on our experimental results, we recommend the use of asymmetric motion filtering for effective salient feature detection, sparse video content representation, and consequently, action classification.

References

1. Shabani, A. H., D. A. Clausi, and J. S. Zelek, “Improved Spatio-temporal Salient Feature Detection for Action Recognition“, British Machine Vision Conference, University of Dundee, Dundee, UK, August, 2011.
2. Shabani, A. H., J. S. Zelek, and D. A. Clausi, “Robust Local Video Event Detection for Action Recognition”, Advances in Neural Information Processing Systems (NIPS), Machine Learning for Assistive Technology Workshop, Whistler, Canada, December, 2010. Details
3. Shabani, A. H., J. S. Zelek, and D. A. Clausi, “Human action recognition using salient opponent-based motion features“,7th Canadian Conference on Computer and Robotic Vision, Ottawa, Ontario, Canada, pp. 362 – 369, March, 2010. Details
4. Shabani, A. H., D. A. Clausi, and J. S. Zelek, “Towards a robust spatio-temporal interest point detection for human action recognition“, IEEE Canadian Conference on Computer and Robot Vision, Kelowna, BC, Canada, Kelowna, British Columbia, Canada, pp. 237-243, February, 2009. Details

Related people

Directors

Students

Alumni

Students

Supervision – Current

PDF

ph.D.

M.A.Sc.

Supervision – Completed

PDF

ph.D.

M.A.Sc.

Currently, I am the Associate Dean – Research & External Partnerships within the Faculty of Engineering, responsible for all research related activities in the largest Engineering school in Canada.

Research interests

My research interests are in the fields of computer vision, image processing, and pattern recognition with an emphasis on the automated interpretation of SAR (synthetic aperture radar) and sports analytics (ice hockey).  I am an active interdisciplinary and multidisciplinary researcher, publishing refereed journal and conference papers in the diverse fields of remote sensing, sports analytics, computer vision, algorithm design, and biomechanics. My research efforts have led to successful commercial implementations including creating and selling a startup company (CREZ).

Students

Supervision – Current

PDF

Ph.D

M.A.Sc

Supervision – Completed

PDF

Ph.D

M.A.Sc