I am a research assistant in the area of AIOps (Artificial Intelligence for IT Operations) at the group of Distributed and Operating Systems DOS at Technical University Berlin, supervised by Prof. Dr. Odej Kao. Currently I am leading the Joint Innovation Hub between Technical University Berlin and Huawei. My work covers broad domain of interdisciplinary topics including machine learning (with particular focus on meta learning and multi-label classification) and IT operations (log analysis for reliability engineering). I am an external collaborator with the Department of Knowledge Technologies at Institute Jožef Stefan (IJS), Ljubljana, Slovenia. I have obtained my Master Thesis at IJS under supervision of Dr. Dragi Kocev and co-supervision of Prof. Dr. Saso Džeroski. Previously, I received my Bachelor in Robotics and Control System Engineering at the Automation System Institute at FEEIT, Skopje, North Macedonia.

Research

My research interest relates to developing novel machine learning methods in the area of log analysis. Some of the learning tasks include intelligent coding, and anomaly detection. Furthermore, I am also interested in explaining the strengths and weaknesses of ML methods through the lens of comprehensible meta learning from complex data types (see as an example my line of studies on the task of multi-label classification).

Innovations

1) Patent 1: DISTRIBUTED TRACE ANOMALY DETECTION WITH SELF-ATTENTION BASED DEEP LEARNING

Journals

A catalogue with semantic annotations makes multilabel datasets FAIR Bogatinovski, J., Kostovska, A., Džeroski, S., Kocev, D., & Panov, P. (2022), Nature Scientific Reports doi.org/10.1038/s41598-022-11316-3

Explaining the Performance of Multi-label Classification Methods with Data Set Properties Bogatinovski, J., Todorovski, L., Džeroski, S., & Kocev, D. (2022) International Journal of Intelligent Systems doi:10.1002/int.22835

Comprehensive comparative study of multi-label classification methods Bogatinovski, J., Džeroski, S., & Kocev, D. (2022) Expert Systems with Applications doi.org/10.1016/j.eswa.2022.117215

Less is more: Selecting the right benchmarking set of data for time series classification Eftimov T., Petelin G., Cenikj G., Kostovska A, Ispirova G., Korosec P., Bogatinovski, J. (2021) Expert Systems with applications doi.org/10.1016/j.eswa.2022.116871

Unification of the evaluation measures for multi-label classification. Eftimov, T., Petkovič, M., Bogatinovski, J., & Kocev, D. (2022)
(submitted)

Benchmarking MLC methods: data from a comprehensive set of experiments Bogatinovski, J., Kostovska A, Tomasic A., Džeroski, S., Kocev, D., Panov P. (2022) (submitted)

Conference proceedings

Leveraging Log Instructions in Log-based Anomaly Detection
Bogatinovski J., Madjarov G., Nedelkoski S., Cardoso J., and Kao O., In 18th Conference on Service Computing, Terassa, Barcelona, (2022)

QuLog: Data-Driven Approach for Log Instruction Quality Assessment
Bogatinovski J., Nedelkoski S., Alexander A., Cardoso J., Kao O.
In 30th International Conference on Program Comprehension (ICPC ’22), May 16–17, 2022, Virtual Event, USA. doi:10.1145/3524610.3527906

Failure Identification from Unstable Log Data using Deep Learning
Bogatinovski J., Nedelkoski S., Li Wu, J Cardoso, Kao O.
In 2022 20th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID), To Appear. IEEE/ACM, May 2022. arXiv

Self-Supervised Log Parsing
Bogatinovski J., Nedelkoski S., Acker A., J Cardoso, Kao O.
In Proceedings of the European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases, ECML-PKDD 2020
arXiv

Multi-Source Anomaly Detection in Distributed IT Systems
Bogatinovski J., Nedelkoski S.
In Proceedings International Conference on Service Oriented Computing (ICSOC 2020)
arXiv

Self-attentive classification-based anomaly detection in unstructured logs
Nedelkoski S., Bogatinovski J., Acker A., Cardoso J., Kao O.
ICDM 2020: IEEE International Conference on Data Mining (ICDM 2020)
arXiv

Self-Supervised Anomaly Detection from Distributed Traces
Bogatinovski J., Nedelkoski S., Cardoso J., Kao O. 2020
IEEE/ACM 13th International Conference on Utility and Cloud Computing UCC 2020
arXiv

Robust and Transferable Anomaly Detection in Log Data using Pre-Trained Language Models
Bogatinovski J., Ott H., Acker A., Nedelkoski S., Kao O.
In Proceedings of the 43th International Conference on Software Engineering (ICSE 2021) Workshop on Cloud Intelligence 2021 Workshop on Cloud Intelligence 2021
arXiv

Feature extraction for heartbeat classification in single-lead ECG.
Bogatinovski, J., Kocev, D., Rashkovska, A. (2019)
In Proceedings of the 42nd International Convention on Information and Communication Technology, Electronics and Microelectronics - MIPRO (pp. 339–345) Opatija Croatia
Exceptional and outstanding paper award.
IEEE Explore.

Artificial Intelligence for IT Operations (AIOPS) Workshop White Paper
Bogatinovski J., Nedelkoski S., Acker A., Schmidt F., Wittkopp T., Becker S., Cardoso J., Kao O.
arXiv

Multi-source distributed system data for AI-powered analytics
Nedelkoski S., Bogatinovski J., Mandapati AK., Becker S., Cardoso J., Kao O.
In Proceedings of European Conference on Service-Oriented and Cloud Computing (ECSOC 2020)
Zenodo

Superiority of Simplicity: A Lightweight Model for Network Device Workload Prediction
In Proceedings of the 15th Conference on Computer Science and Information Systems (FedCSIS) 2020
Acker A., Wittkopp T., Nedelkoski S., Bogatinovski J., Kao O.
arXiv

MicroDiag: Fine-grained Performance Diagnosis for Microservice Systems
Wu L., Tordsson J., Bogatinovski J., Elmroth E., Kao O.
In Proceedings of the 43th International Conference on Software Engineering (ICSE 2021) Workshop on Cloud Intelligence 2021
HAL

Meta learning for multi target regression
Bogatinovski J., Kocev, D., & Džeroski, S. (2018)
In Proceedings of the 10th Jožef Stefan International Postgraduate School Students’ Conference and 12th Young Researchers

Service to the Community

Organization

1) AIOPS 2021 Co-chaired the Second International Workshop on Artificial Intelligence 2021 alongside Prof. Dr. Roberto Natella, collocated with International Conference on Service Oriented Computing (ICSOC 2021), Dubai, UAE (virtual)

2) FAIR MLC 2021 Tutorial at ECML-PKDD 2021 on the topic of bringing the Findability Accessibility Interoperability and Reusability into the multi-label classification domain

3) AIOPS 2020 Leading the organization team of the First International Workshop on Artificial Intelligence 2020, collocated with International Conference on Service Oriented Computing (ICSOC 2020), Dubai, UAE (virtual)

4) ECML-PKDD 2017, Part of the organization team of the most prestigious European Conference on Machine Learning, Skopje, North Macedonia

5) IEEE ICCA 2017, Ohrid, North Macedonia

Program Committee