Contemporary communication networks – from cloud data centers to 5G and IoT – are vastly complex, heterogenous, and dynamic. Manual tuning of such networks becomes impractical. Autonomous networks (alternatively referred to as cognitive networks) are envisioned based on AI-based automation: systems that self-configure, self-heal, and self-optimize in real time. Here, machine learning (ML) is a fundamental network automation facilitator, offering data-driven routing, traffic engineering, fault management, and so on. Yet conventional supervised ML depends on labeled training data, usually not practical networking: it’s time-consuming to get good labels (e.g. “normal” vs “attack” traffic), and the network changes too quickly for static labels to continue to hold.”. Unsupervised ML – that learns structure from unlabeled data – circumvents this drawback. Current surveys report an increasing tendency of using unsupervised learning on raw network data.
For instance, Usama et al. remark that unsupervised techniques can enhance network services like traffic engineering, anomaly detection, application classification, and QoS optimization without labels. Similarly, industry analyses emphasize that unsupervised ML excels when labeled data are scarce or expensive: it can automatically detect patterns and anomalies in raw data. In short, unsupervised learning fits the network automation paradigm by mining the vast amounts of operational telemetry and logs that lack ground-truth annotations. This paper provides a technical overview of unsupervised ML in network automation. We start by revisiting the fundamental principles and techniques of unsupervised learning (Section III).
We then present prominent use cases in networking: Anomaly Detection, Traffic Classification, and Performance Optimization (Section IV), providing concrete models and state-of-the-art outcomes in each domain. We emphasize state-of-the-art research (e.g. deep clustering and generative models in networking) and industrial cases (including industrial deployments). Finally, we present current research directions, outstanding challenges (e.g., interpretability and data drift), and future possibilities (Section V–VII). We support everything with relevant academic and industry literature throughout, for technical thoroughness.
Technical Background
Unsupervised learning is an area of ML concerned with identifying latent structure in unlabeled data. In contrast to supervised learning (which learns a mapping from inputs to pre-existing output labels), unsupervised algorithms can only act on input measurements, finding patterns or compressing information without outside supervision. When applied to networks, this implies learning from raw measurements (e.g., packet counts, flow statistics, logs) without pre-classification of the measurements.
Some typical unsupervised tasks and methods are:
Clustering: Aggregation of similar data points into groups according to feature similarity. Examples include k-means, hierarchical clustering, DBSCAN, spectral clustering, and Gaussian mixture models. Clustering in networks may aggregate flows or network elements by traffic profile or performance metrics.
Dimensionality Reduction / Embedding: Discovering concise feature representations of high-dimensional data. Principal Component Analysis (PCA) and autoencoders (deep neural networks that encode/decode) are commonly employed. For instance, autoencoders can discover a low-dimensional encoding of traffic data that isolates normal patterns and makes anomalies emerge as reconstruction errors.
Generative Models: Neural networks that learn to represent data distributions. Variational Autoencoders (VAE) and Generative Adversarial Networks (GAN) can be trained to produce realistic network data, or identify abnormal samples. Deep Belief Networks (stacked RBMs) are also in this category. They have the ability to learn complex, non-linear structure in traffic or packet features.
Self-Organizing Maps (SOM): A form of neural network that maps high-dimensional data onto a 2D grid, placing similar inputs near adjacent nodes. SOMs have been applied to traffic pattern visualization and network traffic clustering.
Graph and Spectral Methods: Casting the network or traffic flow as a graph, spectral methods (e.g., Laplacian Eigenmaps, graph convolutional nets) may cluster or embed nodes/flows by connectivity or similarity.
These unsupervised techniques enjoy multiple benefits in network environments. Perhaps most critically, they lack the need for labeled data, bypassing expensive human labeling and enabling models to learn from changing traffic. They can take advantage of the large volume of unstructured telemetry (e.g. raw packet traces, NetFlow logs) to identify anomalies or patterns that supervised models may not catch. (For example, a clustering model can alert on an unfamiliar traffic pattern never seen in the training data.) Additionally, unsupervised models tend to need less hand-engineered features: numerous can consume raw feature vectors and develop beneficial abstractions on their own.
Key Algorithms:
In real-world practice, a number of unsupervised algorithms have been widely used in networking:
- K-means and its variants (centroid clustering) for clustering traffic streams or nodes.
- DBSCAN / density-based clustering to detect arbitrary-shaped clusters and mark outliers.
- Hierarchical clustering (agglomerative or divisive) for multi-scale clustering.
- Autoencoders / PCA to determine the “normal” subspace of data and quantify anomaly by reconstruction error. Deep autoencoders (notably VAEs) are widely employed to manage non-linearities.
- Isolation Forest and Local Outlier Factor for detecting outliers in high-dimensional feature spaces (e.g., fine flow features).
- Self-Organizing Maps (SOM): for visual clustering of traffic into a topological grid, which can display patterns and outliers.
By virtue of these and other methods, unsupervised learning can automatically learn discovery of traffic classes or problem states in the absence of supervision. For instance, an industry report points out that unsupervised ML can “act on unlabeled data using algorithms that are particularly good at detecting patterns and locating anomalies”.
Use Cases
Anomaly Detection
Unsupervised learning is extensively applied to identify network anomalies (new attacks or failures) exactly because anomalies tend to be by definition unlabeled and infrequent. In network intrusion detection (IDS), fault management, or fraud detection, supervised models can discover merely those attack types encountered in labeled training data. Unsupervised approaches, on the other hand, can mark outliers or new patterns that vary from typical traffic behavior.
Standard unsupervised anomaly detectors are:
Clustering-Based Detection: Cluster first into common patterns; afterwards, points distant from any cluster (or in very small clusters) are labeled anomalous. For instance, k-means or DBSCAN on flow feature vectors can discern novel flows as outliers.
Autoencoder/Dimensionality Reduction: Train an autoencoder on typical traffic data so that it can learn to reconstruct common inputs. During test time, flows or traffic records that the autoencoder reconstructs with high error are detected as anomalies. Variational Autoencoders (VAEs) have been suggested to more closely model distributions of traffic features.
Tree-based Outlier Detection: Algorithms such as Isolation Forest construct random trees to separate points; anomalies (which are “simpler to separate”) end up nearer to root and can be scored. This approach is efficient for high-dimensional network data. In a web traffic experiment, an Isolation Forest model obtained ~92% F1-score, marking 95% of anomalies.
One-Class SVM / OCSVM: Kernel-based approach that learns the boundary of “normal” class within feature space and marks outlying points as anomalies.
Deep and Hybrid Models: In recent times, anomaly detection has employed deep networks. For example, a self-organizing maps-augmented VAE (SOVAE) was proposed for network IDS: a SOM groups features into group labels, and a VAE that has been trained on these improved features learns normal traffic, which identifies new intrusions accurately.
These unsupervised detectors have been found to be practical. For example, Huang et al. (2025) report that their SOVAE model achieved an F1 score of 0.983 on the UNSW-NB15 dataset and 0.875 on CICIDS2017, substantially outperforming baseline unsupervised methods. Similarly, a recent study of web server logs showed an Isolation Forest detector could achieve ~95% precision and 90% recall on injected anomalies. In conclusion, since they do not need pre-labeling of attack traffic, unsupervised detectors are able to rapidly learn about new threats.
Traffic Classification
administrators frequently have to determine the application or service that is producing traffic (VoIP, video, P2P, etc.) for policy and QoS purposes, but there might not be available labeled packet traces. Unsupervised ML can cluster or classify flows based solely on observed features. For instance, one experiment deployed deep clustering on encrypted IoT traffic: an adversarial autoencoder was employed to learn the embeddings of flow statistics, and a clustering layer mapped flows into groups corresponding to various IoT services. This method allowed classification of traffic with no manually labeled training set.
Other unsupervised classification instances are:
Flow Clustering + Labeling: Cluster flows into clusters through statistics (e.g. packet size histograms, timing) and then deduce cluster labels by external knowledge. Du et al. (2025) suggest such an approach: first, flows are coarsely clustered, and afterwards a graph-matching method maps new clusters against known classes with ~96.8% accuracy in experiments.
Self-Organizing Maps: Traffic patterns are projected onto a 2D SOM grid; similar application patterns fire neighboring neurons. Analysts can then examine cluster centroids to find services, or even automate label assignment to SOM cells.
Prototype Learning: Algorithms such as Gaussian Mixture Models or spectral clustering can identify hidden “service” clusters in traffic data.
Feature Learning + K-means: Deep embeddings (such as from an autoencoder) denoise flow features, and then k-means clusters the embeddings into classes, usually uncovering latent applications.
These methods address issues such as encryption and dynamic service behavior. For example, Du et al.’s graph-clustering pipeline reached 96.8% classification accuracy on live traffic by relating clusters with application profiles. In the real world, operators can use unsupervised classifiers to on-the-fly mark traffic and modify QoS rules, even while new apps emerge.
Performance Optimization
Unsupervised learning also facilitates network performance optimization (e.g. routing, resource allocation, QoS tuning) through patterns in traffic load and network state. For instance, carriers must forecast congestion and adjust resource policies. Orange described a hybrid method where mobile network cells were segmented by performance KPIs (by means of a latent block model), with the groups indicating high-, medium-, and low-congestion cells. Then one prediction model per cluster attained over 90% accuracy in one-hour congestion prediction. This clustering enabled the operator to mitigate false alarms and enhance QoS by handling each group differently.
Another instance is adaptive routing in dynamic networks. Node mobility can result in frequent path changes in mobile ad-hoc or vehicular networks. An unsupervised method pre-clusters nodes by “mobility resilience” (the stability of their connections) and subsequently employs a reinforcement-learning (RL) agent to select paths made of the most stable nodes. Suh et al. (2025) demonstrated that such UL+RL strategy considerably enhances packet delivery (increased by ~35%) and saves energy compared to RL alone for high mobility. Unsupervised clustering effectively minimizes the routing complexity by concentrating on stable subnetworks.
Other performance-oriented applications are:
QoS Class Discovery: Clustering delay/throughput demand flows to define automatically the service classes. It is possible for unsupervised models to divide flows into prototypes with analogous QoS requirements without thresholds specified by a human.
Traffic Engineering: Discovering common patterns of demands by clustering link load vectors, and then applying those prototypes to adjust routing or capacity schedules.
Self-Organizing Networks: In SDN-oriented or self-organizing designs, unsupervised ML can continuously profile slices of the network or tenants of slices, adapting resource allocation as a function of learned load or behavior categories.
In general, by uncovering hidden structure in performance metrics, unsupervised ML allows for more adaptive control policies. Key benefits include mitigating manual parameter tuning and allowing proactive adaptations (e.g., reconfiguring a QoS policy when a particular traffic cluster increases).
Current Research Directions
Work on unsupervised learning for networks is also fast-changing, with a number of promising avenues:
Deep Clustering and Representation Learning: Merging deep neural networks and clustering tasks. Techniques such as DeepCluster learn feature embeddings and cluster labels simultaneously; in networking, they can generate more discriminative traffic features automatically. For example, adversarial training with deep clustering has been adopted to learn robust embeddings of IoT traffic. Self-supervised methods (e.g. contrastive learning) are also starting to be explored, trying to enhance feature learning without labels.
Advanced Generative Models: GANs and VAEs are being adapted to network data. The SOVAE model is representative of this direction by integrating SOM with VAE. Subsequent models could integrate temporal/spatial dimensions (e.g. spatio-temporal autoencoders for mobile traffic), or employ normalizing flows for improved density estimation of network characteristics.
Graph Neural Networks (GNNs): The natural graph structure of given networks makes GNNs a natural unsupervised model. Node or flow graphs (with edges between shared servers or IP subnets) can be embedded through graph autoencoders, for instance. Graph clustering can be used to find communities of flows or devices. Initial work such as Du et al.’s graph-matching classifier suggests this possibility. Graph contrastive learning is an active area of research for network anomaly detection and finding traffic patterns.
Human-in-the-Loop and Explanatory Unsupervised ML: Unsupervised models tend not to be interpretable. Techniques such as EXPLAIN-IT (Morichetta et al.) are beginning to provide explanations for clustering outputs (e.g. explain why flows were clustered together). This is important for network ops, where an engineer will have to rely on the model’s output. Hybrid methods that incorporate regular human inspection to label clusters that were found are also being explored.
Federated and Distributed Unsupervised Learning: With network telemetry going distributed (e.g. base stations, edge devices), federated unsupervised learning is a new concept. Models may cluster data locally and pool patterns without sharing unprocessed data, maintaining privacy while gaining from global structure.
Integration with Reinforcement Learning: With increasing interest in using unsupervised learning alongside RL in an end-to-end manner for network control, there are several examples of using unsupervised clustering to define the state space or options of an RL agent, such as in the UL+DRL routing example. However, on the other hand, unsupervised feature learning can enhance sample efficiency in RL by giving improved representations of network state.
Unsupervised Anomaly Explanation and Causality: Beyond detection, research is moving toward diagnosing anomalies. Causal inference and graph analysis could help identify root causes of anomalies detected by unsupervised methods. Work on counterfactual reasoning in networks may also emerge.
Overall, the frontier is pushing beyond traditional clustering: deep and hybrid methods are being designed specifically for networking. The goal is to build unsupervised models that are robust to non-stationary traffic, automatically interpret their findings, and integrate smoothly into automated decision workflows.
Challenges and Future Work
Despite the promise, applying unsupervised ML in network automation faces challenges. First, there is a problem with evaluation and validation without labels: how do we define the “correctness” of clusters or anomaly scores? Quite often one must resort to proxy metrics or expert labeling. Such uncertainty can slow down deployment of fully unsupervised systems. Scalability is another problem: network data is large and high-dimensional, and so unsupervised models need to be efficient and incremental. Certain methods (such as deep autoencoders) need to be carefully tuned to prevent overfitting noise.
Another challenge is interpretability. Clusters generated by an algorithm might not represent useful classes unless projected by domain knowledge. As explainable ML literature remarks, unsupervised outputs might be black boxes, with new explanations tools needed to understand why flows were clustered or flagged. This is related to trust and accountability: unsupervised techniques can create spurious patterns if not closely monitored. Industry watchers warn that unsupervised ML “takes more expertise” and that its outputs can be “less reliable” than those from supervised classifiers. Human-in-the-loop architectures and confidence indicators therefore tend to be necessary.
Data problems also arise: networks evolve over time (concept drift), and thus unsupervised models need to adapt. For instance, an autoencoder learned on yesterday’s traffic may misclassify today’s valid but new patterns. Online learning and adaptation are significant areas of research. In addition, producing ground truth for benchmarking is itself difficult, so more realistic simulated data or field trials must be used to compare unsupervised approaches.
Lastly, there is an operational issue of integration. Unsupervised ML needs to integrate with network control systems (SDN controllers, orchestration platforms) in real-time. Standardized pipelines from raw logs to model inference to automated action must be supported. Industry roadmaps (such as Cisco’s vision of a “single source of truth” about network state) propose eventual movement toward ML-augmented orchestration, but making unsupervised insights operational is still work in progress.
We anticipate these difficulties being solved in the future by hybrid solutions (human-assisted or semi-supervised ML), ongoing learning frameworks, and improved tooling for ML on networks. As unsupervised methods mature, they will increasingly contribute to the development of genuinely self-driving networks.
References
Usama, Muhammad, et al. “Unsupervised Machine Learning for Networking: Techniques, Applications and Research Challenges.” IEEE Communications Surveys & Tutorials, 2019 (arXiv:1709.06599).
Lewotsky, Kurt. “How Unsupervised Machine Learning Benefits Industrial Automation.” A3 Predictive Maintenance Automation Insights, Mar. 2022.
Chua, Wilson, et al. “Web Traffic Anomaly Detection Using Isolation Forest.” Big Data Analytics and Intelligence, 2023 (MDPI).
Huang, Lingjuan, et al. “Self-Organizing Maps-Assisted Variational Autoencoder for Unsupervised Network Anomaly Detection.” Symmetry, vol. 15, no. 4, 2025, doi:10.3390/sym15040739.
Zhang, Weijie, et al. “Intelligent Unsupervised Network Traffic Classification Using Adversarial Training and Deep Clustering for Secure IoT.” Future Internet, vol. 15, 2023 (MDPI).
Du, Haigang, et al. “A Clustering-Based Approach for Classifying Data Streams Using Graph Matching.” Journal of Big Data, 2025.
Alliot, Arthur. “Combining Clustering and AI for Congestion-Free Mobile Networks.” Hello Future (Orange R&D), June 2024.
Suh, BeomKyu, et al. “A Resilient Routing Protocol to Reduce Update Cost by Unsupervised Learning and Deep Reinforcement Learning in Mobile Ad Hoc Networks.” Electronics, vol. 14, no. 1, 2025.
Cisco Systems. Single Source of Truth in Network Automation. Cisco White Paper, 2024.