inferwire
/
AI·4 min read

nas-torch Searches for Compact Neural Networks on One RTX 3060

A single-author paper pairs a small Transformer with a bee colony algorithm to design compact networks on a consumer GPU, finding a 174,000-parameter CIFAR-10 model in about three hours.

TL;DR

  • nas-torch, an open-source search framework, found a CIFAR-10 classifier with about 174,000 parameters and 84.85 percent accuracy in three hours on an RTX 3060 [^1].
  • The result shows architecture search running on consumer hardware, though hand-designed ResNet-20 still reaches about 91.25 percent with roughly 270,000 parameters [^1].

Background

Neural architecture search (NAS) automates the choice of a network's layers and how they fit together. Early methods were expensive: the reinforcement learning approach of Zoph and Le, in which a controller network proposes architectures and learns from their scores, cost up to 22,400 GPU-days [^1]. Later shortcuts such as ENAS and DARTS cut searches to a few GPU-days, but according to the paper they are tied to specific cell-based templates [^1].

What happened

Romain Amigon of the Université du Québec à Chicoutimi posted the paper to arXiv on July 13 and released the framework, nas-torch, as open source [^1]. It searches in two stages. First, a small Transformer with about 68,000 parameters writes architectures token by token from a vocabulary of 15 building blocks, such as convolutions, residual blocks, pooling and dense layers. It learns by reinforcement from how well each candidate does [^1]. Its reward is accuracy minus a penalty for depth, which steers it toward small networks, and an entropy term pushes it to explore when scores stall [^1]. The best design then seeds an Artificial Bee Colony algorithm, a swarm method in which simulated bees try small changes around promising candidates and abandon regions that stop improving [^1].

To keep the search cheap, each candidate trains for about 10 epochs on 40 percent of the training images and is scored on a separate 10 percent. The test set is used only once, at the end [^1]. On a sample of 15 architectures, this shortcut ranked them much like full 100-epoch training, with a Spearman correlation of 0.721 [^1]. All runs used an NVIDIA GeForce RTX 3060 with 6 GB of memory [^1].

On CIFAR-10, the full hybrid averaged 84.85 percent accuracy across three seeds, with about 174,000 parameters and a search time of about 192 minutes [^1]. Under a matched evaluation budget, random search reached 83.51 percent with about 133,000 parameters in about 29 minutes, and simulated annealing 87.21 percent with about 294,000 parameters in about 109 minutes [^1]. Without the depth penalty, the hybrid reached 86.82 percent at about 229,000 parameters. The bee colony alone, started without the Transformer's blueprint, stalled at 76.19 percent [^1]. Outside image tasks, the framework designed a credit card fraud detector with 4,614 parameters and an F1 score of 0.7178 [^1].

Why it matters

The main result is the cost. The paper puts the search at about 0.16 GPU-days, or roughly 2.08 TFLOPS-days after normalizing for the GPU's compute power, against about 17 for DARTS and over 96,000 for the original reinforcement learning method [^1]. The author attributes that gap to a deliberate trade: nas-torch searches only standard layers rather than the complex cells those methods use, and the paper notes that comparing accuracy with them directly is not meaningful [^1].

The results also show the limits. Simulated annealing beat the hybrid on accuracy and finished faster, though with larger networks, and the author concedes that the small tabular tests don't isolate what the Transformer contributes [^1]. On CIFAR-100, accuracy averaged 52.36 percent, which the paper attributes to its restricted vocabulary of layers. One seed ballooned to about 3.10 million parameters because, the author writes, a validation split of about 50 images per class made the reward too noisy for the depth penalty to hold [^1]. The author presents the work as a proof of concept for automated search on one consumer GPU, not as a bid for state-of-the-art accuracy [^1].

Practical example

Say you need a small image classifier for a device with very little memory, and all you have is a PC with an RTX 3060. With nas-torch, you could let the search run for about three hours instead of designing the network by hand [^1]. Going by the paper, you would get a compact model but give up accuracy compared with hand-designed ResNet-20, so you check the result against your own target [^1]. You would also run a simple baseline: in the paper's CIFAR-10 test, random search came within 1.34 points of the hybrid in about 29 minutes instead of about 192 [^1].

Related gear

We recommend this book because it provides the foundational mathematical and architectural knowledge required to understand the deep learning structures that Neural Architecture Search aims to optimize.

AdvertisementAmazon

Deep Learning

Sources

  1. [1]arXiv — Transformer-Guided Swarm Intelligence for Frugal Neural Architecture Search