C# Wrapper for CNTK Evaluation

CNTK is a great library for using Neural Networks, but the power and flexibility come at a cost of complexity. Sometimes you just want to hand some data to a trained model and get back an answer.

I wrote a little wrapper that I use for just such a purpose and thought I’d share it here. This class links against the CNTK.GPU NuGet C# package, so you get hardware accelerated evaluation using trained models.


First, the class definition and internal members:

public class Function
 CNTK.Function cntkFunction;
 CNTK.DeviceDescriptor Device;
 CNTK.Variable InVar;
 CNTK.Variable OutVar;
 Dictionary<CNTK.Variable, CNTK.Value> InMap;
 Dictionary<CNTK.Variable, CNTK.Value> OutMap;

These members will need to be used when creating or using the model. Speaking of which, the initialization can be done like so:

public Function(string PathToFunction, string InputLabel, string OutputLabel)
 Device = CNTK.DeviceDescriptor.GPUDevice(0);
 cntkFunction = CNTK.Function.Load(PathToFunction, Device);
 InVar = cntkFunction.Arguments.First(x => x.Name == InputLabel);
 OutVar = cntkFunction.Outputs.First(x => x.Name == OutputLabel);
 InMap = new Dictionary<CNTK.Variable, CNTK.Value>();
 OutMap = new Dictionary<CNTK.Variable, CNTK.Value>();

The GPUDevice(0) refers to the first device; if you are working on a multi-gpu system, you probably don’t need this guide; however, you may separately enumerate and query information about the GPUs available.

The InVar and OutVar simply retrieve the variables from the CNTK model. In CNTK parlance, InVar would be the Features and OutVar would be the labels.

With the above variables already created, evaluating a sample is actually quite easy:

public IEnumerable<float> Evaluate(IList<float> Input)
 InMap[InVar] = CNTK.Value.CreateSequence(InVar.Shape, Input, Device);
 OutMap[OutVar] = null;
 cntkFunction.Evaluate(InMap, OutMap, Device);
 return OutMap[OutVar].GetDenseData<float>(OutVar).First();

We create a sequence, assign it to the InVar item in the dictionary, pass that dictionary to CNTK and return the first value of the result. While CNTK itself is fairly straightforward, the beauty of this wrapper is it simply takes a list of input floats and returns a list of floats. Simple, straightforward and, thanks to the magic of hardware acceleration, fast.

Of course, once you start doing this, you start wanting to evaluate several sequences at once. I’ve found, for instance, that I get a speed improvement when going up to 500-1000 samples in parallel. This is only slightly more complicated:

public IList<IList<float>> Evaluate(List<List<float>> Inputs)
  InMap[InVar] = CNTK.Value.CreateBatchOfSequences(InVar.Shape, Inputs, Device);
  OutMap[OutVar] = null;
  cntkFunction.Evaluate(InMap, OutMap, Device);
  return OutMap[OutVar].GetDenseData<float>(OutVar);

Instead of converting a single set of data to a sequence, we convert several sets to a batch. Like I said, it’s much more efficient to process several sets in parallel, so you’ll want to do this if you can.

Of course, if you have a multi-GPU system you’ll need to make adjustments for that, and you’ll need to change the data type if you’re using doubles instead of floats. However, to quickly get you up and running, this is all you need.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s