PROC OLLAMA, Running LLM Locally With SAS code

[GreenCode]

PROC OLLAMA, a SAS procedure written using SAS/toolkit calls llama.cpp to run the large model locally(Windows 11 x64), details: analyserxyz/myrepo1: test

Example code:

options fullstimer;
libname tmplib "c:\temp";
proc ollama out=tmplib.result;
model "c:\temp\gemma-1.1-7b-it.Q4_K_M.gguf";
params "-no-cnv -ngl 99 -p ""Once upon a time""";
run;

Output:

[Vrushin]

is there a way to run this in viya?

[GreenCode]

Unfortunately, there is no SAS/Toolkit module under Viya right now, so this procedure does not work on Viya. It can be replaced by other methods, such as using x, pipe to execute the llama-cli command line to output the result to a file, and then read the result back from the file.

[williamnadolski]

This looks really promising - is there anything that can be done to help prioritize the development of this kind of development for Viya or SAS workbench?

[GreenCode · Accepted solution]

The more people who need this module, the more likely it is that R&D will deliver it as soon as possible, so it is recommended that you or remind all customers who need it to send a request to SAS Technical Support for SAS/Toolkit for Viya 4