Need to change:

• [x] \lambda to \Lambda (lowercase to uppercase) (Typically, \Lambda uppercase denotes the spectrum, while \lambda lowercase denotes an eigenvalue.)
• [x] "$\lambda(\cdot)$ is the eigenvalue of $\cdot$," to "$\Lambda(\cdot)$ is the spectrum of a square matrix,"
• [x] "$\alpha$ is the maximum real part of eigenvalues of $M$." to "$\alpha(\cdot)$ is the spectral abscissa of a square matrix, i.e., the maximum real part of its eigenvalues."
• [x] For imaginary part, \Im, in papers, I generally redefine this from the standard symbol to just "Im", e.g., "Im(A+BXC)", as I think this is usually much clearer. But this is a debatable suggestion.

Machine Learning News: https://groups.google.com/g/ml-news

SIAM: https://www.siam.org/membership/activity-groups/detail/computational-science-and-engineering https://www.siam.org/membership/activity-groups/detail/data-science https://www.siam.org/membership/activity-groups/detail/imaging-science https://www.siam.org/membership/activity-groups/detail/optimization

General things:

• [x] #13
• [x] #20
• [x] What is the deal with X_struct and the separation of variables as opposed to just an input vector?
• [x] #14

Gradient stuff:

• [x] Does the AD enabling have to happen in comb_fn that the user provides?
• [x] #15
• [x] Is adding support for the user to choose between explicit and AD gradients useful? Might be a bit of work to support this though!

Line search stuff

• [x] What is the deal with disabling code 6 termination and how is the BFGS update then ensured to not lose positive definiteness?
• [x] What is the deal with the backtracking line search? Same issue with positive definiteness and this may not work well for nonsmooth problems. The GRANSO line search is a key component of the BFGS-SQP algorithm.
• [x] #16

I went through the private folder and saw that we have four different types of files to update. I did some sample updates so you (a) can review to see if you agree and like my changes and (b) have templates to work from. The types are:

1. direct port from GRANSO (see bfgsDamping.py)
2. direct port from URTM / GRANSO (see isAnInteger.py)
3. port from GRANSO with changes (see linesearchWeakWolfe.py)
4. entirely new file and code (see getCiGradVec.py)

Note that I added a GPL header block linesearchWeakWolfe.py (and bfgsHessianInverse.py) as these individual routines were derived from HANSO which is GPL. But don't add this headers to any other files.

Overall, I tried to make things as compact as possible and only use one AGPL license block. I also eliminated some lines I thought were unnecessary or would be annoying to continually update for future releases. For example, maintaining version numbers in every file is tedious work, so I only put version numbers in the main user-facing routines, i.e., granso, gransoOptions, and gransoOptionsAdvanced.

p.s. Let me know what you think and if you have suggestions! I definitely went back and forth on how to clearly do all of this.

Regarding AUTHORS.md:

1. Do you think we need or want such a detailed file? I was thinking we just I wrote GRANSO and you ported it to Python and added PyTorch support and more example.
2. I wasn't sure how you determined the file groups, as they look different from what I see marked as direct port or not in the private folder.

Probably we should remove a level of indentation in the docstring for each file, since as is, we have double the normal amount of indentation. Obviously, this is a super minor point and not critical.

I'm going to replace in the dummy class with dict for the v1.0.0 release.

How should I define the dict for the opts so that the values in dict can have different data type?

What does "Please see the notes for the data structures used in PyGRANSO" refer to? Is it another document somewhere? Or is this self-referential, i.e., referring to the "Notes on Documentation" section in the readme that this phrase appears in?

I just updated all examples in the ./example folder. Maybe we can find a time to carefully review the README.md tomorrow

Please let me know if there is any other thing to be updated in the source code before the initial public release.

1. "zero dependency on proprietary packages": we should discuss this since GRANSO doesn't really have this because it could be run using OCTAVE
2. Role of the NCVX name in the PyGRANSO package. NCVX and PyGRANSO in the JMLR paper

• [x] #9

• [x] #10 optimization in machine learning" be for NCVX or PyGRANSO?

• [x] Relatedly, is the title of the paper going to change?

• [x] #12

Depending on the answers to these questions, we'll need to change the docstrings in all the files.

https://github.com/sun-umn/PyGRANSO/blob/main/pygranso/private/qpTerminationCondition.py#L226

Use Hreg = torch.tensor(V) @ torch.diag(dvec) @ torch.tensor(V).T instead

1. For avoiding unnecessary AD gradients, the GRANSO line search could be modified so that the AD gradient is only computed on demand, i.e., only compute the gradient when sufficient decrease holds

2. Adding the MPI parallel full BFGS Python code Tim has