Allow optional precompilation of JiT kernels

[zatkins-dev]

I'm running into a bit of a challenge when it comes to MPM perf tests. For FEM, we can use preloading to force CEED kernels to compile. For MPM, however, we

1. can't use preloading, since it would mess with the swarm values
2. have no way to preload between timesteps, when the operator kernels are rebuilt

Do you think we could add something like, e.g. CeedOperatorSetup, that is a no-op if the operator is set up but forces compilation of JiT kernels if they haven't been yet? That way, we could call that outside the log stage

[jeremylt]

I think the big thing is figuring out what all we want to pre-compile and labeling this as a "you probably don't want this" function. CeedOperatorSetup is a mild collision with a function name in the backend API, so maybe CeedOperatorPrecompileKernels?

[zatkins-dev]

That name seems reasonable! And at least for gen operators, there's already a private function that does setup tasks, I don't know about shared and ref

[jeremylt]

Note to me, gen would need to fall this for it's preconditioning fallback operator

[jedbrown]

I think it shouldn't be necessary to recompile if only the restriction sizes have changed. We can recompile and exclude that cost for profiling or we could have an interface that allowed us to avoid unnecessary recompilation.

Mutation is a common source of logic errors. One idea would be to consume a CeedOperator (have the interface check that the refcount is 1), forwarding the previously-compiled kernels from the old CeedOperator to a new CeedOperator that must be equivalent except for the size/values of the restrictions.

[jeremylt]

The big thing that would require recompilation would be the max number of points per element. I think there's some opportunities for optimization on the Ratel side that get much easier if we have the ability to consume and recreate a CeedOperator.