#// TODO list
#///////////////////////////////////////////////////////////////////////////////
#//
#//   Copyright (C) 2008   Georg S. Weber
#//
#//   This file is part of the Essen Modular package programs.
#//
#//   The Essen Modular package programs are
#//   free software:
#//   you can redistribute them and/or modify them
#//   under the terms of the GNU General Public License as published by
#//   the Free Software Foundation, either version 3 of the License, or
#//   (at your option) any later version.
#//
#//   The Essen Modular package programs are
#//   distributed in the hope that they will be useful,
#//   but WITHOUT ANY WARRANTY; without even the implied warranty of
#//   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#//   GNU General Public License for more details.
#//
#//   You should have received a copy of the GNU General Public License along
#//   with the Essen Modular package programs,
#//   see the text file COPYING in this directory.
#//   If not, see <http://www.gnu.org/licenses/>.
#//
#///////////////////////////////////////////////////////////////////////////////
#//
#//   history/version (the version is YYYY-MM-DD of the last change)
#//   2008-03-13: creation (gsw)
#//   2008-04-07: first version for composite N (gsw)
#//
#///////////////////////////////////////////////////////////////////////////////



2008-04-07 (Georg S. Weber):
First version to be able to do not only prime power levels N,
but all levels N (>=2).
Currently, "EssenHeilbronnManin" is broken.
The code definitely needs to be cleaned up, especially "BasmajiSequences.c"
tends to be unrreadable, even for myself.
The following points from the past have been adressed:
2008-03-05-1: done
2008-03-05-5: well, this has not been adressed here, but the Magma algo
              alluded to should be surpassed in speed if one uses the
              Cremona algo's way together with the original proof of Merel
The following new points shall be adressed in the future:
1. Do clean up! (Make work "EssenHeilbronnManin" again among many other things)
2. Right now, the "RetrieveK" lookup-mechanism is incomplete.
   It can and should be done better in the prime power case, and lacks
   completely in the composite case. In the composite case one just needs
   to "handpick" the virtual start matrices to obtain a nice result,
   namely that one can plac all the k for different yModN in one single
   N x N matrix. (A rigorous proof is required. I have one, although it's
   ugly.)
3. Right now, the "cut-off points" optimization is incomplete. For N prime,
   it's OK, but for more general N, it will definitely bring speed benefits.
4. This "cut-off points" optimization relies on facts that do not seem to
   have been recognized by Basmaji, or Blady. More precisely, Bladys
   "Lemma 4.1.3" is wrong (counterexample: p == 2213, (x_1,y_1) == (30, 29),
   (x_2, y_2) == (38, 37), then x_1 x_2 + y_1 y_2 == p, but A_1^1 == D_0^2 !).
   Blady's Definiton 4.1.4 is correct as stated though, but misses one point:
   namely, that "double" matrices occur only within Basmaji series of
   length one; i.e the whole series vanishes then, no "shifted-by-one" start
   matrices occur at all
   Todo: write up a correct proof of this Definition 4.1.4
5. Todo: write up the (simple) argumentation why the cut-off points
         optimization works



2008-03-13 (Georg S. Weber):
many rough edges, code cleanup necessary, some documentation/comments missing,
several places still have potential for further speed optimization,
but essentially the code is functional and the speed is good



2008-03-05 (Georg S. Weber):

1.
By now, the level N has to be a pime power.
One obstacle to be overcome is the lacking abilty to produce the set
of representatives of P^1(Z/NZ) in the composite case (two or more
different prime factors), and the algo to "find" the representives.
This has to be resolved together with the question how to find (and
quickly so) the index of the representative of a start matrix of a
Basmaji series. Currently, this relies on N being a prime power, too.

This is most probably the first restriction to be removed in the future.


2.
Right now, only the Basmaji sequences method is implemented.
This is done in a quite different flavor than the original algo of
Jacques Basmaji himself (see his 1996 thesis) or the recent version
of Guido Blady (for his 2007 thesis).
The main difference to their implementations is that now O(N^2) space
is used (and needed for good speed in a more or less essential way)
instead of only O(N) space, and that some superfluous restrictions
are removed (p >> N is not any longer required).
It would be interesting to have a version more in line with the
original Basmaji algo, needing only O(N) space. The two main obstacles
are that on the one hand, we then need a GCD algorithm, so an external
library would be necessary (preferably GMP); and on the other hand,
that in this case it would be nice to profit from Guido Bladys
optimized lookup table algorithms to check whether two numbers are
coprime.

I have something in mind to do the latter with (far) less space
requirements than his implementation, but before having coded and
tested it, it is too immature to speculate about it.


3.
The HeilbronnManin matrices produced by (all of) the Basmaji sequences
algorithms are the ones described in Loic Merel's 1991 paper
"Operateurs de Hecke pour $\Gamma_0(N)$ et Fractions Continues"
(Ann.Inst.Fourier 41, 1, (1991), 519-537) on page 529, right after
"Defintion 3", and denoted there as set $S_m$.
(And which I chose to call "HeilbronnManin" matrices to pin down a name.)
Merel puts them in correspondence to what he there names "lifting of
Heilbronn-Manin", though he notes that his set $S_m$ of matrices is
strictly greater than those considered by Manin ("Parabolic points and
zeta functions of modular curves" Isvestija 1972).
(See also the 1972 Bourbaki report n 414 of Mazur, about the results
of Manin loc.cit., and more.)
John Cremona added in the 1997 Second Edition of his "Algorithms for
Modular Elliptic Curves" in his chapter 2.4 a section "Heilbronn matrices".
There he described (page 22) an algorithm to produce a set of matrices,
let's call it $H-C_p$, which is "almost" the same set as $S_m$, for
$m$ being a prime $p$.
There are three differences, however.
Firstly, instead of a matrix A, one sometimes gets the matrix -A
(which is not at all that important for the time being).
Secondly, the Cremona algorithm (see there) works with r running through
an arbitrary set of representatives modulo p --- only if this set is
chosen to be [-p/2 .. p/2], the resulting set $H-C_p$ remains comparable
to Merels $S_m$. (Since this is the case in the implementations I know of,
it's not a problem either.)
Thirdly, producing a continued fraction expansion, one sometimes may make
choices, all of which lead to valid results (as continued fractions).
Especially choosing the "nearest_integer(a/b)" when 2* a/b is an integer,
but a/b is not:   since then, q := floor(a/b) and q := ceil(a/b) are both
"nearest_integers". However, if one does not follow the conventions
of Merel (Proposition 1 on page 522, loc.cit.: we need to choose q such
that |q - a/b| < 1 and a - q*b > 0 --- I hope I didn't put in a typo here
:-)))),
then one does get different sets of Heilbronn matrices.
Cremona in loc.cit., page 22, gave explicitly a set R_5, the third matrix
of which is $\smallmatrix{2 & 1 \\ 1 & 3}$.
This is clearly not a matrix with the properties described on the bottom
of the page 21 (Cremona, loc.cit.) before, since the entries on the
antidiagonal are both positive!
Using the "Merel" convention on choices in the continued fractions,
on would have gotten instead the matrix:
$\smallmatrix{2 & 1 \\ -1 & 2}$, which has the desired property, that
the entries on the antidiagonal have different signs whenever both of
them are non-zero.

Now it would be interesting to have both versions of Cremonas algorithm
implemented, especially the one producing the same set $S_p$ as the
Basmaji sequence algorithms (all three possible differences noted above
are pretty easy to avoid).


4.
Cremonas Heilbronn-matrix algorithm does not seem to be able to profit
from giving it O(N^2) space instead of the usual O(N) space, apart from
the obvious possibility to use lookup-tables for multiplication mod N
and for the representatives of Manin symbols (given arbitrary (u:v)
with gcd(u,v,N) == 1).
However, it nevertheless then should beat the Basmaji sequences, when
the level N gets larger, say N greater 10^4.

Another chance to experiment.


5.
In his 1994 paper, Loic Merel gave descriptions of yet other sets
of "Heilbronn" matrices. Magma and Sage both have not only the
"HeilbronnCremona" version (with the differences to "HeilbronnManin"
alluded to above), but also a "HeilbronnMerel" version.
At least the Magma version contains an optimized algorithm to
produce them, credited to Allan Steel.

Although these "HeilbronnMerel" sets are considerably larger, it
might be interesting to introduce Guido Bladys lookup-tables in
their calculation and see what speed we can get.
(It is also interesting in itself to have an alternate set of
"Heilbronn" matrices at hand for correctness and plausibility checks,
Just random checks by hand whether the resulting matrices for
different p commute, is a bit unsatisfactory.)


6.
If one goes for larger levels N, and one cannot (or does not want to)
use O(N^2) space, it becomes more and more vital to speed up the
calculation of the representative Manin symbols of given arbitrary ones
(not being able to do just a look-up).
For large N, it also becomes interesting to be able to reduce modulo
the 2-relations and 3-relations already on the "C side", since this
reduces not only storage, but also the amount of data to be passed
to the caller side (e.g. Magma code).
The "graph trick" due to Xiangdong Wang achieves both of this at once.
However, the literature situation on this subject is suboptimal.
In the 1998 paper of Gerhard Frey and Michael M"uller, in Remark 3.10.3,
it is referred to. The thesis of Michael M"uller referenced in there,
which should contain more details, however still remains unpublished,
at least as far as I know.
Michael M"uller explained it to me (oral communication), and William Stein
and I talked about it in 2000 or so, when William stayed in Bonn at the MPI.
In a preliminary version of his book, William had had a note on this,
but this note did not make in the published version of the book.

I think I'll have to implement the "graph trick" of Xiangdong Wang
myself, and if it proves to be worth mentioning, it might make it into
some future edition of Wil's book.


7.
Xiangdong Wang's "graph trick" works also especially well for higher
weights k > 2. The higher the weight, the greater the benefit (since
the weight then "adds" to the complexity, but does not "multiply" it).
But for now, we are restricted to weight k == 2 anyway.

Change this.
(But probably not before I have had the time to learn how to draw nice
pictures of graphs using Sage.)


8.
Currently, we are restricted to Haupttyp, i.e. trivial Nebentyp.

Change this.


9.
The way the Basmaji sequences are implemented right now, they can
be applied not only to Manin symbols w.t.r. to $\Gamma_0(N)$, but
they can identically be applied to the Manin symbols w.r.t. to
$\Gamma_1(N)$, which need O(N^2) space anyway. The implementation
becomes even easier.

Does anybody need this instead of the Magma "MultiChar" feature?

