Git migration and submodules

Hi,

Kate and Digikam seems to have similar problems to solve when it comes to the
git migration.

Below is an informative response to my idea to use submodules like it is
planned in Digikam.

On Tuesday 21 December 2010 17:55:44 Andreas Pakulat wrote:

> On 21.12.10 15:54:23, Kåre Särs wrote:
> > I have not used git much, but if I'm correctly informed, it is possible
> > to create a 'superproject' that has submodules that point to different
> > git repos.
> >
> > Would this not solve the problem? Kate and kdelibs could point submodules
> > to the same katepart repo with some added logic in the Kate to figure
> > out if the kpart should be compiled...
> >
> > I got the idea from the kde-imaging/digikam mailings list where they are
> > discussing the move to Git.
>
> As Ian said, submodules are anything but easy. In particular one needs
> to be really careful during setting up commits in the parent repository
> and even more careful when working inside a submodule (as usually its at
> a specific revision and not on a branch-head. So comitting will not put
> that commit into any branch).
>
> We're using submodules at work for one project and its still something
> we want to eliminate in the mid-term as we do mess up history in either
> module from time to time and need to rebase stuff and force push to the
> shared repository.
>
> Oh, last but not least, the kde-scm list has long acknowledged that
> submodules in git are not up to the task of grouping repositories for
> KDE. Thats one of the main reasons we're now having a software on
> projects.kde.org that allows arbitrary nesting of "projects".
>

Andreas: Can you give any links or information on the nesting software on
projects.kde.org?

--
Kåre
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Andreas Pakulat]

On 21.12.10 21:50:20, Kåre Särs wrote:

> Hi,
>
> Kate and Digikam seems to have similar problems to solve when it comes to the
> git migration.
>
> Below is an informative response to my idea to use submodules like it is
> planned in Digikam.
>
> On Tuesday 21 December 2010 17:55:44 Andreas Pakulat wrote:
> > On 21.12.10 15:54:23, Kåre Särs wrote:
> > > I have not used git much, but if I'm correctly informed, it is possible
> > > to create a 'superproject' that has submodules that point to different
> > > git repos.
> > >
> > > Would this not solve the problem? Kate and kdelibs could point submodules
> > > to the same katepart repo with some added logic in the Kate to figure
> > > out if the kpart should be compiled...
> > >
> > > I got the idea from the kde-imaging/digikam mailings list where they are
> > > discussing the move to Git.
> >
> > As Ian said, submodules are anything but easy. In particular one needs
> > to be really careful during setting up commits in the parent repository
> > and even more careful when working inside a submodule (as usually its at
> > a specific revision and not on a branch-head. So comitting will not put
> > that commit into any branch).
> >
> > We're using submodules at work for one project and its still something
> > we want to eliminate in the mid-term as we do mess up history in either
> > module from time to time and need to rebase stuff and force push to the
> > shared repository.
> >
> > Oh, last but not least, the kde-scm list has long acknowledged that
> > submodules in git are not up to the task of grouping repositories for
> > KDE. Thats one of the main reasons we're now having a software on
> > projects.kde.org that allows arbitrary nesting of "projects".
> >
>
> Andreas: Can you give any links or information on the nesting software on
> projects.kde.org?

The software hosting projects.kde.org is called redmine, you should be
able to find lots of information on it via the web. Or simply following
the "Help" link on projects.kde.org.

If you click on "Projects" on projects.kde.org you'll see the current
structure, which is similar to what we have in trunk/ today (with
playground, extragear, kdesupport and KDE). This is all just 'virtual',
the actual git repositories (as visible in gitweb.kde.org) are a
flat list.

The structure on projects.kde.org will also be used to create the KDE SC
releases. So for example even if modules like kdeedu or kdegames are
split up into several git repositories, the KDE SC release will still
ship a kdeedu.tar.bz2 (at least thats the current plan as far as I
know) that assembles all of the sources of the various repositories.

In the same way its planned to supply special files for tools like
kdesrc-build to easily build kdeedu and other split-modules.

I'm sure some of this has been written down in either the
GitKDEOrgManual on community.kde.org or on techbase.kde.org in the "plan
for git migration" page (not sure what the actual title is right now).

Andreas

--
Your temporary financial embarrassment will be relieved in a surprising manner.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

> The structure on projects.kde.org will also be used to create the KDE SC
> releases. So for example even if modules like kdeedu or kdegames are
> split up into several git repositories, the KDE SC release will still
> ship a kdeedu.tar.bz2 (at least thats the current plan as far as I
> know) that assembles all of the sources of the various repositories.

Creating tarballs is fine for the main KDE modules, but in digikam we have a
slightly different problem: There is one application with a number of
dependencies, which will be spread out over 7 repos.

We would like to give the user one repository which, when cloned, has all the
code necessary and builds just out of the box.
It's not about tarballs or releases, it's about users who want to run latest
Git code to see if their problem was fixed by a commit.

There will be additionally some very lightweight CMake glue and a README file
in this toplevel directory, and these file also need to have a place
somewhere.

The problem is that git submodules dont track the latest HEAD of the
referenced repos. The issue is discussed here:

http://jonathan-oliver.blogspot.com/2009/10/git-submodules-like-
svnexternals.html

but, unfortunately, and although you'll find patches with Google, not
implemented by git as of yet.
So currently it seems git-submodules is insufficient for our needs and it's
necessary to provide a script in the toplevel directory which pulls all sub-
repos.

Marcel
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[kunal ghosh]

On 12/22/2010 02:57 PM, Marcel Wiesweg wrote:

>> The structure on projects.kde.org will also be used to create the KDE SC
>> releases. So for example even if modules like kdeedu or kdegames are
>> split up into several git repositories, the KDE SC release will still
>> ship a kdeedu.tar.bz2 (at least thats the current plan as far as I
>> know) that assembles all of the sources of the various repositories.
> Creating tarballs is fine for the main KDE modules, but in digikam we have a
> slightly different problem: There is one application with a number of
> dependencies, which will be spread out over 7 repos.
>
> We would like to give the user one repository which, when cloned, has all the
> code necessary and builds just out of the box.
> It's not about tarballs or releases, it's about users who want to run latest
> Git code to see if their problem was fixed by a commit.

we could have one digikam repository that we can give to the user.
inside the .git/ of the repository we can modify the
post-checkout hook provided by git (This is done in the kde hosted
repository).
The hooks are triggered by git automatically when an event occurs (after
checkout, in the case above).

We would have to modify the "post-checkout" hook and the "post-update"
hook for the starters:

post-checkout : would contain additional git commands to pull in the
dependencies.
post-update   : the same thing as above but would update the
dependencies automatically.

http://www.kernel.org/pub/software/scm/git/docs/githooks.html

> There will be additionally some very lightweight CMake glue and a README file
> in this toplevel directory, and these file also need to have a place
> somewhere.
>
> The problem is that git submodules dont track the latest HEAD of the
> referenced repos. The issue is discussed here:
>
> http://jonathan-oliver.blogspot.com/2009/10/git-submodules-like-
> svnexternals.html
>
> but, unfortunately, and although you'll find patches with Google, not
> implemented by git as of yet.
> So currently it seems git-submodules is insufficient for our needs and it's
> necessary to provide a script in the toplevel directory which pulls all sub-
> repos.
>
> Marcel
> _______________________________________________
> Digikam-devel mailing list
> [hidden email]
> https://mail.kde.org/mailman/listinfo/digikam-devel

--
regards
-------
Kunal Ghosh
Dept of Computer Sc.&  Engineering.
Sir MVIT
Bangalore,India

permalink: member.acm.org/~kunal.t2
Blog:kunalghosh.wordpress.com
Website:www.kunalghosh.net46.net

_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Marcel Wiesweg]

> we could have one digikam repository that we can give to the user.
> inside the .git/ of the repository we can modify the
> post-checkout hook provided by git (This is done in the kde hosted
> repository).
> The hooks are triggered by git automatically when an event occurs (after
> checkout, in the case above).
>
> We would have to modify the "post-checkout" hook and the "post-update"
> hook for the starters:
>
> post-checkout : would contain additional git commands to pull in the
> dependencies.
> post-update   : the same thing as above but would update the
> dependencies automatically.
>
> http://www.kernel.org/pub/software/scm/git/docs/githooks.html

Yes this is pretty interesting. Could be the solution for us.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel

[Andreas Pakulat]

[ Leaving out kwrite-devel as this concerns digikam only ]

Please cc me on replies as I'm not subscribed to the list.

On 22.12.10 10:27:46, Marcel Wiesweg wrote:

>
> > The structure on projects.kde.org will also be used to create the KDE SC
> > releases. So for example even if modules like kdeedu or kdegames are
> > split up into several git repositories, the KDE SC release will still
> > ship a kdeedu.tar.bz2 (at least thats the current plan as far as I
> > know) that assembles all of the sources of the various repositories.
>
> Creating tarballs is fine for the main KDE modules, but in digikam we have a
> slightly different problem: There is one application with a number of
> dependencies, which will be spread out over 7 repos.
>
> We would like to give the user one repository which, when cloned, has all the
> code necessary and builds just out of the box.

How about leveraging projects.kde.org infrastructure and help the
sysadmins get those xml files in place which allow tools such as
kdesrc-build to fetch all of those repositories 'at once with minimal
setup effort'?

> It's not about tarballs or releases, it's about users who want to run latest
> Git code to see if their problem was fixed by a commit.

A user having to setup the compilation of 7 repositories is however a
one-time thing with todays build-tools (not cmake, but the meta-tools
like kdesrc-build). On the other hand, keeping this meta-repository
compiling and working and not letting it bitrot is an every-day effort
for all involved developers.

Not to mention that if the fix is on a particular branch then such a
meta-repository won't help users either, as cloning it and updating its
submodules will always only update them to a specific commit from the
corresponding upstream. So the user still has to change into the
subdirectories and adjust the remote and/or branch manually.

> There will be additionally some very lightweight CMake glue and a README file
> in this toplevel directory, and these file also need to have a place
> somewhere.
>
> The problem is that git submodules dont track the latest HEAD of the
> referenced repos. The issue is discussed here:
>
> http://jonathan-oliver.blogspot.com/2009/10/git-submodules-like-
> svnexternals.html
>
> but, unfortunately, and although you'll find patches with Google, not
> implemented by git as of yet.
> So currently it seems git-submodules is insufficient for our needs and it's

Heh, I guess I should've read this part first before writing the above
comments :)...

> necessary to provide a script in the toplevel directory which pulls all sub-
> repos.

Well, instead of writing your own, it may really make sense to help
sysadmins with those xml files and kdesrc-build (or other tools) to use
them. This could also solve the problem of the README, the text could
simply be part of your projects description and the xml file could
contain that description. This way kdesrc-build can generate a README
while setting up the top-level directory.

Andreas

--
You'll never see all the places, or read all the books, but fortunately,
they're not all recommended.
_______________________________________________
Digikam-devel mailing list
[hidden email]
https://mail.kde.org/mailman/listinfo/digikam-devel