[ jEdit-devel ] SF trackers XML export

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

[ jEdit-devel ] SF trackers XML export

Eric Le Lay
Administrator
Dear admins,

have somebody tried the XML export function ?
(http://sourceforge.net/apps/trac/sourceforge/wiki/XML%20export)
Does it produce big files ?
I was looking for a way to browse/analyze open tickets offline and
wondered if XML exports could help...
Anyway, XML exports are restricted to project admins.

Cheers,
Eric

------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
--
-----------------------------------------------
jEdit Developers' List
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jedit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [ jEdit-devel ] SF trackers XML export

Jarekczek
Administrator
The file may be useful. It's 30MB. If you would like to create a tool
that would make a worksheet out of this, it would be very appreciated.
Drop me an email if you want the file. Sf suggests to avoid multiple or
frequent downloads because of server resource consumption. But
performing a conversion say once a week would be beneficial.

Jarek

W dniu 2012-07-25 12:17, Eric Le Lay pisze:

> Dear admins,
>
> have somebody tried the XML export function ?
> (http://sourceforge.net/apps/trac/sourceforge/wiki/XML%20export)
> Does it produce big files ?
> I was looking for a way to browse/analyze open tickets offline and
> wondered if XML exports could help...
> Anyway, XML exports are restricted to project admins.
>
> Cheers,
> Eric
>
> ------------------------------------------------------------------------------
> Live Security Virtual Conference
> Exclusive live event will cover all the ways today's security and
> threat landscape has changed and how IT managers can respond. Discussions
> will include endpoint security, mobile security and the latest in malware
> threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/



------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
--
-----------------------------------------------
jEdit Developers' List
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jedit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [ jEdit-devel ] SF trackers XML export

Jarekczek
Administrator
Eric, awesome job! After weeks I could possibly convert this xml into
something useful, but definitely not in the perfect way you did it.

So Eric converted tracker entries into html files. We are going to
recreate these htmls periodically and publish. Only details are left to
determine.

Eric, the htmls you have are already stripped of the sensitive data,
aren't they? So this material is ready to publish, I mean the "res"
directory.

In what place do you see it? jedit.org/trackers?

Is the history really needed? I would be satisfied with bare headers,
anything more is great but not so crucial. IP addresses must be
definitely removed, these trivial emails may stay.

I think to publish it the fastest possible so that everyone could see
that. Say when it's ready!

Jarek

W dniu 07/27/2012 05:40 PM, Eric Le Lay pisze:

> Hi Jarek,
>
> here's the result...
> I produced a set of html files because I prefer it to a WorkSheet.
>
> It still lacks the history in tickets.
> And all the attachments are online (they are not part of the export and
> I don't really measure what size they would be if we downloaded them
> all).
>
> About sensitive info, I've noticed the project members emails in
> cleartext (but it's [hidden email]: no big news), and maybe IP
> addresses in history items, but I didn't dig more (search for
> field_name=IP).
> I'd like to find a way to strip from all sensitive info, so
> that we could provide a link to the latest export to grab when
> going offline. filter.sh keeps only the trackers content, but as I
> said there are these IP history items to remove, I think.
>
>
> Regards,
> Eric
>
> Le Wed, 25 Jul 2012 14:07:41 +0200,
> Jarek Czekalski <[hidden email]> a écrit :
>
>> Hi Eric
>>
>> That's it. Please don't publish it as a whole as it contains some
>> sensitive data, althout I don't remember what exactly :)
>>
>> Regards
>> Jarek
>>
>> W dniu 2012-07-25 13:41, Eric Le Lay pisze:
>>> Hello Jarek,
>>>
>>> Yes, I would be interested :-)
>>>
>>> I've seen
>>> http://trac.edgewall.org/browser/trunk/contrib/sourceforge2trac.py
>>> which works with the sf dump.
>>> Otherwise, I'll try to write an XSLT directly...
>>>
>>> BR,
>>> Eric
>>>
>>> Le Wed, 25 Jul 2012 12:40:24 +0200,
>>> Jarek Czekalski <[hidden email]> a écrit :
>>>
>>>> The file may be useful. It's 30MB. If you would like to create a
>>>> tool that would make a worksheet out of this, it would be very
>>>> appreciated. Drop me an email if you want the file. Sf suggests to
>>>> avoid multiple or frequent downloads because of server resource
>>>> consumption. But performing a conversion say once a week would be
>>>> beneficial.
>>>>
>>>> Jarek
>>>>
>>>> W dniu 2012-07-25 12:17, Eric Le Lay pisze:
>>>>> Dear admins,
>>>>>
>>>>> have somebody tried the XML export function ?
>>>>> (http://sourceforge.net/apps/trac/sourceforge/wiki/XML%20export)
>>>>> Does it produce big files ?
>>>>> I was looking for a way to browse/analyze open tickets offline and
>>>>> wondered if XML exports could help...
>>>>> Anyway, XML exports are restricted to project admins.
>>>>>
>>>>> Cheers,
>>>>> Eric
>>>>>
>>>>> ------------------------------------------------------------------------------
>>>>> Live Security Virtual Conference
>>>>> Exclusive live event will cover all the ways today's security and
>>>>> threat landscape has changed and how IT managers can respond.
>>>>> Discussions will include endpoint security, mobile security and
>>>>> the latest in malware threats.
>>>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
>>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
--
-----------------------------------------------
jEdit Developers' List
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jedit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [ jEdit-devel ] SF trackers XML export

Eric Le Lay
Administrator
Le Fri, 27 Jul 2012 21:11:21 +0200,
Jarek Czekalski <[hidden email]> a écrit :

> Eric, awesome job! After weeks I could possibly convert this xml into
> something useful, but definitely not in the perfect way you did it.
>
thanks :-)

> So Eric converted tracker entries into html files. We are going to
> recreate these htmls periodically and publish. Only details are left
> to determine.
>
> Eric, the htmls you have are already stripped of the sensitive data,
> aren't they? So this material is ready to publish, I mean the "res"
> directory.

yes.

>
> In what place do you see it? jedit.org/trackers?
>

Seems good!
What I'd like to find there is an archive to download, containing either
the raw filtered.xml or the tar-ed html results.
Are you thinking of publishing the unarchived htmls for direct
browsing ?

> Is the history really needed? I would be satisfied with bare headers,
> anything more is great but not so crucial. IP addresses must be
> definitely removed, these trivial emails may stay.
>

I have modified filter.xsl to remove emails and IP addresses and
anything but the trackers contents.
Only hidden information that I did'nt notice in the trackers could be a
problem now.

> I think to publish it the fastest possible so that everyone could see
> that. Say when it's ready!
>

If it's going to be used on one of our servers, I could put the code in
jedit/WHAT_HAVE_YOU in subversion so that others can enhance it and
the server can pull changes directly.


Regards,
Eric

> Jarek
>
> W dniu 07/27/2012 05:40 PM, Eric Le Lay pisze:
> > Hi Jarek,
> >
> > here's the result...
> > I produced a set of html files because I prefer it to a WorkSheet.
> >
> > It still lacks the history in tickets.
> > And all the attachments are online (they are not part of the export
> > and I don't really measure what size they would be if we downloaded
> > them all).
> >
> > About sensitive info, I've noticed the project members emails in
> > cleartext (but it's [hidden email]: no big news), and maybe
> > IP addresses in history items, but I didn't dig more (search for
> > field_name=IP).
> > I'd like to find a way to strip from all sensitive info, so
> > that we could provide a link to the latest export to grab when
> > going offline. filter.sh keeps only the trackers content, but as I
> > said there are these IP history items to remove, I think.
> >
> >
> > Regards,
> > Eric
> >
> > Le Wed, 25 Jul 2012 14:07:41 +0200,
> > Jarek Czekalski <[hidden email]> a écrit :
> >
> >> Hi Eric
> >>
> >> That's it. Please don't publish it as a whole as it contains some
> >> sensitive data, althout I don't remember what exactly :)
> >>
> >> Regards
> >> Jarek
> >>
> >> W dniu 2012-07-25 13:41, Eric Le Lay pisze:
> >>> Hello Jarek,
> >>>
> >>> Yes, I would be interested :-)
> >>>
> >>> I've seen
> >>> http://trac.edgewall.org/browser/trunk/contrib/sourceforge2trac.py
> >>> which works with the sf dump.
> >>> Otherwise, I'll try to write an XSLT directly...
> >>>
> >>> BR,
> >>> Eric
> >>>
> >>> Le Wed, 25 Jul 2012 12:40:24 +0200,
> >>> Jarek Czekalski <[hidden email]> a écrit :
> >>>
> >>>> The file may be useful. It's 30MB. If you would like to create a
> >>>> tool that would make a worksheet out of this, it would be very
> >>>> appreciated. Drop me an email if you want the file. Sf suggests
> >>>> to avoid multiple or frequent downloads because of server
> >>>> resource consumption. But performing a conversion say once a
> >>>> week would be beneficial.
> >>>>
> >>>> Jarek
> >>>>
> >>>> W dniu 2012-07-25 12:17, Eric Le Lay pisze:
> >>>>> Dear admins,
> >>>>>
> >>>>> have somebody tried the XML export function ?
> >>>>> (http://sourceforge.net/apps/trac/sourceforge/wiki/XML%20export)
> >>>>> Does it produce big files ?
> >>>>> I was looking for a way to browse/analyze open tickets offline
> >>>>> and wondered if XML exports could help...
> >>>>> Anyway, XML exports are restricted to project admins.
> >>>>>
> >>>>> Cheers,
> >>>>> Eric
> >>>>>
> >>>>> ------------------------------------------------------------------------------
> >>>>> Live Security Virtual Conference
> >>>>> Exclusive live event will cover all the ways today's security
> >>>>> and threat landscape has changed and how IT managers can
> >>>>> respond. Discussions will include endpoint security, mobile
> >>>>> security and the latest in malware threats.
> >>>>> http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
> >>
>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
--
-----------------------------------------------
jEdit Developers' List
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jedit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [ jEdit-devel ] SF trackers XML export

Jarekczek
Administrator
W dniu 07/28/2012 03:04 PM, Eric Le Lay pisze:
>> In what place do you see it? jedit.org/trackers?
>>
> Seems good!
> What I'd like to find there is an archive to download, containing either
> the raw filtered.xml or the tar-ed html results.
> Are you thinking of publishing the unarchived htmls for direct
> browsing ?

Initially I published the htmls as http://jedit.org/trackers and as
http://jedit.org/trackers/Export.tgz. So it is immediately available to
anyone.
Eric, are you able to upload to directory
/home/project-web/jedit/htdocs/trackers after you login to sf with ftp
client? rsync is a better method but for the start ftp may also do. It
would be easier for us if you arrange the directory yourself.

>
> If it's going to be used on one of our servers, I could put the code in
> jedit/WHAT_HAVE_YOU in subversion so that others can enhance it and
> the server can pull changes directly.

I think a subdirectory of www (in svn) would be a good place to host the
scripts. However our servers currently are not able to run java at all,
so the files must be generated on our machines and then uploaded.

Jarek


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
--
-----------------------------------------------
jEdit Developers' List
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jedit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [ jEdit-devel ] SF trackers XML export

Jarekczek
Administrator
Hi Eric

W dniu 07/29/2012 12:21 PM, Eric Le Lay pisze:
> But I can't get my hands on fresh export.xml and I'd like the
> transformation to be done automatically.

That's sufficient reason for you to become an admin. You need the
access. Considering your jedit experience I don't think anyone would
mind it.

> We can talk about it in September when Eric Berry has found a new build
> server (it would fit in a daily or weekly build I think).

BTW, the hosting is still the up-to-date topic. We have an offer from
Vampire about 120$ a year. There were voices about free hosting for
opensourcers, but nothing confirmed. It requires probably a bit of
investigations and we may have java hosting.

>> However our servers currently are not able to run java
>> at all, so the files must be generated on our machines and then
>> uploaded.
>>
> I've tried to use gestalt-xslt (native program) as an alternative to
> saxon, but it fails with out of memory error.
> xsltproc is there but it is only for xslt 1.0 and I've used xslt 2.0.
> I could rewrite the transforms but it would be a pain.

Let's forget about it at the moment.

>
> Anyway, we would not be able to generate the
> export.xml directly from a script on the server due to sf restrictions
> so offline transformation it is...

I perform automatic backup of jedit resources including tracker xml
file, using wget script with cookies. The problem is that it asks for my
sf account password, that is jedit admin's password. I don't know about
a safe way to store such a password on any server. But maybe such exists.

We will have soon a new tracker from sourceforge, when they finally
convert all the projects to new platform (still called "beta"). If we
are not satisfied with it, we will switch to bugzilla. It will be easier
when they enable account integration, which is to happen in 1-2 months.
Once you created tools for extracting data from export.xml, converting
the tracker items to another platform seems to be quite feasable. All
this comes to a conclusion that processing export.xml is a temporary
solution and may be as well performed offline.

BTW: notice that I send it through the group also, but I don't see any
secret message in your email.

Jarek


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
--
-----------------------------------------------
jEdit Developers' List
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jedit-devel
Reply | Threaded
Open this post in threaded view
|

Re: [ jEdit-devel ] SF trackers XML export

Jarekczek
Administrator
Jarekczek wrote
W dniu 07/29/2012 12:21 PM, Eric Le Lay pisze:
> But I can't get my hands on fresh export.xml and I'd like the
> transformation to be done automatically.

That's sufficient reason for you to become an admin. You need the
access. Considering your jedit experience I don't think anyone would
mind it.
Done.

Jarek
Reply | Threaded
Open this post in threaded view
|

Re: [ jEdit-devel ] SF trackers XML export

Eric Le Lay
Administrator
In reply to this post by Jarekczek
Hi,
just a followup to say I got it, done it, and I've scheduled myself to
do it about every Friday.

Cheers,
Eric

>
> BTW: notice that I send it through the group also, but I don't see
> any secret message in your email.

No secret: just Reply instead of Reply-all
>
> Jarek
>


------------------------------------------------------------------------------
Live Security Virtual Conference
Exclusive live event will cover all the ways today's security and
threat landscape has changed and how IT managers can respond. Discussions
will include endpoint security, mobile security and the latest in malware
threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/
--
-----------------------------------------------
jEdit Developers' List
[hidden email]
https://lists.sourceforge.net/lists/listinfo/jedit-devel