Discussion:
How can I download all the packages from a remote repository?
youhaodeyi
2008-03-08 03:14:38 UTC
Permalink
Can I download all the packages from a remote repository to my local
repository? For example, there is a repository:
http://repo1.maven.org/maven2/, I want to download all the content to my
local repository. I want to do this for that I have limitation for accessing
Internet, so I need to do this one time to get as much packages as possible.
--
View this message in context: http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-repository--tp15910759s177p15910759.html
Sent from the Maven - Users mailing list archive at Nabble.com.
Wayne Fay
2008-03-08 03:33:32 UTC
Permalink
For the repo1.maven.org repo, you should use rsync to pull it down.
But realistically, this is very bad idea, as "central" is like 10gb
and 98% of it is stuff you will never use. So you will spend an
enormous amount of time, bandwith, and disk space for things you don't
need.

Instead, you should consider a proxy/repo solution like Archiva, Proximity, etc.

Wayne
Post by youhaodeyi
Can I download all the packages from a remote repository to my local
http://repo1.maven.org/maven2/, I want to download all the content to my
local repository. I want to do this for that I have limitation for accessing
Internet, so I need to do this one time to get as much packages as possible.
--
View this message in context: http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-repository--tp15910759s177p15910759.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
youhaodeyi
2008-03-08 04:59:51 UTC
Permalink
I use this command to download:

rsync -avz http://repo1.maven.org/maven2/ ./

and got the error:

http: Unknown Host
rsh.exe: can't establish connection
rsync: read error: Connection reset by peer
rsync error: error in rsync protocol data stream (code 12) at
/tmp/rsync-2.5.6/io.c(177)
Post by Wayne Fay
For the repo1.maven.org repo, you should use rsync to pull it down.
But realistically, this is very bad idea, as "central" is like 10gb
and 98% of it is stuff you will never use. So you will spend an
enormous amount of time, bandwith, and disk space for things you don't
need.
Instead, you should consider a proxy/repo solution like Archiva, Proximity, etc.
Wayne
Post by youhaodeyi
Can I download all the packages from a remote repository to my local
http://repo1.maven.org/maven2/, I want to download all the content to my
local repository. I want to do this for that I have limitation for accessing
Internet, so I need to do this one time to get as much packages as possible.
--
http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-repository--tp15910759s177p15910759.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
---------------------------------------------------------------------
--
View this message in context: http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-repository--tp15910759s177p15911327.html
Sent from the Maven - Users mailing list archive at Nabble.com.
Wendy Smoak
2008-03-08 05:05:35 UTC
Permalink
Post by youhaodeyi
rsync -avz http://repo1.maven.org/maven2/ ./
As others have mentioned, you'd be much better off running a
repository manager app (Archiva, Artifactory, Proximity) and filling
it up with the things you actually need.

If you really must get the whole thing, check the bottom of this page,
it looks like you can rsync from one of the mirrors:
http://maven.apache.org/guides/mini/guide-mirror-settings.html
--
Wendy
Brian E. Fox
2008-03-08 03:28:55 UTC
Permalink
Get a repository manager (proximity/nexus/archiva/artifactory) to act as
a local proxy. Scraping the entire repository is not being a friendly
netizen and will probably get your ip blocked. There are much better
ways.

-----Original Message-----
From: youhaodeyi [mailto:***@gmail.com]
Sent: Friday, March 07, 2008 10:15 PM
To: ***@maven.apache.org
Subject: How can I download all the packages from a remote repository?


Can I download all the packages from a remote repository to my local
repository? For example, there is a repository:
http://repo1.maven.org/maven2/, I want to download all the content to
my
local repository. I want to do this for that I have limitation for
accessing
Internet, so I need to do this one time to get as much packages as
possible.
--
View this message in context:
http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-
repository--tp15910759s177p15910759.html
Sent from the Maven - Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@maven.apache.org
For additional commands, e-mail: users-***@maven.apache.org
youhaodeyi
2008-03-08 07:04:31 UTC
Permalink
Can I write the dependency with wildcard characters? For example,

<dependency>
<artifactId>*</artifactId>
<groupId>*</groupId>
...
</dependency>

Then I will download all the packages from the repository.
Post by Brian E. Fox
Get a repository manager (proximity/nexus/archiva/artifactory) to act as
a local proxy. Scraping the entire repository is not being a friendly
netizen and will probably get your ip blocked. There are much better
ways.
-----Original Message-----
Sent: Friday, March 07, 2008 10:15 PM
Subject: How can I download all the packages from a remote repository?
Can I download all the packages from a remote repository to my local
http://repo1.maven.org/maven2/, I want to download all the content to my
local repository. I want to do this for that I have limitation for accessing
Internet, so I need to do this one time to get as much packages as possible.
--
http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-
repository--tp15910759s177p15910759.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
---------------------------------------------------------------------
--
View this message in context: http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-repository--tp15910759s177p15911893.html
Sent from the Maven - Users mailing list archive at Nabble.com.
Wayne Fay
2008-03-08 14:56:01 UTC
Permalink
Does that sound like a reasonable thing the Maven developers would
have implemented?? "This project depends on all artifactIds in all
groupIds with all/any versions."

No chance.

Wayne
Post by youhaodeyi
Can I write the dependency with wildcard characters? For example,
<dependency>
<artifactId>*</artifactId>
<groupId>*</groupId>
...
</dependency>
Then I will download all the packages from the repository.
Post by Brian E. Fox
Get a repository manager (proximity/nexus/archiva/artifactory) to act as
a local proxy. Scraping the entire repository is not being a friendly
netizen and will probably get your ip blocked. There are much better
ways.
-----Original Message-----
Sent: Friday, March 07, 2008 10:15 PM
Subject: How can I download all the packages from a remote repository?
Can I download all the packages from a remote repository to my local
http://repo1.maven.org/maven2/, I want to download all the content to my
local repository. I want to do this for that I have limitation for accessing
Internet, so I need to do this one time to get as much packages as possible.
--
http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-
repository--tp15910759s177p15910759.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
---------------------------------------------------------------------
--
View this message in context: http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-repository--tp15910759s177p15911893.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
Brian E. Fox
2008-03-08 22:36:10 UTC
Permalink
Umm, see my message below. Scraping the entire repo will get you
blocked. Use a repository manager if you need to be network independent.

-----Original Message-----
From: youhaodeyi [mailto:***@gmail.com]
Sent: Saturday, March 08, 2008 2:05 AM
To: ***@maven.apache.org
Subject: RE: How can I download all the packages from a remote
repository?


Can I write the dependency with wildcard characters? For example,

<dependency>
<artifactId>*</artifactId>
<groupId>*</groupId>
...
</dependency>

Then I will download all the packages from the repository.
Post by Brian E. Fox
Get a repository manager (proximity/nexus/archiva/artifactory) to act as
a local proxy. Scraping the entire repository is not being a friendly
netizen and will probably get your ip blocked. There are much better
ways.
-----Original Message-----
Sent: Friday, March 07, 2008 10:15 PM
Subject: How can I download all the packages from a remote repository?
Can I download all the packages from a remote repository to my local
http://repo1.maven.org/maven2/, I want to download all the content to my
local repository. I want to do this for that I have limitation for accessing
Internet, so I need to do this one time to get as much packages as possible.
--
http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-
Post by Brian E. Fox
repository--tp15910759s177p15910759.html
Sent from the Maven - Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
---------------------------------------------------------------------
--
View this message in context:
http://www.nabble.com/How-can-I-download-all-the-packages-from-a-remote-
repository--tp15910759s177p15911893.html
Sent from the Maven - Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: users-***@maven.apache.org
For additional commands, e-mail: users-***@maven.apache.org
Loading...