MogileFS is our open source distributed filesystem. Its properties and features include:
Application level -- no special kernel modules required.
No single point of failure -- all three components of a MogileFS setup (storage nodes, trackers, and the tracker's database(s)) can be run on multiple machines, so there's no single point of failure. (you can run trackers on the same machines as storage nodes, too, so you don't need 4 machines...) A minimum of 2 machines is recommended.
Automatic file replication -- files, based on their "class", are automatically replicated between enough different storage nodes as to satisfy the minimum replica count as requested by their class. For instance, for a photo hosting site you can make original JPEGs have a minimum replica count of 3, but thumbnails and scaled versions only have a replica count of 1 or 2. If you lose the only copy of a thumbnail, the application can just rebuild it. In this way, MogileFS (without RAID) can save money on disks that would otherwise be storing multiple copies of data unnecessarily.
"Better than RAID" -- in a non-SAN RAID setup, the disks are redundant, but the host isn't. If you lose the entire machine, the files are inaccessible. MogileFS replicates the files between devices which are on different hosts, so files are always available.
Flat Namespace -- Files are identified by named keys in a flat, global namespace. You can create as many namespaces as you'd like, so multiple applications with potentially conflicting keys can run on the same MogileFS installation.
Shared-Nothing -- MogileFS doesn't depend on a pricey SAN with shared disks. Every machine maintains its own local disks.
No RAID required -- Local disks on MogileFS storage nodes can be in a RAID, or not. It's cheaper not to, as RAID doesn't buy you any safety that MogileFS doesn't already provide.
Local filesystem agnostic -- Local disks on MogileFS storage nodes can be formatted with your filesystem of choice (ext3, XFS, etc..). MogileFS does its own internal directory hashing so it doesn't hit filesystem limits such as "max files per directory" or "max directories per directory". Use what you're comfortable with.
Can't execute select:
The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay at Modules/DataBase.pm line 81.
...
2010/11/25 05:59:43 [error] 19835#0: *1 could not find named location "http://www.x0.org.ua/to_nginx?bs=$body_bytes_sent" while sending response to cl$
2010/11/25 06:27:33 [error] 19835#0: *2 could not find named location "http://www.x0.org.ua/to_nginx?bs=$body_bytes_sent" while sending response to cl$
он не может отправить запрос на другой сервер (доменое имя) ?
rtyug добавил 25.11.2010 в 17:36
Andreyka: uplaod в nginx собран?
видимо, нет
/nginx -V
nginx version: nginx/0.8.48
built by gcc 4.1.2 20071124 (Red Hat 4.1.2-42)
configure arguments: --with-http_perl_module --with-http_stub_status_module
для того чтобы он отправил запрос надо собрать с uplaod?
rtyug добавил 25.11.2010 в 19:59
вродебы разобрался
# Premium
location /download_p/ {
alias /home/files/cgi-bin/uploads/;
internal;
#set $limit_rate 500k;
limit_conn one 50;
set $postURI $uri;
set $postIP $remote_addr;
set $postHOST $host;
Хардов нужно много для скорости, а не для количества места на диске....
Может кто чего предложит "быстрое"?
вы хотите файлы хранить?
почему бы не взять распределенную систему с репликацией?
реплицировать на несколько серверов и прикрутить туда nginx?
или разработвать свою систему, на perl в 10 раз быстрее :)
http://code.google.com/p/mogilefs/wiki/HighLevelOverview
http://www.opennet.ru/prog/info/2620.shtml
http://www.danga.com/mogilefs/
MogileFS is our open source distributed filesystem. Its properties and features include:
Application level -- no special kernel modules required.
No single point of failure -- all three components of a MogileFS setup (storage nodes, trackers, and the tracker's database(s)) can be run on multiple machines, so there's no single point of failure. (you can run trackers on the same machines as storage nodes, too, so you don't need 4 machines...) A minimum of 2 machines is recommended.
Automatic file replication -- files, based on their "class", are automatically replicated between enough different storage nodes as to satisfy the minimum replica count as requested by their class. For instance, for a photo hosting site you can make original JPEGs have a minimum replica count of 3, but thumbnails and scaled versions only have a replica count of 1 or 2. If you lose the only copy of a thumbnail, the application can just rebuild it. In this way, MogileFS (without RAID) can save money on disks that would otherwise be storing multiple copies of data unnecessarily.
"Better than RAID" -- in a non-SAN RAID setup, the disks are redundant, but the host isn't. If you lose the entire machine, the files are inaccessible. MogileFS replicates the files between devices which are on different hosts, so files are always available.
Flat Namespace -- Files are identified by named keys in a flat, global namespace. You can create as many namespaces as you'd like, so multiple applications with potentially conflicting keys can run on the same MogileFS installation.
Shared-Nothing -- MogileFS doesn't depend on a pricey SAN with shared disks. Every machine maintains its own local disks.
No RAID required -- Local disks on MogileFS storage nodes can be in a RAID, or not. It's cheaper not to, as RAID doesn't buy you any safety that MogileFS doesn't already provide.
Local filesystem agnostic -- Local disks on MogileFS storage nodes can be formatted with your filesystem of choice (ext3, XFS, etc..). MogileFS does its own internal directory hashing so it doesn't hit filesystem limits such as "max files per directory" or "max directories per directory". Use what you're comfortable with.
Software error:
Can't execute select:
The SELECT would examine more than MAX_JOIN_SIZE rows; check your WHERE and use SET SQL_BIG_SELECTS=1 or SET SQL_MAX_JOIN_SIZE=# if the SELECT is okay at Modules/DataBase.pm line 81.
...
надо добавить
$db->Exec('SET SQL_BIG_SELECTS=1');APC
ху из ху?
PHP APC Extensions?
можно ли сделать на подобе сайта 2spb.ru за ~150$?? (без дизайна)
(закасчик мне заказывал давно, но не было времени сделать)
в логах nginx такая строка:
он не может отправить запрос на другой сервер (доменое имя) ?
rtyug добавил 25.11.2010 в 17:36
uplaod в nginx собран?
видимо, нет
для того чтобы он отправил запрос надо собрать с uplaod?
rtyug добавил 25.11.2010 в 19:59
вродебы разобрался
запрос не хочет отправятся...
я добавил строку
в полностью рабочий конфиг
но запрос НЕ отправился после того как я скачал файл, в логах apache на http://www.site.com/ в error и access нету запроса
Товарищи, подскажите есть:
post_action
location ~* ^/media/.+\.(mp3|wma|ogg|ape|flac)$ {post_action /counter.php?bs=$body_bytes_sent;
}
кроме $body_bytes_sent, как отправить после того как клиент скачет что-то одно из:
1) хеш который был прислан для разрешения скачивания файла (чтобы идентифицировать скачивание и файл пользователя)
2) или имя отдаваемого файла, но лучше п.1
где это найти?
Detektiv, можно сделать, я участвовал в разработке и поддержке одной из крупнейшей банерской сети в западном сементе, можно сделать на основе этой...
там еще есть perl модуль perl_require download.pm;
Покажите Ваш конфиг, тонче его часть, ответственную за скачивание.
вродебы это оно:
там еще есть perl модуль perl_require download.pm;