[Bacula-users] How much space does Bacula need?

[Dagan McGregor]

Hi all,

Just a question in regards to how much space Bacula generally uses, when doing backups and restores.

On the backup server with the storage daemon and director, does Bacula use much in the way of tmp space or disk space allocated for archive files?
Does spooling cause Bacula to use tmp space or so forth?

I have a few servers here that don't have much room in /var and /tmp, and I am just trying to work out where I need to increase the space available on those servers.

Cheers,

Dagan McGregor
Landmark Technologies

[Jari Fredriksson]

Post by Dagan McGregor
Hi all,
Just a question in regards to how much space Bacula
generally uses, when doing backups and restores.
On the backup server with the storage daemon and
director, does Bacula use much in the way of tmp space or
disk space allocated for archive files?
Does spooling cause Bacula to use tmp space or so forth?

Spooling writes the entire data set first to disk in a directory where you configure it to spool. If no spooling is in use, it does not use temporary space. When using spooling, you can configure the max size of the spool file per job. The job will be spooled in chunks of that size, and each chunk will be written to tape or other external media you have configured.

Post by Dagan McGregor
I have a few servers here that don't have much room in
/var and /tmp, and I am just trying to work out where I
need to increase the space available on those servers.

Bacula keeps it's catalog in a persistent storage like a MySQL or PostgrSQL database. The size requirements will vary and depends on retention times, count of media, and files to backup. My /var/lib/mysql/ibdata1 table space (MySQL with InnoDB) seems to be 512MB, but in that database is also a SpamAssassin database. Can't say how much bacula takes, but I guess almost all of it. I backup only 3 machines, so that is a very small system.

[Timo Neuvonen]

Post by Jari Fredriksson

Post by Dagan McGregor
Just a question in regards to how much space Bacula
generally uses, when doing backups and restores.
On the backup server with the storage daemon and
director, does Bacula use much in the way of tmp space or
disk space allocated for archive files?
Does spooling cause Bacula to use tmp space or so forth?

Spooling writes the entire data set first to disk in a directory where you
configure it to spool. If no spooling is in use, it does not use temporary
space. When using spooling, you can configure the max size of the spool
file per job. The job will be spooled in chunks of that size, and each
chunk will be written to tape or other external media you have configured.

Post by Dagan McGregor
I have a few servers here that don't have much room in
/var and /tmp, and I am just trying to work out where I
need to increase the space available on those servers.

Bacula keeps it's catalog in a persistent storage like a MySQL or
PostgrSQL database. The size requirements will vary and depends on
retention times, count of media, and files to backup. My
/var/lib/mysql/ibdata1 table space (MySQL with InnoDB) seems to be 512MB,
but in that database is also a SpamAssassin database. Can't say how much
bacula takes, but I guess almost all of it. I backup only 3 machines, so
that is a very small system.

Catalog is important to remember when estimating system disk space
requirements, especially if long retention times are needed. I think
somewhere in the documentation is a formula how to estimate catalog size.
And I think I've seen some postings on the Bacula list that mention over 10
GB catalog sizes. In my system (a few servers only), catalog is below 500
megabytes.

Then, when making a backup of the catalog, there is a need of approximately
same size of temporary file (ascii dump of the database). If not taken into
account, this might cause the filesystem to fill up. I've sometimes been
thinking, if this file should be written into /temp instead of /var/bacula
where Bacula writes it by default.

--
TiN

[Dagan McGregor]

-----Original Message-----
Of Timo Neuvonen
Sent: Monday, 25 August 2008 2:23 AM
Subject: Re: [Bacula-users] How much space does Bacula need?
Catalog is important to remember when estimating system disk space
requirements, especially if long retention times are needed. I think
somewhere in the documentation is a formula how to estimate
catalog size.
And I think I've seen some postings on the Bacula list that
mention over 10
GB catalog sizes. In my system (a few servers only), catalog
is below 500
megabytes.

Firstly, thanks for all the replies so far!

I will be running a back-up of maybe ~10 CentOS servers, not all of which will be entirely stored on tape, mostly the more important files and database dumps.

The back-up server I have has a 500GB HDD dedicated for the back-ups, and if I rebuild it and install the CentOS .el5. RPMS, which appear to reside in /var/lib/bacula with configs in /etc/bacula, then I think I may need to re-do the partition plan for that machine to allocate space more efficiently.

Because I am going to use bacula to back-up ESX VMs, I was hoping that I could copy the VM files and snapshots to somewhere on /home on the back-up server, have bacula-fd on the back-up server pickup those files to write to tape.
Then inside each VM, run bacula-fd to back-up the key files to the back-up server written to tape.

Pardon my ignorance, but I am assuming here that all spooling only takes place on the server running bacula-sd, the back-up server, as each bacula-fd instance pushes the files across the network to that location. If that is the case, and spooling location is configurable, I can set that to somewhere in /home/bacula and make use of that space.

Network back-up tools in Linux are fairly new to me, and I am trying to get my head around the Bacula architecture before rolling out all the new servers, thus my emails :)

Cheers,

Dagan McGregor
Landmark Technologies

[Ryan Novosielski]

Post by Dagan McGregor

-----Original Message-----
Of Timo Neuvonen
Sent: Monday, 25 August 2008 2:23 AM
Subject: Re: [Bacula-users] How much space does Bacula need?
Catalog is important to remember when estimating system disk space
requirements, especially if long retention times are needed. I think
somewhere in the documentation is a formula how to estimate
catalog size.
And I think I've seen some postings on the Bacula list that
mention over 10
GB catalog sizes. In my system (a few servers only), catalog
is below 500
megabytes.

Firstly, thanks for all the replies so far!
I will be running a back-up of maybe ~10 CentOS servers, not all of which will be entirely stored on tape, mostly the more important files and database dumps.
The back-up server I have has a 500GB HDD dedicated for the back-ups, and if I rebuild it and install the CentOS .el5. RPMS, which appear to reside in /var/lib/bacula with configs in /etc/bacula, then I think I may need to re-do the partition plan for that machine to allocate space more efficiently.
Because I am going to use bacula to back-up ESX VMs, I was hoping that I could copy the VM files and snapshots to somewhere on /home on the back-up server, have bacula-fd on the back-up server pickup those files to write to tape.
Then inside each VM, run bacula-fd to back-up the key files to the back-up server written to tape.
Pardon my ignorance, but I am assuming here that all spooling only takes place on the server running bacula-sd, the back-up server, as each bacula-fd instance pushes the files across the network to that location. If that is the case, and spooling location is configurable, I can set that to somewhere in /home/bacula and make use of that space.
Network back-up tools in Linux are fairly new to me, and I am trying to get my head around the Bacula architecture before rolling out all the new servers, thus my emails :)

Judging from your situation, which is roughly similar to mine, and
depending on your retention times, you can reasonably expect to see a
catalog size of between 250MB and 750MB. Mine is about 400 right now. I
backup about 11-12 machines, none backing up the whole OS. One machine
has many small files though. The backups of the catalog compress with
GZIP down to about 65MB, which is something to consider. Makes life a
little easier in the disk space department (this is the only place I ran
into trouble as I was using file volumes for them).

I think a 500MB range is a fair estimation though.

- --
---- _ _ _ _ ___ _ _ _
|Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Systems Programmer II
|$&| |__| | | |__/ | \| _| |***@umdnj.edu - 973/972.0922 (2-0922)
\__/ Univ. of Med. and Dent.|IST/AST - NJMS Medical Science Bldg - C630

[John Drescher]

Post by Timo Neuvonen
Catalog is important to remember when estimating system disk space
requirements, especially if long retention times are needed. I think
somewhere in the documentation is a formula how to estimate catalog size.
And I think I've seen some postings on the Bacula list that mention over 10
GB catalog sizes. In my system (a few servers only), catalog is below 500
megabytes.

Yep. My catalog is over 20GB

Here is what bacula-web says:

Total clients: 46 Total bytes stored: 17.51 TB
Total files: 4672387 Database size: 20.28 GB

John

[John Drescher]

Total clients: 46 Total bytes stored: 17.51 TB
Total files: 4672387 Database size: 20.28 GB

One thing I should have included which is on the lines of what Ryan
mentioned about compression. My 20.28GB database compresses to 1.75GB

Build OS: x86_64-pc-linux-gnu gentoo
JobId: 12088
Job: BackupCatalog.2008-09-03_14.47.02
Backup Level: Incremental, since=2008-09-03 01:18:34
Client: "dev6-fd" 2.4.2 (26Jul08) x86_64-pc-linux-gnu,gentoo,
FileSet: "Catalog" 2006-07-24 14:04:28
Pool: "BackupCatalogs" (

[Alan Brown]

Post by John Drescher

Total clients: 46 Total bytes stored: 17.51 TB
Total files: 4672387 Database size: 20.28 GB

One thing I should have included which is on the lines of what Ryan
mentioned about compression. My 20.28GB database compresses to 1.75GB

Do you compress before writing to tape or do you let the tape drive do
that?

AB

[John Drescher]

Post by Alan Brown

Post by John Drescher

Total clients: 46 Total bytes stored: 17.51 TB
Total files: 4672387 Database size: 20.28 GB

One thing I should have included which is on the lines of what Ryan
mentioned about compression. My 20.28GB database compresses to 1.75GB

Do you compress before writing to tape or do you let the tape drive do
that?

I store the catalog on a File (under raid 5) device that is on the
same machine as the director but not the same machine as the the one
that has the database.

John

[Thomas Glatthor]

~ 90 clients

65.201 GB saved Data
(101.965.136 Files)

Database size currently 28.365 MB. (postgreSQL)

the database grows over the time up to 40 GB,
after dump and restore is again at ~ 27 GB.
(autovacuum is activated)

Post by John Drescher

Post by Timo Neuvonen
Catalog is important to remember when estimating system disk space
requirements, especially if long retention times are needed. I think
somewhere in the documentation is a formula how to estimate catalog size.
And I think I've seen some postings on the Bacula list that mention over 10
GB catalog sizes. In my system (a few servers only), catalog is below 500
megabytes.

Yep. My catalog is over 20GB
Total clients: 46 Total bytes stored: 17.51 TB
Total files: 4672387 Database size: 20.28 GB
John
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
https://lists.sourceforge.net/lists/listinfo/bacula-users

--
IC3S Information, Computer und Solartechnik AG
Bäckerbarg 6, 22889 Tangstedt, Germany

Telefon [phone]
Telefax +49.4109.555.57
E-Mail ***@ic3s.de
Web http://www.ic3s.de/

Vorstand: Jan Behrmann (Vorsitzender), Stefan Fehlauer
Aufsichtsrat: Sven Niklasson (Vorsitzender)
Dresdner Bank (200 800 00), Kto. 0123 9582 00
Commerzbank (200 400 00), Kto. 8540 288
Amtsgericht Kiel HRB 7590
UST.IdNr. DE184532766

[John Drescher]

Post by Thomas Glatthor
~ 90 clients
65.201 GB saved Data
(101.965.136 Files)
Database size currently 28.365 MB. (postgreSQL)
the database grows over the time up to 40 GB,
after dump and restore is again at ~ 27 GB.
(autovacuum is activated)

I have dumped and restored my postgreSQL database a few times and that
did reduce the size by a few GB (10 to 25%) but I only have done this
3 to 5 times and that was just to move the database to different
servers.

John

[Bob Hetzel]

Post by Dagan McGregor
Hi all,
Just a question in regards to how much space Bacula generally uses, when doing backups and restores.
On the backup server with the storage daemon and director, does Bacula use much in the way of tmp space or disk space allocated for archive files?
Does spooling cause Bacula to use tmp space or so forth?
I have a few servers here that don't have much room in /var and /tmp, and I am just trying to work out where I need to increase the space available on those servers.
Cheers,
Dagan McGregor
Landmark Technologies

There are a few considerations. Make sure you don't fill up where any
of these go...

1) If you intend to do more than one backup at a time you're likely
going to want to turn on spooling.
2) Turning on spooling will then turn on spooling of attributes.
3) The database will need storage to track what's backed up.
4) The database you choose may need temporary storage.
5) Lastly, when you back up the database, if you do a db export to do
the backup make sure that goes somewhere that won't fill up either.

You can probably use defaults for everything if you have a small amount
of files and/or data to back up, or if you have a way of monitoring up
to the minute when file systems get close to full--nightly report
monitoring won't cut it if the temp spaces fills up then files are
removed when backups are done (or crash).

In my case I've got about 130 desktop backup clients averaging around
10GB each, and about 10 servers with about 1 TB of total server data
being backed up. All goes to tape (but spooling is in use). My bacula
configuration is
2x500GB 7,200 RPM SATA drives in a raid-1
this holds /tmp, and /home which has bacula's catalog, temporary
storage for mysql
4x146GB 10,000 RPM SAS drives in a raid-0 config
this holds the spool files and nothing else.

[Hemant Shah]

Subject: Re: [Bacula-users] How much space does Bacula need?
Date: Thursday, September 4, 2008, 6:00 AM

Post by John Drescher

Total clients: 46 Total bytes

stored: 17.51 TB
20.28 GB

Post by John Drescher
One thing I should have included which is on the lines

of what Ryan

Post by John Drescher
mentioned about compression. My 20.28GB database

compresses to 1.75GB
Do you compress before writing to tape or do you let the
tape drive do
that?
AB
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move
Developer's challenge
Build the coolest Linux based applications with Moblin SDK
& win great prizes
Grand prize is a trip for two to an Open Source event
anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
https://lists.sourceforge.net/lists/listinfo/bacula-users

You should always let the tape drive do the compression, the hardware compression on tape will be faster then software compression done by the client. Your backups will be faster too.


Hemant Shah
E-mail: ***@yahoo.com

[Ryan Novosielski]

Post by Hemant Shah

Subject: Re: [Bacula-users] How much space does Bacula need?
Date: Thursday, September 4, 2008, 6:00 AM

Post by John Drescher

Total clients: 46 Total bytes

stored: 17.51 TB
20.28 GB

Post by John Drescher
One thing I should have included which is on the lines

of what Ryan

Post by John Drescher
mentioned about compression. My 20.28GB database

compresses to 1.75GB
Do you compress before writing to tape or do you let the
tape drive do
that?
AB

You should always let the tape drive do the compression, the hardware compression on tape will be faster then software compression done by the client. Your backups will be faster too.

Unless you're not using a tape, which is why I compress in software. :)

- --
---- _ _ _ _ ___ _ _ _
|Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Systems Programmer II
|$&| |__| | | |__/ | \| _| |***@umdnj.edu - 973/972.0922 (2-0922)
\__/ Univ. of Med. and Dent.|IST/AST - NJMS Medical Science Bldg - C630

[Chad Netzer]

Post by Ryan Novosielski

Post by Hemant Shah
You should always let the tape drive do the compression, the hardware compression on tape will be faster then software compression done by the client. Your backups will be faster too.

Unless you're not using a tape, which is why I compress in software. :)

Or if you are using bacula to encrypt the data on each client before
sending it to the storage daemon. Tape compression is practically
useless for already encrypted data.

Chad

[Dagan McGregor]

-----Original Message-----
Of Hemant Shah
Sent: Friday, 5 September 2008 5:05 AM
To: John Drescher; Alan Brown
Subject: Re: [Bacula-users] How much space does Bacula need?

Subject: Re: [Bacula-users] How much space does Bacula need?
Date: Thursday, September 4, 2008, 6:00 AM
Do you compress before writing to tape or do you let the
tape drive do
that?
AB

You should always let the tape drive do the compression, the
hardware compression on tape will be faster then software
compression done by the client. Your backups will be faster too.
Hemant Shah

Just out of interest, are there recommended hardware compression settings to use for LTO-3? I want my LTO-3 tapes using 2:1 compression (~800GB).

Please advise if I need to change something below?

# mt -f /dev/nst0 status
SCSI 2 tape drive:
File number=-1, block number=-1, partition=0.
Tape block size 0 bytes. Density code 0x44 (no translation).
Soft error count since last status=0
General status bits on (1010000):
ONLINE IM_REP_EN

# tapeinfo -f /dev/sg1
Product Type: Tape Drive
Vendor ID: 'IBM '
Product ID: 'HH LTO Gen 3 '
Revision: '73P8'
Attached Changer: No
SerialNumber: '1020008840'
MinBlock:1
MaxBlock:16777215
SCSI ID: 1
SCSI LUN: 0
Ready: yes
BufferedMode: yes
Medium Type: 0x38
Density Code: 0x44
BlockSize: 0
DataCompEnabled: yes
DataCompCapable: yes
DataDeCompEnabled: yes
CompType: 0x1
DeCompType: 0x1
BOP: yes
Block Position: 0

Cheers,
Dagan McGregor

[Hemant Shah]

Subject: Re: [Bacula-users] How much space does Bacula need?
Date: Thursday, September 4, 2008, 6:46 PM
On Thu, Sep 4, 2008 at 11:42 AM, Ryan Novosielski

Post by Ryan Novosielski

Post by Hemant Shah
You should always let the tape drive do the

compression, the hardware compression on tape will be faster
then software compression done by the client. Your backups
will be faster too.

Post by Ryan Novosielski
Unless you're not using a tape, which is why I

compress in software. :)
Or if you are using bacula to encrypt the data on each
client before
sending it to the storage daemon. Tape compression is
practically
useless for already encrypted data.

Why would it be useless for encrypted data? It is not compressed unless you encrypt and compress on client.

Chad
-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move
Developer's challenge
Build the coolest Linux based applications with Moblin SDK
& win great prizes
Grand prize is a trip for two to an Open Source event
anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Bacula-users mailing list
https://lists.sourceforge.net/lists/listinfo/bacula-users

Hemant Shah
E-mail: ***@yahoo.com

[Ryan Novosielski]

Post by Hemant Shah

Subject: Re: [Bacula-users] How much space does Bacula need?
Date: Thursday, September 4, 2008, 6:46 PM
On Thu, Sep 4, 2008 at 11:42 AM, Ryan Novosielski

Post by Ryan Novosielski

Post by Hemant Shah
You should always let the tape drive do the

compression, the hardware compression on tape will be faster
then software compression done by the client. Your backups
will be faster too.

Post by Ryan Novosielski
Unless you're not using a tape, which is why I

compress in software. :)
Or if you are using bacula to encrypt the data on each
client before
sending it to the storage daemon. Tape compression is
practically
useless for already encrypted data.

Why would it be useless for encrypted data? It is not compressed unless you encrypt and compress on client.

I think -- and now I'm pretty sure this depends on your tape hardware as
some of them are a lot better than others -- that encrypting something
tends to randomize it so much that it becomes incompressible.

- --
---- _ _ _ _ ___ _ _ _
|Y#| | | |\/| | \ |\ | | |Ryan Novosielski - Systems Programmer II
|$&| |__| | | |__/ | \| _| |***@umdnj.edu - 973/972.0922 (2-0922)
\__/ Univ. of Med. and Dent.|IST/AST - NJMS Medical Science Bldg - C630

[Dan Langille]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Post by Hemant Shah

Subject: Re: [Bacula-users] How much space does Bacula need?
Date: Thursday, September 4, 2008, 6:46 PM
On Thu, Sep 4, 2008 at 11:42 AM, Ryan Novosielski

Post by Ryan Novosielski

Post by Hemant Shah
You should always let the tape drive do the

compression, the hardware compression on tape will be faster
then software compression done by the client. Your backups
will be faster too.

Post by Ryan Novosielski
Unless you're not using a tape, which is why I

compress in software. :)
Or if you are using bacula to encrypt the data on each
client before
sending it to the storage daemon. Tape compression is
practically
useless for already encrypted data.

Why would it be useless for encrypted data? It is not compressed unless you encrypt and compress on client.

I think -- and now I'm pretty sure this depends on your tape hardware as
some of them are a lot better than others -- that encrypting something
tends to randomize it so much that it becomes incompressible.

Agreed.

[lists]

Post by Ryan Novosielski
I think -- and now I'm pretty sure this depends on your tape hardware as
some of them are a lot better than others -- that encrypting something
tends to randomize it so much that it becomes incompressible.

and what about crompessing before encrypting? sure this should be done
by software only. if its compressed first and then encrypted it would be
smaller too.

or am i wrong?

[Alan Brown]

Post by lists
and what about crompessing before encrypting? sure this should be done
by software only. if its compressed first and then encrypted it would be
smaller too.

Most good encryption methods I'm aware of compress the data as well as
encrypting it.

AB

[Dan Langille]

Post by Hemant Shah

Subject: Re: [Bacula-users] How much space does Bacula need?
Date: Thursday, September 4, 2008, 6:46 PM
On Thu, Sep 4, 2008 at 11:42 AM, Ryan Novosielski

Post by Ryan Novosielski

Post by Hemant Shah
You should always let the tape drive do the

compression, the hardware compression on tape will be faster
then software compression done by the client. Your backups
will be faster too.

Post by Ryan Novosielski
Unless you're not using a tape, which is why I

compress in software. :)
Or if you are using bacula to encrypt the data on each
client before
sending it to the storage daemon. Tape compression is
practically
useless for already encrypted data.

Why would it be useless for encrypted data?

Encrypted data does not compress well.

Post by Hemant Shah
It is not compressed unless you encrypt and compress on client.

This is not true. bacula-fd can compress without encrypting.