ORA-19809, Failed RMAN Backups and Flashback

Flashback Recovery Area is a common gotcha in many locations as time goes by, so I thought it might be worthwhile to review, since for me, it’s all new.  My last two shops, due to design, had no use for it…

payday loans lenders online

Flashback Recovery Area, (FRA) is an intriguing feature and should not be confused as the same as the flashback feature- The ability to flashback to a given time or flashback a table when something has gone wrong has benefits over recovery that is easy for anyone who’s had the to perform either task can appertain.   Per Andy Colvin, as of vervion 11.2, FRA now stands for Flash Recovery Area to eliminate some of the confusion.   As I become more familiar with the FRA, I see a few areas of improvement that I hope Oracle will continue to work on and I will continue to add to my knowledge on.

With more complexity comes more challenges and knowledge of configuration. FRA is no different. It has three main parameters, making it a simple feature to set up:

  • db_flashback_retention_target          Sets retention target time, in minutes.
  • db_recovery_file_dest                          The location of the archive logs used by FRA
  • db_recovery_file_dest_size                The max space allocated to everything residing in the FRA destination, helps deter from overwhelming a disk location or diskgroup.

You manually set the location for the FRA and the destination max size that you wish to allocate to the FRA.

The first parameter only pertains to the retention time for flashback logs, no other FRA data retention.

Oracle sets a standard default of 1440 minutes retention target, (1 day) If you think about how different each database usage is, this could be quite a lot of archive log retention and disk allocation for some.

The db_recovery_file_dest_size parameter is there to help with this. This is the max amount of space you have set aside for this the database. This will deter the database from over- allocating and filling up a disk or diskgroup. In a RAC environment, this is set at the instance level, so you must set it for each node but the values must match for the entire RAC environment- no different parameter values per node.

It's good to know what is contained in your FRA location. Keep in mind that its not just archive logs:

SQL> select * from v$flash_recovery_area_usage;
FILE_TYPE PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES
 ------------ ------------------ ------------------------- ---------------
 CONTROLFILE      0   0    0
 ONLINELOG        0   0    0
 ARCHIVELOG   21.06  12  172
 BACKUPPIECE  29.52   0    6
 IMAGECOPY        0   0    0
 FLASHBACKLOG  9.02  22   28

When you are posed with the challenge of space limitations, it’s good to know how much space is currently being used by the FRA.  In one case, the database with too high a FRA retention, it was a development database and the FRA setting for retention had been brought over from production during a cloning process.

This resulted in the following:

SQL> select  name
, floor(space_limit / 1024 / 1024) "Size MB"
, ceil(space_used  / 1024 / 1024) "Used MB"
from    v$recovery_file_dest
order by name ;

NAME                                                            Size MB    Used MB
------------------------------------------------------------ ---------- ----------
/<dir>/FRA                                                         153600      128475
NAME                                                     TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_flashback_retention_target        integer     10880
SQL> alter system set db_flashback_retention_target=1440;
System altered.

With the FRA retention time put back to the default, the space usage was more in line with a development database, which often has less extra allocation of space than a production one.  With a clone, all three parameter settings must be set to the correct values involving FRA.

Now, here is the challenge, if you up against an ORA-19809 error in RMAN and impacted due to an FRA retention size limit for archive logs, as in the pictured example:

…you can’t just backup the archive logs with a purge option, as follows:

rman target /
 run {
 allocate channel chan1 type disk;
 backup archivelog all delete input format '//arch_%d_%u_%s';
 release channel chan1;
 }

This will fail because performing a backup of these archive logs before purging them takes FRA space and your error states you are at your limit…

You have a couple choices to address this-

you can add more space to FRA and then perform the backup, re-adjusting the parameter post the clean up:

alter system set db_recovery_file_dest_size=scope=both;

or you can just purge everything that is expired..

rman target /
 crosscheck archivelog all;
 delete expired archivelog all; 

If RMAN is the culprit.  Verify that your backups are using FRA space and that they are the reason for your issue.  For a backup that is being written to the FRA location, the standard practice for an RMAN backup is to:

1.  Perform the full backup

2. Purge the previous backup

In the example below, you will note that the backuppiece is utilizing 51% of the FRA space.  If it were to attempt to perform a second backup, THEN purge, it will hit into the FRA limitation FIRST, resulting in an error.

SQL> select * from v$flash_recovery_area_usage;

FILE_TYPE    PERCENT_SPACE_USED PERCENT_SPACE_RECLAIMABLE NUMBER_OF_FILES
------------ ------------------ ------------------------- ---------------
CONTROLFILE                   0                         0               0
ONLINELOG                     0                         0               0
ARCHIVELOG                    0                         0               0
BACKUPPIECE                51.1                         0              87
IMAGECOPY                     0                         0               0
FLASHBACKLOG                  0                         0               0

Then you may have the choice of adjusting or changing the way you do backups if you do not wish to add to the FRA  space allocated, (db_recovery_file_dest_size.)

Let’s say the customer is using image copies and they agree to go to standard RMAN compressed backups, which will save considerable space.

You may look at the retention time and decide it is high, (assume its a development environment and there is no need to keep a full week of backups, which was one of the issues above that was taking up so much space.

rman target/
RMAN> show all;

RMAN configuration parameters are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 7;

Another customer ran into the issue with a failed standby “catch up” of archive logs.  It had ran out of physical space on the primary for archiving and once released, hit the ceiling on max on db_recovery_file_dest_size.  The size max had to be escalated until we could successfully get the system going again and then the standby had to match the primary.

What to remember as you are setting up FRA:

  1. Set a retention time that makes sense for the database usage and the space you are able to allocate towards FRA.
  2. When sizing the db_recovery_file_dest_size, keep in mind that more than archive logs are using the space allocated.  You must have enough space, plus adequate growth, for archive logs, backups, and/or controlfiles, online logs and image copies, depending on your configuration.  Use the scripts to see what you are using and size appropriately.
  3. Have proper monitoring of the FRA usage area so that you can pro-actively address issues.

The more I use it, the more I find value and challenges with it.  If a backup is incorrectly or falls back to the default settings to backup to the FRA location, (the GUI in EM12c can easily set this up, even though the DBA may think they have configured otherwise…)  If you fill up the FRA max size, the archiver hangs, which is a service outage for many companies that in today’s business is not an option.  If the FRA is set to an ASM diskgroup that has limits, just as with disk, the FRA destination size may still be hindered as you attempt to work around a challenging issue.

 

Migrating Jobs in Enterprise Manager 12c- not for the faint of heart… :)

These are instructions that allowed me to migrate over a large library of EM12c jobs from an existing, split EM console, (Linux) and OMS Repository(Windows) to a single server environment without requiring me to recreate them new.

Disclaimer**
The migration is a multi-step process that was performed on:
A NEW EM12c 12.0.1.0 with the latest bundle patch, OS 64-bit Linux
11.2.0.3.0 repository
a FULL backup of the OMS repository before performing.

A full documentation of your EM jobs that are up to date!

The source environment the EM jobs were datapumped from:
11.2.0.3.0 OMS repository on Windows 2008
EM12c, without the bundle patch on Linux, 64 bit.

I have not attempted this on any other platform or environment and I strongly recommend that you start with a fresh OMS repository, new EM12c with the bundle patch and secure backups if you do not want to have to go back and recreate repositories or re-install!

If you do not have a lot of jobs, then just recreate them, but this challenge came about because I had 52 complex EM jobs that recreating would have impacted the production environment and I had to have a way to migrate them with as little downtime as possible.

Challenges ahead
Jobs are identified in the internal OMS tables by TARGET_GUID. As this is a brand new EM and no targets exist or will be migrated as part of the process till the final steps, invalidations and constraint issues should be expected.

High Level Steps
____________________________

Create your datapump dir you wish to use on both machines, if you don’t have one.
Run the export using the par file provided from the source EM12c
Transfer the dump file over to the destination server and into the datapump dir.
Shut down the agent and the OMS on the destination server’s EM
Log into the destination EM12c and run the disable constraints script.
Run the import script using the par file provided. DIsregard the metadata errors, the data should be appended…
Bring up the OMS and the agent.
Log into the EM on the destination server and view the job library, verifying all jobs are present

During outage scenario- Shut down the agents on each of the targets and remove the agent homes.

Deploy the agents from the new EM12c server to the targets.

Edit each job, removing the old db listed in the “General” tab and add the correct database, (same name, but the Target GUID has changed underneath in the new EM when the agent was deployed…)

Save to Library.

Submit each job to the system and you have completed the migration.

Detailed Steps———————

Backup the destination database so you have a recovery point if things go wrong…

————————————————————

First challenge- what tables are involved in the Enterprise Manager jobs?

select table_name from dba_Tables
 where owner='SYSMAN'
 and table_name like '%MGMT%JOB%'
 and table_name not like '%BCN%'
 and table_name not like '%GENSVC%'
 ;

Use the results from this query to build a parameter file for a datapump export:
Export Datapump Parfile

On both databases ensure you have a directory to work from:

SQL> !mkdir <dir>/backup/exp
SQL> create directory sysman_dir as '<dir>/exp';
Directory created.
SQL> grant read on directory sysman_dir to sysman;
Grant succeeded.
SQL> grant write on directory sysman_dir to sysman;
Grant succeeded.

Export out the sysman jobs using the par file as the sysman user. The MGMT_JOB table has fine-grain control on it,
but as we only want the sysman jobs brought over, we do not want to write over the metadata that exists in this table, only append to it.

FTP the file over to the new (destination) EM server.

On  the new server, bring down the local agent:

cd $AGENT_HOME
 cd bin
 ./emctl stop agent
 Oracle Enterprise Manager 12c Cloud Control 12.1.0.1.0
 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
 Stopping agent ..... stopped.

Bring down the console and the OMS:

./emctl stop oms
 Oracle Enterprise Manager Cloud Control 12c Release 12.1.0.1.0
 Copyright (c) 1996, 2012 Oracle Corporation. All rights reserved.
 Stopping WebTier...
 WebTier Successfully Stopped
 Stopping Oracle Management Server...
 Oracle Management Server Successfully Stopped
 Oracle Management Server is Down

Disable the constraints using the script found here:
Disable Constraints
Import in the data to the database using the following parameter file for datapump:
Import Parfile

DIRECTORY=SYSMAN_DIR
TABLE_EXISTS_ACTION=APPEND
DUMPFILE=SYSMAN_DIR:EXPDP_SYSMAN_JOBS.DMP
LOGFILE=SYSMAN_DIR:impdp_sysman_jobs.log
TRANSFORM=oid:n
STATUS=60
TABLES=
( SYSMAN.MGMT_JOB_OUTPUT
, SYSMAN.MGMT_JOB_EXEC_CRED_INFO
, SYSMAN.MGMT_JOB_EXT_TARGETS
, SYSMAN.MGMT_JOB_TARGET
, SYSMAN.MGMT_JOB_LARGE_PARAMS
, SYSMAN.MGMT_NESTED_JOB_TARGETS
, SYSMAN.MGMT_PAF_JOBTYPES
, SYSMAN.MGMT_PAF_JOBTYPE_PARAMS
, SYSMAN.MGMT_USER_JOBS
, SYSMAN.MGMT_JOB_STEP_PARAMS
, SYSMAN.MGMT_JOB_PARAM_SOURCE
, SYSMAN.MGMT_JOB_USER_PARAMS
, SYSMAN.MGMT_JOB_SQL_PARAMS
, SYSMAN.MGMT_JOB_CRED_PARAMS
, SYSMAN.MGMT_JOB_SUBST_PARAMS
, SYSMAN.MGMT_JOB_PROP_PARAMS
…. yeah, the same tables as in the export file!!

Simply execute the following command with your proper information:

impdp sysman@<dbname> parfile=/<dir>/impdb_sysman_jobs.par

If you do not use the parameter in your par file:

TRANSFORM=oid:n

You’ll receive the following upon import:

ORA-31693: Table data object "SYSMAN"."MGMT_JOB_PARAMETER" failed
to load/unload and is being skipped due to error:
 ORA-29913: error in executing ODCIEXTTABLEOPEN callout
 ORA-29400: data cartridge error
 ORA-39779: type "SYSMAN"."MGMT_JOB_VECTOR_PARAMS" not found or
conversion to latest version is not possible

due to the following types:
SYSMAN.MGMT_JOB_VECTOR_PARAMS
SYSMAN.MGMT_JOB_INT_ARRAY
SYSMAN.MGMT_SHORT_STRING_ARRAY

The output from your datapump import should look like the following:

Worker 1 Status:
 Process Name: DW00
 State: EXECUTING
 Object Schema: SYSMAN
 Object Name: MGMT_JOB_HISTORY
 Object Type: TABLE_EXPORT/TABLE/TABLE_DATA
 Completed Objects: 2
 Total Objects: 42
 Completed Bytes: 8,981,768
 Percent Done: 100
 Worker Parallelism: 1
 . . imported "SYSMAN"."MGMT_JOB_PARAMETER" 19.51 MB 254627 rows

 

Once the import has completed

-Re-enable constraints-  19 will remain invalid due to the Target GUID differences from the old OMS repository data and the new one, (the script is down below!)

-Start OMS and the agent.

-Log into the console and verify the jobs are showing in the library:

Once you have your official window to migrate over, you will need to perform the following:

-Shutdown the agents on the servers that are migrating over.

-De-install the Agent Home:

-Remove the agent home:

-Deploy the agent from the new EM12c.

Once the agent is deployed and the targets are uploading to the new OMS, edit each job to update the Target GUID discrepency from the migration.

  • From the summary page, click on ENTERPRISE–> JOB–> LIBRARY
  • For each job, click in the radio button, click on EDIT.
  • On the general tab, you will need to remove the existing host or database, then add the new one that resides in the OMS repository.  You will view it as removing and re-adding the same target, but the Target GUID is different and so the OMS will react as if a full edit to new host or database has been made.
  • Click on “Save to Library”
  • Once you’ve done this for each job, you will return to the main screen and you can then click the “Submit” button for the job.

You can verify submissions by looking at the activity view, filtering by “scheduled”.

Once all jobs have been updated with the correct Target GUIDS, then you can compile the last 19 constraints.
Enable Constraints

 

Did you run into a snag, need to start over?

************************
Use your backup…as this is the OMS you can’t use EM to restore, soooo…

Ensure all the following is shut down

-EM agent.
-The OMS Repository
-The DB Instance, ensuring you pulled your backup tag before you shut it down!

Connect to the new EM throgh RMAN, (RMAN target/)
 startup force mount;
 restore database from tag "";
 restore database from tag "BACKUP_EMREP_000_050912095636";
 recover database;
alter database open resetlogs;
Or go back to a point in time:
run {
 sql "alter session set nls_date_format=''dd-mon-yyyy hh24:mi:ss''";
 set until time '09-may-2012 10:38:00';
 restore database;
 recover database until time '09-may-2012 12:00:00';
 alter database open resetlogs;}
startup oms
startup agent

Remember to recreate your datapump directory you were using, if not a standard one!

OK, back where we started before everything got out of hand! :)
Sometimes the Repository db will not leave a pending state in the EM console after the restore.  If this happens, remove the OMS db from the EM console and simply re-add.  It will then update everything and all targets will again upload correctly.

 

 

Miracle Oracle World 2012 Recap

I ended up so wrapped up in everything going on when I returned home that I never did get my Miracle Oracle World 2012 post out on my blog, (bad DBA Goth Princess!:)) so here it is!

Miracle Oracle World, (aka MOW2012, hashtag #MOW2012) is a yearly event held by Mogens Noorgard and his company Miracle, (http://www.miracleas.dk/ ) at Hotel Legoland in Billund, Denmark each year. This year was no exception and I was thrilled when Mogens asked me while in attendance at RMOUG’s Training Days if I would come to Denmark and do my presentation on EM12c, (then let me choose one more topic to speak on, which turned out to be ASH Analytics.)

I’d never been to Denmark and it sounded like a wonderful opportunity, but there were clashes in schedules personally and I wasn’t sure if I’d be able to attend at first, but Tim was adamant about me going to MOW this year and stayed home to take care of responsibilities so I could speak in Denmark.

Upon reaching Copenhagen, Mogens picked me up at the airport, (still impressed he was able to locate me in the chaos that is any airport!) and after a short time to recover from the long fight at Mogen’s house, I loaded up into the car, driven by Mogens and accompanied by Cary Millsap, his wife, Mindy and their five year old daughter, Cat.

I have to give a lot of credit to young, Miss Cat. She was so well behaved and content to just ride along, watching movies and playing games, etc. She’s very close to both her parents and was quite enjoying her *special* trip to Denmark. It’s a 3 ½ hour trip to Billund from the town near Copenhagen where Mogens resides and the trip was pleasant, good conversation and humor when Cary attempted to calculate the miles per hour of the windmills by circumference, length and speed of the blades.

Upon arriving in Billund, we toured the Hotel Legoland for a bit, but I was glad to have the Millsap’s drop me off at the house at Lilandia and let me recuperate. Lilandia is a huge waterpark across from Hotel Legoland with what looks like small cul de sac’s of houses, different sizes and designs. The one I was in was four bedrooms and I was to share my room with Heli Helskyaho, who I was so looking forward to catching up with since I hadn’t seen her since RMOUG training days! My fellow house guests were Alex Gorbachev, Christian Antognini and Jonathan Lewis.

Most of the folks attend the party house and I heard it was quite the event, but I won’t lie, I tried to catch up on some sleep after being up for 36 hrs straight! Not much jet lag but I was up and spoke to Jonathan when he came in at around 8pm, Alex when he showed up at Midnight and then chatted with Christian when he was up at 2am working on his presentation slides. Jet lag is a funny thing… :)

I did get enough sleep to be up and ready before 7am and Jonathan happened to be up at the same time. He asked if I would like some company on my walk over to the conference site and I was happy to have such an excellent fellow along for the 1 mile walk. As much as I enjoy Jonathan’s presentations, there has got to be something said for one on one discussions with him. He can converse on any number of subjects and we were at the hotel Legoland in no time at all.

I appreciate Cary Millsap and his daughter, Cat allowing me to sit with them at breakfast. This helped alleviate some of my separation anxiety from my own children just watching Cat interact with her father. I ran into Oyvind Isene from Norway’s Oracle User Group, who I’d had the pleasure of meeting last year at Oracle Open World and he was someone I could always count on to put up with my incessant chatting for the two days of the MOW conference… :)

Choosing sessions was more difficult for me than I think for most other folks at the conference. The thing that really impressed me about MOW was that it was a mixed platform conference. This doesn’t mean there were a few MySQL sessions scattered among the Oracle, but that there was a full set of SQL Server presentations. For me, having as many years in SQL Server as I do Oracle, it was a real treat. I missed out on a few I would have liked to attend, but as always happens, conflicted with others I had promised attendance to. To attend Ron Crisco’s session, (someone I had spoken to and emailed with working with the Method R tool suite, but had never actually met…) meant I missed Thomas Kejser’s “SQL 2012 OLTP Improvements” session. This went on for most of the day even my own presentations competing with a few sessions I wanted to attend.

Christian Antognini’s “End to End Response Time Analysis” was engaging- anything to do with waits and response is going to interest me. His data was flawless and everyone in the room left with a few more ways to address performance challenges after seeing his examples and demos.

I then attended Dan Norris presentation on Exa-environment platform integration techniques. Dan didn’t have a large crowd in his session but this did allow for the conversation to be very specific between the attendees and Dan, which in turn satisfied those that attended his session greatly. Dan has a lot of experience and insight and I was impressed with the level of interest from those that had questions about Exa* products.

I proceeded to attend a SQL Server session next by Alexei Khalyako on SQL 2012 High Availability and Disaster Recovery. I hadn’t worked in this area of SQL Server for a couple years and was quite impressed with some of the new features Microsoft has introduced into the SQL Server product, including Availability Groups Multiple Secondary’s for HA and DR, Advanced Replication and new clustering features.

The next session, by Thomas Kejser, I found very intriguing. It was Hadoop and Big Data vs. Datawarehousing from a SQL Server perspective. I’ve heard plenty of presentations and read a number of papers on it from the Oracle side, but really enjoyed hearing a view of it via SQL Server world. The data behind the presentation was accurate and Thomas is a solid presenter, so the session was one of my favorites. I made enough of an impact, (can you say mouthy?) that Thomas came over promptly after his presentation was done and we continued to talk in a small group about big data and what options DBA’s had to handle it, no matter what the platform or technology choice.

My ASH Analytics session was at 4pm, last session of the day and it went as well as I could have expected. I was missing my demo for this session, the server I’d set it up on was down for maintenance, (I know, why did I trust a server to be up or even available??) I’d already run into the complication when I discovered much of my data I had been collecting over the last couple months missing from my external drives. I unfortunately didn’t realize this until I went to enter all of the data I thought I had, on slides just two weeks prior to the conference. I came up with challenges recollecting this data as my current EM12c environments are on 10g RAC databases. Anyone who has attempted to use ASH Analytics on a 10g environment will learn- it’s not going to happen. You can install the package to bring up the ASH Analytics page, you just won’t have any data in 95% of the panes, so nice try. Lucky for me, I was able to capture most of the data I needed to come up with a presentation just in time- still not the quality I would have liked.

Of course, my ASH Analytics session was attended by Cary Millsap, Alex Gorbachev, Uri Shaft, John Beresniewicz and other folks, many on the DBA God list… :) I still appreciated their time, I was thrilled to have them attend, even if it was not my best session. Afterwards, we talked for quite some time, (missing demos have a tendency to grant that time… ) and Cary asked if I would like to attend his “Mastering Trace Data” Class that he was giving the next day. The opportunity to take it offsite was on the table for me, but to take it in person was too much to pass up, so I agreed.

The afternoon ended with after-first-day-conference drinks at Hotel Legoland’s bar while waiting for dinner to be served. Heli Helskyaho had arrived at that point, was happy to see her, (and this also meant I was no longer the only female presenter at the conference! :) ) I hung out with her, Oyvind Isene and Christian Antognini, speaking to many others throughout the time there until they informed us dinner was ready.

Dinner was a fun affair, great food, great company and again, Cary and Mindy Millsap’s daughter, Cat was well-behaved, (more so than some of the adults? :) ) Alex Gorbachev, along with our common banter, discussed his recent election to the IOUG Board of Directors and had a good conversation about RMOUG’s relationship with the IOUG.

Post dinner, Heli Helskyaho and I went over to the “Party House” which was four doors down from our own. There was eating and drinking, although not as much eating and drinking as the first night, as I heard there was a roast pig that I chose to miss to catch up on sleep. We spent most of the evening talking with John Beresniewicz, Tuomas Pystynen and Uri Shaft. There were many others, but so many, I gave up on remembering faces, let alone names. Alex Gorbachev took my “hoppy” beer off my hands at one point and replaced it with red wine, which turned into my drink of choice for the evening.

The second day started with a lovely breakfast with Heli, Tuomas Pystynen, Jonathan Lewis and others. I took the first hour off, prepped for my EM12c presentation in a quiet location and it was worth the time. My session was in full attendance and I was content with the outcome. The Q&A was not as intensive as my RMOUG session, but the questions were well thought out and discussion was actively participated in. The presentation was well received and only makes me more anxious for KSCOPE 2012 in San Antonio come June!

I attended Cary Millsap’s “Mastering Trace Data” class for the rest of the day. This was a great refresher course for me after taking the extended class over a year ago and reviewing the book used for the class back in December. Cary goes over the basic information so that anyone can easily utilize the Method R tool suite along with offering clear reasons to trace, showing how having this knowledge can make us better DBA’s.

The conference ended with a speech by Mogens Noorgard, awarding participants and those that came up with a creative way to explain the huge pool that had been a hit with the sauna partiers at the party house to Lilandia management.
Dan Norris had arranged with Mogens that I would drive back with him and his group. It was a good drive back to Malov, Denmark, near Copenhagen. His friend Anne is lovely, easy to talk to and Tuomas Pystynen is fun to be around, (except for those salted, black licorice fish candy he bought and had me try! :) )

I spent the last 24hrs at the Yellow Mansion, home of the Oak Table. Dan, Anne and Tuomas made sure I arrived at the airport on time to catch my plane and upon arriving at my connection at London/Heathrow, I entered the lift and heard a familiar voice. Looking over the crowd entering, I noticed Mark Rittman and yes, the person that was missing at MOW was on her way to Calloborate in Las Vegas- Debra Lilley. We shared a few words, hugs and kisses, then I was off on my last leg of my flight home to Denver.

My only regret regarding Miracle Oracle World is that I didn’t have more time in Denmark. Upon returning, viewed some of the pictures Dan Norris’ friend Anne took in Copenhagen after they dropped me off at the airport. Even one more day in the city would have been wonderful!