| Author |
Message |
Robert Myers
Guest
|
 Posted:
Sat Jan 15, 2005 10:44 pm Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
On 15 Jan 2005 13:59:47 -0800, jgrove24@hotmail.com wrote:
| Quote: |
Actually, I'm read usenet in GOOGLE, what the FBI wanted is basically
GOOGLE features.
|
The problem is that the part of the problem you can see--the part the
looks like the internet and google--is sufficiently non-trivial that
it is successfully masquerading as the whole problem to you.
The FBI wants data relevant to terrorism to be seen instantaneously
(as opposed to the currently claimed 24-hour cycle, which at that I
don't believe)--google can't do that.
There's lots of stuff on the web, the so-called deep web, that you
can't see with google, and the Virtual Case File would have to look
like a deep web so that it didn't pose unacceptable security risks.
Google can't manage document access or audit changes and versions.
The tools open source contributors use can do many of those things
nicely, but if FBI agents could function at that level of competence,
they could probably build the system themselves on the fly.
I wonder if something very similar to the false intuition you have
about this problem doesn't explain the whole wasted $170 million. It
looks easy, so it must be easy. Put it out on an RFP, take a
plausible bid, and wait for your own ready-made personal internet and
google to appear. Piece of cake.
RM |
|
| Back to top |
|
 |
Guest
|
| Posted:
Sat Jan 15, 2005 10:44 pm Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
jgrove24@hotmail.com wrote:
| Quote: | Robert Myers wrote:
Greetings,
News articles say FBI is probably going to scrap its $170 Million
Virtual Case File system and start over from scratch.
The problem is characterized as a software issue. Not being an
enterprise architect, I would have thought the problem was an
enterprise
architecture issue: hardware, software, and mostly organizational
wetware.
I'm posting here because of the number of posters and lurkers with
significant architectural experience in mainframe and enterprise
systems.
The features the FBI looked for are easy using UNIX, and a tool like
Documentum. They were looking to manage CASE FILES, you know like
paper
documents. And the ability to search or GREP in UNIX. A Grand Jury
examination of this folly is in order.
JG
|
Actually, I'm read usenet in GOOGLE, what the FBI wanted is basically
GOOGLE features.
JG |
|
| Back to top |
|
|
Nick Maclaren
Guest
|
| Posted:
Sat Jan 15, 2005 10:44 pm Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
In article <o16iu015p74cbkvqjlqcv34kf7tpccamj8@4ax.com>,
Robert Myers <rmyers1400@comcast.net> wrote:
Indeed, no. When it comes to IT projects, I agree that the British
government is a world leader in incompetence.
Regards,
Nick Maclaren. |
|
| Back to top |
|
|
del cecchi
Guest
|
| Posted:
Sun Jan 16, 2005 6:59 am Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
"Robert Myers" <rmyers1400@comcast.net> wrote in message
news:o16iu015p74cbkvqjlqcv34kf7tpccamj8@4ax.com...
| Quote: |
snip
... (where is IBM's muscle
when we need it?), and the endless denial that goes on about how badly
huge sums of money are spent on systems that don't work using
technology that was figured out four decades ago.
RM
I think the Brooks act and other regulations give the job to the low |
bidder, eh? No good bureaucrat wants to be pilloried or sent to jail
for awarding no bid contracts or violating procurement regulations in
other ways. Me either.
Check out the new system for registering firearms in Canuckistan.
del cecchi |
|
| Back to top |
|
|
del cecchi
Guest
|
| Posted:
Sun Jan 16, 2005 7:05 am Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
"Bernd Paysan" <bernd.paysan@gmx.de> wrote in message
news:ovmnb2-rqa.ln1@vimes.paysan.nom...
| Quote: | Robert Myers wrote:
The security regimes with which I have been acquainted lean heavily
on
strict compartmentalization by need to know. That means that access
paths all have to be laid out rigidly and with significant
foreknowledge about what metadata need to be visible--and it means
the
metadata have to be tediously created by hand. I don't know how the
FBI has proposed to deal with compartmentalization, but you can't
just
dump everything out there in a flat resource space, index it, and
let
people do their own searches.
As far as I know the FBI (and even more the secret services) have
troubles
with "left hand doesn't know what right hand does". This is a
consequence
of security regimes like the "need to know" principle. Since the FBI
is a
federal police, the "need to know" regime should happen at the border
to
the FBI, not within. Crimes that are no concern to the FBI don't go
into
the FBI database, and crimes that do should allow to be linked
relatively
freely. Organized crime needs correlation between seemingly unrelated
cases. Organized crimes also requires surveillance of the officers in
charge, as the combination of "need to know" and corruption is an easy
way
for organized crime to continue their work even though the FBI has
been set
on them.
|
Sorry Charlie. You must not have gotten the memo about the wall between
criminal and national security work at the FBI. Sent by Garfolo(Not
sure about the spelling, worked for Reno)
| Quote: |
The "need to know" principle does not help civil liberties. The
Gestapo was
clearly organized that way, too. It's a two-sided sword - it helps to
hide
crimes by those organizations as well as it helps to hide well enough
organized criminals.
|
You also don't know about the fruit of the poisoned tree and all that.
American Blind justice is almost too bizarre to think about.
| Quote: |
Systems like that need boundaries. Some material needs to be
classified
internally, too. However, it's dangerous to allow material to be
locked in
(like the notebook from the suspected 9/11 highjacker which was caught
a
month before). I'm quite sure that the inner boundaries, the virtual
cubicle walls inside the FBI are there for the same reasons many
programmers in classical corporate environments don't want other
people to
look into their sources: They are there as job protection mean, to
hide
errors, not to help nation security or civil liberties.
|
Bzzt, thanks for playing. Read the accounts of the arrest and trial of
Z Massoui ( the purported 20th hijacker). His hard drive couldn't be
searched. Would violate his rights.
| Quote: |
Sure, the database should not be searched that easily. At least a
"four
eyes" approach or something like that is needed. Internal
communication
within the team may be concealed from non-members (the same thing is
even
possible on Sourceforge). For things like that (mailing lists), you
don't
need special infrastructure, not even for a very tough security regime
on
those mails (OpenPGP or S/MIME is clearly sufficient).
I don't think you should talk about stuff you don't know about. They |
already have a database. Hell, they probably have 17 databases. All
living on servers somewhere that are updated overnight.
snip |
|
| Back to top |
|
|
Guest
|
| Posted:
Sun Jan 16, 2005 1:51 pm Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
If Documentum is the answer, you have to wonder aloud what the question
might be. The government is attempting to retrieve cross-referenced
data from multiple sources; listeners if you will.
Having used Documentum, I don't see the slightest correlation. It is a
static wannabe document organizer. In a previous employment, they
organized the data once then again - painfully, and unforgivingly. You
might as well pour your documents in cement.
Nor are utilities like grep in the ballpark of heterogenuous,
cross-database data mining - uh-uh.
Finally, if they can't understand the requirements of what they want -
then managing [people AND technology] will bring any such system to its
knees during peak vacation times. $170M is a drop in the bucket for a
meta-enterprise system requirement such as this. |
|
| Back to top |
|
|
Robert Myers
Guest
|
| Posted:
Sun Jan 16, 2005 7:31 pm Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
On Sat, 15 Jan 2005 19:59:23 -0600, "del cecchi"
<dcecchi.nojunk@att.net> wrote:
| Quote: |
"Robert Myers" <rmyers1400@comcast.net> wrote in message
news:o16iu015p74cbkvqjlqcv34kf7tpccamj8@4ax.com...
snip
... (where is IBM's muscle
when we need it?), and the endless denial that goes on about how badly
huge sums of money are spent on systems that don't work using
technology that was figured out four decades ago.
RM
I think the Brooks act and other regulations give the job to the low
bidder, eh? No good bureaucrat wants to be pilloried or sent to jail
for awarding no bid contracts or violating procurement regulations in
other ways. Me either.
Check out the new system for registering firearms in Canuckistan.
|
A billion-dollar mess, by one estimate, with an early-on claim that it
would cost taxpayers net $2 million:
http://www.baselinemag.com/print_article2/0,2533,a=130871,00.asp
http://www.cbc.ca/news/background/guncontrol/
The infamous Harvard Business Review article, "IT Doesn't Matter"
might have stated more honestly: "The costs of making enterprise IT
work in the way it was once expected to work have become unacceptable,
the struggle has become boring, and the field is no longer hot, so
it's time to declare victory and go home."
As to IBM's leadership, or lack of it, IBM knows how to market when it
wants to. It has, for example, built a marketing program around "Deep
Computing" that pushes the odd notion that all the different things it
is selling under that umbrella are somehow related to Deep Blue
winning a chess match against Kasparov. What IBM is selling, really,
is the idea that enormous computing resources can successfully be
brought to bear to solve technical problems (that may not even
resemblance computer chess). IBM is raising the level of what
customers should expect from technical computing.
IBM or anyone else is doing that for its old bread and butter
business? Nup. It's all become a commodity and thus it's all about
cost now, and your post underscores that. If it were such a $%!&
commodity, the director of the FBI should be able to log onto Dell's
web site and order what the FBI needs.
RM |
|
| Back to top |
|
|
mike
Guest
|
| Posted:
Mon Jan 17, 2005 7:55 am Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
Hold on a second!!
I just said that real time index maintenance was not the issue, I did not
mean to imply that GOOGLE and a few million dollars would be a full function
case system for the FBI. I have not even seen a clear statement of system
requirements.
However, just thinking out loud, you can not just let every FBI agent look
at every file with no controls and no log of access. If Al Queda manages to
suborn just one agent they could find out exactly what the FBI was looking
at and the names of all informants and puff! the whole FBI is useless.
I would expect the FBI needs a system that will look at each case and notify
the agents on that case of other cases and documents that might be relevant.
Further, supervisors should get detailed reports on system usage so they can
verify the appropriateness of data requests as relevant to the assignments
of their staff.
Also, the system should have a "work flow management" component like call
centers and large help desk operations use, so that new data entry
transactions needing investigation or analysis are automatically assigned
and progress tracked. The FBI is a big organization and without such a
facility, too much will fall between the cracks.
One of the biggest problems with government system procurement is that many
of these features can be found in commercial packages. When available, they
should be used as subsystems rather than reinvented.
Mike Sicilian
"Stephen Fuld" <s.fuld@PleaseRemove.att.net> wrote in message
news:QvIGd.33750$w62.13950@bgtnsc05-news.ops.worldnet.att.net...
| Quote: |
"mike" <mike@mike.net> wrote in message
news:14IGd.10035$Vj3.7515@newssvr17.news.prodigy.com...
There are many organizational and cultural hurdles for governmental
organizations to overcome when developing large systems. However this
posting implies one problem that does not exist.
With the Internet there is no way for GOOGLE to know immediately when a
web page is created or updated. Therefore, GOOGLE must scan the whole
net, or as much as it can access, every few days or weeks to keep its
indexes fairly current. Google can assign a high priority to some sites
based on user access frequency and site volatility. These sites can be
scanned every day, hour or even every few minutes. On the other hand
changes to less important sites will not be reflected in the index for
weeks or even months. There is no possibility for a "real time" index.
In a closed government system like the FBI case database the software
that enters or modifies a case file can send a message directly to the
indexing engine. Further unlike GOOGLE the indexing engine does not have
to waste cycles re-indexing old entries that have not been changed.
Therefore a near real time index, within a few hundred milliseconds, can
be maintained.
Precisely. And the other technical problems seem fairly easily solvable.
For instance, I don't doubt that by sending some money to Google for a
special system for them selves, Google could easily take each query,
optionally look up all the words in a thesaurus and modify the queries to
include either the original word or the synonym. As for getting all of
the existing paper information into the database, Google has just
commenced scanning a huge amount of data, from several libraries into
searchable form, so this is right up their alley.
Some people mentioned the security issues. ISTM that either you allow FBI
agents access to the data or you don't. If you do, then there is no
problem. If you don't, then you can't say that you have "broken the stove
pipes" or "broken down the walls", as they claim to be doing. If there is
some particular need, again, having some code on each document and making
the search or display software aware of the codes seems like it could be
made to work.
While I agree that trying to use the existing Google system exactly as it
is now wouldn't work, but ISTM that it is so close, and the modifications
should cost a tiny fraction of the $157 million, that it is worth trying.
--
- Stephen Fuld
e-mail address disguised to prevent spam
|
|
|
| Back to top |
|
|
Stephen Fuld
Guest
|
| Posted:
Mon Jan 17, 2005 7:55 am Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
"mike" <mike@mike.net> wrote in message
news:14IGd.10035$Vj3.7515@newssvr17.news.prodigy.com...
| Quote: | There are many organizational and cultural hurdles for governmental
organizations to overcome when developing large systems. However this
posting implies one problem that does not exist.
With the Internet there is no way for GOOGLE to know immediately when a
web page is created or updated. Therefore, GOOGLE must scan the whole
net, or as much as it can access, every few days or weeks to keep its
indexes fairly current. Google can assign a high priority to some sites
based on user access frequency and site volatility. These sites can be
scanned every day, hour or even every few minutes. On the other hand
changes to less important sites will not be reflected in the index for
weeks or even months. There is no possibility for a "real time" index.
In a closed government system like the FBI case database the software that
enters or modifies a case file can send a message directly to the indexing
engine. Further unlike GOOGLE the indexing engine does not have to waste
cycles re-indexing old entries that have not been changed. Therefore a
near real time index, within a few hundred milliseconds, can be
maintained.
|
Precisely. And the other technical problems seem fairly easily solvable.
For instance, I don't doubt that by sending some money to Google for a
special system for them selves, Google could easily take each query,
optionally look up all the words in a thesaurus and modify the queries to
include either the original word or the synonym. As for getting all of the
existing paper information into the database, Google has just commenced
scanning a huge amount of data, from several libraries into searchable form,
so this is right up their alley.
Some people mentioned the security issues. ISTM that either you allow FBI
agents access to the data or you don't. If you do, then there is no
problem. If you don't, then you can't say that you have "broken the stove
pipes" or "broken down the walls", as they claim to be doing. If there is
some particular need, again, having some code on each document and making
the search or display software aware of the codes seems like it could be
made to work.
While I agree that trying to use the existing Google system exactly as it is
now wouldn't work, but ISTM that it is so close, and the modifications
should cost a tiny fraction of the $157 million, that it is worth trying.
--
- Stephen Fuld
e-mail address disguised to prevent spam |
|
| Back to top |
|
|
mike
Guest
|
| Posted:
Mon Jan 17, 2005 7:55 am Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
There are many organizational and cultural hurdles for governmental
organizations to overcome when developing large systems. However this
posting implies one problem that does not exist.
With the Internet there is no way for GOOGLE to know immediately when a web
page is created or updated. Therefore, GOOGLE must scan the whole net, or
as much as it can access, every few days or weeks to keep its indexes fairly
current. Google can assign a high priority to some sites based on user
access frequency and site volatility. These sites can be scanned every day,
hour or even every few minutes. On the other hand changes to less important
sites will not be reflected in the index for weeks or even months. There is
no possibility for a "real time" index.
In a closed government system like the FBI case database the software that
enters or modifies a case file can send a message directly to the indexing
engine. Further unlike GOOGLE the indexing engine does not have to waste
cycles re-indexing old entries that have not been changed. Therefore a near
real time index, within a few hundred milliseconds, can be maintained.
Mike Sicilian
"Robert Myers" <rmyers1400@comcast.net> wrote in message
news:c76ju01ic19dv5h8efai6c021cqvol77f1@4ax.com...
| Quote: | On 15 Jan 2005 13:59:47 -0800, jgrove24@hotmail.com wrote:
Actually, I'm read usenet in GOOGLE, what the FBI wanted is basically
GOOGLE features.
The problem is that the part of the problem you can see--the part the
looks like the internet and google--is sufficiently non-trivial that
it is successfully masquerading as the whole problem to you.
The FBI wants data relevant to terrorism to be seen instantaneously
(as opposed to the currently claimed 24-hour cycle, which at that I
don't believe)--google can't do that.
There's lots of stuff on the web, the so-called deep web, that you
can't see with google, and the Virtual Case File would have to look
like a deep web so that it didn't pose unacceptable security risks.
Google can't manage document access or audit changes and versions.
The tools open source contributors use can do many of those things
nicely, but if FBI agents could function at that level of competence,
they could probably build the system themselves on the fly.
I wonder if something very similar to the false intuition you have
about this problem doesn't explain the whole wasted $170 million. It
looks easy, so it must be easy. Put it out on an RFP, take a
plausible bid, and wait for your own ready-made personal internet and
google to appear. Piece of cake.
RM |
|
|
| Back to top |
|
|
krasicki
Guest
|
| Posted:
Mon Jan 17, 2005 7:55 am Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
Very good points.
Right the first part of the problem is a canonic case file, the second
part is making it virtual, the third part - realtime, the fourth -
realtime rollback when it's wrong, and so on.
Architecturally, we are talking about the largest in-memory database
ever attempted and we're talking about synchronization features that
haven't been dreamed of with the initial investment. As you say, this
is almost certainly vendor driven, government payola driving the train.
One of the most severe problems that we as a nation face in truly
beginning to prepare ourselves against the war on terror (let's face it
- the Bush administration are klowns) is that the RFPs for far too many
systems have overt, vendor driven agendas built into the spec. Often
the vendor insists only their inside consultants can do the work. The
contract award this becomes self-insulating for the vendor or family of
vendors and no fresh air ever gets in to disinfect the project from
poor management, honest review, and so on.
And you are right in observing that Google is not the answer. |
|
| Back to top |
|
|
Martin Rodgers
Guest
|
| Posted:
Mon Jan 17, 2005 1:08 pm Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
mike wrote:
| Quote: | Further, supervisors should get detailed reports on system usage so they can
verify the appropriateness of data requests as relevant to the assignments
of their staff.
|
That sounds like profiling of FBI agents. Oh, the irony!
--
http://www.wildcard.demon.co.uk You can never browse enough
Will write code that writes code that writes code for food |
|
| Back to top |
|
|
Stephen Fuld
Guest
|
| Posted:
Mon Jan 17, 2005 1:09 pm Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
"mike" <mike@mike.net> wrote in message
news:21JGd.18525$by5.14061@newssvr19.news.prodigy.com...
| Quote: | Hold on a second!!
I just said that real time index maintenance was not the issue, I did not
mean to imply that GOOGLE and a few million dollars would be a full
function case system for the FBI.
|
OK.
| Quote: | I have not even seen a clear statement of system requirements.
|
Nor have I, but we can do some intelligent speculation.
| Quote: | However, just thinking out loud, you can not just let every FBI agent
look at every file with no controls and no log of access.
|
I mentioned in my post the idea of codes to limit access. That doesn't seem
like a major issue. And today, Google can log each query and does for
statistical purposes.
| Quote: | If Al Queda manages to suborn just one agent they could find out exactly
what the FBI was looking at and the names of all informants and puff! the
whole FBI is useless.
|
What are the controls today? I remember the one agent complaining that she
couldn't see if other cities had people with arabic names taking flight
lessons and this was not due to some security issues, but because the system
simply couldn't support it. Clearly, some people need access to all the
data and if they are compromised, then we have a Robert Hanson type
situation.
| Quote: | I would expect the FBI needs a system that will look at each case and
notify the agents on that case of other cases and documents that might be
relevant.
|
I'm not sure how you could code the test for "relevant". But let's assume
you could do that. Then when a new case is created, or when there are
substantial updates, a "relevant information" query could be run. Then the
software could automatically send an e-mail to the agents on the other
cases.
| Quote: | Further, supervisors should get detailed reports on system usage so they
can verify the appropriateness of data requests as relevant to the
assignments of their staff.
|
Sure. A simple extension of the logging function seems to meet that
requirement.
| Quote: | Also, the system should have a "work flow management" component like call
centers and large help desk operations use, so that new data entry
transactions needing investigation or analysis are automatically assigned
and progress tracked. The FBI is a big organization and without such a
facility, too much will fall between the cracks.
One of the biggest problems with government system procurement is that
many of these features can be found in commercial packages. When
available, they should be used as subsystems rather than reinvented.
|
Yes. I absolutely agree. And it is even the case that it may be faster to
slightly modify an existing system than to code from scratch.
--
- Stephen Fuld
e-mail address disguised to prevent spam |
|
| Back to top |
|
|
Andrew Reilly
Guest
|
| Posted:
Mon Jan 17, 2005 4:45 pm Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
On Mon, 17 Jan 2005 06:30:37 +0000, mike wrote:
| Quote: | In a closed government system like the FBI case database the software that
enters or modifies a case file can send a message directly to the indexing
engine.
|
kqueue, famd, snapshots ?
No, I know that's probably not the whole answer...
--
Andrew |
|
| Back to top |
|
|
Bernd Paysan
Guest
|
| Posted:
Mon Jan 17, 2005 4:54 pm Post subject:
Re: [OT?] FBI Virtual Case File is even possible? |
|
|
Stephen Fuld wrote:
| Quote: | Precisely. And the other technical problems seem fairly easily solvable.
For instance, I don't doubt that by sending some money to Google for a
special system for them selves, Google could easily take each query,
optionally look up all the words in a thesaurus and modify the queries to
include either the original word or the synonym.
|
Not just the query, the index, too. Certainly, the thesaurus must map equal
words when querying, but when they go in as equals in the map (index), the
map is denser and smaller.
| Quote: | While I agree that trying to use the existing Google system exactly as it
is now wouldn't work, but ISTM that it is so close, and the modifications
should cost a tiny fraction of the $157 million, that it is worth trying.
|
ISTM, too.
--
Bernd Paysan
"If you want it done right, you have to do it yourself"
http://www.jwdt.com/~paysan/ |
|
| Back to top |
|
|
|
|
|
|
FAQ
Memberlist
Register
Profile
Log in to check your private messages
Log in